You can create boundaries and continue to improve your GAL and not hurt the chances of recon M if you choose to do so despite what reaction you might generate from your H in the process.
Let him get angry. Let him make you try to feel guilty. But, do not give in to him. Show him what he is losing out on!!!!