Ruby, I agree with you that your H needs to feel the absence of the connection with you sometimes. He is still trying to figure himself out. He needs to do a little more work on himself. Maybe it is good that he feels that you don’t care when you give him some space (personally I think that this is not what he actually feels.) He needs to be afraid of losing you.
I agree with what needgrace said too, it takes some strength and courage to come back to M.
It looks like you are on a very delicate path right now. I wish you all the patience and wisdom.
M:50 H:52 S28 (my S from previous marriage) M:17 + 3 BD: 06/12 S: 06/12 - H works in another state