You will want what you want, and you can continue to hope that things turn around but have no expectation and pull back on the mindreading about things he does and says. I think your best bet is to back off and let him come to you, if and when he has a change of heart.
I would guess that he feels a lot of shame related to your extra-marital stuff, and every time he sees you, it's all stirred up again. Fade into the background except when you need to deal with things about your S. Be only your son's mother right now and give your H the chance to heal.
Yes, continue to work on yourself, become the better option, as I said before, he knows where to find you.
Me 57/H 58 M36 S 2.5yrs R 12/13
Let me give up the need to know why things happen as they do. I will never know and constant wondering is constant suffering. Caroline Myss