When people are involved in affairs, then insofar as their sex life with their spouses go, they can either lose interest or increase interest. It sounds like your H has increased interest.
For now I will wait for H to contact me to hang out. If we hang out I will try my best to leave at a decent hour to come home by myself. I just hate rejecting him b/c I feel he won't want to hang out again...like I will be pushing him away. How do I stop the spiral downward from this...?
Recognize that your conclusion that he will see it as a rejection and he will feel like not hanging out with you is, at this moment, a conclusion you've arrived at based on your assumptions of what he'll think, and not really a matter of evident reality. Also, lacking in that thought is the fact that even if he does feel cut off, that very well may have a positive effect on your situation, as it may influence him to reflect on what he may be losing.