misshim, did someone say being physical with your H was the wrong thing to do? IMO, it is not. BUT, you have to be emotionally prepared for your mood swings as well as his. Also, if he has been sexually intimate with someone else you have to be cautious...
Remember that a man usually needs the physical aspect to feel the emotional connection, while a woman needs the emotional connection to want the physical side.
You and H had a good day together, and a good night together. It's typical for a big withdrawal from them now. They get too close for comfort and have to step away again. Leave him be for now, and he'll be contacting you soon.
Be a buddy, be a safe place, let him garnish himself with all the good parts of the life he is leaving, let it call him back, but do not pursue him. My 2 cents.
Live your life while you are still living. Riding the trail less traveled.