My H felt the same way after we were close for the first time. He left saying that he'd need at least two sessions with his shrink after what happened. The second time he didn't state any kind of remorse and the third he left smiling. Time is key.
I think it's natural for him to say you guys shouldn't have done what you did but on the positive side he said he had fun. Just let it go and don't mention it again. The next time you see him casually try to be a little affectionate again. Maybe touch his shoulder, accidentaly touch his hand...something subtle right?
I hope the house business goes well and yes breathe and relax.
Jen *The more anger towards the past you carry in your heart, the less capable you are of loving in the present*