Hey girl, You sound wonderful. Keep doing what you are doing.
He appears to be walking on eggshells right now, trying to do the right thing but not quite sure what to do. That explains the confused signals he is giving you. That also explains being down after shopping. You not reacting was the exact right thing to do. He feels low enough if he really regrets the PA and the D.
Okay, the physical contact thing. Ugggg....
As a man, we are always reminded that all men are jerks b/c all we want is sex, sex, sex. To be physical on date #1, never mind all that R stuff. We are also told the woman wants a R first and the "love" (and sex) comes later. Maybe H is reading up and wants to build the R slowly before it goes to the physical level? Why not? If all he wants is sex is that who you want back? I bet not.
As for the moving back part, I too am confused. That should be YOUR choice, not his. He had the PA, not you. If he wants to come back he should earn it. Yes, you forgave him and that is good but he needs to earn your trust (and you mentioned trust in your first post and you mention the OW again above). Let him earn his way home by demonstrating he has earned your trust. You two need to fall in love all over again and build a new R. The old R is dead, and his past behavior unacceptable - make that clear or he may stray again. Without full trust, there is no hope for a good R.