I would really pay attention to what his physical contact with you could be trying to say. Perhaps you can ask him what his R's in the past were like....did the women dictate when they had sex? (this was the case with my H) Did they possibly put him down in the bedroom (my H also experienced this). Perhaps he wanted it more back then and was also told that was not normal...something yet again my H went through...to the point he learned to squealch it.
Now, this doesn't mean YOU have done this. And I know you stated that for a while things were pretty good...but here's something else to chew on. After my H and I married his behavior did a 180 on me...it was as if his mind went in to automatic defensive overdrive. He wasn't trying to hurt me...he was trying to prevent anyone treating him the way he had been treated repeatedly in the past....he wasn't even aware he was doing this. But, while in his defensive posture he was effectively pushing me away and out of his life....he truly didn't even know he was doing it. When it was pointed out to him in our C's office he almost broke down...I could see the pain in his eyes and it wasn't caused by what others had done to him, the pain was due to the realization of what he was doing to me.
My H is still very subtle about his advances...most of the time they could still easily be misconstrued but I have learned to ask and clarify....and in returned he is slowly learning it's ok to be more bold, and in fact...I want him to be bold.