May I ask how long you rejected him, and how long you've now been working to correct that and be affectionate toward him?
Don't underestimate the power of that rejection, and what it may have done to his willingness to open up to you. While it does not justify an affair, you are probably right in your assessment that he's keeping you away now as a defense mechanism over your past rejection of him.
It's going to take time for him to see that you are sincere in this, and that you're not going to revert back to rejecting him sexually and romantically.
