I agree that you really need to talk first. There seems to be a really big communication gap between you and H. I think that continued references to sex or even attempts at initiation will make him withdraw even more, unless you first have a really good conversation.
I know I'm not being very helpful here, but that is what I noticed with H. He felt attacked by all the little 'digs' as he called them.
