Why don't you start by talking to him about everything discussed here, what you feel, what you want, and then ask and LISTEN to everything he says. He knows full well what is bothering him. You need to find a way to get it out of him. If he isn't talking, there is a reason for it. Ask him what that reason is.
