"It's easy for me to forget what he is going through."
That's why he needs validation, encouragement and compliments from you. Physical love is a big way that men receive that validation. How about making slow movements towards him first. Rather than giving him a hug, how about lightly holding his hand when you're talking and then smoothly move it away.
Then change it to light touches on his shoulder, then eventually a hug. He's not letting you touch him because he's afraid of being rejected again. Show him it's safe to do so.
M-43 W-40 2D - 9 and 5
Emotion, yet peace. Ignorance, yet knowledge. Passion, yet serenity. Chaos, yet harmony. Death, yet a new life.