Thanks, lc4. You don't have to be a long time poster because you're living my situation.
Anyway, I guess I was mistakenly avoiding these talks because I thought it would makes things worse and would be pursuing or not giving him space but now I wonder if I made things worse by avoiding him because he seemed to pull back more when I didn't want to talk.