If I were in your shoes, I wouldn't bring up the issue again. You are already aware of his chronic complaining about things and not taking any action to correct/fix his complaints.
He already thinks that his son doesn't care about him and let's face it, no matter what you say, he's already formed his own opinion of the situation. Your child is very small at the moment and can only show his father that he cares by pulling on his pants leg. If he were older, he would be able to say I love you daddy...but he can't.
Some people are constant complainers and do nothing to help themselves. I still think it's best to just let things be. The next time he raises this issue w/you, why not just ask him "what makes you think your son doesn't care about you?" Ask the question and put it back on him to tell you just exactly why he feels that way.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.
