are you spending time with his daughter but he isn't spending time with your son?
I do believe it is time for a direct talk with him. I tend to drop the hints, and get upset when I don't get the response I want, then I realize, I just need to be very direct with my communication.
Maybe the two of you just need to get on the same page.
