In that list, you haven't mentioned if you were paying him more attention. I don't mean going overboard, but just doing little things here and there.
IMO I wouldn't leave and have just your son and him together. You want to build the bond between your H not force him to spend time with your son. How often do the three of you do things together?
M-43 W-40 2D - 9 and 5
Emotion, yet peace. Ignorance, yet knowledge. Passion, yet serenity. Chaos, yet harmony. Death, yet a new life.