Once you DO decide some "family time" is in order (and it's not time yet), YOU should initiate it -- not respond to his "beckon call" thing.

Puppy