Joking aside I think you are doing a good job. You realize where you need to make changes (as far as asking to do things together) it seems that he makes time for you three to spend time together as a family (like how he said he was going to eat at home and then stayed there to eat with you guys) so if you stop asking altogether it seems he will pick up the rope and make ways for you to spend time together, even if not in "public" for awhile. But that's okay.
