You had a legitimate reason not to be able to hang today; you had plans. Don't stress over it.
I would also encourage you (again) not to fret over a possible negative impact to the kids re: you guys hanging as a family. Carve out time when possible to spend with H, sure. But the family is a dynamic I'm betting he's missing. So I think you should feel free to pitch an idea that involves all of you.
M: 40 H: 44 Married 14 years S11 & D6; D20 & D19 from previous M 2BDs/PAs, 8 years apart Piecing: April 2014