Perhaps... and this is a perhaps... he thinks he will still be expected to plan something. "Hanging out" may be too much for him right now? You did ask, but you didn't ask him about doing something you asked him to come hangout. I can't imagine my W and "hanging out" right now.
Come up with an action plan for that day. Then invite him, but not in a "oh god please come" way... just, "hey, D and I are doing this today want to come?" and the implied message is we are going with our without you.
Married 6 together 8 Me:38 W:31 second marriage for both SS12, SD10, S6 Bomb: 9/8/11 (day before our 5 yr ann) W moved out: 2/18/12 D final: 11/12/12 Share S 50/50. Spend as much time as I can with SS & SD