Jpeg, I think you need more info before you can really answer. Like Ancaire mentioned is his plan to see you or the kids?
My personal approach (and please remember to take it with a grain of salt) is to not ignore him or say yes and not tell him the kids won't be home. Both seem passive aggressive to me. I leave it up to you for your response if he wants to have dinner with only you. My only recommendation is to keep your reply consistent with what you've already been telling him. oh, and make sure you are direct without any beating around the bush.
Also, does making dinner hold any significance to him? He could have said he was going to bring carry out or meet the family at a restaurant...
Me-38 W-38; T-15 M-12; S10 and S6 BD - July 2016; EA confirmed; confronted Aug 2015 EA dissolved 12/2016
Never make someone a priority when all you are to them is an option - Maya Angelou