I would say plan the vacation for your and your children, but leave him the invitation to join. He can book his own flight and accommodations. So you are not so hung up on basing your decisions on what he’s “hinting” you.
On another thread I read that you fear that if you act okay without him, he might be comfortable to leave. A wise man named Steve85 once told me that that fear will be your downfall. I had the same fear. But you need to overcome it, because you will be okay if he leaves. That’s where you’d want to be, a place of strength where you will be fine with or without him. It’s his decision ultimately, and you can’t subject yourself to living under that fear.