Money became a thing with my H too. In all fairness, he has valid concerns/complaints. Nothing I can do about the past, so I let it go. I'm making different choices in some cases now.
My D's already know that there isn't going to be enough money for cllege. They will have to work at least part-time and go after scholarships (not likely with their grades), loans and or grants.
Would it be nice to be able to pay for it? Sure. Would the money I spend going on trips and vacations be better spent? Honestly, I don't know. Because of some of the issues I have with D's, I will say that some of these trips have allowed for more discussion and a better connection about what they want to do etc. That's really valuable in my world. They know I'll do everything I can to help, but if they want it, they have to be willing to find solutions too.
I thought it was funnny (in that dark and twisted sort of way) what your H said about his not being happy so nobody would be. We had an issue with that this summmer (except that I didn't het all worked up about it). Yes, H went with us for a week. It was a learning experience for me and it worked out ok.
Try not to get into his head (re: what he's really thinking). Whatever it is, just trust yourself that you can handle anything.