Whether or not he can afford it is irrelevant. If you don't want him to go then you need to say so. Set the boundary. Yes it is nice for children to have a 2 parent family, but you are not in a relationship with him, and you are not a family in any functional sense! If you continue to play family with him then the crazyness will continue! Cut the cord!!!!!
"You can't stop the waves, but you can learn to surf." Jon Kabat-Zinn