So by me saying I'm not going, it means the trip isn't happening as he has envisioned it, four full weeks.
Of course it isn't. He doesn't get to lie and cheat and hold the door open for his mistress AND get the sorts of things that happen in committed marriages. He wants a committed marriage and everything that goes with it, then he actually commits to the marriage.
Otherwise, you proceed as a separated couple who have to cohabit. Which means separate vacations with the children.