He's gaslighting you. He made up his mind to leave before all this but is now trying to find a way to make this your fault. That way he can lessen his guilt about leaving his family.
Also, though he wants to leave (likely to be with an OW) he also wanted you to be pining for him at home so he has a Plan B in case things don't work out with the OW. By being strong you are depriving him of having his cake and eating it too.
Are you going on holiday to visit family? If not, do you have a sibling or good friend you can invite to go with you so you and the kids still have a good time?
