Set your boundaries, let him know what they are, and enforce them. He'll respect that. It may not change what he's going to do, but at least he'll know the consequences.
If I KNEW my W was going to go visit an OM, when she came back, all her stuff would be neatly packed and stuck in the garage. Heck, I might even be so bold as to move it all into a motel room, paid for in advance. When she came home, I'd just hand her the key and say, "Have a nice life."
Don't tolerate crap behaviour. Give yourself the respect you deserve, and demand it from him.
