Ok - now take a deep breath. Keep someone with you if you are worried he will become violent. If he asks you why, stay CALM and say simply "It's apparent that you prefer to be with OW and I have no intention of standing in your way. I have higher standards for a marriage."
Stick to the high road, don't let him bait you, and STAY SAFE.
