JoJo, Is he still living with you? From your posting I have gotten the impression that you had a conversation this morning about the excuses for contacting the OW? Am I interpreting that correctly?
I'm with Jack on this one. If you are going to set boundaries, then you will need to enforce them, no waffling one bit. You state your case once and from there, it's all action. No more threats, no more discussions. It will be up to him to figure it out and come to the realization that you mean business about the ow and the contact. The more you waffle and threat and do not act, the more he'll continue to test the waters, i.e., just like a child.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.
