Well done. The obvious course of action is as clearly stated.. DISTANCE yourself for at least three days. Do NOT ACCEPT ANY of his calls, texts, or anything. If he comes by you LEAVE.
This is your chance to SHOW him you are NOT a doormat... you show him by action - by shutting him out.
This "talk" thing if his is interesting.. what on earth does he expect to accomplish by talking.
His problem isn't with talking.. he can't put himself into the deep end of the pool and make a solid commitment to either of you.. he is playing both of you and hates it... he thought it was fun for a while, but now he realizes how stressful and unplesasant it is. Talking isn't a solution - him acting like a man is.
I would tell him that when he tries to "lets talk" again. I would tell him "Talk isn't what we need... YOU need to either make an honest commitment to your marriage or make a complete mess of two marriages and run away with this floozy... like that's going to last. Marriage is about honesty, commitment, and hard work... if you're ready for that, you send her packing and I am here willing to be a family with you... otherwise you need to start packing and get ready to be a part time dad"
And hang up or leave (if in person)
It's at a point now that you can throw a few truth bullets at him like that.
