In my opinion when he thinks that you are not their he pulls you back in case it does not work with him and the O/W.
Act like you are moving on with your life dont text him and go dark as heck. Tell him you met some one and you want to see where it goes.
That with him texting you and calling it is not fair to the other man.
You said it yourself when you stop being their for him he starts calling again and you start going out again then he runs back to the O/W. you have your own answer what you need to start doing.
Me 37 Waw 32 son2 bomb 8/11/09 O/M 12/25/09 Divorce filed 8/25/09 divorce finale 6/16/10 Divorce putt on hold 6/16/10 Divorce postponed STBXW idea 8/8/10