OK I will give you my opinion. As long as there is an OW he is still in replay. He has not yet done the work to look inside himself. His R with the OW is not bringing him happiness and he is looking back to you to try to find it. You in effect are becoming the NEW OW. This is not where you want your M to go. Proceed with caution. IMHO all is not as how you think it is. He must get rid of the OW, go through OW withdrawal, depression stage and then withdrawal stage. Finally there would be acceptance. Once he enters depression stage you can start the rebuilding phase. Right now I would keep you boundaries in place, watch, wait and listen.