I can't say for sure but my H said the same thing many times to me last year... that he thought about getting back together with me (as if HE is the only one who had a say in the matter, lol) but he couldn't because he was afraid OW would be thrown in his face.
IMO I think it is the way for the WAS to push more blame on the LBS when an affair is involved. To this day I still don't understand how a WAS can even fathom in their wildest dreams their affair would not have to be addressed if they did reconcile.
I am not saying you have to discuss it every day for the rest of your lives but it is something that would need to be addressed IMO.