You should call him on his crap. He's having an affair even if it's only EA at this point and that's why he needs to go. You won't tolerate living in an open marriage(if you won't, that is). Don't be afraid to speak the truth to his face when he says things like that. He won't like it but it will eat at him later on.
By the way, please keep to one thread so we can follow along. It's hard to follow a person's sitch when they keep starting new threads all over the place. Can you link your main thread?