Yes, do what is right for you and your situation. It sounds like you know the outcome could be the end of the M and that is a good thing. If you set a boundary you have to know what the outcomes could be and be ready for it.
At this point in my sitch, this is H's second A and there will have to be a LOT more work for him and us before I would think of taking him back. And of course, your H doesn't like the word affair, because it is WRONG and he now has to redefine the dictionary on what A means! LOL Sometimes I really despise the nuttiness of all this MLC crud.
Sometimes the consequences can shine a light down their narrow tunnel. Sometimes not. Hang in there 3!
Me:49 H:47 S: 16 T:27 M:25 My EA: 2001 His PA: 10/2007, 6/2013 Separated, but H still in house
Find your Shambala: a place of peace and happiness.