You do have a mean MLCer - so was my h, and I empathize. Now he is showing small signs of realising that this behaviour is not OK. Intermittent, but, for example he admitted to his middle son a while back that he had not been a very good father for some time. Wow!
As for the OW - of course it takes two, but the OP has usurped the spouse's role in the family - with the co-operation of the spouse. But that is what they: are a usurper, and they know it.
I am not saying we own our spouse's of course, but in relationships we have rights, roles and responsibilities, to ourelves, to each other, to the wider family, and to the community.
To abrogate those [our spouse] and to usurp them [OP] is not acceptable. They know it deep down, and that is why your h is trampling over ALL boundaries. Partly it is control, and partly it is a refusal to see ANY boudnaries, because then he might have to see SOME, and they might not be the ones he chooses.
On a more general point, I have a feeling that they more p*ssy they are with us whey the r with OP is going less well [although they would deny this]. Looking back with my h I can see a pattern there - does anyone else have this experience?
