You have an answer for everything on the surface but you fail to see the root issue that barricades every aspect of your life.
If we can see it (strangers) and clearly your W sees it why can't you?
Again, setting boundaries is not about "ditching" anybody, finding somebody else, being mean or defending your stance. Until you really understand that AND implement the mindset I am not sure anything will ever change for YOU as a man.