Hi W2be, and welcome to piecing. I am pretty new to the board. I am in a marriage that survived nearly exactly what you are going through, about 9 yrs. ago. I am here to see if can make more improvements on our communication.
I had very similiar situation, and things never changed until I finally stood up one day (like I read you have been doing) and finally said, enough is enough. I tell you, it tore my heart out. I finally just 'assumed' he was doing everthing heart wrenching thing I could think of..(and prob. was..) and of course there was a young single woman, no kids, who could was fancy free). Once I set up that barrier.. then things started to change. I had tried to many times before, but my pain would always make me let him back in. After the last time, then I started controlling where I wanted my life to go. I could survive and did. We did reconcile some time after this.