HaWho's husband lived in the basement. He was gutting it out for the kids. This went on for years and years. He would get a bit better, than disappear again. Finally, one day he walked out of the basement and said he was leaving and has made her life a living hell. He manipulates the kids and violates court orders, he had hidden money and bank accounts too. Was socking it away for years. She didn't save herself or her kids any pain.
I think you are thinking this will go on forever and be tenable at the level it is. I'm trying to tell you that HaWho thought the same. I thought the same. Then bam! I don't recall her H having an OW. He was just the troll in the basement.
They say if you need to go to therapy with someone and they won't go, then you go. Find a really good marriage and family therapist. Don't tell him. Just go. Go when the kids are at school. Go and explain your situation and the dynamic in the home. Ask them to help you find a path to happiness together or a path to happiness apart.
When I say you are internalizing the things he says, I mean you are accepting the merit of them and finding yourself at fault for his accusations. Remember on this site they say believe none of what you hear and half of what you see. Stop listening to his words. They are meaningless.
Do I think you should separate and explore a different kind of life, absolutely. Do I think you have to make that decision for yourself. Absolutely. The consequences are yours and the choice must be yours.