I think a big problem here is that you are listening to what he says and internalizing it. When I found out about the affair I told him to leave the house and never come back. He said he would sleep in the driveway, and did one night. I got scared and let him stay.
He said that we should get divorced and never tell the children. He said he would work away a week and come back a week. I told him he was insane. He then accused me of blowing up the family. He said that whatever happened to this kids was on me. At times he has tried to get me to move closer to where he lives, even though he lives in driving distance and never comes to see the kids anymore.
I bet your kids know far, far more than you think. My kids knew so many things and were keeping it in. When the you know what hit the fan they came forward to tell me what they knew, including things I did not. They could accurately pinpoint exactly when the wheels came off. I was still in a fog about that.
I'm not trying to tell you to leave him. I'm really not. But I am telling you to stop living this way and to stop living in fear. Your fear is what is holding you back. We have all been there. We all get it. What if your wrong. What if you are all happier. What if he finally gets the damage he is doing. What if he finally appreciates how bad it has become.
Your kids are older, at least some of them as I recall. You can go to counseling with them and with him to work on family dynamics. In the course of that, you might allay some of your fears about the kids knowing and even how they might process their feelings.
My H had multiple affairs. He used the kids as an excuse to conduct as least one of them. When they found out, he ran from the house like a chicken and wouldn't talk to them about their feelings. When he showed up months later with his tail between his legs they forgave him. They gave him another shot, and another, and another. He keeps letting them down. They will keep giving him chances.
Your kids hating you is not going to happen. They may be hurt, they may be afraid, but they would adjust. What they may not forgive is the example of parents gutting it out in a miserable family. One icy cold function after another. I know you love him and want him work, but it really does take two. You are in control of whether you keep your heart open for him.
He is a grudge-holder and a score-keeper. When is that going to end? You said before you would separate to consider saving your marriage. I think you should give that some thought. You think you are saving your marriage by staying, but I told you my example, and please read up on HaWho. See what happened with her.
You don't know what will happen, no one does. You just know what you are doing now is not working. How can a scab ever form and become a scar if you peck at the wound every single day. How will you have peace. How will he have a chance to see what life would be without you.