It's not unforgivable. Thankfully, our Lord is so much more forgiving than many humans prove to be. You have been living with a man who, by his own admission, does not intend to ever forgive your ONS. That is unfortunate for him.
IMHO, you are currently in a position where you can begin to heal and move forward in building a happier life. Your H sounds as if he has a lot of ill thinking toward you, and I think physical separation was the correct move from you. I agree with 25 that you need to go no contact with him, b\c of his punitive statements and coldness could lead him to mental and emotional abusive tactics. I mean, he has taken his anger out on you for a long time, so he may not want to let go. You are not obligated, in no sense of speaking, to take any type of bullying, threats, stalking, or other bad behavior from him.
I hope you will not take responsibility for his happiness, b\c it's not your job. Focus on yourself and do what it takes for you to find peace and happiness.
((Hugs))
It is not about what you feel should work in your M. It is about doing the work that gets the right results. Do what works!