In your last paragraph you stated two options you had. Do you realize that one of those options was your choice and the other is you giving in to his choice? I think you need to GAL and make the choice that is best for you and your kids. His anger and blaming you is him. It is not you, he just can't see that. No matter how hard it is you need to let him have his opinion and not buy into the blame game he is playing. You don't have to argue, or admit he is right or wrong. Just accept that his opinion or thoughts are exactly that, HIS!
"Our life is what our thoughts make it."
Marcus Aurelius