Peace, you are human and have done all you can. Do what you must for you. It is not ok for him to point his anger at you.
Go as dim as you can and take care of you and your kids.