Lisa, please, for your sanity, for your self respect DO NOT let him talk to you that way. This is not a matter of pride or of pushing your rights. You are a human being and you make mistakes like all the rest of us. The troubles in your marriage have partly been your responsibility and I don't even want to speculate how much percentage-wise. He has at least an equal share of responsibility too.
Some have advised that he might be looking at himself and, not seeing anything good in himself, that you will stop loving him and will abandon him some day. Maybe in his mind he is trying to abandon you before you can do that to him. The fact is, he committed to you. But when you had kids together you both committed your lives to them. You are keeping up your end of the bargain, he is not. You have five beautiful kids now and if you are going to be any sort of mother for them you've got to pull yourself together. You cannot let him tear you down like that or there will be nothing left of you to give to your kids. I'm barely managing my two pre-teen boys. I can't imagine having to care for five with such an age range as yours, but I know you can do it.
When you want to sell something, you look at it and think of what would make someone else NEED what you are selling. You try to put yourself in a prospective customer's shoes and figure out what some salesperson would have to say to convince you that you need what is being sold. You have to psych yourself up to sell it, to ignore rejection and negative perceptions. Your husband is trying to sell himself on not needing you anymore. What he is saying betrays his thoughts about himself. He needs to keep having rah-rah sessions to keep up the energy to sell the death of your family as something that is OK and that he needs, and he keeps inviting you to those rah-rah sessions. Don't go anymore.
Tell him you are sorry he feels that way, but that as long as he wants to express his emotions in such negative and hurtful ways you will not be a part of that. Learn to say assertively "do not talk to me this way." Create boundaries. It's the only way you'll survive.
I came to the realization that I was avoiding doing some things that would benefit my kids because I didn't want her to hate me. I further hoped that by not doing things that I knew needed to be done, that would also cause her pain, that would help me to buy her love. She wasn't interested in buying, and she hated me anyway. It is a tough and stinky cycle to be in.
Try not to give up, but you need to take care of those kiddos and you need his help to do it. Right now that takes the form of financial support. He knows it and he doesn't like it so he makes you feel guilty about it. You'll lose your guilt once the courts and/or child welfare agencies straighten him out.
I am sorry for your difficulties, but you need to look past his pettiness and think of yourself. Don't beat yourself up and don't let him do it either.