Arsh, I'm online very briefly but want to say two things -
I don't think you need to tell your husband that you have to let go or that you're letting go. You have no choice and he doesn't seem to care much about how you feel so I anticipate this won't really help much.
In terms of negotiating custody and finances I wouldn't say "it's up to him as to how he likes to proceed" or "he's free to do as he wishes" because this gives him all the power to take your children or push the finances in his favor. Maybe I'm misunderstanding? But it seems the only response if he's not cooperating would be to tell him mediation won't work and you'll need to proceed on to a contested divorce via lawyers to get this sorted out.
I hope these next steps go as well as possible for you. Any judge or court should be able to take into account the need for such young children to reside primarily with their mother if your husband chose to walk away to free himself from responsibility, if you can document his aggressive abusive behavior (maybe start doing voice recordings?), and considering your baby is probably still nursing and your toddler probably still needs assistance with getting dressed, toilet usage, etc...
