You're not a mess. Unfortunately, most of us did not have the type of parents who validated OUR feelings which helped us become defensive.
It's a tough row to hoe but if you try to remember what he says about you is his opinion (at the moment) or his way to inflict pain on you because he's in pain, and does NOT define you or who you want to be, then it may be easier to refrain from defending yourself and easier to validate his FEELINGS or just to step back and disengage.
You don't have to correct his opinion of you or set him straight. In fact, at this point in your relationship, you absolutely can't. Each attempt to do so will fail and only make him angrier because you are telling him HE'S WRONG.
Your boundaries have to be expressed in a non-threatening way and not as an ultimatum. You may have to email them. I don't recommend you speak to him about boundaries until you are a little more dispassionate about them.
Boundaries let someone know what you need and what will happen in they're not respected. For example, your custody arrangements are all over the place right now. That's fertile ground for problems and disappointments.
Try to find something that comes close to working for both of you. If he can be scheduled a regular time, it's fair for you to get 24 hours notice and if you don't, to make other plans. It's also kind to try to be accommodating and enlist his help in working out what's best for him.
No matter what happens to your M, it's imparitive that the two of you start setting a better example for your children on how adults and parents find solutions to their differences.
M:63 H:53 S:41, SS:28, SS:25, SD:23 M:15 T:16
Bomb:12/17/11, "I think we should go our separate ways." H moves to his mother's house, 4/1/12 12/21/12: H moves back home, piecing