He thinks you're crazy? No he doesn't. He's just projecting. He knows exactly what's what, but maybe if he makes you doubt yourself you might back off. He made a decision to go against the agreement and do what he wanted instead of taking the kids feelings or your wishes into consideration.
Lissie that was really a sad thing to read about your son saying he was scared to tell his Father how he felt. Since M trusts you and feels safe with you, you have no choice but to do the right thing for him and your little girl, there is no other choice. J put you in this position! All you ever asked was that he do right by the kids.