Heather,
Your H is a weirdo, as it concerns the kids. To think that he can be alone with the kids and you can't--and then to manipulate you into thinking that it's all due to some potential infidelity (iow, hitting your ouch button a-GAIN) is asinine.

I'm sorry that you are going through this. It is abusive. My heart breaks for your kids because he will continue to use them for his own selfish and destructive purposes--namely to pump up his flagging esteem and to tear down yours.

Hang in there girl. You are a warrior and you know what you're doing.