It sounds like very controlling behavior to me. If he is not living in the house with you and the kids, why does he have access?
I know this is easier said than done, but trust me, he has to see that you are serious too. If you become strong, so does he.
Im still standin better than I ever did looking like a true survivor feeling like a little kid Im still standin after all this time and Im picking up the pieces of my life without you on my mind..