I had to fight constantly for my custody time. It served two purposes: It let her know that she was not to fark with me on that issue. And it let my kids know that I cared enough about them to fight for my time to spend with them.
As for his suspicion of the non-existent infidelity, let him be suspicious. Not your job to validate your fidelity to him anymore. And you might also want to think about whether he is projecting his own guilt onto you.
I agree with Hairdog. Heather, don't your papers state custody arangements as well as the right to have other relationships? In my S papers that was included. Your H does not have a right to control your time with the children or your non-existent, but inevitable R with new people. If he wants to have some control over that than he needs to be working on the M. I never had to deal with that because H was so checked out emotionally with his depression and what not. I'm sorry this is such a difficult road for you. He certainly is quite abusive as Corri re-iterated. Stay strong.