Now - make SURE on that night that when he comes over, you are dressed up as if for a date - perfume, hair, etc. As soon as he gets there, kiss the kids and scoot out the door - be mysterious about where you're going ("meeting a friend").
And when you come home, be happy and "up" as if you'd been out having a blast.
As for the custody stuff - try not to let it turn into a huge fight. Don't be a doormat but try to minimize conflicts. He's looking for fights to pick with you to justify his bad behavior ("see how unreasonable she is?"). Don't give him the ammo.
Oh - and if he has an attorney, you need to see one too. At least find out what your rights are.
