IMPO, you are making this separation easy for him by having him have full use of the house and family whenever he wishes. I also think it is bull that the current pattern with seeing the kids is the only one that would work. If you were D (yeah, I know, you aren't married), but if you were D, you would find an arrangement to make this work. You are using this as an excuse to see him and control him.
You will never win your SO by trying to limit his options for happiness without you, or by preventing him from seeing how happy he might be without you. Love is not driven by fear.
His "you aren't mad at me" comment is probably a bit of projection. He may be mad at you for not maintaining your boundary or mad at himself for getting sucked back in. I would be very surprised if you don't now see him backing off, or after a few more rounds of sex, backing off. I'd also be very surprised if you don't feel a bit miserable for awhile -- not because you miss him, but because you don't like feeling like your old self.
And PLEASE tell me that you are practicing safe sex for your sake and the sake of your children.
