Want2 Be, start coming up missing (to him).. stay out a late night here and there. Of course have the kids taken care of by someone, but just say you are 'going out'.
I guarantee he will get the word. It doesn't even have to be anywhere risky, LOL, you can shop at Meier's all night and then sit and read or have coffee!
Just dont' say when you've been doing. You know in your conscience, that you have done nothing.

Dont' know if this suggestion will help. But he will surely hear about it from someone. If he asks about it, blow him off. I did this a few times, went out with married friends, completely innocent. Boy, the reaction was priceless.
Cheers