Why don't you just talk to him? There are ways to confront without issuing ultimatums. Right now, I tend to agree -- if he knows that you know, then he's cake-eating, and he probably feels your condoning his adultery.

Puppy