YOu are also assuming that he will think you have only one source... All you have to tell your H is

"A LOT of people know a LOT more about what you're doing than YOU THINK... We are NOT stupid..."

That's an incredibly vague statement that should intimidate him quite a bit...

WS' dont' like it when you bust open their affair to the public...