The next time your husband grills you about your whereabouts, simply say "You gave up your right to know where I am all the time when you decided to have an affair. Please don't ask me again -- it's disrespectful, and -- frankly -- not very attractive."
And then leave the room. (or say "goodbye" and hang up the phone).
