Wasn't it you Purg w/ the 24 hour rule? Or perhaps it was Perseverance?? Anyhow, no matter what you do, sit on it for at least a day so you have a clearer head in how you want to handle things.
I am in the camp of not saying anything. But, what about if you are hanging out w/ the kids or having dinner all together and ask the kids how the gym was in front of him? "uh what gym mom?" I know this is being slightly manipulative but it should get the cat out of the bag if that is something you truly feel you need.