When he comes over, just look him in the eyes, and say as ominously as you can "Have a safe trip. We need to meet as soon as you get back and unpack; there's something I need to discuss with you, and it's important."
And leave it at that. Basically a grown-up version of "Young man, just WAIT 'til your father gets home!"
Alternate version:
Look him dead in the eyes and say "I HOPE you're not going on this trip with your girlfriend. That would be incredibly disrespectful to me, our marriage and our family."
Either way, the idea is to have them talking about YOU -- even if it's unflattering (and it will be) -- their entire time, instead of about fun fantasy.
