Lol! It freaks them out when it occurs to them that maybe what's sauce for the gander is sauce for the goose too!
When my husband had his affair, one day I was driving and he had to answer my cell phone for me. After, he started looking through the list of incoming calls and asking me who this number was and who that one was! Me, who had never even looked at another man in 26 years!!!!!! It was purely his own guilt speaking.
So.....your H probably had a little moment of wondering if YOU could be out doing something naughty. Let him wonder - it's good for him to feel the shoe on the other foot!