No doubt your remark did not undermine anything in him, but I wonder if some teeny percent of him now does not trust you. Kind of like I read on another thread somewhere that a woman ONE TIME made a slight negative remark about the flowers her husband gave her and he never gave her flowers again.
Frankly, I think you owe him an apology for that remark.
Actually, I did apologize to him right after the fact and I made it clear that I felt sorry for the other guy because he was clearly too out-of-shape to be working in a vitamin store and probably needed the stuff etc. Really I am a very sexually validating person to be in a relationship with. For example, not too long after the vitamin store incident we were having sex while his son was distracted with video games in the basement and afterwards I said mock crankily "I was trying to be quiet but you made me scream. I hope you are proud of yourself. Probably you should go check and see if your son thinks you murdered me."
"Tell me, what is it you plan to do with your one wild and precious life?" - Mary Oliver