Maybe someone could enlighten me, but I don't see how questioning this "relationship" calls into question your manhood.
By all accounts, this guy is putting the moves on your wife, he has a history of this behavior. In what kinda of twisted mind does that make you week for not tolerating it? i think you look 10X more weaker for tolerating it. I think if she respected you then she would take what you say to heart. What else is making her not respect you?
I get that, if nothing is going on, your wife might be hurt that you don't trust her. But that has nothing to do with being weak.
Patience is bitter, but its fruit is sweet. --Jean Jacques Rousseau.