a. Suppose there were teenagers throwing rocks at your house and you knew where they lived? Would you DO something about that?
b. Suppose someone broke into your house, took your stereo, and left his name and number... would you DO something about that?
c. Suppose someone broke into your house, vandalized the place top to bottom, and left their name and number.. would you go to their HOME and DEMAND they leave your home alone and NEVER COME NEAR it again?
Infidelity of this sort here you are dealing with is no different than any other domestic violation that should be dealt with as a crime. Infidelity USED to BE a criminal act.
Do NOT just sit there and let this man violate your wife while she's vulnerable.
On this point. You haven't said much about the STATE of your MARRIAGE when all of this started. I suspect you two were NOT getting along all that well... this is VERY OFTEN how affairs manage to sneak into the home. You leave your wife vulnerable to an affair by neglecting her or hurting her in some way, and a predator WILL SENSE this and TAKE ACTION... YOU need to PROTECT YOUR WIFE AND YOUR FAMILY... this is your HOME
DO SOMETHING about it... march over there and tell this CREEP to STEER CLEAR
If you have ANY LEVERAGE you can put up on the table to make sure he knows you mean business you USE IT.