Hello?...... STD's anyone? Sex with OM one day and sex with you another day?! And you allow this? WTF?!
Do you enjoy dipping your stick in another man's vanilla? Sorry for the graphic description but what you're doing just sounds gross, it's like your both sharing her just for sex, I'm sure you can practically smell each other on your wife by now.
No more sex.
Take a break, it's ok, you will survive.
Rethink what you're doing and regroup, what's going on right now isn't working so we you can stop anytime now and start focusing on doing something that does work.