If I were in your shoes I would feel the same way. Actually, I'd probably have a man to man discussion with him that he'd have a hard time getting up from. However, that doesn't make it right.
Besides, I fear that it would push your wife away from you were she to find out.
