You are asking a moral question and not a DB question.
That's the knot in your stomach, your instinct telling you something is wrong. Stress is trying to tell you something. Watching your wife leave your home to spend "time" with another man should be against your morals. You don't control your spouse but why enable behavior that is harmful to all parties involved? Is it loving to your woman to not speak up and act in her, your marriage and family's best interests? You made a vow in front of both families & God to "love, honor and cherish" her, think thru that and make sure your actions are lining up with your word.
You can still continue to DB and see positive results while setting boundaries about three people in your marriage. You have to be willing to prepare for either outcome. It's not as simple as - "If I do this then that will happen." Focus on what you can control.
M22,H45,W45 S21/18D12 Retain faith that you will prevail in the end, regardless of the difficulties and at the same time confront the most brutal facts of your current reality, whatever they might be.