Sorry Rop but this is inevitable. By not watching the kids you are only prolonging something that will eventually happen. Did you actually say, "You are destroying the family." I think she already knows that the words are unnecessary. Don't bother telling her, she will only find some way to turn it around to you and make you to blame for it.
Don't worry. The time with OM won't be perfect, and even if it starts like some perfect fantasy reality eventually sets in (I've watched my friends go through with this over and over...). But don't worry about that... just make sure your time with the kids is TOTALLY GREAT. So they go back raving about what a great time they had with dad. Make sure you do something really special with them. Try to enjoy your time with them. Wives and girlfriends come and go, but your children are always your children. Don't miss out on the time with them just to try and keep your wife away from OM. You will rob yourself of memories that are both fleeting and irreplacable.
There is no arriving, ever. It is all a continual becoming.