I know how hard it is, but think of it this way - YOU are going with him this time. That probably tears the OW up knowing that. And if you got the same bed - it will be erasing memories and replacing them with new ones for him.
Just remember that this is your vacation. Not OW's.
