Sage, I for one am with you on NOT being okay with your H working on a full time, ongoing basis where OW works.
I was thinking about your dillemma and Dr. Phil's face came to mind. You DO know what he would say to your H right? "WHAT are you thinking??? That your W should be okay knowing you're seeing this ow every day at work...GET REAL!"
But we do know that Dr. Phil has his own ideas and his stance on contact with former OP is simple: Don't do it...get another job, move, whatever it takes.
This is going to be very trying for you (is H taking the consulting gig for sure? How long will it last?). H may be on the lookout for signs of "irrationality" and "weakness" or whatever on your part.
What to do? Act as if (there's zip going on there, H is fully trustworthy)? And come on here to vent? Do more stuff for yourself?