Am I jumping to conclusions?

Yes. Though I had that sneaky suspicion too.

If they are together I will be upset but how should I act?????

Well, if H's gonna share himself with the OW, then go ahead and get one of those jobs where you "share yourself" at Fritz.

Just kidding... I dunno, because it depends on your sitch... but I still think that, since most affairs crash and burn sooner or later... then the longer and more contact they have together, the closer they come to their probable demise... just let it crash and burn, and in the meantime, get that life carved out for yourself where H gets the clues that you're moving on.