Yes, well, even paranoid people have enemies, right? But it is possible to snatch defeat from the jaws of victory, and it looks like you are working on that.
I was unsure after my H's affair too. I thought I would never relax, but it did go away. Of course, you are there to read the signals. Working late does not always mean an affair. Have you tried asking him if he is in contact with her again? Is her number in his phone?