I'm having a problem understanding the problem here. It probably would have been better to simply state things directly rather than asking if he had a problem, but it isn't too late to do so.
"H, I just want to let you know that the Bears game customers kept us really late last night, we had to throw them out. I hope you didn't think I was blowing you off, because I really would have preferred to spend that time at home."
Now, as to your intuitions, my guess is you are right. His pissiness is likely not because you worked late, but because he feels like crap for doing something he "shouldn't", whether it was looking at porn, thinking about calling OW, or actually calling her. My point is, yeah, he probably was up to no good in some way, but you have no idea what way. The question is, what are you going to do about it?
If nothing, then relieve the tension YOU are feeling because of your own fears about working late (as above), and then let it go. You will feel better. Just do it simply and directly and be done.
It is probably good for you to give him a little space with respect to whatever crawled up his behind, that way he can't as easily project his uncomfy feelings onto you.
