Nothing in what you wrote suggests to me that it was not OW-related.
Did you say to him, anything like: "Hey, I feel uneasy and scared. It would help me a lot if you share with me roughly what is going on. Let me in. If it involves OW, let's trust that we can work through that together. Let's not hide from each other, OK?"
