That is a really upsetting thing to assume, that he doesn't want to be unfaithful to OW! Everything I've read from you in the past week or two has said that he is done with OW, doesn't even like her anymore. Once you quoted him as saying he feels sorry for her. Those two things don't go together.
I think you are assuming something that he is not at all thinking. There's only one way to know what's in another person's head. That is to ask.
You have been very upfront with him asking other questions lately. You should ask about this. Just tell him what you told us about accidently leaning against him and how he pushed you away, and it hurt your feelings. Then ask him why he pushed you away. You can even tell him what you thought it meant so that he has some vision of how the affair has affected you and undermined your confidence.
I remember how insecure I was before Retrouvaille. I tried to get my Husband to understand, but it didn't work. At Retrouvaille when the other couples talked about how they felt when the spouse was having the affair, then he understood. It was clearer when it wasn't us.
Sounds like maybe he was just in a bad mood. It's OK to ask. It's better than assuming something bad.
