Actually, I just asked him. He said that I have only rejected him once. That was when it had been awhile and I said "No." in anger.

I then clarified and asked if there were any subtle signs that he was sending me that I was missing. He said no.