Hi Sue, You probably keep jumping to assumptions because the beginning of all this mess caught you by suprise. So now you are super sensitive to everything he says and does.
It's OK to pay attention to what he says and does, just don't belive it or at least react to it. And I agree that he probably thinks he has no right to be looking at you that way right now.