I don't think it's sinister. He is just that kind of person. When his brother got a new GF, the first thing he did was look her up on facebook - and make lots of snarky comments about her to me. Even now, when he comes round, he checks out any packages that have come in (the cleaner / nanny tell me). I'm sure it probably triggers an "urgh, she's wasting money again" voice in his head, but frankly, (mostly) I don't give AF.
I've spoken to various people on the topic of right of entry and unfortunately it's a really grey area. He has a right of entry. I have a right to privacy. It all depends on how reasonable he is being. I can go to court and get a court order, but that requires proof that he is threatening. He cannot however, legally look at documents which are not in plain site, which means he cannot look at things which are on my computer, laptop or phone. I can't prove anything though.
He is definitely anxious, but he won't speak to anyone. I know he hasn't because the things he tells me are just plain wrong.
Yes, it would be easier if he couldn't enter. He is here again at the moment. He organised someone to come in and quote on replacing the fence. It just so happened that I am WFH today.