Do you suspect that neighbour girl / OW is part of this "group" now?
One thing that seems fairly common is, while there is an OP it is unlikely that things will progress to R. On a positive note, most As end a natural death.
As far as trusting him, what specifically do you not trust him with? Yes, you have mentioned some reasons, I just want you to list them again in case there are anything else that might have been missed.
Then maybe I can offer some suggestions on how you could 180 on your trust of him.
Good on being open. While it may not seem like he is interested in what you are doing, you would likely be surprised what they DO notice.
