OK first, sorry you find yourself in a sitch that finds yourself here. It WILL get better, eventually.
Second, your texas lilt is coming through in your posts, so I'm doing my best to decipher things.
From what I can understand, your H's behaviour with his best friend's W 4.5yrs ago, if not out of place, certainly borders on the... bizarre. Certainly, I do not understand the context, but on the surface it seems like an EA, but it just seems strange that even his buddy appeared to be OK with it. ???
And now with this young long distance, now neighbour. Well...
If he's being secretive about his R with the neighbour, best guess is there is at least an EA. If he can't be completely open with his R with the neighbour, then he is intentionally hiding something he is uncomfortable with letting you know. Even if he rationalizes the withholding as due to you being jealous.
That said, you indicate that, as best as I can determine, you had a PA with your H's brother? If so, while two wrongs don't make a right, two wrongs still make two wrongs. It adds an added layer of messy to the whole sitch, on top of it all.
Aside from his fondness of his friend and W that you mention from 5 years ago, was there anything at that time that was possible traumatic in his life, or anything from then to now that you can think of?
So at this point, it is really important that you do not justify or rationalize your PA. It happened. Own it completely as your choice and responsibility. Any issues your H has regarding it, are valid.
Your issues with your H and his likely EAs (and possibly PA; you likely don't know for sure) is HIS choice and HIS responsibility and any issue you have regarding his secrecy ARE valid, also.
With just that, the two of you are going to have to do some fairly work on trust in order for this M to be saved. But before that can happen, you need to take care of yourself, while ensuring you do not get yourself into any further EAs.
You will want to work on some 180s (what do you think those might be?) and also GAL, but DO NOT be mysterious, considering the context. YOU NEED to be like an open book. When looking at the 180s, you will need to look back to 5 years and figure out how your H became more interested in his buddy and buddy's W than in you and his family. How do you think that happened?
