LL, Part of it may be that he just "doesn't get it". He may just not see anything wrong at all with him hanging out with his buddy.
Another thing may be the fact that you two still have a lot to work through. This is a long and complicated process. The time he spends with his buddy is time where he can be "stress" free. Time that he can just relax and be himself - without having to worry about what he says or does that could possibly be taken wrong.
He's probably still working his way back to being comfortable again with your M - after everything that the two of you have been through - it will take some time for everyone to heal.
He might not even be concious of the fact that he's spending so much time with his buddy - it may just be the most natural thing for him to do right now - it probably does make him more at ease and relieve his stress from the whole sitch - but he may not even know that is why he's hanging out with his buddy.
Suffice to say, I wouldn't worry too much about it. If he's hanging out with the buddy - then he sure doesn't have time to spend with an OW. If you press him about it - it will probably push him away because he won't understand. If you just let it go... then try to provide a relaxed and comfortable atmosphere for him when the two of you are together - then I think he will naturally begin to gravitate to you more as he starts to feel "safer" being with you... when your company becomes more and more comforting - and not "stress" territory.
Hope you didn't take that wrong. I'm not trying to say you are stressing him out - but we've all got to realize that our spouses ARE uncomfortable around us right now because of everything that we are going through - and they feel like the villian - so, they are feeling guilty and feel pressured to do something about it.
All I can figure is that it will take a while to make him feel more relaxed again... that he is not under a microscope.
I sure hope that helps... and hope that I did not offend you in any way.