First, it is never "being silly" to feel uncertain about someone who has once proven they can lie and cheat...it's natural. (I should know; I was the cheater, and I wouldn't have trusted me. )
I personally think your best bet is to tell your husband how you feel about this. make sure it doesn't sound like an attack. Just tell him you want him to have time with the guys, but the old familiar loop makes you uncomfortable. He obviously loves you enough to have come home and leave the OW, and probably isn't dumb enough to make the same mistake again.
As you learned through the whole ordeal of the A, communication is key. Let him prove to you that he can be trusted again.
) 
