What you are really raging against here is the loss of "innocence" and the fact that your M is forever altered by your H's affair. I get that totally. We cannot ever recapture that innocence. And that stinks!!!!
As for you asking H about his affair(s), I think it is within your right to do so as you are not yet fully confident and secure. Perfectly understandable.
I'd suggest that you print off some articles from DearPeggy.com for the wayward spouse to read so your H can understand why you have those moments and feelings. I think H can stand reading these short articles. This isn't War & Peace, y'know.
