I hate to tell you this but your H is sending you RED FLAGS of an A.
Personally, I don't know that I'd confront the OW at this point. You need to start snooping, and hard. It's very common for a cheater to tell you things the OW never said...to deflect the blame off of him and make you look/feel like a fool. She probably NEVER said that to begin with.
You need to get ahold of his cell phone bills and see who he's talking to. You need to really start monitoring credit card activity and bank ATM withdrawls (account for all money to make sure it looks reasonable).
Your H has just done several things that IMPO would send me straight into detective mode (and I'm getting pretty darned good at that myself).
#1 He purchased condoms (when you two don't use them)
#2 He told you two of them were thrown away (yeah, right). Why throw any away?
#3 He accused you of snooping and invading his privacy (cause you caught him with a condom in his wallet and knew he'd bought 3 of em).
#4 This may be nothing, but your H coming home really early when he thought you wouldn't be around, combined with the other things is simply something to pay really close attention to.
#5 He supposedly discussed your M with another woman, if nothing else this is highly suspect and inappropriate.
If I were you....I'd...
Get copies of his cell phone records (for a few months back)
Install a keylogger on your computer to monitor his activities online. You'd be absolutely amazed how many people have their own outside e-mail addresses to keep A's underground. If he has an account you aren't aware of this will give you his User Name & Password so you can see what's up.
I'm really hoping he's not up to something, but you need to find out and ASAP. If he is having an A it could explain many of the problems you are experiencing in your M now.