It does really sound like this is one sided on your H's part. It reminds me in a way of that Chevy Chase "Vacation" movie where he keeps seeing Christi Brinkly and has these images of her which are not reality. I know its wierd but that is what popped into my head.
Hopefully OW is telling the truth. It does sound like it. When I confronted OW she went out of her way to tell me how great of friends they were and what a fabulous guy he was. It sounded so fake. OW in your case doesn't want anything to do with him.
I don't know what to say, but I am thinking of you.
Me: 46 FWS: 36 Married and Divorced 4/07, Pregnant 7/07,False R 7/07 Baby Girl born 3/08 Kicked him out because OW: 7/08 5/10 He realized what he had and lost. Moved home! REMARRIED 3/14/11!!