God, I'm so sorry for your struggle. I hate living in the unknown like that. I sort of have a different take on things right now. So I will start with what I believe is likely and what I don't believe is likely.
What is most likely is that the panties belonged to XW. I just cannot imagine too many men stupid enough to bring another woman's underwear into the house after having an A? I mean think about it. Man has sex with OW, then asks her for her panties, slips them into his pants pockets, sneaks them into the door past W, then slips them in a drawer where W can easily find them? It just doesn't make a heck of a lot of sense to me. Unless your H is dealing with some severe mental issues, I just don't see this as likely.
BUT, what is bothering me is something you said, "I don't feel comfortable with his answer, but I don't think I'll ever get a different answer." Hope, tell me something... if I were to force you to give me an answer right this second, how would you answer this: do you not believe him because your gut is telling you that or do you not believe him because you are feeling insecure about the dating site debacle? And woman to woman, you know what I am asking you... are your hairs up or are you just feeling paranoid by everything and anything.