Quoting eskb then smokry: My advice is to come clean. If he knew nothing about it, I'd suggest not telling him. But it sounds like he knows in his gut that you did sleep with him, and it will continue to knaw away at him.
But I'm afraid if I confirm it- it will continue to knaw away at him even more!
Just my 2 cents but I totally agree with Brian...the "not knowing for sure" is absolutely horrendous...get it out there so you can both deal with it constructively. It's not going to be easy for either one of you and there's no way I can promise you a totally positive outcome but if you don't come clean, it will always, always, always be a barrier between you two -- albeit unspoken.
Sage
Relax. Appreciate. Be calm. Laugh. Enjoy. Be secure. Be loving. Be loved. Don't personalize. Don't ASSume. Accept. Be grateful.