You need to be really clear about your needs right now, in my opinion. To your H, it seems very logical that you may be having an affair. It is very difficult to understand why your spouse would leave, especially if there is no one else. And, many counselors will actually push you to believe that because it is such a common reason for this type of behavior. I am not accusing you, I am simply telling you, although it seems a horrible accusation and place for your husband's brain to go, so many of us have been there, and it is where your brain and heart go out of fear and out of trying to answer "why is this happening?" because it makes no logical sense to us.
So, my suggestion is, to not get defensive when he makes those types of accusations but to be very open and honest with him regarding what your needs are. Maybe you could say something like: "There is no one else. I can understand your need to find a reason for this but I am continuing to stay faithful to you and our marriage. I really need space right now. I think our marriage has a chance, but right now the best thing you can do is give me some time and space. When you contact me, accuse me, etc. it just increases the negative feelings I'm having. When I have some time and space, I can start to deal with those negative feelings and think about the good things about our marriage. I know you want answers right now. I can't give them to you and I don't think we need to make any decisions right now. I want to heal and for us to heal. Here is what I need to do that....."
This might sound crazy... but maybe you should ask him to read Divorce Remedy. The book outlines what you need from him right now.
Very little is needed to make a happy life; it is all within yourself, in your way of thinking. -Marcus Aurelius