H still hiding A

[Beans]

My H started his A 6 months ago. Admitted to me in January saying he couldn't live a lie anymore. He said that they have been friends for about a year and that she said all the right things. He also told me he had and A 4 years ago. He was surprised when I said that I wanted to work on forgivness, he said he thought I would say it was over after he exposed the A. We spend the next month talking about things. mostly it became a blamfest, him blaming me for not being loving enough or attentive enough to his needs, and his saying that all he did was try to make me happy, with no regard for himself. He needed to start thinking of himself for a change. I admitted to him and apologized for my part of our relationship issues. He also left 4 years ago for several months, and when he came back he didn't want to talk about this "crazy time". I thought If I gave him space he would eventually tell me about that "crazy time. since he has come back I have had trouble telling him I love him. And he claims that is the reason he needed to leave me.And that if I was afraid of him leaving again that I should of made him feel more loved and appreciated. He also said couldn't you tell I have been unhappy for the last 4 years? Honestly, no he put on a pretty good front! He moved out 1 month ago, and is still seeing the OW. He as only told a few people that he has moved out and only when they call him. He has told noone of his A. He tells me he doesn't care what I tell other people but he feels this is only between him and me. He gets upset that other people want an explaination from him, recently his brother asked him to explain face to face. But my H hasn't made any contact with him. It's like what he doesn't admit he doesn't have to deal with? The not knowing what his intentions are about us are unbearable. As of last week I would text him a joke or a goodnight, sometimes he would reply sometimes not. So this week I decided to not contacting him and he has called a couple of times and it always ends in a argument and him saying "why do you think I hate you?". He has said this the last 3 conversations. I told him "Honestly,I don't know how you feel about me" to which he ends the conversation.
I know this topic will come up again, he claims he is coming over this weekend to clean up some stuff in the garage. We will see if he follows thru. Anyways, thanks for letting me vent. I am really confused right now. I don't even know how to act around him. Has anyone else felt like this? Your best friend of 20 years and suddenly you don't know what to say or how to act around them? Do I start contacting him again? at least we were civil when I was contacting him, but then again. I feel like I'm persuing him.
How do I handle his unpredictable behavior? I don't even know how to respond to some things he says? Confused!