Need Help handling affair thats still going on

[kjensen]

OK
I left a brief non-emotional thank you to my H for doing a lot of chores yesterday, unasked, in our house. He emailed me that he got the note and that he had wanted to help. He then asked me to not leave my books about spouses having affairs around so our girls can see them. He felt it was disrespectful to him... My heart started pounding and I felt shame reading his remarks. I keep the books by my nightstand and I don't think the girls look at what I'm reading b/c I have a stack of about 10 books and I usually face the covers/spines upside down-so they are not readable-maybe I got sloppy-I really don't know...So I neutrally emailed a short (three sentences)apology and said I didn't mean to be disrespectful. Then I broke down and called him to apologize.
I know that was entirely wrong.
He was cold and said we weren't supposed to be talking(that was my idea, not his). I apologized again and thanked him again for doing the chores and said I did appreciate his help. I asked him if he had decided to keep seeing the Ow(b/c in our last conversation he agreed to tell me). He said he felt very detached, he wasn't sure, and he wasn't going to have a heavy conversation. I just said a yes or no would be fine, but I got nothing and said goodbye.
Ok.. So I know all of you veterans are going to think I'm a weak and stupid person walking into that and breaking down already to call him. I will say my only defense(if there is one!) is that I have an extrememly vivid imagination and it took all my efforts last night and early this morning NOT to drive by the OW house to see if his car was there. But I didn't drive by. I tried calming myself and my imagination. I know I should assume he is continuing the A, but in a way that lets my imagination run wilder. Anyway. I'm starting the "no contact" again.
I've been reading "Not Just Friends" and I certainly am going through much of what a betrayed spouse goes through-the post traumatic/hyperaroused/suspicious state of mind. It seems hard to get past b/c there is no communication and the A is most likely continuing. I will keep trying.