It really seems like you base how you are feeling on whether he is affectionate or withholding. I know this, because I have done it as well. Really try to work on detaching. It's not about not caring, it's about not having your emotional well being be tied to what your loved one is doing. It's soooo hard.
Really think this through. I set my boundries and H is moving out. I'm very ok with it. I have a serial cheater on my hands, and one of his reasons for not giving me access is that he is afraid he will do it again. He knows I'm DONE as in divorce, if it happens again. (I only found out about all of them 2 months ago) So he needs to go figure out if he wants to work at being faithful. If it was just the one OW, things would be different.
Really be wary of who you discuss this with. I have 3 people and one is my IC. The one friend I do talk about this with would never tell me what to do and would support me if I always stayed with him and looked the other way...which I wouldn't, but that is the kind of friend she is. Only you have to live with your choices. We all "think we know'...but we don't.
We all backslide and lash out or talk about the R when we shouldn't, but I made a deal with myself day one never to threaten divorce. I know even today, I wouldn't divorce him. Even when I just set my boundries, of course my hope was that my H would do them, thus stay at home. But I knew I would rather him leave than watch him tell me he is done cheating, sleep with me, ML, and seem sneaky and still FB friends with OW. If he was still sleeping on the couch, being mean, it would be different. He was 75% in, 25% out. I needed to open the cage door.
I've had several discussions with H over the past 5 days, and it's amazing how things are when discussed calmly. He actually feels horrible about not being able to comply with my boundries. Wants to go figure out what is wrong with him, so we can move forward. Now, when I initially stated them 2 weeks ago angerly, emotionally, upset, sobbing...he could care less and I was a crazy, horrible wife and he said should just D my crazy butt. It will never work!!
I really feel for you! It's so hard. But I think the most important thing is "what is your goal?"
M: 9 yrs T: 13 yrs H:34 Me:35, S4, D2, S 7 months Dday 11/12-PA & multiple PAs Dbing 12/12 S 1/13 7/13 H moved back in basement. 8/13 #3 born 10/13 still cheating 10/13 He moves across country, I file for D