KC, in my experience the reason you also feel bad after contacting him is that the contact doesn't end up what you hope it would be.
I know this from experience. With my pre-married, long time on again, off again R. I would reach out hoping she'd say things like "I miss you." "I cant' live without you!" "We can we be together?" And when it would invariably not be that, I would be bummed that I broke NC.
I should also say that I also felt that way after she initiated contact. After my initial excitement, when the contact itself didn't live up to my expectations, then I felt like crap again. I got to the point where I avoided all of her initiated contact. This was pre cellphone days, but I would let all calls (before caller ID) go to answering machine to screen them.
My life was so much better when I was NC and wasn't responding to her initiated contact.
M(53), W(54),D(19) M-23, T-25 Bomb Drop - Dec.23, 2017 Ring and Piecing since March 2018