Maybe we shouldn't take it personally, but unfortunately I think most of us do. I think this often becomes a downward spiral. Wife doesn't want to, we see it as a rejection and take it personally. Pull back a little which causes them to pull back a little. Etc, etc.
Sex is such a taboo subject and often difficult to talk about, so it falls by the wayside and lurks in the shadows. I know in my situation, I rarely got upset about anything and wouldn't argue. If something irritated me, I would just push it to the side and get over it. Sex was one thing I would make a point of bringing up when things weren't going right. She would get mad at me for always bringing this up. I think she ended up viewing this as sex being the only thing that I cared about. Communication is key.