It doesn't make you crazy. But it ("doing the friend thing") also doesn't work, in my experience.
The reason I asked you about how it went down -- and the reason I was able to predict it so accurately -- is because I know how us humans (especially us MEN), tick. We are, when all is said and done, basically "path-of-least-resistance" creatures, and we will largely do what we are ALLOWED to do, and won't make difficult choices until we are forced to. Another poster, Robx, calls this "crisis" -- that a walkaway spouse won't react until they are faced with crisis (this can be the credible threat of divorce or separation, or exposure of their affair, or the chance of losing their spouse).
See, I have learned that our spouses -- heck, really EVERYONE IN OUR LIVES -- treat us the way we have TAUGHT them to treat us. You have taught your husband how to treat you over time.
Does this excuse his infidelity? Nope -- no way. That is HIS choice, and he'll be held accountable to it, both here and in the afterlife, I believe. All YOU can do is decide what your own Boundaries of Personal Integrity are, and learn to communicate them and enforce them.
What you are communicating now to your husband, whether it's your intention or not, is that you are needy, desperate, fearful, and you will put up with his behavior -- including his infidelity -- so long as he throws you a bone (sex, jewelry) every now and again.