You didnt fail. It takes 2. You might have enabled his behavior by not setting and enforcing boundaries, but you didnt fail. It's goodtimes and badtimes for BOTH partners. Not just one. You H not following through with your original agreement is grounds for an annulment. Not to mention all the other sick stuff your H has done that would make a priest kick out a stain glass window.
Gee. It's a wonder why you were craving attention and RESPECT from another man. Dont feel bad. It's understandable. Sleeping with him originally was not a good decision but liking the continued attention is understood.
Sounds like you got your answer from H. Why continue working on a R/M that is not WANTED by both parties? That is the question that many of us have had or are in the process of dealing with on this site.
Be strong and do what you know you have to do. In your case, it is true what your friends and family have been telling you. YOU do deserve better. Many WAS's hear that to justify their cheating or hurting, which in most cases is just an excuse to STOP doing the WORK. You on the other hand, have been trying to do the WORK.