There is nothing wrong with you. However, your H's actions have, as you admit, lessened your self-esteem. Can you get to an IC who maybe specializes in healing from infidelity? Maybe re-read DR's section on that. Again. Think about your boundaries. How long will you stand for his behavior?
I can't remember, and I'm on slow internet, so I apologize if this has been addressed upthread - have you consulted an L just for information on your rights and possible steps you could take to protect yourself and your kids?