Gabriel, Not sure if your really contemplating sole custody or not, but I think of it like this. I know I am the more responsible, committed parent, but taking them away from her would do damage to the boys and I just couldnt do that. If there's abuse, neglect or drug abuse then yes one should seek custody. From what your telling me it doesnt sound like this is the case.
Dont take offense at this, but are you or you XW drawing your son into the situation? I ask this because I have a 5 year old boy also. He is not going through the same emotions your boy seems to be going through. Sometimes I think my 5S cant remember us ever being together, simply because he never talks about it, even though 1 year ago we were together. He did tell me one day that he wished Mommy and I didnt argue. Makes me sad to know thats what he remembers.
You have to give till it hurts, then turn around and give a little more. Expect that she will tell you lies or tell you she's going to do something then not do it. Expect it, it will happen. She's going to start dating, if she isn't already and she'll lie to you about that, either because she doesnt want to hurt you or just feels its none of your business.