I worried about about my son being from a broken home as well. But living with my ex was a lot more broken then my current situation. My son is not witnessing a really unhealthy dynamic and poor role model on a daily basis.
My ex is also a disney dad or maybe just a really good babysitter. While I still get annoyed, I am happy at the very unique bond I have with my son. It is a very special bond single parents seem to have with their children. I love that I get to make the decisions I feel are best for son without any input or argument as well.
Letting my ex go and focusing on how I was going to be the parent my son needed is empowering. I do not think my son is suffering for it. He would have suffered by witnessing and copying a very selfish man and how he treated me. Now, he will only see me with a partner that is really a partner.