How much of your time is spent really talking with and interacting with your children? How much of your anger at your husband do they see, hear, feel? What could you be doing to comfort them instead of going over and screwing him? How much of your daily focus is on him instead of on them? How much of YOUR life is about HIM?
I'm playing devil's advocate here because it is very easy at this stage to just do the bare minimum for the kids because it so hard just to survive yourself. You have your red flag and that is a wake-up call. You can't put it off on your husband because right now, he is worthless to you and the kids because he is acting like one himself. So you do it. You just do it. You cut out all the bullsh*t in your life, trim the fat, take out the trash, get rid of the riff-raff, whatever the case may be. You stop everything that isn't about the kids and ya'll getting through this TOGETHER (YOU AND THEM) and you stop it now. Blaming your husband, even though justified, isn't going to help your boy.