I'd say do what is good for you and protects you. If you don't your no good to yourself, children or husband. As a matter of fact you'll be in a bad place, resentful and angry.
What would that look like to you, to be in a position to protect yourself? Punishing him is not the goal, protecting you and your children is.
You can not change your past, but you can ruin a perfectly good present by worrying about the future.
