Kids need their Daddies, especially sons!!!! Don't ever let your W's attitude and behavior push you into thinking otherwise. Your sons might choose to be with her now and then, and I think its great you are open to that. Maybe you can have a little rule with them, about when the decision needs to be made. And maybe tuck in a "Are you sure, because we were going to do ____ when we got to our house?" I wouldn't do that in front of W, lol, and don't turn it into bribery, but you will find a good balance.
Fathers are so very important. Especially dedicated, honest, open, caring, trustworthy, and loving fathers like yourself. Your kids would be devastated if you dropped out of their lives. Truly.