Your advice and your answers are all in your post...
You are making decisions in your mind and that is great. I mean this in complete respect, are you asking us for permission to do what you feel is "right" and in your best interest and the best interest of the kids?
What are you committed to? Your W's sense of comfort or your responsibility to you and your kids?
You know how the saying goes. You cannot help anyone else until you first help yourself.
Like QOS said, good boundary. How else can that show up in your life?
