In terms of your safe-harbour idea, its about being the light-house for your family and anyone that you choose to be in your circle.
You are just doing what you always did - nothing. Letting your WW tell you where you can go, where you can't go. Sandi said it best, once a H gives his balls to his W, then he cannot get them back. He needs to grow a new pair.
So if you really want to go, then go alone. Face the loneliness. If she is willing to let your D go with you, then that's also ok.
You need to do what makes you happy.
Just cos things are going right, doesn't mean that they were always wrong.