I wish I was quicker to 'think on my feet' when the sitch arises. I'm always afraid I'm going to say or do the wrong thing.
B/c you are worried what HE will say/do in response. The questions is whether a given comment/action is right FOR YOU. To he!! with what H might think or do. As long as you are doing the right thing for you, his reaction doesn't matter. Work on detaching and this will come.
