Not sure what to tell you. In my case, XH started instituting the physical part of the R on his terms....sleepover nights, etc. I too am afraid to say NO once in a while because I am afraid to lose him.
I just keep thinking....he is not mine to lose....he did D me. He is my XH.....not sure - just try giving her some space. I wish mine would want to work on the R and start over....but, not sure if he will ever be.