You're not letting HIM go - you're letting go... it's a big difference. You have to let him go - even when you're with him he is never Yours, in a way. (I'm saying this from my own learning experience - so, trust me, I know how it feels) -

But it will do you - Him - your D - and your R SOOOO much good. There is no other way. I believe you can do this!