I'm sorry those thoughts intruded on your activity. It sucks that those feelings can just pop up anywhere at anytime. For me, it's the lack of control over those thoughts that's driving me crazy right now.
You should NOT feel guilty about thinking about D. I agree with naej, everyone's situation is different (while eerily similar) and you need to do what's best for you. I'm struggling with the boundaries issue myself and am also seriously considering leaving. You are not alone in those thoughts.
Keep up the good work, you are doing whatever it takes to be good to yourself!
If you love somebody, set them free. http://tinyurl.com/2empx2g