There are a million ways to break one heart, aren't there?
You have a right to look after YOUR best interests. He has certainly thought about nothing but his own. Don't let him guilt you into second guessing yourself. He's just trying to manipulate you into feeling like you did something wrong and that it's all your fault. We know this game. Don't believe it. Stick to what you know.
