Quote: When I ask him what he wants from me he only says "tell me to F off and then I wouldn't be the bad guy" He also says he doesn't know why I am still around.
This kind of jumped out at me. If YOU'RE the one who didn't listen, didn't respond, etc., why is HE feeling like the bad guy? Why is he wondering why you're still around? Hmmmm... this strikes me as fishy. I would think that someone in his position would be full of righteous anger or at least indignation. He's trying to push you away at this point-- sorry, does not compute. I haven't read your whole story, so maybe I'm unaware of some important detail.
I hate to sound like NOP here, but are you sure he's got no one else in the picture?