This guy wants sex of a kind you don't want to give him. He is uncommunicative. He is emotionally abusive. He is emotionally unavailable. He is manipulative. He is depressed. He is someone you don't trust. He is someone you have known only a short while, and you've never had a good R with him.
What exactly do you get out of this R? You get a chance to relive old crap from your XM and from your childhood. That is why you keep going back. Nothing in what you right about your XF suggests that you love him. In one way, he is right, it is all about you and all about your unresolved problems that you get to continue to cycle through with him. It is not about him except that if he didn't recreate those problems for you, you would not be drawn to him.
You acknowledge that he is abusive, yet you continue to return to him. This is simple self abuse. And, contrary to what you have indicated here, in the past you have indicated major problems when you were growing up. The co-dependent behavior here is extreme, unhealthy, and very obvious.
The part that is really beating you down is returning to him to beat yourself up over and over. Stop it and get help, professional help. Quit letting your pride and ego get in the way of living a decent life.