That's easy. The answer's as obvious as the nose on your face.

Because you're a better person than he is. Unlike The Bloke, you can't simply cut another human being out.

To me, the only thing problematic is that it's holding you back.

Why not do both? Care for him AND move forward?