I think you need to stick to your guns when it comes to finances and what you want. Who cares if he gets mad at some point you have to draw a line in the sand. Get some respect back from him by not letting him bully you and make you feel bad by saying he knew you would leave him high and dry what about what he has done to you.
I know you want to be nice and not make him mad but at some point if he keeps pushing you will have to.
