You are correct...he has to deal with his impotency issues, you can be supportive, but it is up to him.
Stay firm, stand your ground, do a 180 from your past behavior and you will fell your self-esteem greatly improve too. You'll become more confident the more you stand up for yourself and stop allowing him to behave this way with you.