This is the biggest obstacle to overcome. Fear. So I don't in any way blame you for feeling so trapped. You're the only one who can get you out of that trap though.
If you are going to be by yourself or with another, you don't want to live as a prisoner of fear.
Think about it this way: you are in a position where you can take a calculated stand. You don't need us to tell you that if you don't act, you have in a sense acted, and this will have its own consequences. If you want to act, and you know what you want to say - but can't because of your fear, then yes, I think you owe it to yourself to communicate this. Maybe you need to write a letter just so you can say everything without your fear preventing you from getting everything you want to say out. Now, I'm not sure if by confrontation you mean that you are going to give him an ultimatum - because I don't really think that that's wise unless you have made sure that you can be on your own. You need to be able to follow through - and even then it can backfire.
I'm not sure if I've helped you any here. I wish you the best.
“No problem can be solved from the same level of consciousness that created it. ” – Albert Einstein