Happy to Be Me, I completely understand all the feelings you are having right now. I was having them myself not long ago.
You hit something on the head that was an issue for me, too: thinking about his reaction to setting boundaries. He will, no doubt, react and you have to be prepared for it. But it still needs to be done. For me, it took realizing I had nothing to lose and everything to gain by setting boundaries. A lot of it has to do with how you say things. Try to do it with an open heart but not emotional. He will see and feel that you are being genuine. He may try and put you on the defensive, but you don't need to defend your boundaries. They are there for you not him.
