Hey my friend. I always say you feel what you feel. And no feeling is wrong. It's all in what you do with them, right?
If you dont feel like talking to him, then dont. That is your right. As long as it is not as a ploy or to punish, it's a decision you've made for you at this time. And you're right, if you are not comfortable with what you might say, best not to say anything for now.
I am thinking he is feeling a small shift in you and he isnt liking it much. Too bad for him. You do what is best for you. You matter, too, my friend.
And sometimes it is ok to do something different.
Never apologize here, or to yourself for your thoughts and feelings. You are entitled to feel negative, so long as you dont live there, right?