Although yesterday's conversation was painful, it was good because feelings were communicated and there was understanding between you. The same is true of what you are thinking. Talk to him about it. Make your feelings known, listen to his. Be clear so each of you knows what is going on. Silence is not your friend here. You and he have a lot to talk about if you are going to find a new balance for the relationship.