I'm sorry to keep returning to the topic of the mother stuff. But I believe you mentioned he has issues with a controlling and manipulative mother. Ask your therapist how this can impact his perception of you and your relationship. This can feed into his concern about agendas and trust in general and especially in relationships. I think you're going to have to get actually okay with letting go of this situation because that cannot be faked very well. Yet letting go is also hard to achieve quickly. So in the meantime, stay away from him as much as possible. When you interact, stay positive. But don't follow up the interaction with extremes of excitement (pressure) about where this positive interaction could lead. You are going to have to fake detachment until you become okay with detachment it seems like.
Make birthday plans that there is no way he could participate if he wanted to. If there's any other holidays coming up soon, make other plans for that too and let him know it. Otherwise he will be obsessing about that ahead of time and making an issue in his mind that you are trying to get him to do that event with you. You need to plan that he will be absent and not invite. Invite=pressure, agenda in his mind. Same with calling or returning every phone call. Start not calling ever and only return some phone calls with a text. You have to find a way to deal with the anxiety and get the courage to detach. I know it's easier said than done. I'm failing half the time! When I don't fail, I can see it takes the pressure off and taking the pressure off helps. People can't think clearly if they are locked in a battle in their minds with you over whether you are pressuring them.
