I think you should sit down with him and explain that he is old enough to make his own decisions on things like this
This is pretty much the basis from which I approached my convo with him a couple of weeks ago when I decided to stop trying to control when they saw H. My S15 in particular whilst wanting to be independent does not like to be completely left to his own decisions (hence why he goes w/o food if I am not around to cook it for him). This may sound cruel but this is one of those times that he has to make that decision for himself. If he goes to breakfast with H I will be bitterly disappointed BUT I'm not going to hold it against my son (as I may have done in the past). If however he ends up staying all day with H as he invariably does when they get together in this way then I might have to voice my unhappiness to them both.
Me 43 XH 45 M 2.7.88 Divorce 7.10.09 Kids D20,S17 & D15