You could compromise on the time AND offer him some extra time with S later in the week to make up for you taking some of his to go elsewhere. That way he is not denied access to time with S. but you can still have your afternoon with S.