Quote
to which I responded sorry we are busy

This is how you DB. Calm, measured, responses which feel wrong but are actually the best way forward.

I guarantee it made him stop and think “Wonder what she’s doing if she says she is too busy for me to come over.”

You’re the prize here. Start acting like it, and maybe he will realise it too.

Well done Pattnee.