If it is YOUR time, you just tell him you aren't interested.

That's the ? here is I don't know if you mean he is taking them for his time and invited YOU, or is HE inviting HIMSELF into YOUR time with them?