To someone who is really looking for a reason to be offended, saying something about you staying away so he can have as much time as he can could be viewed as you giving 'permission' for him to have extra time, or that you're really being put out by staying away. idk it's probably nothing, but clean seems best.
