Yes, I can see the harm. You might escalate to ringing his doorbell and running away.