In my humble opinion it doesn't matter If you think you're nice or being mean. I was kind and friendly when refusing to give answers to my W and she still said I was a jerk & mean. Honestly I think it has to do with you just not giving them the answer or information they're looking for, not the kindness or demeanor. They are going to see you as they do regardless of you how truly behave. Do you, It's about you now. That lesson took me some months. It's only about you now!
