People will say what they want about you but they don't know the whole, true truth to the sitch. You'll just have to ignore them totally. I've faced similar situations and I just completely ignored negative chatter that was going on around these parts. In due course, some people do eventually come around to you. Stay the course.
