I so used to struggle with this one in the past. Okay, I'll admit that people I know and enjoy being around -- their opinion does matter to me, but those that only know of me through gossip I couldn't care less. Just ask them over at whaleshares -- apparently I'm a scammer over there haha. But then half of that is my fault. If I took a leaf out of your book I'd just have left them alone instead of stirring the pot.
Self image is a real thing. Those that matter, yep, I care a lot. Those that don't? I may as well be a two headed monster to them for all I care.
I think the best thing you can do for yourself is just keep bettering yourself. Some people, you know? You could cure world hunger, find the cure for aids, and significantly uplift the living situation of half the world -- and someone would still find fault.
Can't please everyone! :)
I actually find that it is often the opinions of people close to me that frustrate me the most lol - because yes, they DO matter and sometimes I wish their opinions didn't affect me as much as they do! (My brothers as an example).
But yes, I have learnt to care a lot less about the rest... it is liberating haha!