If something is being said about you, it is your business, especially if it's slanderous. Having said that, I mostly don't care what people say about me, except maybe friends and family.
Here's my motto!!
I do agree with you. What other people think of you doesn't matter a hoot.. However, if they do more than think it.. if they say it or spread negative stuff around about you then yes, that is a different matter.