Do I have to go to the dentist to have a wisdom tooth extracted even if it is growing straight?

Recently, while looking in the mirror, I discovered that a new wisdom tooth had grown in the back of my molar.
Fortunately, there was no pain at all, and it grew in nicely in a straight direction, just like the other teeth.
Everyone around me is making a fuss, saying that wisdom teeth will eventually rot if left alone and that I must get them extracted no matter what.
It doesn't hurt, and as long as you brush your teeth well, isn't it fine not to pull it out?
Is there anyone else like me who hasn't had their straight-growing wisdom teeth extracted and is just keeping them?