Is it best to extract impacted wisdom teeth when things are peaceful?

I recently went for a scaling and was told that my wisdom tooth is positioned nicely. However, it doesn't hurt at all right now. The dentist is scaring me by saying that if I wait until I get older to have it extracted, recovery will be slow and it could damage the adjacent teeth... I am terrified of having a healthy tooth pulled out when it doesn't hurt. Would it be too foolish to just try to hold out until it starts hurting?