11.4 Advantages and disadvantages of KNN

Like all predictive algorithms, KNN has some advantages and some disadvantages.

Advantages of KNN

Advantages of the KNN are the following. First, the algorithm is simple. Because it uses simple comparisons to find similar records in the training data, KNN is sometimes very effective at making good predictions.

Second, KNN can be effective at capturing complex interactions among variables without having to define a separable statistical model (no coefficients and weights).

Third, use of KNN requires no assumptions about the data. This is different than some other algorithms. For example, we learned that MLR works best when the data is normally distributed, when the relationships between predictors and outcome variables are approximately linear, and so forth. KNN can be effective without requiring such assumptions be met.

Disadvantages of KNN

A disadvantage of the KNN algorithm is that it does not create a generalized separable model. There is no summary equations or trees that can be produced by the training process that can be quickly applied to new records. Instead, KNN simply uses the training data itself to perform prediction.

KNN provides no insight about the relative importance of each predictor.

Another significant disadvantage of KNN, is that the algorithm is computationally intensive. Computational effort of the algorithm increases greatly as more predictors, p, are considered and when the number of training records increase. The algorithm must compute the distance and find the nearest neighbors in all the training data for each prediction. This takes time and can be especially slow when there are a large number of training records that must be examined for each record to be predicted.

Including too many predictors will also make all instances in the training data far away from the record to be predicted. This is because expected distance to nearest neighbor increases with p. When the number of predictors is high, all records end up "far away" from each other because you are squaring and summing so many differences.

In situations where the number of records cannot be saved in active memory, the time costs of having to find many records in secondary storage such as hard drives can make the process slow because the mechanical process of spinning a hard drive to find the training records adds to the length of time required by the algorithm.

Because of this, KNN is best used when the number of training records is not great, when the number of predictors is not great, and when the training data can be saved in active computer memory.