9.1 Background

A classifier algorithm is an algorithm that has been trained to find records that contain a specific characteristic of interest (value) for the outcome variable. 

A perfect classifier will accurately predict which records will be positive and which records will be negative.  But typically, classifiers are not perfect. This means the analyst must be able to evaluate how accurate the classifier is at predicting the characteristic of interest. This chapter describes common tools the analyst can use to evaluate the quality of the classifier.

Correct model evaluation is the key to making progress in predictive accuracy. Multiple algorithms are available for classification (to predict the category). Different combinations of input variables may be used. Finally, in many cases, there are a variety of settings that can be selected for each algorithm. So correct assessment of your results is essential.

After the machine learning algorithm has learned from the data, it predicts which instances (records) are positive and which instances are negative:

In this context, the term “positive” means the characteristic that your model is looking for. It does not mean that the outcome is positive for the population. For example, assume your model is trying to predict which people in the population have breast cancer. Since breast cancer is what the model is looking for, it is considered the positive outcome.