Sometimes the hardest part of deciding what model you want to use, is deciding what metrics are important to you, as they can give you conflicting stories; especially if you’re unsure what they’re trying to tell you!

In this blog post I will go through the main metrics for Classification models, what they tell you and how to use them.

Classification

We should probably start off with what a classification problem is. In simple terms it is where we want a model to assign a categorical label to new data. Be that a yes or a no, or saying whether something is a cat a dog or a rat. There is no sliding scales or ambiguity we are looking for direct answers. To this end we want metrics which allow us to assess how well out model can apply these labels. I will discuss what I deem to be the 3 main metrics plus one graphic which are essential metrics for classification. These are Accuracy,Recall,Precision,F1 and the Confusion Matrix. For all examples below we will be presume we are performing a binary classification (vegetarian or not), as this makes explaining easier. Furthermore, if truth be told when we have more than two classes to classify, the metrics work as an average of several binary classifications (average of the metric values for vegetarian or not, Meat eater or not, Vegan or not for example).

It is useful before we start to define four terms (the two classes being vegetarian or not vegetarian) and understand that with these metrics we always need to think of context, what do we want to know? To this end we shall presume we want to predict if someone is vegetarian so that will be our positive :

• True Positive: an instance (person) that has been correctly predicted as vegetarian
• True Negative: an instance (person) that has been correctly predicted as not vegetarian
• False Positive: an instance (person) that has been incorrectly predicted as vegetarian
• False Negative: an instance (person) that has been incorrectly predicted as not vegetarian

Accuracy

This is the easiest and most straight forward metric to understand and in essence is the foundation of Recall and Precision. It is simply how many times did the model predict correctly as a percentage of all predictions? This can be summed up with the below equation:

Accuracy = Number of Correct Predictions/Number of Predictions

This will always be a number between 0 and 100 (the same is true for Precision and Recall). The downside of being the most simple to understand, is that it doesn’t really provide much context to the predictive powers of the model and can be misleading. For example, it is stated by The Vegetarian Society states that in 2018/19 4.5% of the UK population was vegetarian (https://vegsoc.org/facts-and-figures/#:~:text=Number%20of%20UK%20vegetarians%201%204.5%25%20of%20the,age%20of%2018%20months%29%20in%20the%20UK.%20%5B2%5D), if we wanted a model that predict if someone was vegetarian or not in 2018 and the model predicted everyone as non vegetarian and then used it to classify 100 people, it would give an impressive 95% accuracy. But it didn’t predict a single vegetarian as vegetarian, this is where Recall and Precision can be useful.

Recall

Recall tells us the number of times we correctly predicted a positive instance out of all the positives (how many times a person was vegetarian and we predicted them as a vegetarian):

Recall = Number of True Positives/ Number of True Positives         and False Negatives

This would give us a recall of 0.0, indicating awful model performance and rightly so as it didn’t classify a single person correctly as vegetarian. So lets change out model to predict everyone as a vegetarian, our accuracy goes down to 4.5%, but our recall goes up to 1.0 which means we have an amazing classifier by that metric. But we’ve labelled 95.5% of the people incorrectly. This is why we have Precision which is usually used in tandem with Recall to get a holistic view of Classification models.

Precision

Precision tells us how accurate we were in classifying a data point as positive instance. If we said a person was vegetarian, how many times were they actually a vegetarian instead of not a vegetarian and we just incorrectly identified them we classified as a vegetarian:

Precision = Number of True Positives/ Number of True Positives and False Positives

If we assume the model has classified everyone as vegetarians again, we would have a Precision of 0.045 which is an awful score. It is vital that when creating a model, we look for not only a high accuracy but also as best a trade off between Recall and Precision as can be achieved, although there are some exceptions to this which we will discuss below.

F1 Score

It is obvious that we would want to maximize both Recall and Precision for a model, but usually when a model improves at one it comes at the cost of a reduction in the other, hence why it can be useful when trying to find an optimal blend of the two to use something called F1 which is defined below:

F1 = 2 * ((Precision * Recall)/ (Precision + Recall))

It is worthwhile noting that because Precision and Recall are always between 0 and 1 Precision + Recall will always be more than Precision * Recall, hence why this works. Further more, the reason why we take a harmonic mean over a simple average, is that it enhances the effect of extreme values. For example if we have a precision of 1.0 and a recall of 0.0 then a simple average of 0.5 but an F1 of 0.0. The bottom line is that if we want a model with a balance of precision and recall, of which a good proportion of the time we do, then we want to maximize F1 score.

When to use Accuracy, Precision and Recall

The imaginary vegetarian dataset I gave earlier ( 9 vegetarians out of 200 people) is an imbalanced dataset; it has significantly more instance of one class over another. Accuracy is misleading when used on imbalanced datasets and so should be used in conjunction with Recall and Precision to assess our models. This allows you to avoid scenarios stated above. This doesn’t mean you should only use Recall and Precision on imbalanced data as they have their specific use cases there as well. As stated above in the section on F-Score, on the whole we want to find a balance, but there are cases where we want to optimize one over the other.

Optimize Recall

We want to optimize for recall when we want to minimize the number of false negatives our model predicts. A common use case for this is medical image diagnostic models, in particular cancer diagnosis. We definitely want to prioritize keep false negatives to a minimum here, as we would rather tell someone they have cancer when they don’t, then the inverse which would leave someone without treatment.

Optimize Precision

We optimize for Precision when we want to minimize the number of false positives our model predicts. This is common for recommendation systems, like suggested media on YouTube or Netflix. We would rather not recommend something you won’t like then, not show you something you may like, as this recommends as much content that you do like as possible.

Confusion Matrix

Personally, I find the confusion matrix the most powerful ‘metric’ to be used, with the addition of being the easiest to understand. It’s not strictly a metric though, and is more a graphical representation of the other metrics discussed above. An example of a confusion matrix can be seen below:

We have counts of the ‘fundamental’ metrics we discussed earlier (true positive etc.). When assessing a confusion matrix we want to see large numbers across the diagonal, as this is where we have correctly predicted instances which is our accuracy (sometimes this is accompanied by a heatmap where more extreme shades of color dictate the highest and lowest values). Another useful feature is that we have Recall and Precision, being a product of the bottom left ‘L’ of the table and the inverse of this respectively. It’s the way that a confusion matrix gives a an easy to digest summary of the other metrics that make it a really powerful tool for assessing your models.