Precision & Recall: When Conventional Fraud Metrics Fall Short
Highlights:
- Addressing Imbalanced Data: In fraud detection, imbalanced datasets—where safe transactions vastly outnumber fraudulent ones—can bias machine learning models toward classifying everything as safe, causing businesses to overlook critical fraud risks.
- Optimizing Strategy with Precision & Recall: By utilizing precision and recall instead of one-size-fits-all metrics, businesses can strike a profitable balance that minimizes false positives (incorrectly flagging legitimate transactions) while maximizing the capture of actual fraud.
It’s pretty easy to grab a cup of flour from a basket but a lot harder to find the salt!
This skewed data means you could label every transaction as safe, and you’d be correct more than 90% of the time. On the surface, that might seem fantastic.
But here's the catch: the goal is not to miss those rare grains of salt — the fraudulent transactions.
How Precision and Recall Help Overcome Imbalanced Data
The challenge is to strike a balance. The aim is to accurately catch the fraudulent transactions — those elusive grains of salt — while also ensuring there aren't too many genuine transactions mistakenly classified as fraudulent — effectively confusing flour for salt.
This balance is where precision and recall come into play, acting as our guides through the world of imbalanced fraud datasets. Machine learning is a powerful tool in the ever-evolving battle against fraud, but its effectiveness is contingent on a delicate balancing act. You want to accept as many safe transactions as possible while simultaneously declining as much fraud as possible.
A lot of businesses turn to conventional fraud metrics to set that all-important approval-decline threshold. But a one-size-fits-all approach is not always the most profitable.
Let’s look at how a generic fraud strategy comes up short when compared to a customized option.
How Imbalanced Data Impacts Machine Learning
When it comes to fraud detection, machine learning and data go hand-in-hand. The value of each individual component is dependent on the quality of the other.
And one factor that can negatively impact machine learning outcomes is imbalanced data.
Imbalanced data means that there are far more legitimate, safe transactions than fraudulent ones. This causes a machine learning model to lean heavily toward the majority — which is the safe class.
To put it simply, it's like having a big basket of flour with just a tiny pinch of salt hidden in it.
In simple terms, the two can be defined in the following way:
-
Precision helps ensure safe transactions are not mistakenly classified as fraudulent. It's like ensuring that when you try to pick out the grains of salt from the basket, they are actually salt and not flour.
-
Recall ensures that actual instances of fraud are not missed amongst the safe transactions. It's akin to ensuring you don't miss any grains of salt in the flour.
Before we look at the calculations for each measurement, let’s first understand the variables involved and their place in our flour vs. salt analogy:
| TRUE POSITIVE | A correctly identified fraudulent transaction | Salt identified as salt |
| TRUE NEGATIVE | A correctly identified legitimate transaction | Flour identified as flour |
| FALSE POSITIVE | A legitimate transaction incorrectly identified as fraudulent | Flour confused for salt |
| FALSE NEGATIVE | A fraudulent transaction incorrectly identified as legitimate | Salt overlooked amongst the flour |
Now, let’s look at the calculations for precision and recall — and how they impact the salt vs. flour analysis.
Precision is calculated using the following formula: Precision = True positives / (True positives + False positives).
Precision tells you the percentage of correctly identified fraud cases out of all the cases predicted as fraud. A high precision value indicates a low rate of false positives, reducing the chances of wrongly flagging legitimate transactions as fraudulent.
Recall is calculated using the following formula: Recall = True positives / (True positives + False negatives).
Recall informs you about the percentage of correctly identified fraud cases out of all the actual fraud cases. A high recall value indicates a low rate of false negatives — meaning you capture as many fraudulent transactions as possible.
Considering the specific business challenges and costs associated with each type of error, it’s essential for a fraud detection system to strike the right balance between precision and recall. And there are two ways to do that — with one approach being more profitable than the other.