Why Standard Fraud Detection Fails: Balancing Precision and Recall To Protect Your Business

Highlights: 

• Beyond F1 Scores Alone: Relying on one-size-fits-all metrics like the F1 score can lead to suboptimal decision-making, as it fails to account for unique business costs like customer retention and reputation.
• The Need for a Customized Strategy: Optimal fraud detection requires balancing precision and recall based on specific industry priorities—such as protecting customer loyalty in competitive markets versus minimizing high-value losses in niche sectors.

In the first part of this series, we discussed, when used for fraud detection, how machine learning models often struggle with imbalanced data as legitimate transactions significantly outnumber fraudulent ones. This is why it’s important to use precision and recall metrics to strike a balance between accurately catching fraud while minimizing false positives for legitimate transactions.

In this final part of the series, we’ll continue the conversation by examining the positive impact of incorporating precision and recall into fraud detection strategies. 

Some fraud detection systems address precision and recall with the F1 score. 

The F1 score is calculated with the following formula: F1 score = 2 x (Precision x Recall) / (Precision + Recall).

The F1 score does provide a balanced measure of performance. 

However, it is a one-size-fits-all approach. The F1 score does not account for the unique business challenges and costs associated with false positives and false negatives. 

And that’s a problem.

If you ignore the unique features of your business and rely solely on the F1 score, you’ll have suboptimal decisions that harm your business long-term.  

A Better Approach

To make informed decisions, you need to consider a more holistic approach that takes into account the specific costs and challenges your business faces. This includes evaluating trade-offs between not just precision and recall but also factors like customer retention, reputation management, and potential financial losses.

By considering the broader context and aligning the threshold selection with your business objectives, you can develop a fraud detection system that not only maintains accuracy but also addresses the specific challenges of your industry.

To introduce a better, more profitable alternative to the F1 score, we’ll look at the risk management priorities for two different businesses. 

To create a level playing field for our comparison, we'll use the same set of example transactions for both cases. The common dataset comprises 2,300 transactions — including 300 fraudulent ones.

With identical recall, precision, and F1 score in both scenarios, we can direct our focus toward the specific challenges that each business faces when setting their threshold values.

Imagine a bustling pizza shop. 

Surviving as a restaurant in a highly competitive industry is challenging. 

The customer is always right, and the owner is accustomed to "eating" the cost of a pizza now and then to keep the clientele happy. Whether it's due to a cold pizza or bad customer service, they prefer to err on the side of keeping their customers satisfied. 

Dealing with fraudulent transactions adds another layer of complexity.

Because a primary concern for the pizza shop is to maintain the loyalty of its returning customers, incorrectly flagging a legitimate customer's transaction as fraudulent can have severe consequences. After all, a customer who gets declined for purchasing a pizza can easily walk away and buy from a competitor instead. 

In data science speak, this translates to a strong requirement for high-precision fraud detection. 

We can put some numbers to this by exploring the specific costs associated with different types of fraud detection errors. 

In this scenario, we assume an average transaction value of $50 with a profit of $25 per transaction. 

We estimate the average loss for false positives to be -$150. This takes into account not only the potential loss of a loyal customer but also the associated merchant risks and the negative impact on the shop's reputation. 

On the other hand, false negatives occur when the fraud detection system fails to identify a fraudulent transaction. For the pizza shop, the estimated cost of false negatives is -$60. This includes the financial loss associated with the fraudulent transaction itself and potential risks such as chargebacks or legal complications. 

While true positives and true negatives do not directly incur any costs in this simplified example, they still play significant roles. 

True positives represent the successful identification of fraudulent transactions, safeguarding the pizza shop from potential financial losses and ensuring the security of its operations. True negatives denote legitimate transactions correctly identified as non-fraudulent, contributing to the overall profitability of the pizza seller.

Using these costs, along with the standardized precision and recall values established for this test, we can build a net profit curve for the pizza business. 
The optimum threshold is the point at which the net profit is at a maximum (i.e. normalized net profit = 1.0). Given our proposed business impacts, this value is 0.83 for the pizza shop. Adjusting the threshold to be lower or higher increases the number of false negatives or false positives respectively, which decreases the overall profit for the business.

The peak of the profit curve is significantly skewed towards higher precision than the F1 score would lead us to believe. 

In fact, using the F1-predicted threshold of 0.5 would result in a normalized net profit of 0.94, representing a 6% loss in revenue for this business. 

This means that for the pizza shop, prioritizing precision over recall is crucial to maximizing revenue.

Now, picture a high-end ruby emporium — a business with a very different story. 

This industry operates with a smaller number of high-value transactions, catering to an elite group of customers. For the ruby seller, the consequences of allowing even a single fraudulent transaction to pass can be catastrophic, resulting in significant financial losses and irreparable damage to their reputation. 

Maximizing the identification of fraudulent transactions is paramount for mitigating risks, even if it means rejecting a higher number of legitimate customers. 

Unlike the pizza shop, where losing a customer can be particularly detrimental due to the presence of numerous competitors, the ruby emporium operates in a niche market with limited alternatives. A declined customer is more likely to reach out to the business directly to seek a resolution rather than immediately switching to a competitor. 

As before, let’s explore the specific costs associated with different types of errors in the ruby industry. 

In our simplified example, each transaction has an average value of $2,000 with a profit of $700. The average loss for false positives is -$100. This includes the costs associated with investigating flagged transactions and potential delays in processing legitimate transactions. 

On the other hand, the estimated cost of false negatives is -$2,500. This includes the financial loss incurred from the fraudulent transaction itself, potential business damages, and associated risks. The higher cost compared to the average transaction value reflects the potential impact on the seller's reputation and additional consequences. 

Similar to the pizza shop example, true positives and true negatives do not directly incur any costs in this simplified scenario. However, they are crucial for the successful identification of fraudulent and non-fraudulent transactions, respectively, ensuring the financial security and integrity of the ruby emporium.

Using these costs we can build a net profit curve for the ruby emporium. 
In this case, the optimum threshold is significantly skewed towards lower thresholds, reflecting the increased importance of high recall in the business. 

If the F1-predicted threshold of 0.5 was used in this case, the normalized net profit would be 96%, representing a 4% decrease in revenue. Given the price of the individual items sold, that lost revenue could quickly become game-changing for this business! 

The analysis indicates that ruby emporium's priority should be to maximize the identification of fraudulent transactions, even if it means rejecting some legitimate customers. By emphasizing recall, the ruby seller can effectively mitigate risks and protect their business from significant financial losses and reputational damage.

Customized vs. Standardized

The presented figure captures the essence of our analysis: 
Our analysis showcases the importance of a customized strategy. It is essential for businesses to go beyond the simplistic evaluation provided by the F1 score to strike the right balance between precision and recall. 

This approach enables businesses to:

• Safeguard customer trust

• Protect the business’s reputation

• Mitigate financial risks

• Optimize profitability 

In the dynamic world of business, fraud can be a lurking threat. The need for precision and recall in fraud detection is clear, but it's even more important to understand how these metrics relate to your specific industry. 

Here are some practical steps to take control of your fraud detection strategy. 

1. Assess your environment: Delve deep into your business landscape. Understand your industry, customer behavior, and the potential costs tied to fraud. 

2. Tailor your approach: Find the right balance between precision and recall based on your unique business priorities. Consider whether you operate in a highly competitive market or in a niche with limited alternatives. 

3. Analyze cost: Examine your transaction data to identify the financial implications of different types of errors, such as false positives and false negatives. 

4. Create a custom solution: Collaborate with experts to create a customized fraud detection solution that aligns perfectly with your business requirements. 

5. Implement and train: Ensure your team is well-versed in using the chosen fraud detection system and implement it effectively. 

6. Monitor regularly: Keep a close eye on your system's performance and be prepared to fine-tune your thresholds as your business evolves. 

7. Stay informed: Stay updated on the latest developments in fraud detection to adapt to emerging threats. 

With the right strategy, precision, and recall, you can protect your business from fraud while maintaining customer trust. 

And Equifax can help. 

Whether you have a restrictive recall requirement or prioritize perfect precision, we can customize our machine learning to align with your business’s goals. Reach out to our team today to explore how we can customize a fraud detection solution tailored to your business. 

Don't let fraud undermine your success. Take control today and secure your business's future.

Disclaimer: The examples and financial results presented are based on simulated scenarios and hypothetical cost assumptions. Actual business results and ROI may vary based on unique business factors and market conditions.