---
title: "How to rank fraud detection models using custom cost metrics"
description: "Accuracy is misleading for fraud detection because a model labeling everything legitimate can score 96% while catching zero fraud when fraud is only 4% of transactions. The post runs a hand-built Python grid search across algorithm, hyperparameters, feature set, class weight, and decision threshold, ranking by a custom cost metric balancing fraud savings against a US 6 analyst cost per flag. On a 480-transaction test split from a 2,400-transaction dataset with 4% fraud, the winning logistic regression caught 15 of 19 frauds, saving US 3,578 over a baseline that caught none. The winner’s evidence is passed to a local 1-billion-parameter LLM that drafts analyst notes, keeping the LLM downstream of the decision to avoid hallucination. The same workflow scales to Red Hat OpenShift AI by swapping local components for enterprise services like AutoML with AutoGluon and model serving via vLLM."
---

# How to rank fraud detection models using custom cost metrics

[Red Hat](https://yomu.fyi/company/red-hat) · Isaac Tigges, Neha Malepati · Sep 28, 2026

**Type:** Tutorial

## Summary

Accuracy is misleading for fraud detection because a model labeling everything legitimate can score 96% while catching zero fraud when fraud is only 4% of transactions. The post runs a hand-built Python grid search across algorithm, hyperparameters, feature set, class weight, and decision threshold, ranking by a custom cost metric balancing fraud savings against a US 6 analyst cost per flag. On a 480-transaction test split from a 2,400-transaction dataset with 4% fraud, the winning logistic regression caught 15 of 19 frauds, saving US 3,578 over a baseline that caught none. The winner’s evidence is passed to a local 1-billion-parameter LLM that drafts analyst notes, keeping the LLM downstream of the decision to avoid hallucination. The same workflow scales to Red Hat OpenShift AI by swapping local components for enterprise services like AutoML with AutoGluon and model serving via vLLM.

## Context

Fraud detection models default to accuracy as a metric, but on imbalanced data where fraud is rare (e.g., 4%), a model labeling everything legitimate achieves 96% accuracy while catching zero fraud. The post targets this pitfall by demonstrating a custom cost metric and systematic model search.

## Approach / What changed

Hand-built Python grid search that sweeps across algorithm choice (logistic regression, decision trees, random forests), hyperparameters, feature sets (from 3 to 12 columns), class weighting, and decision thresholds. Candidates are ranked by a custom cost function: each caught fraud saves its transaction amount, each flag costs US$6 for analyst review. The winner’s evidence is passed to a local 1-billion-parameter LLM via Ollama for drafting analyst notes. The workflow can scale to Red Hat OpenShift AI using AutoML (AutoGluon on Kubeflow Pipelines) and vLLM for model serving.

## Takeaways

- Accuracy is misleading for imbalanced fraud detection because the majority class dominates; a custom cost metric that prices both missed fraud and false alarms is necessary.
- A grid search over algorithm, hyperparameters, feature set, class weight, and threshold can find a better model than shipping defaults, as the winning logistic regression used all 12 features and 4x class weighting.
- Keep LLMs downstream of the classifier: use them to explain decisions based on evidence from the model, not to make the classification, to avoid hallucination.

## FAQ

### Why is accuracy misleading for fraud detection?

When fraud is rare, e.g., 4% of transactions, a model labeling everything legitimate achieves 96% accuracy while catching zero fraud. Accuracy does not expose this failure because it is dominated by the majority class.

### What metric does the post recommend instead of accuracy?

A custom cost metric that balances savings from caught fraud against the cost of false positives. In the demo, each caught fraud saves its transaction amount and each flag costs US$6 for analyst review, giving a single dollar-value score for each candidate.

### How does the LLM fit into the fraud detection workflow?

The LLM is used downstream of the classifier decision. It receives the risk score and evidence from the winning model and drafts an analyst note, rather than making the classification itself, which avoids hallucination.

**Tags:** [Fraud Detection](https://yomu.fyi/topic/fraud-detection), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Python](https://yomu.fyi/topic/python)

- Source: [Red Hat](https://developers.redhat.com/articles/2026/09/28/how-rank-fraud-detection-models-using-custom-cost-metrics)
- Source URL: https://developers.redhat.com/articles/2026/09/28/how-rank-fraud-detection-models-using-custom-cost-metrics
- Ingested by Yomu: 2026-09-28T15:00:39.896Z

[Read original post](https://developers.redhat.com/articles/2026/09/28/how-rank-fraud-detection-models-using-custom-cost-metrics)