Loading post…
Red HatIsaac Tigges, Neha MalepatiTutorial
How to rank fraud detection models using custom cost metrics
Summary
Accuracy is misleading for fraud detection because a model labeling everything legitimate can score 96% while catching zero fraud when fraud is only 4% of transactions. The post runs a hand-built Python grid search across algorithm, hyperparameters, feature set, class weight, and decision threshold, ranking by a custom cost metric balancing fraud savings against a US 6 analyst cost per flag. On a 480-transaction test split from a 2,400-transaction dataset with 4% fraud, the winning logistic regression caught 15 of 19 frauds, saving US 3,578 over a baseline that caught none. The winner’s evidence is passed to a local 1-billion-parameter LLM that drafts analyst notes, keeping the LLM downstream of the decision to avoid hallucination. The same workflow scales to Red Hat OpenShift AI by swapping local components for enterprise services like AutoML with AutoGluon and model serving via vLLM.
Context
Fraud detection models default to accuracy as a metric, but on imbalanced data where fraud is rare (e.g., 4%), a model labeling everything legitimate achieves 96% accuracy while catching zero fraud. The post targets this pitfall by demonstrating a custom cost metric and systematic model search.
Approach / What changed
Hand-built Python grid search that sweeps across algorithm choice (logistic regression, decision trees, random forests), hyperparameters, feature sets (from 3 to 12 columns), class weighting, and decision thresholds. Candidates are ranked by a custom cost function: each caught fraud saves its transaction amount, each flag costs US$6 for analyst review. The winner’s evidence is passed to a local 1-billion-parameter LLM via Ollama for drafting analyst notes. The workflow can scale to Red Hat OpenShift AI using AutoML (AutoGluon on Kubeflow Pipelines) and vLLM for model serving.
Takeaways
Accuracy is misleading for imbalanced fraud detection because the majority class dominates; a custom cost metric that prices both missed fraud and false alarms is necessary.
A grid search over algorithm, hyperparameters, feature set, class weight, and threshold can find a better model than shipping defaults, as the winning logistic regression used all 12 features and 4x class weighting.
Keep LLMs downstream of the classifier: use them to explain decisions based on evidence from the model, not to make the classification, to avoid hallucination.
FAQ
Why is accuracy misleading for fraud detection?
When fraud is rare, e.g., 4% of transactions, a model labeling everything legitimate achieves 96% accuracy while catching zero fraud. Accuracy does not expose this failure because it is dominated by the majority class.
What metric does the post recommend instead of accuracy?
A custom cost metric that balances savings from caught fraud against the cost of false positives. In the demo, each caught fraud saves its transaction amount and each flag costs US$6 for analyst review, giving a single dollar-value score for each candidate.
How does the LLM fit into the fraud detection workflow?
The LLM is used downstream of the classifier decision. It receives the risk score and evidence from the winning model and drafts an analyst note, rather than making the classification itself, which avoids hallucination.
Source and freshness
- Source
- developers.redhat.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.