---
title: "Fine-tuning LLMs for risk assessment"
description: "Stripe makes thousands of near real-time risk decisions every second, balancing fraud detection against false positives that disrupt legitimate businesses. Zero-shot prompting achieved 0.487 precision, domain-guided prompting 0.706, and human reviewers 0.790 at the same recall. The team applied supervised fine-tuning (SFT) to an open-weight 4B-parameter LLM, serializing features like card payments and account activity into text records, training it on labels, and deriving a continuous fraud score from the next-token probabilities normalized over two label tokens. SFT increased average precision by approximately 24% over domain-guided prompting and 70% over zero-shot prompting, and consistently outperformed XGBoost across multiple risk tasks. Linear probing of hidden states showed that SFT made benign signals more linearly decodable, with ROC-AUC improving from 0.583 to 0.642, indicating the model better recognizes legitimate business activity and reduces false positives."
---

# Fine-tuning LLMs for risk assessment

[Stripe](https://yomu.fyi/company/stripe) · Hangzhi Guo · Oct 9, 2026

**Type:** Problem & solution

## Summary

Stripe makes thousands of near real-time risk decisions every second, balancing fraud detection against false positives that disrupt legitimate businesses. Zero-shot prompting achieved 0.487 precision, domain-guided prompting 0.706, and human reviewers 0.790 at the same recall. The team applied supervised fine-tuning (SFT) to an open-weight 4B-parameter LLM, serializing features like card payments and account activity into text records, training it on labels, and deriving a continuous fraud score from the next-token probabilities normalized over two label tokens. SFT increased average precision by approximately 24% over domain-guided prompting and 70% over zero-shot prompting, and consistently outperformed XGBoost across multiple risk tasks. Linear probing of hidden states showed that SFT made benign signals more linearly decodable, with ROC-AUC improving from 0.583 to 0.642, indicating the model better recognizes legitimate business activity and reduces false positives.

## Context

Stripe processes trillions of dollars annually and makes thousands of risk decisions per second in near real-time, requiring high precision to prevent fraud while avoiding disruption to legitimate businesses. LLMs bring world knowledge and can integrate heterogeneous features through serialization, but they are word predictors not designed for rigorous statistical analysis. Initial zero-shot and domain-guided prompting both underperformed human review, motivating supervised fine-tuning.

## Approach / What changed

The team applied supervised fine-tuning (SFT) to an open-weight 4B-parameter LLM. Business features including card payments, money movement, and account activity were serialized into text-based records. The model was trained to predict the review label (fraud or non-fraud) from each record. A continuous fraud score was derived by normalizing the next-token probabilities for the two label tokens. Linear probing was applied to hidden states to analyze how SFT changed representations of risk and benign signals.

## Takeaways

- Supervised fine-tuning of a 4B-parameter LLM on risk review labels increased average precision by approximately 24% over domain-guided prompting and 70% over zero-shot prompting, and outperformed XGBoost across multiple tasks.
- SFT made benign signals more linearly decodable from the model’s hidden representations (ROC-AUC improved from 0.583 to 0.642), indicating the model learned to better recognize legitimate business activity.
- A continuous fraud score is derived by normalizing the next-token probabilities for the fraud and non-fraud label tokens, enabling ranking and threshold selection rather than using the predicted label directly.

## FAQ

### How does the model produce a continuous score if trained on discrete labels?

The model outputs the label token as the prediction, but the fraud score is computed from the next-token distribution. Log probabilities of the fraud and non-fraud tokens are normalized to obtain a continuous score between 0 and 1, used for ranking accounts and setting thresholds.

### Does fine-tuning teach the model new concepts beyond output adjustment?

Yes, linear probing experiments showed that SFT makes benign signals more linearly decodable from hidden states, with ROC-AUC improving from 0.583 to 0.642. This indicates the model learns to recognize characteristics of legitimate businesses, not just adjust output probabilities.

### How are heterogeneous features incorporated into the LLM?

Features such as numerical, categorical, text, and time-series data are serialized into a text-based business record. This allows the LLM to reason jointly over all feature types without requiring feature engineering or type-specific encodings.

**Tags:** [Fine-Tuning](https://yomu.fyi/topic/fine-tuning), [Fraud Detection](https://yomu.fyi/topic/fraud-detection), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Risk Modeling](https://yomu.fyi/topic/risk-modeling)

- Source: [Stripe](https://stripe.dev/blog/training-llms-for-risk-assessment)
- Source URL: https://stripe.dev/blog/training-llms-for-risk-assessment
- Ingested by Yomu: 2026-10-12T18:00:18.317Z

[Read original post](https://stripe.dev/blog/training-llms-for-risk-assessment)