---
title: "Putting a Transformer Inside Every Card Authorization"
description: "Ramp’s new card fraud model combines a small transformer pre-trained without fraud labels with an XGBoost stage to score each authorization in real time. The transformer reads a cardholder’s recent transactions and learns to predict the next transaction, producing embeddings and surprisal scores that capture how anomalous each field is. The model caught 32% more fraud dollars than its predecessor while flagging the same number of transactions, with average precision 12% higher than XGBoost alone. Most of the dollar gain comes from weighting XGBoost training examples by dollar amount, while the transformer improves ranking and catches more first fraudulent charges on a card. The transformer trains in under 90 minutes on four L40S GPUs and runs on CPU within the authorization latency budget after optimizations cut median model step time from 145 to 54 milliseconds."
---

# Putting a Transformer Inside Every Card Authorization

[Ramp](https://yomu.fyi/company/ramp) · Dwight Temple, Simon Miller · Oct 7, 2026

**Type:** Problem & solution

## Summary

Ramp’s new card fraud model combines a small transformer pre-trained without fraud labels with an XGBoost stage to score each authorization in real time. The transformer reads a cardholder’s recent transactions and learns to predict the next transaction, producing embeddings and surprisal scores that capture how anomalous each field is. The model caught 32% more fraud dollars than its predecessor while flagging the same number of transactions, with average precision 12% higher than XGBoost alone. Most of the dollar gain comes from weighting XGBoost training examples by dollar amount, while the transformer improves ranking and catches more first fraudulent charges on a card. The transformer trains in under 90 minutes on four L40S GPUs and runs on CPU within the authorization latency budget after optimizations cut median model step time from 145 to 54 milliseconds.

## Context

Ramp’s card fraud detection needed to understand transaction context beyond simple counts, as fraud often appears as a change in routine. The previous model used XGBoost alone, which could not capture sequence-level patterns or differentiate between unfamiliar merchants. The new model combines a transformer with XGBoost to score each authorization in real time.

## Approach / What changed

A small transformer pre-trained without fraud labels reads a cardholder’s recent transactions and produces embeddings and surprisal scores for each field. The transformer uses three tasks: next-transaction prediction, masked transaction prediction, and a swap-detection task where pieces of a purchase are swapped from another transaction. These outputs are fed to an XGBoost model that also uses dollar-weighted training. The transformer runs on CPU during authorization, and optimizations such as removing padding and moving lookup tables out of garbage collector reduced median model step time from 145ms to 54ms without changing scores.

## Takeaways

- The transformer is pre-trained without fraud labels using three tasks: next-transaction prediction, masked transaction prediction, and a swap-detection task where pieces of a purchase are swapped from another transaction.
- The combined model caught 32% more fraud dollars than the previous XGBoost-only model while flagging the same number of transactions, with most of the dollar gain from dollar-weighted training and the transformer improving ranking by 12% average precision.
- The transformer is kept small to enable frequent retraining; it trains in under 90 minutes on four L40S GPUs and runs on CPU. Optimizations such as removing padding and moving lookup tables out of garbage collector reduced median model step time from 145ms to 54ms without changing scores.

## FAQ

### How does the transformer handle rare or unseen merchants?

Frequent merchants get their own vector; less common merchants are hashed into shared slots within their category; the rarest fall back to one token per category, so an unseen merchant still carries its category's meaning. This prevents needing to retrain when the merchant vocabulary changes.

### What latency optimizations were made to serve the transformer in real time?

Padding was removed, the merchant lookup table was moved out of Python's garbage collector, and small requests were built directly with numpy. These changes cut median model step time from 145ms to 54ms without altering any scores.

### How does the swap-detection task improve the model?

The swap task asks the transformer to identify which field (merchant, amount, country, or card presentation) of a purchase was swapped from another transaction. It adds less than 1% to training time and drives deeper understanding of user spending patterns.

**Tags:** [Fraud Detection](https://yomu.fyi/topic/fraud-detection), [Real-time](https://yomu.fyi/topic/real-time), [Transformer](https://yomu.fyi/topic/transformer), [XGBoost](https://yomu.fyi/topic/xgboost)

- Source: [Ramp](https://builders.ramp.com/post/sequence-aware-card-fraud)
- Source URL: https://builders.ramp.com/post/sequence-aware-card-fraud
- Ingested by Yomu: 2026-10-08T00:00:17.274Z

[Read original post](https://builders.ramp.com/post/sequence-aware-card-fraud)