# Build Your Own Transaction Foundation Model for Financial Intelligence

[NVIDIA Developer Blog](https://yomu.fyi/company/nvidia-developer-blog) · Benjamin Wu · Jun 16, 2026

**Type:** Tutorial

## Summary

Production financial intelligence systems frequently rely on hand-engineered tabular features and rule sets that are brittle, expensive to maintain, and blind to historical sequential patterns. NVIDIA demonstrates an accelerated reference pipeline that replaces standard BPE tokenization with a GPU-based domain tokenizer, converting raw transactions into semantic tokens with an 8,192-token context window. Using the NeMo AutoModel library, a compact 29-million parameter decoder-only transformer is pretrained from scratch on unlabeled transaction sequences using causal language modeling. Learned sequence embeddings are extracted, compressed using PCA, and concatenated with raw tabular features to train a downstream GPU-accelerated XGBoost fraud detection model. On the IBM TabFormer benchmark dataset, this combined approach achieves a 41.76% lift in Average Precision over the baseline model relying solely on raw tabular features.

## Context

Production financial intelligence workflows for tabular transaction data often rely on hand-engineered features and rule sets that are brittle, costly to maintain, and incapable of capturing sequential patterns across customer histories.

## Approach / What changed

NVIDIA introduces an end-to-end accelerated pipeline that uses cuDF and cuML for GPU data processing and custom domain tokenization, pretrains a compact 29M-parameter transformer decoder from scratch with NeMo AutoModel, extracts embeddings, and combines them with raw tabular features in a downstream GPU-accelerated XGBoost classifier.

## Takeaways

- Custom domain tokenization yields roughly 12 semantic tokens per transaction with a 6,251-symbol vocabulary, fitting over three times more transaction history into a fixed context window than GPT-2 BPE.
- The compact Llama decoder architecture uses roughly 29 million parameters, 8 transformer layers, a hidden size of 512, Grouped-Query Attention, SwiGLU activations, RMSNorm, and an 8,192-token RoPE context window.
- Combining 13 raw tabular features with 64-dimensional PCA-reduced foundation model embeddings increases downstream test Average Precision by 41.76% over the XGBoost baseline on the IBM TabFormer dataset.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python)

[Read original post](https://developer.nvidia.com/blog/build-your-own-transaction-foundation-model-for-financial-intelligence)