Build Your Own Transaction Foundation Model for Financial Intelligence

Summary

Production financial intelligence systems frequently rely on hand-engineered tabular features and rule sets that are brittle, expensive to maintain, and blind to historical sequential patterns. NVIDIA demonstrates an accelerated reference pipeline that replaces standard BPE tokenization with a GPU-based domain tokenizer, converting raw transactions into semantic tokens with an 8,192-token context window. Using the NeMo AutoModel library, a compact 29-million parameter decoder-only transformer is pretrained from scratch on unlabeled transaction sequences using causal language modeling. Learned sequence embeddings are extracted, compressed using PCA, and concatenated with raw tabular features to train a downstream GPU-accelerated XGBoost fraud detection model. On the IBM TabFormer benchmark dataset, this combined approach achieves a 41.76% lift in Average Precision over the baseline model relying solely on raw tabular features.

Takeaways

1. 1

   Custom domain tokenization yields roughly 12 semantic tokens per transaction with a 6,251-symbol vocabulary, fitting over three times more transaction history into a fixed context window than GPT-2 BPE.

2. 2

   The compact Llama decoder architecture uses roughly 29 million parameters, 8 transformer layers, a hidden size of 512, Grouped-Query Attention, SwiGLU activations, RMSNorm, and an 8,192-token RoPE context window.

3. 3

   Combining 13 raw tabular features with 64-dimensional PCA-reduced foundation model embeddings increases downstream test Average Precision by 41.76% over the XGBoost baseline on the IBM TabFormer dataset.