Synthetic Data Generation for Financial AI Research with NVIDIA NeMo

Summary

Real-world financial natural language processing datasets often overrepresent common events like earnings while underrepresenting rarer occurrences such as credit-rating changes. To address this imbalance, an iterative generation pipeline produced over 500,000 unique headlines across 13 categories using NVIDIA NeMo Data Designer, NeMo Curator, and Nemotron 3 Nano. Instead of relying on a single large batch that yielded 65 percent near-duplicates, the workflow executes iterative cycles of category-weighted generation, rule-based filtering, global semantic deduplication, and dynamic category reweighting. Farthest-from-centroid few-shot example selection guided subsequent rounds toward novel semantic spaces across 82 total iterations. Executed over approximately six days on an eight-way NVIDIA B200 node, the resulting corpus enabled fine-tuning compact student language models to achieve 95 percent of teacher model performance.

Takeaways

1. 1

   A single-pass naive batch of 50,000 headlines resulted in 65% near-duplicates, whereas an iterative generation loop sustained yield and reached 502,536 unique headlines over 82 iterations.

2. 2

   Global semantic deduplication against the full accumulated corpus, rather than local batch-level deduplication, is required to prevent cross-batch collisions as synthetic datasets scale.

3. 3

   Selecting farthest-from-centroid few-shot examples with cross-iteration similarity cutoffs actively steers model generation into novel semantic regions across rounds.