Loading…
Synthetic Data Generation for Financial AI Research with NVIDIA NeMo
NVIDIA Developer BlogElizabeth Goodman
Summary
Real-world financial natural language processing datasets often overrepresent common events like earnings while underrepresenting rarer occurrences such as credit-rating changes. To address this imbalance, an iterative generation pipeline produced over 500,000 unique headlines across 13 categories using NVIDIA NeMo Data Designer, NeMo Curator, and Nemotron 3 Nano. Instead of relying on a single large batch that yielded 65 percent near-duplicates, the workflow executes iterative cycles of category-weighted generation, rule-based filtering, global semantic deduplication, and dynamic category reweighting. Farthest-from-centroid few-shot example selection guided subsequent rounds toward novel semantic spaces across 82 total iterations. Executed over approximately six days on an eight-way NVIDIA B200 node, the resulting corpus enabled fine-tuning compact student language models to achieve 95 percent of teacher model performance.
Context
Fine-tuning language models for financial natural language processing is constrained by limited, imbalanced real-world data that overrepresents earnings and stock price movements while underrepresenting rarer events such as credit-rating changes, product approvals, and labor issues.
Approach / What changed
An iterative synthetic data pipeline orchestrates Nemotron 3 Nano via vLLM and NeMo Data Designer across 12 topics plus an Other category. Each cycle generates batches, applies rule-based text filters, runs global semantic deduplication against the accumulated corpus via NeMo Curator, selects farthest-from-centroid few-shot examples, and dynamically adjusts category weights.
Takeaways
- A single-pass naive batch of 50,000 headlines resulted in 65% near-duplicates, whereas an iterative generation loop sustained yield and reached 502,536 unique headlines over 82 iterations.
- Global semantic deduplication against the full accumulated corpus, rather than local batch-level deduplication, is required to prevent cross-batch collisions as synthetic datasets scale.
- Selecting farthest-from-centroid few-shot examples with cross-iteration similarity cutoffs actively steers model generation into novel semantic regions across rounds.
Related reading
Optimizing a Neural Reconstruction Pipeline Using NVIDIA Nsight Developer Tools
NVIDIA Omniverse NuRec reconstructs dynamic 3D environments from camera and lidar sensor data for autonomous vehicle simulation workflows, but high computational demands historically caused long reconstruction times. Engineers used NVIDIA Nsight Systems and Nsight Compute to systematically profile the PyTorch-based training pipeline and optimize its underlying CUDA kernels. Nsight Systems revealed GPU underutilization caused by numerous small kernels and blocking synchronization calls, leading to a fused interpolate kernel that accelerated execution from 4.184 milliseconds to 83.81 microseconds. Profiling with Nsight Compute uncovered resource mismatch in the renderBackward kernel, prompting the team to split it into dedicated camera and lidar implementations with tuned register and shared memory allocations. These adjustments raised kernel occupancy from approximately 15% to between 30% and 50% while decreasing the longest lidar kernel runtime from 31 milliseconds to 18 milliseconds.
Tanya Lenz