Loading…
Topic
4 posts about Synthetic Data. Every summary links to the original.
See the idea
Compare two ways to adapt a model to a new language or domain: generate a broad synthetic corpus like AI4Bharat's, or fine-tune on one narrow task and evaluate it.
Arabic is not a single target for a language model. This explainer maps its four parts — dialect surface, data pipeline, model, and evaluation — so you can tell where an Arabic feature really succeeds or fails.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
A joint open-source release from NVIDIA and IIT Madras’s AI4Bharat introduces Nemotron-pretraining-Indic-v1, a corpus of approximately 2.8 trillion tokens across all 22 Indian languages listed in India’s Constitution. The corpus includes native-script data for all languages and romanized text for twelve, spanning 34 language-script configurations, built from 15 curated English source subsets covering mathematics, code, and question-answer content. A GPU-accelerated pipeline using NeMo Data Designer and Gemma-4-31B-it translates documents, assesses quality with a Multidimensional Quality Metrics rubric, repairs low-scoring outputs, and converts a subset to romanized text. For ten additional languages, translation uses Indic Translate from Bodhan AI with adapted document-windowing for longer texts. Every record retains a source identifier for provenance, and the full generation recipe including prompts and quality checks is released openly so developers can adapt it to their own source material.
Flo Health, a women’s health app with 81 million monthly active users, aimed to improve generative AI responses for medical safety and usefulness while operating at scale. Relying on general-purpose models created cost, dependency, and domain-expertise limitations, so Flo built a post-training pipeline with Databricks AI Runtime. The team defined 26 AI judges across three categories: 12 medical-safety judges acting as hard constraints, 6 micro judges for nuanced behaviors, and 8 usefulness criteria. Using these judges as a reward signal, they generated over 15,000 synthetic questions from privacy-first principles and applied reinforcement fine-tuning with supervised fine-tuning (SFT) and direct preference optimization (DPO). After fine-tuning on 32 NVIDIA H100 GPUs using PyTorch and Fully Sharded Data Parallel (FSDP), medical safety scores rose from 34% to 61%, usefulness from 42% to 76%, and the combined metric improved nearly threefold to 47%.
AutoSynthData, built by ServiceNow CoreAI, generates training data for enterprise agents by targeting capability gaps from evaluation, using a stronger teacher such as Qwen3.8-27B to demonstrate successes. It uses a target model’s failures and a stronger teacher’s successes to decide what to learn next. The system produces tasks composing a system specification, user prompt, and verifier, ensuring feasibility, realism, and difficulty against the target environment. After sample-level checks and batch-level review, the dataset grows via target and multiply phases, generating 2,000 samples in 18 hours for the Hybrid domain. In EnterpriseOps Gym, synthetic SFT on Gemma-4-26B-A4B-it improved mean Pass@1 by 7.2 points (35% relative) in Hybrid and from 18.77% to 27.18% in ITSM, showing the approach works across domains.
The post analyzes limitations of confidence-based ranking in discrete diffusion, where sequences are generated by writing multiple token positions per step from per-position distributions. It shows that a step matches the training distribution only when the written positions are conditionally independent given fixed tokens, and no product of per-position distributions can match a dependent group. Additionally, per-position distributions do not determine group dependence: two joint distributions can have identical per-position marginals but differ in value combinations. On the synthetic ScanAndAdd task, every group of two or more undetermined positions written by a confidence ranking is dependent, and the generated distribution is 29 times the sampling-noise floor in total variation, with per-sample metrics at 1.0. The post also includes updates on two related readings: on-policy distillation for reasoning models and self-supervised learning with Transformers.
The best engineering reads, once a week.
