Hugging FaceWill Jennings, Kiran Praveen, Sriharsha Niverty, Raviraj Joshi, Markus Kliegl, Yev Meyer, Utkarsh Vaidya, Niket AgarwalAnnouncement
A joint open-source release from NVIDIA and IIT Madras’s AI4Bharat introduces Nemotron-pretraining-Indic-v1, a corpus of approximately 2.8 trillion tokens across all 22 Indian languages listed in India’s Constitution. The corpus includes native-script data for all languages and romanized text for twelve, spanning 34 language-script configurations, built from 15 curated English source subsets covering mathematics, code, and question-answer content. A GPU-accelerated pipeline using NeMo Data Designer and Gemma-4-31B-it translates documents, assesses quality with a Multidimensional Quality Metrics rubric, repairs low-scoring outputs, and converts a subset to romanized text. For ten additional languages, translation uses Indic Translate from Bodhan AI with adapted document-windowing for longer texts. Every record retains a source identifier for provenance, and the full generation recipe including prompts and quality checks is released openly so developers can adapt it to their own source material.
AI4Bharat at IIT Madras and NVIDIA built Nemotron-pretraining-Indic-v1, a corpus of approximately 2.8 trillion tokens covering all 22 languages in India’s Constitution. The core problem is that Indian language technology trails English because of insufficient training data for Indian languages. The corpus includes native-script data for all 22 languages and romanized text for twelve, spanning 34 language-script configurations. Source material comes from 15 curated Nemotron-CC-v2 subsets covering web text, mathematics, code, and question-answer content. Every record retains a source document identifier for provenance.
A four-stage GPU-accelerated pipeline using NeMo Data Designer and Gemma-4-31B-it: (1) document-level translation from English, (2) automated quality assessment using a Multidimensional Quality Metrics rubric, (3) targeted repair of low-scoring outputs using assessment feedback, and (4) script conversion to romanized text for twelve languages. For ten additional languages, translation uses Indic Translate from Bodhan AI with adapted document-windowing for longer texts.
The corpus provides approximately 2.8 trillion tokens across 22 Indian languages, comprising 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted using the Gemma-4 tokenizer.
The pipeline uses a four-stage workflow: document-level translation, quality assessment with a 75/100 MQM-inspired threshold, targeted repair using assessment feedback, and script conversion for twelve languages.
Romanized text is derived from accepted native-script translations rather than directly from English, reducing hallucinated additions and residual English in initial human comparisons.
The corpus contains approximately 2.8 trillion tokens: 2,601.0 billion native-script tokens and 224.6 billion romanized tokens. Token counts use the Gemma-4 tokenizer. Counts describe the retained corpus including accepted rewrites, not a training run's token budget.
Translation quality is assessed by the same Gemma-4-31B-it model using a rubric inspired by Multidimensional Quality Metrics (MQM). Outputs must score at least 75 out of 100 to be retained; lower-scoring outputs are repaired using assessment feedback and reassessed before acceptance. Sampled human review complements automated checks.
For the twelve dual-script languages, romanization is done from accepted native-script translations rather than directly from English. This two-step approach produced fewer hallucinated additions and less residual English in initial human comparisons. A subset of native-script translations is converted to Latin script and checked against the original native-script input.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Compare two ways to adapt a model to a new language or domain: generate a broad synthetic corpus like AI4Bharat's, or fine-tune on one narrow task and evaluate it.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
