--- title: "Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline" description: "A joint open-source release from NVIDIA and IIT Madras’s AI4Bharat introduces Nemotron-pretraining-Indic-v1, a corpus of approximately 2.8 trillion tokens across all 22 Indian languages listed in India’s Constitution. The corpus includes native-script data for all languages and romanized text for twelve, spanning 34 language-script configurations, built from 15 curated English source subsets covering mathematics, code, and question-answer content. A GPU-accelerated pipeline using NeMo Data Designer and Gemma-4-31B-it translates documents, assesses quality with a Multidimensional Quality Metrics rubric, repairs low-scoring outputs, and converts a subset to romanized text. For ten additional languages, translation uses Indic Translate from Bodhan AI with adapted document-windowing for longer texts. Every record retains a source identifier for provenance, and the full generation recipe including prompts and quality checks is released openly so developers can adapt it to their own source material." --- # Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline [Hugging Face](https://yomu.fyi/company/hugging-face) · Will Jennings, Kiran Praveen, Sriharsha Niverty, Raviraj Joshi, Markus Kliegl, Yev Meyer, Utkarsh Vaidya, Niket Agarwal · Oct 16, 2026 **Type:** Announcement ## Summary A joint open-source release from NVIDIA and IIT Madras’s AI4Bharat introduces Nemotron-pretraining-Indic-v1, a corpus of approximately 2.8 trillion tokens across all 22 Indian languages listed in India’s Constitution. The corpus includes native-script data for all languages and romanized text for twelve, spanning 34 language-script configurations, built from 15 curated English source subsets covering mathematics, code, and question-answer content. A GPU-accelerated pipeline using NeMo Data Designer and Gemma-4-31B-it translates documents, assesses quality with a Multidimensional Quality Metrics rubric, repairs low-scoring outputs, and converts a subset to romanized text. For ten additional languages, translation uses Indic Translate from Bodhan AI with adapted document-windowing for longer texts. Every record retains a source identifier for provenance, and the full generation recipe including prompts and quality checks is released openly so developers can adapt it to their own source material. ## Context AI4Bharat at IIT Madras and NVIDIA built Nemotron-pretraining-Indic-v1, a corpus of approximately 2.8 trillion tokens covering all 22 languages in India’s Constitution. The core problem is that Indian language technology trails English because of insufficient training data for Indian languages. The corpus includes native-script data for all 22 languages and romanized text for twelve, spanning 34 language-script configurations. Source material comes from 15 curated Nemotron-CC-v2 subsets covering web text, mathematics, code, and question-answer content. Every record retains a source document identifier for provenance. ## Approach / What changed A four-stage GPU-accelerated pipeline using NeMo Data Designer and Gemma-4-31B-it: (1) document-level translation from English, (2) automated quality assessment using a Multidimensional Quality Metrics rubric, (3) targeted repair of low-scoring outputs using assessment feedback, and (4) script conversion to romanized text for twelve languages. For ten additional languages, translation uses Indic Translate from Bodhan AI with adapted document-windowing for longer texts. ## Takeaways - The corpus provides approximately 2.8 trillion tokens across 22 Indian languages, comprising 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted using the Gemma-4 tokenizer. - The pipeline uses a four-stage workflow: document-level translation, quality assessment with a 75/100 MQM-inspired threshold, targeted repair using assessment feedback, and script conversion for twelve languages. - Romanized text is derived from accepted native-script translations rather than directly from English, reducing hallucinated additions and residual English in initial human comparisons. ## FAQ ### How big is the Nemotron-pretraining-Indic-v1 corpus? The corpus contains approximately 2.8 trillion tokens: 2,601.0 billion native-script tokens and 224.6 billion romanized tokens. Token counts use the Gemma-4 tokenizer. Counts describe the retained corpus including accepted rewrites, not a training run's token budget. ### How is translation quality guaranteed for the synthetic data? Translation quality is assessed by the same Gemma-4-31B-it model using a rubric inspired by Multidimensional Quality Metrics (MQM). Outputs must score at least 75 out of 100 to be retained; lower-scoring outputs are repaired using assessment feedback and reassessed before acceptance. Sampled human review complements automated checks. ### How is romanized text generated from native-script data? For the twelve dual-script languages, romanization is done from accepted native-script translations rather than directly from English. This two-step approach produced fewer hallucinated additions and less residual English in initial human comparisons. A subset of native-script translations is converted to Latin script and checked against the original native-script input. **Tags:** [AI4Bharat](https://yomu.fyi/topic/ai4bharat), [MCP](https://yomu.fyi/topic/mcp), [NVIDIA](https://yomu.fyi/topic/nvidia), [Synthetic Data](https://yomu.fyi/topic/synthetic-data) - Source: [Hugging Face](https://huggingface.co/blog/nvidia/scaling-indic-pretraining) - Source URL: https://huggingface.co/blog/nvidia/scaling-indic-pretraining - Ingested by Yomu: 2026-10-16T03:00:14.087Z [Read original post](https://huggingface.co/blog/nvidia/scaling-indic-pretraining)