Loading…
Topic
1 post about AI4Bharat. Every summary links to the original.
See the idea
Compare two ways to adapt a model to a new language or domain: generate a broad synthetic corpus like AI4Bharat's, or fine-tune on one narrow task and evaluate it.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
A joint open-source release from NVIDIA and IIT Madras’s AI4Bharat introduces Nemotron-pretraining-Indic-v1, a corpus of approximately 2.8 trillion tokens across all 22 Indian languages listed in India’s Constitution. The corpus includes native-script data for all languages and romanized text for twelve, spanning 34 language-script configurations, built from 15 curated English source subsets covering mathematics, code, and question-answer content. A GPU-accelerated pipeline using NeMo Data Designer and Gemma-4-31B-it translates documents, assesses quality with a Multidimensional Quality Metrics rubric, repairs low-scoring outputs, and converts a subset to romanized text. For ten additional languages, translation uses Indic Translate from Bodhan AI with adapted document-windowing for longer texts. Every record retains a source identifier for provenance, and the full generation recipe including prompts and quality checks is released openly so developers can adapt it to their own source material.
The best engineering reads, once a week.
