NVIDIA and IIT Madras’s AI4Bharat lab have announced Nemotron-pretraining-Indic-v1, approximately 2.8 trillion synthetic tokens spanning 22 Indian languages, alongside an open, GPU-accelerated generation pipeline. The October 16 announcement describes both as releasing soon. That availability distinction matters: researchers can examine the published methodology, but should not treat the announcement as confirmation that the full corpus is already downloadable.
The authors’ technical article inventories 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted with the Gemma-4 tokenizer. All 22 languages have native-script data; twelve also have Latin-script variants. Here, “native-script” describes the writing system, not naturally authored provenance: the content is translated from curated English documents. Romanization changes the script of an Indic-language translation; it does not turn it back into English.
The core contribution is therefore a reproducible approach to expanding technical-language coverage, not simply a large token total. Translations and script variants can derive from the same source document, so neither record counts nor token counts establish an equivalent number of independent examples. The authors explicitly distinguish retained corpus inventory from a training run’s token budget. For model builders, the experimental question is how a chosen mixture changes capability, language fidelity and retention of existing skills—not whether a larger inventory automatically produces a better model.
The twelve-language workflow starts with complete English documents from 15 Nemotron-CC-v2 subsets, including mathematics, code and question-answer material. NeMo Data Designer orchestrates translation, assessment and targeted repair. Gemma-4-31B-it translates documents while prompts distinguish prose to translate from equations, code, URLs and formatting to preserve. The same model then judges each translation against its English source using an MQM-inspired rubric covering meaning, fluency, terminology, style and formatting. Outputs need at least 75 out of 100 to enter the accepted pool; that is a rubric threshold, not a 75% accuracy measurement. Feedback-guided rewriting recovered 438.4 billion tokens across native-script and romanized data, already included in the corpus total rather than added on top. Accepted translations also seed a separately checked romanization stage.
The other ten languages use a different route because comparisons exposed long-document quality and script-stability problems. Indic Translate supplies document-level translation, with Gemma-4-31B-it performing assessment. Eight languages retain whole-document processing. Longer Santali documents use overlapping windows followed by reconstruction, while Manipuri passes through Bengali script before conversion to Meitei Mayek. These ten configurations retain the same score threshold but produce native-script data only and do not use feedback-based rewriting. That asymmetry should remain visible in any downstream experiment: coverage of a language does not imply identical generation, repair or human-validation procedures across the dataset.
The strongest reported outcome is a continued-pretraining comparison for Nemotron 3.5 Super VL. The configured stage used 100 billion training tokens, with 10 billion allocated to synthetic and natural Indic text and the remaining 90 billion maintaining the preceding mixture’s proportions. All five reported Indic knowledge and reasoning benchmarks improved, by 4.98 to 9.82 percentage points. This supports the combined stage-level recipe, not isolated causation by the synthetic corpus. Benchmark language coverage differs, and GSM8K excludes Odia because evaluation exceeded the serving context limit. A separate comparison of final model generations includes subsequent training effects: five of six task benchmarks improve, while IFEval 1.5 falls 2.04 points. Those results should not be collapsed into a blanket claim of universal improvement.
For practitioners planning a multilingual training study, the authors recommend fixed-budget comparisons, per-language reporting, explicit natural/synthetic and native/romanized shares, and evaluation of English retention alongside script quality. Shared-source variants also need attention when constructing held-out splits; keeping a source identifier is useful provenance, but source-ID duplicate removal is not semantic deduplication. The translator serving as its own judge introduces another limitation, and sampled human review varies by language. Naturally authored Indic material remains necessary for local knowledge and cultural context. Once released, pinning dataset revisions and inspecting streamed samples can make experiments reproducible without immediately downloading the full inventory. The consequential question is whether independent teams can reproduce the gains under controlled mixtures—and identify where automated translation expands access while preserving errors that model-based scoring fails to catch.
Follow Us