Post-Train NVIDIA Cosmos 3 in One Day Using Agent Skills

Summary

Adapting vision reasoning models for specialized physical AI video tasks often demands days of manual container configuration, training script authoring, and hyperparameter tuning. NVIDIA Cosmos 3 Nano, an omnimodal mixture-of-transformers model, pairs an autoregressive Reasoner tower with an iterative diffusion Generator tower for video question answering. By integrating NVIDIA TAO agent skills with a coding agent like Codex, developers can automate data patching, evaluation, and Low-Rank Adaptation (LoRA) post-training. In experiments using the Woven Traffic Safety dataset, LoRA fine-tuning boosted zero-shot baseline accuracy from 54.41% to 87.14% while using approximately seven times fewer GPU hours than full-parameter supervised fine-tuning. Combining this agent workflow with TAO AutoML further elevated model accuracy to 93.35% before automated deployment via NVIDIA NIM.

Takeaways

1. 1

   LoRA post-training required approximately seven times fewer GPU hours than full-parameter supervised fine-tuning for Cosmos 3 Nano.

2. 2

   LoRA adaptation raised zero-shot baseline exact-match accuracy from 54.41% to 87.14%, with TAO AutoML further pushing peak accuracy to 93.35%.

3. 3

   Unlike Llama-style VLM NIMs that serve bare adapters, the Cosmos 3 Reasoner NIM requires a fused checkpoint merging base weights with the LoRA adapter.