Post-Train NVIDIA Cosmos 3 in One Day Using Agent Skills

Summary

Adapting vision reasoning models for specialized physical AI video tasks often demands days of manual container configuration, training script authoring, and hyperparameter tuning. NVIDIA Cosmos 3 Nano, an omnimodal mixture-of-transformers model, pairs an autoregressive Reasoner tower with an iterative diffusion Generator tower for video question answering. By integrating NVIDIA TAO agent skills with a coding agent like Codex, developers can automate data patching, evaluation, and Low-Rank Adaptation (LoRA) post-training. In experiments using the Woven Traffic Safety dataset, LoRA fine-tuning boosted zero-shot baseline accuracy from 54.41% to 87.14% while using approximately seven times fewer GPU hours than full-parameter supervised fine-tuning. Combining this agent workflow with TAO AutoML further elevated model accuracy to 93.35% before automated deployment via NVIDIA NIM.

Context

Adapting vision reasoning models to production video tasks typically consumes days of manual effort spent formatting data, configuring containers, writing training scripts, and sweeping hyperparameters.

Approach / What changed

Automating Cosmos 3 Nano Reasoner post-training on the Woven Traffic Safety dataset using NVIDIA TAO agent skills, a coding agent like Codex, LoRA fine-tuning, TAO AutoML parameter sweeps, and NIM deployment.

Takeaways

• LoRA post-training required approximately seven times fewer GPU hours than full-parameter supervised fine-tuning for Cosmos 3 Nano.
• LoRA adaptation raised zero-shot baseline exact-match accuracy from 54.41% to 87.14%, with TAO AutoML further pushing peak accuracy to 93.35%.
• Unlike Llama-style VLM NIMs that serve bare adapters, the Cosmos 3 Reasoner NIM requires a fused checkpoint merging base weights with the LoRA adapter.

Related reading