Loading…
NVIDIA Cosmos Reason 2 Brings Advanced Reasoning To Physical AI
Hugging FaceTsung-Yi Lin, Debraj Sinha
Summary
NVIDIA released Cosmos Reason 2, an open reasoning vision-language model designed to improve multi-step planning, uncertainty handling, and physical AI decision-making. Available in 2B and 8B parameter configurations, the model increases input context support to 256K tokens, up from 16K in the prior version. The release adds spatial perception capabilities including optical character recognition, 2D and 3D point localization, bounding boxes, and trajectory generation for robotics. In domain adaptation tests on autonomous vehicle video captioning, fine-tuning the 8B model improved BLEU scores by 10.6% and LingoQA scores by 13.8%. The model leads the Physical AI Bench and Physical Reasoning leaderboards for open visual understanding models.
Context
Vision-language models struggle with tasks that require planning multiple steps ahead, managing uncertainty, adapting to new physical situations, and understanding how objects move across space and time.
Approach / What changed
NVIDIA released Cosmos Reason 2, an open reasoning vision-language model in 2B and 8B parameter sizes. It extends context input to 256K tokens and supports spatio-temporal reasoning, OCR, 2D/3D point localization, bounding box coordinates, and trajectory generation for robotic planning and video analytics.
Takeaways
- Cosmos Reason 2 expands long-context processing capability to 256K input tokens, up from 16K tokens in Cosmos Reason 1.
- The model introduces spatial understanding capabilities including OCR, 2D and 3D point localization, bounding box coordinates, and trajectory data for robotic control.
- Fine-tuning Cosmos Reason 2-8B on annotated autonomous vehicle videos improved BLEU scores by 10.6%, MCQ-based VQA by 0.67 percentage points, and LingoQA by 13.8%.
Related reading
Post-Train NVIDIA Cosmos 3 in One Day Using Agent Skills
Adapting vision reasoning models for specialized physical AI video tasks often demands days of manual container configuration, training script authoring, and hyperparameter tuning. NVIDIA Cosmos 3 Nano, an omnimodal mixture-of-transformers model, pairs an autoregressive Reasoner tower with an iterative diffusion Generator tower for video question answering. By integrating NVIDIA TAO agent skills with a coding agent like Codex, developers can automate data patching, evaluation, and Low-Rank Adaptation (LoRA) post-training. In experiments using the Woven Traffic Safety dataset, LoRA fine-tuning boosted zero-shot baseline accuracy from 54.41% to 87.14% while using approximately seven times fewer GPU hours than full-parameter supervised fine-tuning. Combining this agent workflow with TAO AutoML further elevated model accuracy to 93.35% before automated deployment via NVIDIA NIM.
Tanya Lenz