Loading…
How to Run an Autoresearch Workflow with RL Agent Skills and NVIDIA NeMo
NVIDIA Developer BlogTanya Lenz
Summary
Setting up reinforcement learning workflows and running repetitive experimentation often consumes significant engineering time before meaningful metrics emerge. To address this, an autonomous autoresearch workflow uses a frontier coding agent running Codex with GPT 5.5 to manage NVIDIA NeMo RL and NeMo Gym on an NVIDIA Brev GPU instance. The agent executes full-stack setup, creates a novel NeMo Gym visual counting environment, tunes Qwen3-VL-2B-Instruct accuracy from 25.0% to 96.9%, and implements an off-policy reinforcement learning algorithm directly from a research paper. Reusable skills, including brev-etiquette, session-memory, and autoresearch, enforce local conventions and prevent context drift during long executions. This setup offloads environment configuration, dependency resolution, and experiment tracking while preserving human oversight for goals and strategy.
Context
Reinforcement learning research requires substantial experiment infrastructure, environment setup, dependency management, and iterative experimentation before researchers can obtain meaningful metrics.
Approach / What changed
Deploying Codex with GPT 5.5 on NVIDIA Brev GPU instances alongside NVIDIA NeMo RL and NeMo Gym, guided by structured agent skills for instance etiquette, durable session memory, and iterative hypothesis testing.
Takeaways
- Codex created a custom NeMo Gym visual counting task and boosted Qwen3-VL-2B-Instruct accuracy from 25.0% to 96.9%.
- Three structured agent skills (brev-etiquette, session-memory, and autoresearch) provide durable memory and prevent filesystem scattering or context drift.
- Codex translated an off-policy RL algorithm (OAPL) from a research paper into code, achieving higher accuracy with fewer steps than DAPO on the Qwen3-1.7B model.
Related reading
Creating the NVIDIA Nemotron 3 Ultra NVFP4 Checkpoint with NVIDIA Model Optimizer
Moving large model weights efficiently requires quantization, but mapping weights to NVFP4's eight positive values risks severe precision loss or outlier-induced underflow. To address this, NVIDIA quantized the 550B Nemotron 3 Ultra model using NVIDIA Model Optimizer with a multi-precision strategy rather than quantizing all layers uniformly. The recipe applies four-over-six adaptive block scaling to Mixture of Experts routed weights, while assigning FP8 to shared experts and retaining BF16 for sensitive attention linears and embeddings. This process reduced the checkpoint size from 1,121 GB down to 352.3 GB and enabled single-checkpoint execution across Hopper using W4A16 and Blackwell using native W4A4. As a result, the model achieves up to 5.9x higher inference throughput than the GLM-5.1 754B FP4 model on decode-heavy workloads while matching BF16 benchmark accuracy.
Michelle Horton