DeepMath: A lightweight math reasoning Agent with smolagents

Summary

Mathematical reasoning in large language models often suffers from lengthy chain-of-thought traces and frequent arithmetic mistakes. To address these issues, DeepMath pairs a Qwen3-4B Thinking base model with a sandboxed Python execution environment built using the smolagents library and vLLM backend. The system offloads deterministic calculations by emitting concise Python snippets, executing them safely with restricted module imports and no network access, and folding returned values back into the context. Training utilizes Group Relative Policy Optimization on the Tool-Integrated Reasoning subset of OpenMathReasoning with temperature scheduling and length constraints. Across benchmarks including MATH500, AIME, HMMT, and HLE, DeepMath reduces output token lengths by up to 66% while increasing overall problem-solving accuracy.

Context

Mathematical problem-solving in large language models typically generates lengthy chain-of-thought traces that remain prone to arithmetic and numerical errors. Existing approaches often do not focus on reducing trace verbosity or training models to favor concise, computation-driven traces executed within a constrained, auditable environment.

Approach / What changed

DeepMath pairs Qwen3-4B Thinking with a sandboxed Python executor using the smolagents library and vLLM inference engine. The model emits short Python snippets that run in a restricted sandbox without file I/O or network access under strict timeouts and module allow-lists, inserting execution results directly into the reasoning trace. DeepMath is fine-tuned using Group Relative Policy Optimization (GRPO) via a modified TRL framework on the Tool-Integrated Reasoning subset of OpenMathReasoning. GRPO training balances an accuracy reward (+1) and a code-use reward (+1, weighted 10:1 vs. accuracy), restricts completion candidates to 5,000 tokens, applies linear temperature scheduling from 1.2 down to 0.7, and leverages four in-context examples.

Takeaways

• DeepMath integrates a sandboxed Python executor with Qwen3-4B Thinking via smolagents and vLLM, offloading deterministic calculations and reducing output trace lengths by up to 66%.
• GRPO fine-tuning balances a +1 accuracy reward with a +1 code-snippet reward weighted 10:1 vs accuracy, alongside linear temperature scheduling from 1.2 to 0.7 and a 5,000-token candidate cap.
• Benchmarks on MATH500, AIME, HMMT, and HLE show that combining both agentic execution and GRPO training achieves higher accuracy than using either agentic inference or GRPO alone.

Related reading

• Hugging Face · Dec 4, 2025

  We Got Claude to Fine-Tune an Open Source LLM

  Hugging Face Skills equips coding agents like Claude Code, OpenAI Codex, and Google's Gemini CLI to manage end-to-end language model fine-tuning. Using the hf-llm-trainer skill, an agent validates dataset formats, selects cloud hardware, configures authentication, and submits jobs to Hugging Face Jobs. Supported techniques include Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO) for models ranging across various parameter sizes. The integration incorporates Trackio for real-time monitoring and automates LoRA configuration for larger models. Once training completes, the agent pushes artifacts to the Hugging Face Hub and can convert models to GGUF format for local execution.

  ben burtenshaw, shaun smith

• Red Hat · Aug 26, 2026

  GRPO fine-tuning on Red Hat OpenShift AI: Reinforcement learning from verifiable rewards with Training Hub