Jupyter Agents: training LLMs to reason with notebooks

Summary

Small language models often struggle to compete with large frontier models on complex, agentic data science tasks. To improve notebook-based reasoning, researchers simplified agent scaffolding down to roughly two hundred lines of code with dedicated execution and final answer tools, boosting baseline easy accuracy on the DABStep benchmark from 44.4 percent to 59.7 percent. They constructed a curated training dataset by deduplicating two terabytes of Kaggle notebooks, automatically fetching five terabytes of linked datasets, and scoring educational value and relevance using Qwen3-32B. Question-answer pairs grounded in verified execution traces were generated to fine-tune compact Qwen3-4B thinking and instruct models. The team released the trained models alongside the dataset and execution sandboxes, establishing a foundation for reinforcement learning and distillation on notebook workflows.

Takeaways

1. 1

   Streamlining agent scaffolding to roughly 200 lines without external framework dependencies raised Qwen3-4B baseline accuracy on DABStep easy tasks from 44.4% to 59.7%.

2. 2

   Kaggle notebook data curation required aggressive filtering, where deduplication eliminated approximately 90% of raw notebooks and Qwen3-32B educational scoring removed another 70%.

3. 3

   Training data was created by generating question-answer pairs grounded in real code execution traces across executable Kaggle notebooks paired with their downloaded datasets.