NotionGhita Houir AlamiExplainer
Notion developed Auto, an upgraded model router, to deliver high-quality AI results while making customers’ AI credits go further. To evaluate models, Notion categorized knowledge-work tasks into clusters and created an evaluation dataset by sampling hundreds of real-world tasks from the 20 largest clusters. They built Gymmy, a simulated environment running Notion’s application code, databases, and search services in isolated sandboxes loaded with saved workspace data, allowing consistent comparison of multiple models including GPT-6 Astra, Opus 5.5, GPT-6.1 Sol, and Auto. Notion used pairwise comparisons with Elo scores to calibrate a single LLM judge, achieving over 80% agreement between pairwise and pointwise evaluations, validated by customer thumbs-up and thumbs-down feedback. On average, Auto delivers roughly 90% of frontier model performance for about 90% lower cost, enabling users’ AI credits to go up to 10x further for everyday tasks like polishing emails, tracking task progress, and drafting performance reviews.
Notion developed Auto, a model router that selects the best LLM for each task to balance quality, speed, and cost. This post describes the evaluation methodology used to improve Auto, including task clustering, a simulated environment (Gymmy), pairwise comparisons with Elo scores, and calibration of a single LLM judge. Results show Auto delivers roughly 90% of frontier model performance at about 90% lower cost.
Notion built Auto, a model router that selects the appropriate LLM for each task. To evaluate and improve Auto, they created task clusters from common Notion AI use cases, built Gymmy (a simulated Notion environment that runs Notion’s application code, databases, and search services in isolated sandboxes), used pairwise comparisons with Elo scores to calibrate a single LLM judge, and then used pointwise evaluation to score thousands of responses in parallel.
Notion’s evaluation methodology used pairwise comparisons with Elo scores to calibrate a single LLM judge, achieving over 80% agreement between pairwise and pointwise evaluations.
Auto delivers roughly 90% of the performance of frontier models like GPT-6 Astra and Opus 5.5 for about 90% lower cost, allowing AI credits to go up to 10x further.
The evaluation process involved creating task clusters from real-world Notion tasks and replaying them in Gymmy, a simulated environment that runs Notion’s actual application code.
Notion used pairwise comparisons with Elo scores to rank models, then iterated on the judge prompt and thinking level until the LLM judge's rankings closely matched the Elo rankings. This allowed pointwise evaluation of thousands of responses in parallel.
Gymmy is a simulated Notion environment that runs Notion's application code, databases, and search services in isolated sandboxes. Each sandbox was loaded with a saved copy of a workspace's data, allowing different models to attempt the same task from the same starting state for consistent evaluation.
Auto delivers roughly 90% of the performance of frontier models for about 90% lower cost. For everyday tasks, this can make users' AI credits go up to 10x further compared to using frontier models.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
An adversarial prefix is a challenge prepended to a prompt to see whether a model holds its original decision. This diagram maps the layers that turn that pressure test into a measurable, reviewable signal.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
