---
title: "Inside Notion’s upgraded model router, Auto"
description: "Notion developed Auto, an upgraded model router, to deliver high-quality AI results while making customers’ AI credits go further. To evaluate models, Notion categorized knowledge-work tasks into clusters and created an evaluation dataset by sampling hundreds of real-world tasks from the 20 largest clusters. They built Gymmy, a simulated environment running Notion’s application code, databases, and search services in isolated sandboxes loaded with saved workspace data, allowing consistent comparison of multiple models including GPT-6 Astra, Opus 5.5, GPT-6.1 Sol, and Auto. Notion used pairwise comparisons with Elo scores to calibrate a single LLM judge, achieving over 80% agreement between pairwise and pointwise evaluations, validated by customer thumbs-up and thumbs-down feedback. On average, Auto delivers roughly 90% of frontier model performance for about 90% lower cost, enabling users’ AI credits to go up to 10x further for everyday tasks like polishing emails, tracking task progress, and drafting performance reviews."
---

# Inside Notion’s upgraded model router, Auto

[Notion](https://yomu.fyi/company/notion) · Ghita Houir Alami · Oct 20, 2026

**Type:** Explainer

## Summary

Notion developed Auto, an upgraded model router, to deliver high-quality AI results while making customers’ AI credits go further. To evaluate models, Notion categorized knowledge-work tasks into clusters and created an evaluation dataset by sampling hundreds of real-world tasks from the 20 largest clusters. They built Gymmy, a simulated environment running Notion’s application code, databases, and search services in isolated sandboxes loaded with saved workspace data, allowing consistent comparison of multiple models including GPT-6 Astra, Opus 5.5, GPT-6.1 Sol, and Auto. Notion used pairwise comparisons with Elo scores to calibrate a single LLM judge, achieving over 80% agreement between pairwise and pointwise evaluations, validated by customer thumbs-up and thumbs-down feedback. On average, Auto delivers roughly 90% of frontier model performance for about 90% lower cost, enabling users’ AI credits to go up to 10x further for everyday tasks like polishing emails, tracking task progress, and drafting performance reviews.

## Context

Notion developed Auto, a model router that selects the best LLM for each task to balance quality, speed, and cost. This post describes the evaluation methodology used to improve Auto, including task clustering, a simulated environment (Gymmy), pairwise comparisons with Elo scores, and calibration of a single LLM judge. Results show Auto delivers roughly 90% of frontier model performance at about 90% lower cost.

## Approach / What changed

Notion built Auto, a model router that selects the appropriate LLM for each task. To evaluate and improve Auto, they created task clusters from common Notion AI use cases, built Gymmy (a simulated Notion environment that runs Notion’s application code, databases, and search services in isolated sandboxes), used pairwise comparisons with Elo scores to calibrate a single LLM judge, and then used pointwise evaluation to score thousands of responses in parallel.

## Takeaways

- Notion’s evaluation methodology used pairwise comparisons with Elo scores to calibrate a single LLM judge, achieving over 80% agreement between pairwise and pointwise evaluations.
- Auto delivers roughly 90% of the performance of frontier models like GPT-6 Astra and Opus 5.5 for about 90% lower cost, allowing AI credits to go up to 10x further.
- The evaluation process involved creating task clusters from real-world Notion tasks and replaying them in Gymmy, a simulated environment that runs Notion’s actual application code.

## FAQ

### How did Notion calibrate a single LLM judge for evaluation?

Notion used pairwise comparisons with Elo scores to rank models, then iterated on the judge prompt and thinking level until the LLM judge's rankings closely matched the Elo rankings. This allowed pointwise evaluation of thousands of responses in parallel.

### What is Gymmy and how was it used?

Gymmy is a simulated Notion environment that runs Notion's application code, databases, and search services in isolated sandboxes. Each sandbox was loaded with a saved copy of a workspace's data, allowing different models to attempt the same task from the same starting state for consistent evaluation.

### What cost and quality trade-off does Auto achieve?

Auto delivers roughly 90% of the performance of frontier models for about 90% lower cost. For everyday tasks, this can make users' AI credits go up to 10x further compared to using frontier models.

**Tags:** [Cost Optimization](https://yomu.fyi/topic/cost-optimization), [Evaluation](https://yomu.fyi/topic/evaluation), [LLMs](https://yomu.fyi/topic/llm), [Model Routing](https://yomu.fyi/topic/model-routing), [Notion](https://yomu.fyi/topic/notion)

- Source: [Notion](https://www.notion.com/en-gb/blog/inside-notions-upgraded-model-router-auto)
- Source URL: https://www.notion.com/en-gb/blog/inside-notions-upgraded-model-router-auto
- Ingested by Yomu: 2026-10-21T00:00:24.354Z

[Read original post](https://www.notion.com/en-gb/blog/inside-notions-upgraded-model-router-auto)