---
title: "Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine"
description: "Modal and CMU built Quail, a query-aware inference layer for AI-SQL that combines a query planner with an inference engine to process structured LLM transformations of database entries. On a multi-join query, Quail achieves over a billion tokens per minute per H100 GPU, over 10x faster than vLLM, costing under 6 cents per billion tokens on Modal. The gain comes from using the query plan to order requests for better KV cache management, including a revised Hydragen-style cascade attention and batching small requests to reduce host overhead. On a new AI-SQL benchmark, Quail runs 1.84x faster than vLLM, but falls behind on agent trace processing. The authors see this as part of a trend toward custom inference engines for specific workloads, with a reusable core like databases post-DataFusion, and note that much work was done by prompting coding agents."
---

# Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine

[Modal](https://yomu.fyi/company/modal) · Sep 24, 2026

**Type:** Problem & solution

## Summary

Modal and CMU built Quail, a query-aware inference layer for AI-SQL that combines a query planner with an inference engine to process structured LLM transformations of database entries. On a multi-join query, Quail achieves over a billion tokens per minute per H100 GPU, over 10x faster than vLLM, costing under 6 cents per billion tokens on Modal. The gain comes from using the query plan to order requests for better KV cache management, including a revised Hydragen-style cascade attention and batching small requests to reduce host overhead. On a new AI-SQL benchmark, Quail runs 1.84x faster than vLLM, but falls behind on agent trace processing. The authors see this as part of a trend toward custom inference engines for specific workloads, with a reusable core like databases post-DataFusion, and note that much work was done by prompting coding agents.

## Context

AI-SQL workloads use SQL to programmatically produce prompts for LLMs, enabling fuzzy queries on semi-structured data in business intelligence platforms. These workloads require GPUs for LLM inference but differ from chatbot and agent workloads in that they involve many small requests with known structure, making them amenable to query-aware optimization.

## Approach / What changed

Quail combines a query planner with an inference engine to order requests for better KV cache management, using a revised Hydragen-style cascade attention and batching small requests to reduce host overhead and improve GPU utilization.

## Takeaways

- Quail uses a query planner to order requests for better KV cache management, achieving over a billion tokens per minute per H100 GPU on a multi-join query.
- The system employs a revised Hydragen-style cascade attention and batches small requests to reduce host overhead and improve GPU utilization.
- On a new AI-SQL benchmark, Quail runs 1.84x faster than vLLM geometrically averaged over tasks, but falls behind on agent trace processing due to limited cross-document prefix sharing.

## FAQ

### How does Quail achieve higher throughput than vLLM?

By using the query plan to order requests for better KV cache management, including a revised Hydragen-style cascade attention and batching small requests to reduce host overhead.

### What is the cost per billion tokens on Modal?

Under 6 cents per billion tokens on Modal.

### What is a limitation of Quail compared to vLLM?

It falls behind on agent trace processing due to limited cross-document prefix sharing.

**Tags:** [AI](https://yomu.fyi/topic/ai), [GPU](https://yomu.fyi/topic/gpu), [Inference](https://yomu.fyi/topic/inference), [LLMs](https://yomu.fyi/topic/llm), [SQL](https://yomu.fyi/topic/sql)

- Source: [Modal](https://modal.com/blog/quail-billion-tpm)
- Source URL: https://modal.com/blog/quail-billion-tpm
- Ingested by Yomu: 2026-09-27T00:00:15.639Z

[Read original post](https://modal.com/blog/quail-billion-tpm)