Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine

Summary

Modal and CMU built Quail, a query-aware inference layer for AI-SQL that combines a query planner with an inference engine to process structured LLM transformations of database entries. On a multi-join query, Quail achieves over a billion tokens per minute per H100 GPU, over 10x faster than vLLM, costing under 6 cents per billion tokens on Modal. The gain comes from using the query plan to order requests for better KV cache management, including a revised Hydragen-style cascade attention and batching small requests to reduce host overhead. On a new AI-SQL benchmark, Quail runs 1.84x faster than vLLM, but falls behind on agent trace processing. The authors see this as part of a trend toward custom inference engines for specific workloads, with a reusable core like databases post-DataFusion, and note that much work was done by prompting coding agents.

Takeaways

1. 1

   Quail uses a query planner to order requests for better KV cache management, achieving over a billion tokens per minute per H100 GPU on a multi-join query.

2. 2

   The system employs a revised Hydragen-style cascade attention and batches small requests to reduce host overhead and improve GPU utilization.

3. 3

   On a new AI-SQL benchmark, Quail runs 1.84x faster than vLLM geometrically averaged over tasks, but falls behind on agent trace processing due to limited cross-document prefix sharing.

FAQ