Reducing Text2SQL latency with parameterized query templates

Summary

In Text2SQL systems, generating SQL on the fly with frontier foundation models can cause response times of 25 to 30 seconds due to 60,000-token prompts and model latency. To solve this without degrading accuracy, an intelligent caching layer was built using parameterized SQL query templates paired with question embeddings. When an incoming question matches a cached template via semantic search above a confidence threshold, named entity recognition extracts values to populate template placeholders, bypassing the initial SQL generation step. In production, this approach reached a 60% cache hit rate after two weeks, cutting per-hit token consumption by roughly 90% and reducing overall latency by 80% to under 5 seconds.

Context

Text2SQL systems in production suffered from 25 to 30 second end-to-end latencies, token costs that scaled linearly with traffic, and throttling limits caused by generating SQL queries on the fly using large prompts with frontier LLMs.

Approach / What changed

The architecture introduces a parameterized SQL query template cache indexed by question embeddings. On incoming queries, semantic search retrieves matching templates, and lightweight entity extraction fills template placeholders to execute SQL directly, falling back to full LLM generation and self-improving cache population on misses.

Takeaways

• Caching parameterized SQL templates instead of full question-answer pairs prevents data staleness while targeting the most token-intensive pipeline step.
• On cache hits, skipping the roughly 60,000-token SQL generation prompt reduced per-request token usage by about 90% and latency from 25–30 seconds to under 5 seconds.
• With a 60% production cache hit rate achieved after two weeks, blended token consumption across all traffic dropped by over 50%.

Related reading

• Yelp · Mar 27, 2026

  Building Biz Ask Anything: From Prototype to Product

  Yelp expanded Yelp Assistant to business pages by developing a question-answering architecture that extracts concise, evidence-backed answers from reviews, photos, and structured facts. The production system replaces static snapshots with near-real-time streaming pipelines for reviews and structured properties alongside weekly batch ingestion for menus and website data. For each incoming query, the system evaluates intent, retrieves relevant context through keyword and embedding searches, and constructs targeted prompts with dynamically selected few-shot examples. Synthesized answers are augmented with supporting visuals and citations before being streamed token-by-token to users. Generating suggested questions directly from specific business content rather than generic categories improved user engagement by roughly fifty percent and reduced unanswerable queries by twenty-six percent.

  Maria Christoforaki, Group Tech Lead; Shree Shalini Pusapati, Software Engineer

• Sentry · Jul 16, 2026