Evaluation-First AI Agents: How Zepto Scales Customer Support on Databricks and MLflow

Summary

Zepto describes its customer-support system, processing more than 100,000 tickets daily, and the assurance problems that emerged as volume, categories, and customer behavior changed. Its response is an evaluation-first architecture on Databricks and MLflow, using development and production loops connected by a quality gate. Agent traces capture prompts, completions, retrieved documents, tool calls, latencies, and decision paths through MLflow, OpenTelemetry spans, and Unity Catalog-backed Delta tables; stakeholders define evaluation pillars and gates around a living golden dataset. The dataset grew from 500 examples with an eight-point development–production accuracy gap to 5,247 examples with a 0.4-point gap over six months, while prompt optimization and multiple scorers support iteration. Reported outcomes include 80% plus of tickets fully managed with human oversight, a 65% reduction in support cost or tickets, 20% higher CSAT, 8% better accuracy, and faster development and resolution.

Takeaways

1. 1

   LLM judges require calibration: the framework recommends measuring their agreement with human labels, using ensembles for high-stakes cases, and recalibrating them as base models change.

2. 2

   Stratified sampling focused on high-risk flows produced an order of magnitude better issue detection per dollar than naive uniform sampling.

3. 3

   Automating detection, labeling, retraining, deployment, and monitoring saved about 155 hours per month, equivalent to two full-time engineers redirected to feature work.