---
title: "Evaluation-First AI Agents: How Zepto Scales Customer Support on Databricks and MLflow"
description: "Zepto describes its customer-support system, processing more than 100,000 tickets daily, and the assurance problems that emerged as volume, categories, and customer behavior changed. Its response is an evaluation-first architecture on Databricks and MLflow, using development and production loops connected by a quality gate. Agent traces capture prompts, completions, retrieved documents, tool calls, latencies, and decision paths through MLflow, OpenTelemetry spans, and Unity Catalog-backed Delta tables; stakeholders define evaluation pillars and gates around a living golden dataset. The dataset grew from 500 examples with an eight-point development–production accuracy gap to 5,247 examples with a 0.4-point gap over six months, while prompt optimization and multiple scorers support iteration. Reported outcomes include 80% plus of tickets fully managed with human oversight, a 65% reduction in support cost or tickets, 20% higher CSAT, 8% better accuracy, and faster development and resolution."
---

# Evaluation-First AI Agents: How Zepto Scales Customer Support on Databricks and MLflow

[Databricks](https://yomu.fyi/company/databricks) · Gireesh Sreedhar KP, Deepak Dhankani, Eash Sharma · Sep 9, 2026

**Type:** Explainer

## Summary

Zepto describes its customer-support system, processing more than 100,000 tickets daily, and the assurance problems that emerged as volume, categories, and customer behavior changed. Its response is an evaluation-first architecture on Databricks and MLflow, using development and production loops connected by a quality gate. Agent traces capture prompts, completions, retrieved documents, tool calls, latencies, and decision paths through MLflow, OpenTelemetry spans, and Unity Catalog-backed Delta tables; stakeholders define evaluation pillars and gates around a living golden dataset. The dataset grew from 500 examples with an eight-point development–production accuracy gap to 5,247 examples with a 0.4-point gap over six months, while prompt optimization and multiple scorers support iteration. Reported outcomes include 80% plus of tickets fully managed with human oversight, a 65% reduction in support cost or tickets, 20% higher CSAT, 8% better accuracy, and faster development and resolution.

## Context

Zepto needed to maintain reliable customer support as it handled more than 100,000 AI-agent tickets daily, expanded into additional product categories, experienced demand spikes, and served a more diverse, multilingual customer base. Failures could occur at any stage of multi-step agent workflows, while fixes were slow, internal errors were hidden by final answers, and reliability was difficult to assure as agents evolved.

## Approach / What changed

Zepto made evaluation core AI infrastructure using Databricks and MLflow. The framework connects development and production loops through a quality gate, enables detailed MLflow tracing with OpenTelemetry and Unity Catalog-backed Delta tables, defines stakeholder-specific evaluation pillars, maintains a living golden dataset, and uses automated prompt optimization, multiple scorers, monitoring, and feedback to guide deployment and iteration.

## Takeaways

- LLM judges require calibration: the framework recommends measuring their agreement with human labels, using ensembles for high-stakes cases, and recalibrating them as base models change.
- Stratified sampling focused on high-risk flows produced an order of magnitude better issue detection per dollar than naive uniform sampling.
- Automating detection, labeling, retraining, deployment, and monitoring saved about 155 hours per month, equivalent to two full-time engineers redirected to feature work.

**Tags:** [Databricks](https://yomu.fyi/topic/databricks), [MLflow](https://yomu.fyi/topic/mlflow), [Model Evaluation](https://yomu.fyi/topic/model-evaluation)

- Source: [Databricks](https://www.databricks.com/blog/evaluation-first-ai-agents-how-zepto-scales-customer-support-databricks-and-mlflow)
- Source URL: https://www.databricks.com/blog/evaluation-first-ai-agents-how-zepto-scales-customer-support-databricks-and-mlflow
- Ingested by Yomu: 2026-09-09T04:00:54.026Z

[Read original post](https://www.databricks.com/blog/evaluation-first-ai-agents-how-zepto-scales-customer-support-databricks-and-mlflow)