Evaluating AI at Scale: How Thumbtack Approaches Reliability, Safety, and Quality in GenAI

Summary

Thumbtack evaluates generative AI features across customer and service professional experiences to prevent unsupported claims, tone inconsistencies, and safety issues. The engineering organization transitioned from decentralized evaluation efforts to a dedicated cross-functional Evals team that provides shared infrastructure while preserving product team autonomy. System architectures combine rule-based checks, AI-as-a-judge scorers powered by DeepEval, trust and safety assessments, and crowdsourced human reviews. Teams leverage three parallel orchestration paths, including an MLflow-tracked stack, nightly Databricks batch jobs integrated with data warehouses, and multi-layer human-in-the-loop pipelines. Future milestones target centralized LLM judge registries, full task journey evaluations, automated judge-writing workflows, and cost-aware evaluation orchestration.

Context

Generative AI outputs can yield unsupported claims, tone shifts, intent misunderstandings, safety risks, and temporal drift. Early decentralized evaluation efforts caused duplicated work and siloed knowledge across product teams.

Approach / What changed

Thumbtack established a dedicated Evals team and a hybrid evaluation architecture combining rule-based checks, AI-as-a-judge metrics via DeepEval, trust and safety reviews, and crowdsourced human audits across three orchestration paths using MLflow and Databricks.

Takeaways

• Transitioning from decentralized team evaluations to a centralized Evals team reduced duplicated engineering effort while letting product teams keep ownership of iteration cycles.
• Thumbtack wraps DeepEval GEval scorers with MLflow tracking to capture traces, scores, judge models, and metadata for prompt and model comparisons.
• Nightly Databricks jobs sample warehouse conversation data to run ConversationalGEval scorers, automatically pushing failing cases to human review sheets and Slack alerts.

Related reading

• Airbnb · Jul 28, 2026

  Eval-driven development: Lessons from evaluating GenAI at scale

  Airbnb outlines its framework for eval-driven development (EDD), treating GenAI evaluation as a continuous engineering discipline rather than an afterthought. The strategy combines programmatic checks, calibrated LLM-as-a-judge evaluators, and human review to detect subtle quality regressions. By inspecting intermediate agentic execution traces and real-world failure modes, teams ensure generative models remain reliable and aligned with product requirements.

  Rohit Girme

• Thumbtack · May 22, 2026

  AI as the Next Abstraction Layer: How I see engineering evolving at Thumbtack

  Thumbtack is transitioning its engineering workflow from ad-hoc tool experimentation to end-to-end agentic software development. Unlike previous deterministic abstractions such as compilers or high-level languages, artificial intelligence introduces non-determinism where identical prompts can produce varying code and trade-offs. Because consumer marketplace experiments still require weeks for behavioral signals to stabilize, accelerated implementation primarily helps teams prepare cleaner experiment designs and parallelize hypotheses within experimentation bandwidth constraints. Consequently, day-to-day engineering responsibilities shift upstream from manual coding toward problem framing, system architecture, validation, and co-designing features with product managers. Teams maintain technical sharpness by rigorously reviewing generated code while accepting that long-term specialization above the abstraction layer will become standard across the organization.