Evaluating AI at Scale: How Thumbtack Approaches Reliability, Safety, and Quality in GenAI

Summary

Thumbtack evaluates generative AI features across customer and service professional experiences to prevent unsupported claims, tone inconsistencies, and safety issues. The engineering organization transitioned from decentralized evaluation efforts to a dedicated cross-functional Evals team that provides shared infrastructure while preserving product team autonomy. System architectures combine rule-based checks, AI-as-a-judge scorers powered by DeepEval, trust and safety assessments, and crowdsourced human reviews. Teams leverage three parallel orchestration paths, including an MLflow-tracked stack, nightly Databricks batch jobs integrated with data warehouses, and multi-layer human-in-the-loop pipelines. Future milestones target centralized LLM judge registries, full task journey evaluations, automated judge-writing workflows, and cost-aware evaluation orchestration.

Takeaways

1. 1

   Transitioning from decentralized team evaluations to a centralized Evals team reduced duplicated engineering effort while letting product teams keep ownership of iteration cycles.

2. 2

   Thumbtack wraps DeepEval GEval scorers with MLflow tracking to capture traces, scores, judge models, and metadata for prompt and model comparisons.

3. 3

   Nightly Databricks jobs sample warehouse conversation data to run ConversationalGEval scorers, automatically pushing failing cases to human review sheets and Slack alerts.