# Evaluating AI at Scale: How Thumbtack Approaches Reliability, Safety, and Quality in GenAI

[Thumbtack](https://yomu.fyi/company/thumbtack) · Thumbtack Engineering · Apr 29, 2026

**Type:** Problem & solution

## Summary

Thumbtack evaluates generative AI features across customer and service professional experiences to prevent unsupported claims, tone inconsistencies, and safety issues. The engineering organization transitioned from decentralized evaluation efforts to a dedicated cross-functional Evals team that provides shared infrastructure while preserving product team autonomy. System architectures combine rule-based checks, AI-as-a-judge scorers powered by DeepEval, trust and safety assessments, and crowdsourced human reviews. Teams leverage three parallel orchestration paths, including an MLflow-tracked stack, nightly Databricks batch jobs integrated with data warehouses, and multi-layer human-in-the-loop pipelines. Future milestones target centralized LLM judge registries, full task journey evaluations, automated judge-writing workflows, and cost-aware evaluation orchestration.

## Context

Generative AI outputs can yield unsupported claims, tone shifts, intent misunderstandings, safety risks, and temporal drift. Early decentralized evaluation efforts caused duplicated work and siloed knowledge across product teams.

## Approach / What changed

Thumbtack established a dedicated Evals team and a hybrid evaluation architecture combining rule-based checks, AI-as-a-judge metrics via DeepEval, trust and safety reviews, and crowdsourced human audits across three orchestration paths using MLflow and Databricks.

## Takeaways

- Transitioning from decentralized team evaluations to a centralized Evals team reduced duplicated engineering effort while letting product teams keep ownership of iteration cycles.
- Thumbtack wraps DeepEval GEval scorers with MLflow tracking to capture traces, scores, judge models, and metadata for prompt and model comparisons.
- Nightly Databricks jobs sample warehouse conversation data to run ConversationalGEval scorers, automatically pushing failing cases to human review sheets and Slack alerts.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Observability](https://yomu.fyi/topic/observability), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://medium.com/thumbtack-engineering/evaluating-ai-at-scale-how-thumbtack-approaches-reliability-safety-and-quality-in-genai-f75d0211ac54)