# Training Orchestrator: Unifying Model Training at Yelp

[Yelp](https://yomu.fyi/company/yelp) · Ying Wang and Nathan Sponberg, Software Engineer · Jul 14, 2026

**Type:** Problem & solution

## Summary

Yelp developed Training Orchestrator to standardize machine learning pipeline execution and eliminate divergent training scripts across its applied engineering teams. Previously, monolithic Spark scripts coupled training logic directly to cluster runtimes, preventing local testing, slowing debug cycles, and causing inconsistent configuration drift across projects. The new framework introduces a declarative configuration layer backed by Pydantic models, enabling static type checking and step schema validation before runtime execution starts. Training workflows are defined as modular step hierarchies that the orchestration engine compiles into directed acyclic graphs for execution in topological order. By injecting Spark and MLflow contexts into steps, the architecture ensures reproducible tracking, automated Slack alerts, and parity across local development and production environments.

## Context

Applied machine learning teams at Yelp maintained separate Spark-based training batches, scripts, and configurations. Over time, these diverged, resulting in code duplication, configuration drift, scattered validation checks, and fragile custom scripts for monitoring and notifications. Monolithic training scripts were tightly coupled to Spark cluster runtimes and job submission mechanisms, preventing local runs and making unit and integration testing difficult.

## Approach / What changed

Yelp built Training Orchestrator, a declarative, configuration-driven platform that separates training code from execution infrastructure. Pipelines are defined using Pydantic configurations that enforce type safety and validate step input and output schemas upfront. The orchestrator automatically builds a directed acyclic graph from declared step dependencies, executes steps in topological order, injects Spark and MLflow execution contexts, and standardizes Slack notifications.

## Takeaways

- Decoupling training logic from Spark cluster submission allows developers to run identical pipeline configurations locally or in Jupyter, making unit testing and local debugging feasible.
- Training Orchestrator uses Pydantic to validate orchestrator parameters and step input/output schemas at instantiation, catching configuration mismatches before initiating distributed compute jobs.
- Defining workflows declaratively lets the engine construct a directed acyclic graph that enforces prerequisite completion, topological step execution, and shared MLflow tracking across all steps.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Developer Experience](https://yomu.fyi/topic/developer-experience), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Python](https://yomu.fyi/topic/python)

[Read original post](https://engineeringblog.yelp.com/2026/07/training-orchestrator-unifying-model-training-at-yelp.html)