Lessons Learned From Running Apache Airflow at Scale

Summary

Shopify describes operating Apache Airflow 2.2 on Kubernetes with the Celery executor and MySQL 8 for data extraction, machine-learning training, Iceberg maintenance, and DBT modeling. In its largest environment, Shopify runs over 10,000 DAGs, averages over 400 tasks running at a given moment, and executes over 150,000 runs per day, while facing slow file access, metadata growth, ownership tracking, DAG-author control, traffic bursts, and resource contention. The authors describe replacing per-pod GCSFuse access with an in-cluster NFS volume synchronized to GCS, retaining GCS as the user-facing source of truth; they also use 28-day metadata retention, a YAML manifest, DAG policies, and scheduling controls including pools, priority weights, and Celery queues. These measures are presented as ways to improve file performance, limit database growth, identify owners, enforce constraints, and reduce workload interference, though the retention policy does not support durable history-dependent features such as long-running backfills.

Context

Shopify's Airflow deployments grew to over 10,000 DAGs in the largest environment, with more than 400 tasks running at a given moment and over 150,000 runs per day. This growth exposed slow file access, accumulating metadata, difficulty associating DAGs with users and teams, broad author permissions, irregular traffic, and resource contention between workloads.

Approach / What changed

Shopify runs Airflow 2.2 on Kubernetes with the Celery executor and MySQL 8. Its scaling measures include an in-cluster NFS server synchronized with GCS, a 28-day metadata retention DAG, per-environment YAML manifest files, DAG policies that enforce namespace constraints, standardized schedule generation, and resource controls using pools, priority weights, Celery queues, and isolated workers.

Takeaways

• An in-cluster NFS server mounted by scheduler and worker pods reduced the performance bottleneck caused by GCSFuse, while a synchronization script kept the volume aligned with GCS.
• Shopify retained Airflow metadata for 28 days to balance incident investigation and job-performance history against database load; this means long-running backfills and other durable-history features are not supported.
• Manifest files connect DAG namespaces to owners and source repositories, while DAG policies enforce restrictions such as namespace-prefixed IDs, allowed Celery queues, and permitted pools.

Related reading