# The Journey of Deploying Apache Airflow at Grab

[Grab](https://yomu.fyi/company/grab) · Chandulal Kavar · Jul 14, 2020

**Type:** Problem & solution

## Summary

Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.

## Context

Multiple teams ran their own Airflow setups without dedicated expertise, leading to resource inefficiency, frequent job failures, and duplicated efforts around scaling, upgrading, and logging.

## Approach / What changed

A centralized team built a multi-instance orchestration platform on Amazon EKS using Terraform and Helm, isolating each team to a dedicated Kubernetes namespace with its own RDS, Redis, and customized Docker images built from a shared base image.

## Takeaways

- Airflow workers scale dynamically using Kubernetes Horizontal Pod Autoscalers driven by CPU and memory metrics.
- Instances are categorized into small, medium, and large sizing tiers with tailored RDS, Redis, and component resource allocations.
- Airflow 1.10.x lacked native support for Datadog metric tags, prompting internal patching and upstream contributions ahead of Airflow 2.0.0.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [CI/CD](https://yomu.fyi/topic/ci-cd), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kubernetes](https://yomu.fyi/topic/kubernetes)

[Read original post](https://engineering.grab.com/the-journey-of-deploying-apache-airflow-at-Grab)