Loading post…
GrabChandulal KavarProblem & solution
The Journey of Deploying Apache Airflow at Grab
Summary
Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.
Context
Multiple teams ran their own Airflow setups without dedicated expertise, leading to resource inefficiency, frequent job failures, and duplicated efforts around scaling, upgrading, and logging.
Approach / What changed
A centralized team built a multi-instance orchestration platform on Amazon EKS using Terraform and Helm, isolating each team to a dedicated Kubernetes namespace with its own RDS, Redis, and customized Docker images built from a shared base image.
Takeaways
Airflow workers scale dynamically using Kubernetes Horizontal Pod Autoscalers driven by CPU and memory metrics.
Instances are categorized into small, medium, and large sizing tiers with tailored RDS, Redis, and component resource allocations.
Airflow 1.10.x lacked native support for Datadog metric tags, prompting internal patching and upstream contributions ahead of Airflow 2.0.0.
Source and freshness
- Source
- engineering.grab.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.