From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines

Summary

Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.

Takeaways

1. 1

   YARN Distributed Shell enabled Slack to run arbitrary shell scripts uploaded to S3 in isolated YARN containers via standard YARN REST APIs without custom execution infrastructure.

2. 2

   Migrating to REST-based job submission decoupled Airflow workers from cluster nodes, allowing jobs to persist and be queried even when client orchestrator pods restart.

3. 3

   Phased rollout across environments (Dev, CommDev/GovDev, Prod) and progressive deprecation per operator type prevented downtime across eight data regions.