# From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines

[Slack](https://yomu.fyi/company/slack) · Mahendran Vasagam · May 5, 2026

**Type:** Problem & solution

## Summary

Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.

## Context

Slack's data platform accumulated over 700 SSH-based Airflow operators to execute jobs directly on production AWS EMR clusters. This pattern created security risks, key distribution overhead, and poor audit granularity. Operationally, commands ran on master nodes, and pod restarts severed SSH connections, causing orphaned zombie jobs or unverified job statuses. It also blocked modernization efforts like Spark on Kubernetes, EMR on EKS, and Slack's Whitecastle initiative to migrate EMR clusters to child accounts.

## Approach / What changed

Slack migrated all jobs to a REST-based architecture using Quarry, an internal job submission gateway providing unified REST APIs, service-to-service token authentication, and server-side lifecycle tracking. For non-Hadoop workloads and arbitrary CLI commands, Slack leveraged YARN Distributed Shell to run scripts uploaded to Amazon S3 within isolated YARN containers with explicit resource limits. Migrations rolled out progressively across Dev, GovDev/CommDev, and Production environments by operator type.

## Takeaways

- YARN Distributed Shell enabled Slack to run arbitrary shell scripts uploaded to S3 in isolated YARN containers via standard YARN REST APIs without custom execution infrastructure.
- Migrating to REST-based job submission decoupled Airflow workers from cluster nodes, allowing jobs to persist and be queried even when client orchestrator pods restart.
- Phased rollout across environments (Dev, CommDev/GovDev, Prod) and progressive deprecation per operator type prevented downtime across eight data regions.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Migrations](https://yomu.fyi/topic/migration), [REST APIs](https://yomu.fyi/topic/rest-api)

[Read original post](https://slack.engineering/from-ssh-to-rest-a-security-driven-modernization-of-slacks-emr-data-pipelines)