Loading…
From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines
SlackMahendran Vasagam
Summary
Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.
Context
Slack's data platform accumulated over 700 SSH-based Airflow operators to execute jobs directly on production AWS EMR clusters. This pattern created security risks, key distribution overhead, and poor audit granularity. Operationally, commands ran on master nodes, and pod restarts severed SSH connections, causing orphaned zombie jobs or unverified job statuses. It also blocked modernization efforts like Spark on Kubernetes, EMR on EKS, and Slack's Whitecastle initiative to migrate EMR clusters to child accounts.
Approach / What changed
Slack migrated all jobs to a REST-based architecture using Quarry, an internal job submission gateway providing unified REST APIs, service-to-service token authentication, and server-side lifecycle tracking. For non-Hadoop workloads and arbitrary CLI commands, Slack leveraged YARN Distributed Shell to run scripts uploaded to Amazon S3 within isolated YARN containers with explicit resource limits. Migrations rolled out progressively across Dev, GovDev/CommDev, and Production environments by operator type.
Takeaways
- YARN Distributed Shell enabled Slack to run arbitrary shell scripts uploaded to S3 in isolated YARN containers via standard YARN REST APIs without custom execution infrastructure.
- Migrating to REST-based job submission decoupled Airflow workers from cluster nodes, allowing jobs to persist and be queried even when client orchestrator pods restart.
- Phased rollout across environments (Dev, CommDev/GovDev, Prod) and progressive deprecation per operator type prevented downtime across eight data regions.
Related reading
Slack ·
Streamlining Security Investigations with Agents
Slack's Security Engineering team handles billions of daily security events and needed a reliable way to streamline on-call alert triage. An initial prototype relying on a single 300-word prompt produced inconsistent results and frequently reached spurious conclusions without properly challenging assumptions. To gain precise control, the team decomposed the workflow into chained model invocations with structured JSON outputs organized across three agent personas: a Director, four domain experts, and a Critic. Domain experts gather raw evidence through tool calls, the Critic evaluates finding quality and synthesizes a timeline, and the Director steers investigation phases using tiered model costs. The multi-agent system enables engineers to supervise investigations via a real-time dashboard while uncovering emergent issues like credential exposures across process ancestry chains.
Dominic MarksSlack ·
Managing context in long-run agentic applications
Long-running multi-agent systems struggle with context management because accumulating raw message histories degrades inference quality, increases latency, and exceeds context window limits. In a collaborative security investigation platform, passing unrestricted history can also introduce confirmation bias across specialized agents. To maintain coherence across unbounded rounds of investigation, the system eliminates raw message history carryover between invocations. Instead, it coordinates agents through three structured context channels: a Director's Journal for orchestration memory, a Critic's Review that scores findings to filter hallucinations, and a Critic's Timeline of validated chronological events. This architecture provides agents with tailored context without overwhelming their inference capacity.
Dominic MarksGrab ·
Taming the monorepo beast: Our journey to a leaner, faster GitLab repo
Grab's decade-old Go monorepo grew to 12.7 million commits and 250GB of Git data, causing Gitaly replication delays of up to four minutes that routed all read traffic exclusively to the primary node and slowed developer operations. After staging tests proved that shallow history reduced replication lag from hundreds of seconds to under three seconds, standard rewriting tools like git filter-repo and git rebase failed due to complex merge histories and repository scale. To overcome runner memory limits and lengthy git garbage collection cycles, the engineering team implemented a custom two-phase migration script. The script selectively migrated 2,000+ critical dependency tags and one month of recent history, flattening merge commits, embedding legacy hashes for traceability, and reducing total commit volume by 99.9%.
Nagendra GangwarSlack ·
Slack AI: The Path to Multi-Cloud
Slack evolved its Slack AI serving infrastructure across multiple phases to handle enterprise LLM workloads reliably and securely. The initial deployment on AWS SageMaker provided zero-knowledge escrow VPC isolation and FedRAMP compliance, but engineers faced scaling latency, GPU scarcity, and significant operational overhead. Slack then migrated live traffic to Amazon Bedrock to leverage managed Model Units and eliminate model release lag without customer-facing incidents. However, fixed Provisioned Throughput commitments and regional peak traffic variations created persistent underutilization challenges. Consequently, Slack expanded into a multi-cloud orchestration architecture that normalizes disparate provider APIs, integrates unified cross-cloud telemetry, and routes traffic dynamically around latency spikes and outages.
Shaurya Kethireddy