Shipyard: How We Built Slack’s Next-Generation EC2 Platform

Summary

Slack developed Shipyard to modernize its EC2 infrastructure by replacing continuously updated, mutable instances with immutable deployable artifacts. The platform adopts a layered image architecture starting with a standardized base image called slack-zero, which is built using AWS Image Builder and consumed by downstream service-specific machine images. Rather than running periodic Chef jobs in the background, configuration management is restricted to image baking and initial instance provisioning phases. Fleet management is handled through the Peekaboo inventory system, metric-driven progressive rollouts via Gondola, and automated instance rotation managed by a lifecycle component known as the Reaper. Developers validate cookbook updates prior to merging using Ship Quick, which provisions real infrastructure across dedicated worker fleets to verify provisioning workflows.

Context

Continuously updating long-lived EC2 instances across tens of thousands of machines led to configuration drift, complex service-level deployments, and operational overhead. While containerization addressed these issues for certain workloads, critical components such as Kubernetes worker nodes and egress network stacks could not migrate easily.

Approach / What changed

Slack built Shipyard, an EC2 platform that treats instances as immutable artifacts using a layered AMI model anchored by a shared base image named slack-zero. Configuration execution was moved from periodic background runs to image baking and provisioning phases, complemented by automated instance recycling through the Reaper, fleet visibility via Peekaboo, and pull request testing through Ship Quick.

Takeaways

• Base images named slack-zero are built using AWS Image Builder, which updates SSM parameters and triggers EventBridge and Lambda workflows to rebuild downstream service AMIs.
• The Reaper component enforces instance rotation and immutability with configurable rate limits across services, regions, and availability zones, alongside an emergency pause mechanism using an S3 control object.
• Ship Quick tests cookbook changes on real infrastructure by routing jobs to Longshoremen worker fleets split between vanilla Ubuntu for base images and slack-zero for service images.

Related reading

• Slack · May 5, 2026

  From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines

  Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.

  Mahendran Vasagam

• Grab · Jan 6, 2020

  Plumbing At Scale

  Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling.

  Karan Kamath

• Slack · May 28, 2026

  Slack AI: The Path to Multi-Cloud

  Slack evolved its Slack AI serving infrastructure across multiple phases to handle enterprise LLM workloads reliably and securely. The initial deployment on AWS SageMaker provided zero-knowledge escrow VPC isolation and FedRAMP compliance, but engineers faced scaling latency, GPU scarcity, and significant operational overhead. Slack then migrated live traffic to Amazon Bedrock to leverage managed Model Units and eliminate model release lag without customer-facing incidents. However, fixed Provisioned Throughput commitments and regional peak traffic variations created persistent underutilization challenges. Consequently, Slack expanded into a multi-cloud orchestration architecture that normalizes disparate provider APIs, integrates unified cross-cloud telemetry, and routes traffic dynamically around latency spikes and outages.

  Shaurya Kethireddy

• Dropbox · Mar 25, 2026

  Reducing our monorepo size to improve developer velocity

  Dropbox reduced its core server monorepo size from 87GB to 20GB after approaching GitHub's 100GB limit and suffering from hour-long clone times. The excessive bloat was traced to Git's delta compression heuristics interacting poorly with internationalization directory paths, generating inefficient diffs across different languages. By collaborating with GitHub to run an aggressive server-side repack with tuned window and depth settings, Dropbox cut clone times to under 15 minutes.

  Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra