Slack AI: The Path to Multi-Cloud

Summary

Slack evolved its Slack AI serving infrastructure across multiple phases to handle enterprise LLM workloads reliably and securely. The initial deployment on AWS SageMaker provided zero-knowledge escrow VPC isolation and FedRAMP compliance, but engineers faced scaling latency, GPU scarcity, and significant operational overhead. Slack then migrated live traffic to Amazon Bedrock to leverage managed Model Units and eliminate model release lag without customer-facing incidents. However, fixed Provisioned Throughput commitments and regional peak traffic variations created persistent underutilization challenges. Consequently, Slack expanded into a multi-cloud orchestration architecture that normalizes disparate provider APIs, integrates unified cross-cloud telemetry, and routes traffic dynamically around latency spikes and outages.

Context

Early in 2023, Slack faced the challenge of serving LLMs at enterprise scale while meeting security, FedRAMP compliance, and reliability standards. Managing custom model serving on AWS SageMaker created severe operational friction, including scaling latencies, shortages of Nvidia A100 and H100 GPUs, idle compute over-provisioning to satisfy peak workday surges, and model feature lag behind managed cloud offerings.

Approach / What changed

Slack transitioned from self-managed AWS SageMaker containers in escrow VPCs to Amazon Bedrock, utilizing Model Units for Provisioned Throughput on interactive features and On Demand compute for batch workloads. The architecture subsequently evolved into a multi-cloud, multi-provider orchestration system equipped with an API normalization layer, cross-cloud telemetry, and dynamic routing that treats latency spikes as soft failures.

Takeaways

• Shifting to Amazon Bedrock abstracted GPU hardware into Model Units for deterministic token throughput, but multi-month commitments and global traffic peaks still caused resource underutilization during off-peak windows.
• Operating across multiple cloud providers requires a normalization layer to unify distinct API error responses and rate-limiting behaviors, alongside combined telemetry dashboards to prevent blind spots for on-call teams.
• Slack treats performance degradation, such as p90 latency spikes, as soft service failures within its dynamic routing layer to maintain responsiveness rather than relying solely on binary uptime metrics.

Related reading

• Grab · Jan 16, 2026

  From deployment slop to production reality: How BriX bridges the gap with enterprise-grade AI infrastructure

  Internal AI prototypes frequently fail enterprise rollouts due to diverging versions, security oversights, hardcoded credentials, and infrastructure bottlenecks. BriX addresses this deployment gap by turning AI rollout into a configuration-driven platform rather than an engineering rewrite. Built on a synchronous streaming architecture, it routes user prompts through a React frontend using Server-Sent Events, a FastAPI gateway, and LangGraph orchestration. The platform integrates model switching, centralized prompt locks, and standardized Model Context Protocols for governed enterprise data access.

  Sneh Agrawal

• Slack · May 5, 2026

  From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines

  Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.

  Mahendran Vasagam

• Grab · Jul 24, 2026

  Agent platform (Part 1): How we help Grab build and run AI agents at scale

  Grab scaled its internal AI tooling by transforming the infrastructure behind a single technical support bot into a standardized agent framework named LLM-Kit. The original Go-based bot revealed critical bottlenecks, including the lack of automated evaluation, painful provider migrations, fragmented observability, and excessive production scaffolding overhead. To address these frictions, engineers developed a FastAPI template pre-wired with LangGraph ReAct agents, remote Model Context Protocol servers, and built-in ROUGE, BLEU, and LLM-as-judge evaluation suites. The framework centralizes model access through an OpenAI-compatible GrabGPT Gateway and unifies observability across services, outbound calls, and agent steps using OpenTelemetry. This extracted scaffolding reduced day-one production setup time from two weeks to roughly an hour while supporting hundreds of production agents.

  Kendrick Tan

• Grab · Dec 5, 2024

  How we seamlessly migrated high volume real-time streaming traffic from one service to another with zero data loss and duplication

  Grab split a backend service's read and write functionalities into separate services to allow independent scaling. Migrating the write path required transferring processing from 16 source Kafka streams—averaging 20,000 reads per second into DynamoDB tables and output streams—with zero data loss or duplication. Standard feature flags were ruled out because rollout propagation delays could introduce minutes of duplicate or missing data during flag toggling. Instead, engineers extracted processing logic into a shared monorepo commons package that used coordinated timestamps to trigger simultaneous cutovers across both services. Temporary validation sinks verified processing accuracy in production prior to the cutover, completing the stream-by-stream migration across three weeks without downtime.

  Md Riyadh