# Slack AI: The Path to Multi-Cloud

[Slack](https://yomu.fyi/company/slack) · Shaurya Kethireddy · May 28, 2026

**Type:** Problem & solution

## Summary

Slack evolved its Slack AI serving infrastructure across multiple phases to handle enterprise LLM workloads reliably and securely. The initial deployment on AWS SageMaker provided zero-knowledge escrow VPC isolation and FedRAMP compliance, but engineers faced scaling latency, GPU scarcity, and significant operational overhead. Slack then migrated live traffic to Amazon Bedrock to leverage managed Model Units and eliminate model release lag without customer-facing incidents. However, fixed Provisioned Throughput commitments and regional peak traffic variations created persistent underutilization challenges. Consequently, Slack expanded into a multi-cloud orchestration architecture that normalizes disparate provider APIs, integrates unified cross-cloud telemetry, and routes traffic dynamically around latency spikes and outages.

## Context

Early in 2023, Slack faced the challenge of serving LLMs at enterprise scale while meeting security, FedRAMP compliance, and reliability standards. Managing custom model serving on AWS SageMaker created severe operational friction, including scaling latencies, shortages of Nvidia A100 and H100 GPUs, idle compute over-provisioning to satisfy peak workday surges, and model feature lag behind managed cloud offerings.

## Approach / What changed

Slack transitioned from self-managed AWS SageMaker containers in escrow VPCs to Amazon Bedrock, utilizing Model Units for Provisioned Throughput on interactive features and On Demand compute for batch workloads. The architecture subsequently evolved into a multi-cloud, multi-provider orchestration system equipped with an API normalization layer, cross-cloud telemetry, and dynamic routing that treats latency spikes as soft failures.

## Takeaways

- Shifting to Amazon Bedrock abstracted GPU hardware into Model Units for deterministic token throughput, but multi-month commitments and global traffic peaks still caused resource underutilization during off-peak windows.
- Operating across multiple cloud providers requires a normalization layer to unify distinct API error responses and rate-limiting behaviors, alongside combined telemetry dashboards to prevent blind spots for on-call teams.
- Slack treats performance degradation, such as p90 latency spikes, as soft service failures within its dynamic routing layer to maintain responsiveness rather than relying solely on binary uptime metrics.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [LLMs](https://yomu.fyi/topic/llm), [Reliability](https://yomu.fyi/topic/reliability), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://slack.engineering/slack-ai-the-path-to-multi-cloud)