---
title: "What does 99.9% uptime mean for inference?"
description: "The post defines three inference reliability tiers by the failure domains each must survive: 99% requires surviving node-level GPU failures like ECC errors and thermal throttling through automated health checks and fast replica replacement within a single data center. 99.9% requires surviving a full data-center failure by deploying weights across two facilities with enough capacity on each side and routing live traffic to both continuously, not a cold standby. 99.99% requires multi-region deployment with AZ redundancy and reserved failover capacity. The authors argue that vague SLA definitions hide which failure domains are covered, and that infrastructure ownership—whether a provider controls its hardware or rents from a hyperscaler—determines how quickly failures are resolved."
---

# What does 99.9% uptime mean for inference?

[Together AI](https://yomu.fyi/company/together-ai) · Jul 16, 2026

**Type:** Explainer

## Summary

The post defines three inference reliability tiers by the failure domains each must survive: 99% requires surviving node-level GPU failures like ECC errors and thermal throttling through automated health checks and fast replica replacement within a single data center. 99.9% requires surviving a full data-center failure by deploying weights across two facilities with enough capacity on each side and routing live traffic to both continuously, not a cold standby. 99.99% requires multi-region deployment with AZ redundancy and reserved failover capacity. The authors argue that vague SLA definitions hide which failure domains are covered, and that infrastructure ownership—whether a provider controls its hardware or rents from a hyperscaler—determines how quickly failures are resolved.

## Context

The post breaks down what 99%, 99.9%, and 99.99% uptime actually require for GPU inference, focusing on the distinct failure domains at each level: node, data center, and region. It contrasts passive and active health checks, discusses the importance of continuous multi-datacenter traffic routing, and emphasizes infrastructure ownership and chip-to-token visibility as key differentiators for providers.

## Takeaways

- 99.9% inference uptime requires model weights deployed across two facilities with enough capacity on each side to absorb the full load and live traffic routing to both, not a cold standby.
- A provider renting infrastructure from a hyperscaler does not control power, cooling, or transit failure domains, meaning they file tickets instead of resolving issues directly.
- Four nines (99.99%) requires multi-region deployment with AZ redundancy and reserved capacity in the failover region sized to absorb a full regional outage.

## FAQ

### What is the difference between active and passive health checks for GPU inference?

Passive health checks use hardware telemetry without capacity overhead but miss failures that only appear under real GPU load. Active health checks catch those but require unused capacity; Together AI integrates checks with the scheduler to run in workload gaps.

### Why does infrastructure ownership matter for inference SLAs?

A provider that owns its hardware has chip-to-token visibility and one ticket covers hardware, network, storage, and software. Renting from a hyperscaler means filing a ticket with the infra provider, adding queue time and uncertainty in response.

### How should failover for 99.9% uptime be tested?

Failover should be tested with live traffic continuously, not just periodic drills. The key question is whether the provider runs traffic to both facilities at all times or maintains a cold standby.

**Tags:** [GPU](https://yomu.fyi/topic/gpu), [Inference](https://yomu.fyi/topic/inference), [Reliability](https://yomu.fyi/topic/reliability), [SLA](https://yomu.fyi/topic/sla), [Uptime](https://yomu.fyi/topic/uptime)

- Source: [Together AI](https://www.together.ai/blog/99-9-uptime-for-inference)
- Source URL: https://www.together.ai/blog/99-9-uptime-for-inference
- Ingested by Yomu: 2026-09-19T10:13:40.937Z

[Read original post](https://www.together.ai/blog/99-9-uptime-for-inference)