---
title: "We Were Wrong About GPUs"
description: "Fly.io’s GPU bet was intended to give internet-facing apps fast AI/ML inference through Fly GPU Machines, which pair a hardware-mapped Nvidia GPU with a Fly Machine. Building them required dedicated GPU servers, Intel Cloud Hypervisor with PCI passthrough, driver work, orchestrator changes, security assessments, and handling large model-weight files. The product has not found product-market fit because most software developers want LLM APIs and services such as OpenAI, Anthropic, Claude, GPT, Replicate, or RunPod rather than raw GPUs, while serious GPU users seek much larger clusters and lightweight users may be too uncertain a market. L40S demand exists, but those GPUs are not driving the core business. Fly.io will keep existing GPU Machines but does not plan a near-term v2, concluding that the bet was worth making without compromising its isolation or developer experience."
---

# We Were Wrong About GPUs

[Fly.io](https://yomu.fyi/company/fly-io) · Feb 14, 2025

**Type:** Incident / postmortem

## Summary

Fly.io’s GPU bet was intended to give internet-facing apps fast AI/ML inference through Fly GPU Machines, which pair a hardware-mapped Nvidia GPU with a Fly Machine. Building them required dedicated GPU servers, Intel Cloud Hypervisor with PCI passthrough, driver work, orchestrator changes, security assessments, and handling large model-weight files. The product has not found product-market fit because most software developers want LLM APIs and services such as OpenAI, Anthropic, Claude, GPT, Replicate, or RunPod rather than raw GPUs, while serious GPU users seek much larger clusters and lightweight users may be too uncertain a market. L40S demand exists, but those GPUs are not driving the core business. Fly.io will keep existing GPU Machines but does not plan a near-term v2, concluding that the bet was worth making without compromising its isolation or developer experience.

## Context

Fly.io expected internet-facing application developers to want GPUs for AI/ML inference, but found that most developers instead want access to LLMs through APIs. The company also encountered difficult Nvidia driver support, security risks around GPU memory transfers, uncertain demand for smaller virtual GPUs, and hardware requirements that did not align cleanly with its market.

## Approach / What changed

Fly.io built GPU Machines as Fly Machines with hardware-mapped Nvidia GPUs, using dedicated GPU servers and Intel Cloud Hypervisor with PCI passthrough. It funded security assessments, modified orchestration to provide Nvidia drivers, and worked for months to adapt Nvidia host drivers, while preserving the existing Fly Machine isolation and developer experience.

## Takeaways

- GPU Machines required dedicated servers because GPU and non-GPU workloads were not mixed, leaving GPU infrastructure substantially less utilized and less cost-effective than ordinary servers.
- Nvidia’s supported paths involved Kubernetes with a shared Linux kernel or conventional hypervisors such as QEMU, but neither matched Fly Machines’ desired millisecond startup experience.
- Fly.io will retain GPU Machines but scale back its ambitions, viewing the bet as valuable learning while preserving its isolation model and developer experience.

**Tags:** [AI](https://yomu.fyi/topic/ai), [GPU](https://yomu.fyi/topic/gpu), [Security](https://yomu.fyi/topic/security)

- Source: [Fly.io](https://fly.io/blog/wrong-about-gpu)
- Source URL: https://fly.io/blog/wrong-about-gpu
- Ingested by Yomu: 2026-09-02T12:01:00.403Z

[Read original post](https://fly.io/blog/wrong-about-gpu)