Fly.io has GPUs now

Summary

Fly.io announces GPU support for its cloud, extending its near-user deployment model to AI and machine-learning workloads at the edge. The offering attaches Nvidia A100 GPUs with 40GB or 80GB of RAM, CUDA support, eight AMD EPYC CPU cores, and volumes up to 500GB; L40s are also listed as coming soon and then noted as available. A sample fly.toml deploys Ollama with a 100GB model volume, while auto-start and auto-stop settings can reduce min_machines_running to zero when idle. GPU apps can scale across supported regions, retaining the same public IP address and TLS certificates, with an Amsterdam example using fly scale count 2 --region ams. Listed prices are $2.50 per hour for A100 40GB and $3.50 for A100 80GB, with reserved-instance and dedicated-host discounts available.

Takeaways

1. 1

   A100 GPU options include 40GB for $2.50 per hour and 80GB for $3.50 per hour; the source also says L40s are now available for $2.50 per hour.

2. 2

   GPU applications can be scaled across supported regions while retaining the same public IP address and TLS certificates; the example adds Amsterdam with fly scale count 2 --region ams.

3. 3

   Setting auto_stop_machines and auto_start_machines to true with min_machines_running = 0 stops GPU charges when the application has no users requesting sandwich recipes.