GPUs on Fly.io are available to everyone!

Summary

Fly.io has made GPU instances available to everyone, adding datacenter-grade accelerators to its globally distributed compute platform. Users can select Ampere A100 variants with 40GB or 80GB of memory, or a Lovelace L40s with 48GB, by setting an app or process’s vm.size to the corresponding GPU kind. The post demonstrates an Ollama deployment using an L40s, a 100GB persistent volume mounted at /root/.ollama, and HTTP service settings that allow machines to stop and start automatically with zero minimum running machines. These instances are positioned for large language model inference, text transcription, and image generation; lower-cost A10 GPUs are planned for the next few weeks, while some organizations may need additional verification based on account age and payment history.

Takeaways

1. 1

   Available GPU options are the Ampere A100 with 40GB or 80GB of memory and the Lovelace L40s with 48GB.

2. 2

   The Ollama example mounts a 100GB volume at /root/.ollama and sets min_machines_running to 0, allowing the service to scale to zero.

3. 3

   Lower-cost A10 GPUs are expected in the next few weeks, and some organizations may require additional verification based on account age and payment history.