We're Cutting L40S Prices In Half

Summary

Fly.io lowered the hourly price of NVIDIA L40S GPUs to $1.25, making them available at the same price as A10s after customer data showed that A10s—not the more powerful cards—were its most popular GPU. The post argues that most users adding GPUs to fast networks are running inference rather than training, making networking, memory, object storage, and request latency more important than access to an H100. The L40S is described as an AI-optimized L40 with AI compute performance comparable to an A100, while the L40 retains graphics hardware and is designed for rack use with more memory and lower energy consumption than gaming GPUs. The intended combination places model parameters, datasets, and compute near one another on Fly.io’s Anycast network and Tigris object storage. Examples include Llama 3.1 70B, Flux, Whisper, SegAlign, and DOOM Eternal.

Takeaways

1. 1

   A10 is Fly.io’s most popular GPU despite being its least capable offering, because it is sufficient for random inference and mid-sized generative AI workloads such as Mistral Nemo and Stable Diffusion.

2. 2

   The L40S combines AI compute performance comparable to an A100 with the graphics capabilities associated with the L40, and is priced at $1.25 per hour.

3. 3

   The post distinguishes batch-oriented training from transaction-like live inference, which is more sensitive to networking, reliability, and end-user HTTP response times.