Scaling Large Language Models to zero with Ollama

Summary

Running GPU nodes for language-model inference can be costly because they often sit idle between requests, so this guide configures Ollama on a Fly.io Machine that scales to zero when idle. Ollama is described as a wrapper around llama.cpp, enabling self-hosted models with GPU acceleration, while about 70 lines of Go add authentication to protect the endpoint from unauthorized use. Deployment uses an A100-40GB GPU, a private Flycast address, automatic start and stop settings, and a 100GB persistent volume mounted at /root/.ollama so downloaded models survive stopped Machines. A stopped Machine no longer incurs GPU charges, but scaling to zero suspends it on the same host and may fail to wake during capacity issues. The setup supports CLI and API requests, WireGuard access, custom Modelfiles, and JSON-mode function-calling workflows.

Takeaways

1. 1

   Fly.io scaling to zero stops the Machine on its existing host rather than moving it to new capacity; a stopped Machine may be unable to wake if there is a capacity issue.

2. 2

   A persistent volume mounted at /root/.ollama keeps downloaded Ollama models available across Machine stops, avoiding a re-download on every startup.

3. 3

   Ollama's JSON mode can support function-calling workflows, but handling cases where no function should be called requires LangChain or manual iteration.