---
title: "Scaling Large Language Models to zero with Ollama"
description: "Running GPU nodes for language-model inference can be costly because they often sit idle between requests, so this guide configures Ollama on a Fly.io Machine that scales to zero when idle. Ollama is described as a wrapper around llama.cpp, enabling self-hosted models with GPU acceleration, while about 70 lines of Go add authentication to protect the endpoint from unauthorized use. Deployment uses an A100-40GB GPU, a private Flycast address, automatic start and stop settings, and a 100GB persistent volume mounted at /root/.ollama so downloaded models survive stopped Machines. A stopped Machine no longer incurs GPU charges, but scaling to zero suspends it on the same host and may fail to wake during capacity issues. The setup supports CLI and API requests, WireGuard access, custom Modelfiles, and JSON-mode function-calling workflows."
---

# Scaling Large Language Models to zero with Ollama

[Fly.io](https://yomu.fyi/company/fly-io) · Dec 6, 2023

**Type:** Tutorial

## Summary

Running GPU nodes for language-model inference can be costly because they often sit idle between requests, so this guide configures Ollama on a Fly.io Machine that scales to zero when idle. Ollama is described as a wrapper around llama.cpp, enabling self-hosted models with GPU acceleration, while about 70 lines of Go add authentication to protect the endpoint from unauthorized use. Deployment uses an A100-40GB GPU, a private Flycast address, automatic start and stop settings, and a 100GB persistent volume mounted at /root/.ollama so downloaded models survive stopped Machines. A stopped Machine no longer incurs GPU charges, but scaling to zero suspends it on the same host and may fail to wake during capacity issues. The setup supports CLI and API requests, WireGuard access, custom Modelfiles, and JSON-mode function-calling workflows.

## Context

GPU nodes are expensive and commonly remain idle between uses. Ollama does not provide authentication by default, creating a risk that unauthorised internet users could consume GPU time and incur costs. Scaling to zero reduces charges when the Machine is not running, while local model storage avoids repeated downloads after shutdown.

## Approach / What changed

The deployment runs the ollama/ollama image on a Fly.io Machine with an A100-40GB GPU, a private Flycast address, automatic start and stop, and zero minimum running Machines. A 100GB volume stores models at /root/.ollama. A small Go shim adds authentication, and clients can access the service through the CLI, API, or WireGuard.

## Takeaways

- Fly.io scaling to zero stops the Machine on its existing host rather than moving it to new capacity; a stopped Machine may be unable to wake if there is a capacity issue.
- A persistent volume mounted at /root/.ollama keeps downloaded Ollama models available across Machine stops, avoiding a re-download on every startup.
- Ollama's JSON mode can support function-calling workflows, but handling cases where no function should be called requires LangChain or manual iteration.

**Tags:** [Fly.io](https://yomu.fyi/topic/fly-io), [GPU](https://yomu.fyi/topic/gpu), [LLMs](https://yomu.fyi/topic/llm), [Ollama](https://yomu.fyi/topic/ollama)

- Source: [Fly.io](https://fly.io/blog/scaling-llm-ollama)
- Source URL: https://fly.io/blog/scaling-llm-ollama
- Ingested by Yomu: 2026-09-03T01:23:37.408Z

[Read original post](https://fly.io/blog/scaling-llm-ollama)