---
title: "Run a Local LLM on a CPU With BitNet and a Sprite"
description: "Microsoft’s BitNet uses 1.58-bit quantization and ternary weights to run a 2.4 billion parameter model on a CPU, fitting in 1.1GB of RAM and achieving 50 tokens per second. The author deploys this model on a Sprite, a hardware-isolated compute unit that provides a URL with TLS, sleeps when idle, and wakes on request with a 1.2-second model load time. A Python HTTP server wraps llama-cli as a subprocess, exposing an OpenAI-compatible API at /v1/chat/completions, a status page, and a self-addressed client at /client.py. Checkpoints capture the entire environment atomically, allowing safe rollback when experimenting with different models, kernel parameters, or prompt formats. The Sprite registers as a service on port 8080, making the inference endpoint available to any other Sprite in the account without infrastructure changes."
---

# Run a Local LLM on a CPU With BitNet and a Sprite

[Fly.io](https://yomu.fyi/company/fly-io) · Nov 6, 2026

**Type:** Tutorial

## Summary

Microsoft’s BitNet uses 1.58-bit quantization and ternary weights to run a 2.4 billion parameter model on a CPU, fitting in 1.1GB of RAM and achieving 50 tokens per second. The author deploys this model on a Sprite, a hardware-isolated compute unit that provides a URL with TLS, sleeps when idle, and wakes on request with a 1.2-second model load time. A Python HTTP server wraps llama-cli as a subprocess, exposing an OpenAI-compatible API at /v1/chat/completions, a status page, and a self-addressed client at /client.py. Checkpoints capture the entire environment atomically, allowing safe rollback when experimenting with different models, kernel parameters, or prompt formats. The Sprite registers as a service on port 8080, making the inference endpoint available to any other Sprite in the account without infrastructure changes.

## Context

Deploying a 1.58-bit quantized LLM on a Sprite for shared inference endpoints in agent swarms, leveraging CPU-only inference and checkpoint-based experimentation.

## Approach / What changed

Clone the BitNet repository, compile with SIMD kernels for x86-64 with i2\_s quantization, fix a const-correctness bug in ggml-bitnet-mad.cpp, download the pre-quantized GGUF model from microsoft/bitnet-b1.58-2B-4T-gguf, build a Python HTTP server that wraps llama-cli as a subprocess and exposes an OpenAI-compatible API at /v1/chat/completions, a status page, and a self-addressed client at /client.py, then register the server as a Sprite service on port 8080.

## Takeaways

- BitNet’s 1.58-bit quantization enables a 2.4B parameter model to run on CPU at ~50 tokens/sec with 1.1GB RAM.
- Running on a Sprite provides a shared inference endpoint with TLS, sleep-wake lifecycle, and checkpoint-based experimentation.
- The server self-serves its own client at /client.py, allowing any Sprite to call it with a single curl command.

## FAQ

### What is the performance of the model?

The model generates approximately 50 tokens per second with 1.4 seconds end-to-end latency, using 1.3GB total RAM (model + KV cache).

### How do I set up the model on a Sprite?

Clone the BitNet repo, compile with cmake and libomp-dev, install dependencies, fix a const-correctness bug, download the pre-quantized GGUF, build a Python server wrapping llama-cli, and register as a Sprite service on port 8080.

### Why run a local model on a Sprite instead of using a frontier model?

The Sprite provides a shared inference endpoint with TLS for agent swarms, sleeps when idle to avoid paying for idle compute, and checkpoints allow atomic rollback for safe experimentation.

**Tags:** [BitNet](https://yomu.fyi/topic/bitnet), [LLMs](https://yomu.fyi/topic/llm), [Quantization](https://yomu.fyi/topic/quantization), [Sprite](https://yomu.fyi/topic/sprite)

- Source: [Fly.io](https://fly.io/blog/local-llm-on-a-sprite)
- Source URL: https://fly.io/blog/local-llm-on-a-sprite
- Ingested by Yomu: 2026-11-06T18:00:14.263Z

[Read original post](https://fly.io/blog/local-llm-on-a-sprite)
