New in llama.cpp: Model Management

Summary

llama.cpp server now includes a router mode that enables dynamic loading, unloading, and switching between multiple LLMs without restarting the server. The architecture runs each model in an isolated process to ensure a single model crash does not affect other active instances. Running llama-server without specifying a model activates auto-discovery across the cache directory or a designated folder of GGUF files. In addition to on-demand loading and least-recently-used eviction capped by default at four models, the server provides endpoints for manual loading, unloading, and listing model statuses. Models inherit global configuration options or use dedicated configuration presets while also integrating directly into the built-in web UI.

Context

Users requested Ollama-style dynamic model management in llama.cpp server to switch between models, conduct A/B testing, or support multi-tenant deployments without restarting the server.

Approach / What changed

llama.cpp introduced router mode with a multi-process architecture where each model runs in an isolated process. It incorporates GGUF file auto-discovery, on-demand loading, request routing via the model field, LRU eviction, and configuration presets.

Takeaways

• Router mode uses a multi-process architecture where each model operates in its own process, isolating crashes between instances.
• Models load on demand upon request and are evicted using an LRU strategy once the maximum capacity, configured by --models-max with a default of 4, is reached.
• Models inherit global server flags like context size and GPU offloading by default, but can be customized with per-model settings using INI preset files.

Related reading

• Hugging Face · Dec 4, 2025

  We Got Claude to Fine-Tune an Open Source LLM

  Hugging Face Skills equips coding agents like Claude Code, OpenAI Codex, and Google's Gemini CLI to manage end-to-end language model fine-tuning. Using the hf-llm-trainer skill, an agent validates dataset formats, selects cloud hardware, configures authentication, and submits jobs to Hugging Face Jobs. Supported techniques include Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO) for models ranging across various parameter sizes. The integration incorporates Trackio for real-time monitoring and automates LoRA configuration for larger models. Once training completes, the agent pushes artifacts to the Hugging Face Hub and can convert models to GGUF format for local execution.

  ben burtenshaw, shaun smith

• Hugging Face · Nov 20, 2025

  Introducing AnyLanguageModel: One API for Local and Remote LLMs on Apple Platforms