New in llama.cpp: Model Management

Summary

llama.cpp server now includes a router mode that enables dynamic loading, unloading, and switching between multiple LLMs without restarting the server. The architecture runs each model in an isolated process to ensure a single model crash does not affect other active instances. Running llama-server without specifying a model activates auto-discovery across the cache directory or a designated folder of GGUF files. In addition to on-demand loading and least-recently-used eviction capped by default at four models, the server provides endpoints for manual loading, unloading, and listing model statuses. Models inherit global configuration options or use dedicated configuration presets while also integrating directly into the built-in web UI.

Takeaways

1. 1

   Router mode uses a multi-process architecture where each model operates in its own process, isolating crashes between instances.

2. 2

   Models load on demand upon request and are evicted using an LRU strategy once the maximum capacity, configured by --models-max with a default of 4, is reached.

3. 3

   Models inherit global server flags like context size and GPU offloading by default, but can be customized with per-model settings using INI preset files.