Loading post…
Hugging FaceMarc Sun, Arthur Zucker, LysandreAnnouncement
Transformers now runs llama.cpp quants
Summary
Transformers now supports running GGUF quantized models locally, reusing llama.cpp’s ggml kernels for efficient inference on Apple Silicon. The GGUF format packages weights, tokenizer, and chat template in one file, with quantization levels like Q4_K_M that trade precision for memory. Users can load a checkpoint from the Hub with from_pretrained and generate text using the standard Transformers API, or serve it via an OpenAI-compatible endpoint. The initial focus is on the Qwen3.5 architecture, with benchmarks on a MacBook Pro M2 Max showing token-generation rates close to llama.cpp, including prefill time. The integration uses packed quantization kernels and generation-loop optimizations to reduce overhead, though it is currently limited to single conversations on MPS without batching support.
Context
Local AI inference on laptops has become practical thanks to projects like llama.cpp and its GGUF format, but using these quantized models with the Transformers library required dequantization or separate tools. This work integrates GGUF support directly into Transformers, reusing ggml kernels for efficient packed inference on Apple Silicon.
Approach / What changed
Reuse llama.cpp’s underlying ggml kernels via the kernels library, including quantization, layer kernels (ggml-norm, ggml-attn, ggml-gated-delta-net, topk), and generation-loop optimizations (dropping unnecessary attention mask, deferring stopping check). Load GGUF checkpoints with from_pretrained and run packed inference on Apple Silicon MPS.
Takeaways
GGUF packages model weights, tokenizer, and chat template in one file with quantization levels like Q4_K_M that mix tensor precisions.
Transformers’ GGUF support on Apple Silicon uses packed quantization kernels to keep weights in quantized form during inference, avoiding dequantization overhead.
The generation loop optimizations (mask removal and deferred stopping check) benefit all Transformers models, not just GGUF, by reducing CPU-GPU synchronization.
FAQ
How do I load a GGUF model in Transformers?
Pass the Hub model_id and the gguf_file parameter to from_pretrained. For example: AutoModelForCausalLM.from_pretrained('unsloth/Qwen3.5-4B-GGUF', gguf_file='Qwen3.5-4B-Q4_K_M.gguf'). No extra configuration is needed; the library automatically loads compatible ggml kernels.
What hardware is supported for packed GGUF inference?
The initial target is Apple Silicon Macs using the MPS backend. The packed inference path is MPS-only for now; other devices fall back to dequantization.
How does Transformers' GGUF performance compare to llama.cpp?
Benchmarks on a MacBook Pro M2 Max show token-generation rates close to llama.cpp across three checkpoints, with Transformers including prefill time in its measurement while llama-bench reports decode-only throughput.
Source and freshness
- Source
- huggingface.co
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.