---
title: "Transformers now runs llama.cpp quants"
description: "Transformers now supports running GGUF quantized models locally, reusing llama.cpp’s ggml kernels for efficient inference on Apple Silicon. The GGUF format packages weights, tokenizer, and chat template in one file, with quantization levels like Q4_K_M that trade precision for memory. Users can load a checkpoint from the Hub with from_pretrained and generate text using the standard Transformers API, or serve it via an OpenAI-compatible endpoint. The initial focus is on the Qwen3.5 architecture, with benchmarks on a MacBook Pro M2 Max showing token-generation rates close to llama.cpp, including prefill time. The integration uses packed quantization kernels and generation-loop optimizations to reduce overhead, though it is currently limited to single conversations on MPS without batching support."
---

# Transformers now runs llama.cpp quants

[Hugging Face](https://yomu.fyi/company/hugging-face) · Marc Sun, Arthur Zucker, Lysandre · Sep 22, 2026

**Type:** Announcement

## Summary

Transformers now supports running GGUF quantized models locally, reusing llama.cpp’s ggml kernels for efficient inference on Apple Silicon. The GGUF format packages weights, tokenizer, and chat template in one file, with quantization levels like Q4\_K\_M that trade precision for memory. Users can load a checkpoint from the Hub with from\_pretrained and generate text using the standard Transformers API, or serve it via an OpenAI-compatible endpoint. The initial focus is on the Qwen3.5 architecture, with benchmarks on a MacBook Pro M2 Max showing token-generation rates close to llama.cpp, including prefill time. The integration uses packed quantization kernels and generation-loop optimizations to reduce overhead, though it is currently limited to single conversations on MPS without batching support.

## Context

Local AI inference on laptops has become practical thanks to projects like llama.cpp and its GGUF format, but using these quantized models with the Transformers library required dequantization or separate tools. This work integrates GGUF support directly into Transformers, reusing ggml kernels for efficient packed inference on Apple Silicon.

## Approach / What changed

Reuse llama.cpp’s underlying ggml kernels via the kernels library, including quantization, layer kernels (ggml-norm, ggml-attn, ggml-gated-delta-net, topk), and generation-loop optimizations (dropping unnecessary attention mask, deferring stopping check). Load GGUF checkpoints with from\_pretrained and run packed inference on Apple Silicon MPS.

## Takeaways

- GGUF packages model weights, tokenizer, and chat template in one file with quantization levels like Q4\_K\_M that mix tensor precisions.
- Transformers’ GGUF support on Apple Silicon uses packed quantization kernels to keep weights in quantized form during inference, avoiding dequantization overhead.
- The generation loop optimizations (mask removal and deferred stopping check) benefit all Transformers models, not just GGUF, by reducing CPU-GPU synchronization.

## FAQ

### How do I load a GGUF model in Transformers?

Pass the Hub model\_id and the gguf\_file parameter to from\_pretrained. For example: AutoModelForCausalLM.from\_pretrained('unsloth/Qwen3.5-4B-GGUF', gguf\_file='Qwen3.5-4B-Q4\_K\_M.gguf'). No extra configuration is needed; the library automatically loads compatible ggml kernels.

### What hardware is supported for packed GGUF inference?

The initial target is Apple Silicon Macs using the MPS backend. The packed inference path is MPS-only for now; other devices fall back to dequantization.

### How does Transformers' GGUF performance compare to llama.cpp?

Benchmarks on a MacBook Pro M2 Max show token-generation rates close to llama.cpp across three checkpoints, with Transformers including prefill time in its measurement while llama-bench reports decode-only throughput.

**Tags:** [GGUF](https://yomu.fyi/topic/gguf), [llama.cpp](https://yomu.fyi/topic/llama-cpp), [Local AI](https://yomu.fyi/topic/local-ai), [Quantization](https://yomu.fyi/topic/quantization), [Transformers](https://yomu.fyi/topic/transformers)

- Source: [Hugging Face](https://huggingface.co/blog/transformers-llama-cpp-quants)
- Source URL: https://huggingface.co/blog/transformers-llama-cpp-quants
- Ingested by Yomu: 2026-09-22T12:00:12.762Z

[Read original post](https://huggingface.co/blog/transformers-llama-cpp-quants)