# Apriel-H1: The Surprising Key to Distilling Efficient Reasoning Models

[Hugging Face](https://yomu.fyi/company/hugging-face) · Torsten Scholak, Oleksiy Ostapenko, Raymond Li, Luke Kumar, Joel Lamy-Poirier · Nov 19, 2025

**Type:** Problem & solution

## Summary

Retrofitting efficiency into existing large language models often fails when naive distillation disrupts complex multi-step reasoning mechanisms. The Apriel-H1 project addresses this challenge by converting a 15B full-attention reasoning model into a Mamba hybrid using targeted distillation on high-quality SFT reasoning traces rather than diffuse pretraining data. Built on the open-source Fast-LLM framework, the process applies reverse KL divergence alongside a staged layer replacement strategy combining Leave-One-Out analysis and dynamic loss evaluations. Across fifty total blocks, replacing thirty attention layers with Mamba mixers yielded the flagship Apriel-H1-15b-Thinker-SFT checkpoint. This configuration achieved a 2.1x throughput increase with minimal benchmark degradation after 76.8B training tokens.

## Context

Teams possessing strong reasoning models often lack compute for multi-trillion-token pretraining or architectural co-design from scratch. Replacing quadratic attention layers in an existing 15B model with linear state space models through distillation frequently causes severe reasoning quality degradation when relying on standard pretraining data distributions.

## Approach / What changed

The authors developed a staged distillation pipeline using the open-source Fast-LLM framework to replace attention layers with Mamba-1 mixers. Distillation was executed using reverse KL divergence on high-quality teacher SFT reasoning traces. Layer selection utilized Leave-One-Out analysis on MMLU for the initial 25 layers, followed by dynamic MIL-Mamba-Replacement loss scoring for deeper progressive replacements up to 40 layers, followed by a final SFT pass.

## Takeaways

- Distilling reasoning capabilities into linear Mamba hybrids succeeds when training on structured multi-step reasoning traces rather than broad pretraining data.
- Reverse KL divergence outperformed forward KL by enforcing mode-seeking behavior that aligns student predictions with high-confidence teacher reasoning paths.
- Progressive conversion using Leave-One-Out analysis and dynamic short-step loss evaluations enabled replacing up to 40 of 50 attention layers for up to 3.4x throughput.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance)

[Read original post](https://huggingface.co/blog/ServiceNow-AI/apriel-h1)