Apriel-H1: The Surprising Key to Distilling Efficient Reasoning Models

Summary

Retrofitting efficiency into existing large language models often fails when naive distillation disrupts complex multi-step reasoning mechanisms. The Apriel-H1 project addresses this challenge by converting a 15B full-attention reasoning model into a Mamba hybrid using targeted distillation on high-quality SFT reasoning traces rather than diffuse pretraining data. Built on the open-source Fast-LLM framework, the process applies reverse KL divergence alongside a staged layer replacement strategy combining Leave-One-Out analysis and dynamic loss evaluations. Across fifty total blocks, replacing thirty attention layers with Mamba mixers yielded the flagship Apriel-H1-15b-Thinker-SFT checkpoint. This configuration achieved a 2.1x throughput increase with minimal benchmark degradation after 76.8B training tokens.

Context

Teams possessing strong reasoning models often lack compute for multi-trillion-token pretraining or architectural co-design from scratch. Replacing quadratic attention layers in an existing 15B model with linear state space models through distillation frequently causes severe reasoning quality degradation when relying on standard pretraining data distributions.

Approach / What changed

The authors developed a staged distillation pipeline using the open-source Fast-LLM framework to replace attention layers with Mamba-1 mixers. Distillation was executed using reverse KL divergence on high-quality teacher SFT reasoning traces. Layer selection utilized Leave-One-Out analysis on MMLU for the initial 25 layers, followed by dynamic MIL-Mamba-Replacement loss scoring for deeper progressive replacements up to 40 layers, followed by a final SFT pass.

Takeaways

• Distilling reasoning capabilities into linear Mamba hybrids succeeds when training on structured multi-step reasoning traces rather than broad pretraining data.
• Reverse KL divergence outperformed forward KL by enforcing mode-seeking behavior that aligns student predictions with high-confidence teacher reasoning paths.
• Progressive conversion using Leave-One-Out analysis and dynamic short-step loss evaluations enabled replacing up to 40 of 50 attention layers for up to 3.4x throughput.

Related reading

• Hugging Face · Dec 23, 2025

  AprielGuard: A Guardrail for Safety and Adversarial Robustness in Modern LLM Systems

  Modern language model systems face security and safety threats spanning prompt injection, jailbreaks, and tool manipulation across long-context agentic workflows. To address these vulnerabilities, researchers introduced AprielGuard, an 8B parameter causal decoder-only transformer based on a downscaled Apriel-1.5 Thinker Base model. The model detects 16 safety risk categories alongside binary adversarial attacks across standalone prompts, multi-turn dialogues, and agentic workflows. Trained on synthetic datasets created with Mixtral-8x7B, NVIDIA NeMo Curator, and SyGra, AprielGuard operates in both explainable reasoning and fast non-reasoning modes. Evaluation shows high precision and recall on safety and adversarial benchmarks, though trade-offs exist between explainability latency and non-reasoning performance.

  Jaykumar Kasundra

• Apple · Aug 26, 2026

  PROOF-Gen: From Optimized Data to Better Distillation