---
title: "Differential Transformer V2"
description: "Differential Transformer V2 introduces an architectural refinement to differential attention that pairs and subtracts query heads within the same grouped-query attention group. The architecture doubles query heads while preserving key-value head counts, which avoids increasing key-value cache demands and removes the need for custom kernels. DIFF V2 replaces the per-head RMSNorm of its predecessor with a per-token, per-head projected lambda passed through a sigmoid function. This formulation expands the context root-mean-square range to between zero and root two, preventing massive gradient magnifications, mitigating attention sinks, and stabilizing large learning rate pretraining. Ablations confirm that sharing keys and values across subtracted heads and constraining lambda scaling are necessary to prevent training instability and elevated loss."
---

# Differential Transformer V2

[Hugging Face](https://yomu.fyi/company/hugging-face) · Li Dong · Jan 20, 2026

**Type:** Explainer

## Summary

Differential Transformer V2 introduces an architectural refinement to differential attention that pairs and subtracts query heads within the same grouped-query attention group. The architecture doubles query heads while preserving key-value head counts, which avoids increasing key-value cache demands and removes the need for custom kernels. DIFF V2 replaces the per-head RMSNorm of its predecessor with a per-token, per-head projected lambda passed through a sigmoid function. This formulation expands the context root-mean-square range to between zero and root two, preventing massive gradient magnifications, mitigating attention sinks, and stabilizing large learning rate pretraining. Ablations confirm that sharing keys and values across subtracted heads and constraining lambda scaling are necessary to prevent training instability and elevated loss.

## Context

Differential Transformer V1 suffered from slower decoding due to doubled value cache loading, required custom attention kernels, and used a per-head RMSNorm that magnified context vectors by up to roughly 100x during uniform attention, causing gradient spikes and numerical instability.

## Approach / What changed

DIFF V2 doubles the query heads to 2h while keeping key-value heads at h\_kv, aligning dimensions to execute via standard FlashAttention. It subtracts contiguous paired query heads within each GQA group that share identical keys and values, scaled by a token-and-head projected sigmoid lambda.

## Takeaways

- DIFF V2 preserves the standard Transformer key-value head count and output projection dimensions, avoiding key-value cache expansion and custom decoding kernels.
- Lowering the context root-mean-square lower bound to zero via sigmoid-scaled differential attention eliminates attention sinks and stabilizes pretraining gradients.
- Ablations show that failing to share keys and values across subtracted head pairs or omitting sigmoid activation on lambda leads to higher loss and training instability.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance)

- Source: [Hugging Face](https://huggingface.co/blog/microsoft/diff-attn-v2)
- Source URL: https://huggingface.co/blog/microsoft/diff-attn-v2
- Ingested by Yomu: 2026-08-27T20:00:56.638Z

[Read original post](https://huggingface.co/blog/microsoft/diff-attn-v2)