Loading post…
Hugging FaceYousaf ShahExplainer
Granite 4.1 LLMs: How They’re Built
Summary
Granite 4.1 is a family of dense decoder-only language models in 3B, 8B, and 30B sizes, trained on approximately 15T tokens. Its five-phase pre-training pipeline moves from broad web-scale data to progressively curated mixtures, increases math and code emphasis, and extends context from 4K to 512K tokens through staged long-context training. Supervised fine-tuning uses about 4.1M curated samples evaluated with LLM-as-Judge and rule-based checks, while reinforcement learning uses on-policy GRPO with DAPO loss. The 8B instruct model matches or surpasses the earlier Granite 4.0-H-Small 32B-A9B MoE model despite its simpler dense architecture and fewer parameters, and the Granite 4.1 models are released under Apache 2.0.
Context
The work addresses the challenge of building high-quality small language models, emphasizing rigorous data curation rather than relying only on increased compute or model scale. The stated goals include strengthening math, coding, instruction following, general chat, tool use, and long-context performance.
Approach / What changed
Granite 4.1 uses dense decoder-only transformers with GQA, RoPE, SwiGLU, RMSNorm, and shared input/output embeddings. Training combines five pre-training phases over approximately 15T tokens, staged long-context extension to 512K tokens, supervised fine-tuning on curated data assessed by LLM-as-Judge and rule-based filters, and multi-stage reinforcement learning using on-policy GRPO with DAPO loss.
Takeaways
The five pre-training phases progressively shift from broad web data toward higher-quality math, code, technical, instruction, and reasoning mixtures.
Long-context extension proceeds through 32K, 128K, and 512K stages; the 512K stage uses 80% books and 20% code repository data for the 8B and 30B models.
The 8B instruct model reportedly matches or surpasses Granite 4.0-H-Small, a 32B-A9B MoE model, while using a dense architecture with fewer parameters.
Source and freshness
- Source
- huggingface.co
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.