Granite 4.2 LLMs: How They're Built

Summary

Granite 4.2 is IBM’s first family of dense, decoder-only reasoning language models, offered in 3B, 8B, and 30B sizes. The models use a shared design built around Grouped Query Attention, Rotary Position Embedding, SwiGLU, RMSNorm, bfloat16 precision, and separate input/output embeddings; each has a 131,072-token sequence length. Pre-training starts from scratch on approximately 15 trillion tokens across five phases, with the final phase extending the context window to 512K tokens. Supervised fine-tuning uses about 7.2 million samples spanning agentic and non-agentic data, followed by multi-stage reinforcement learning; the 8B and 30B models also receive agentic RL in sandboxed environments for tool use, code execution, terminal control, and web search. All models support thinking, non-thinking, and low-effort thinking modes, native tool calling, and Apache 2.0 licensing.

Takeaways

1. 1

   The three models use dense decoder-only transformers with Grouped Query Attention, 40 or 32 attention heads, 8 KV heads, RoPE with θ = 10,000,000, SwiGLU, RMSNorm, and bfloat16 precision.

2. 2

   The SFT mixture contains approximately 7.2 million samples, with 31.6% agentic and 68.4% non-agentic data; quality checks include LLM judging, heuristic filtering, and SHA-256-based local and global deduplication.

3. 3

   Native tool calling follows the OpenAI function-calling format through compatible endpoints, enabling integration with agentic harnesses; all models are released under the Apache 2.0 license.