Systems Engineering Playbook: Optimizing Qwen 3.5-397B MoE on Ironwood (TPU7x)

Summary

Serving Qwen3.5-397B, a 397-billion-parameter sparse Mixture-of-Experts model, on Ironwood (TPU v7x) requires fitting roughly 400 GB of weights into HBM while sustaining utilization across contrasting workloads. The report applies a modular, model-agnostic playbook built from reusable JAX/Pallas kernels and hardware-aware cost models, focusing new work on Gated DeltaNet linear attention and Attention Data Parallelism while integrating changes into vLLM and SGLang. The architecture activates exactly 17 billion parameters per token and combines 15 repeating blocks of three GDN layers and one GQA layer. Benchmarks use a single host with four Ironwood chips, eight logical devices, 8K/1K prefill-heavy and 1K/8K decode-heavy workloads, and concurrency tiers from 64 to 512. Between April and June 2026, stated optimizations improved inference performance by approximately 4.7x for 512-concurrency prefill-heavy workloads and 3.1x for decode-heavy workloads.

Takeaways

1. 1

   Qwen3.5-397B has 397 billion total parameters but activates exactly 17 billion per token, routing through 512 experts with the top 10 selected and one shared expert path always executed.

2. 2

   The evaluation covers 8K-input/1K-output prefill-heavy and 1K-input/8K-output decode-heavy workloads at concurrency levels of 64, 128, 256, and 512 on eight logical Ironwood devices.

3. 3

   The numerical verification layer reports zero deviation between Pallas-lowered gating weights and the high-precision Float32 reference path while monitoring softmax ranges and expert load balance.