How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin

Summary

Serving agentic AI at more than 3,000 output tokens per second while retaining a 100K-token KV cache creates a small-batch tensor-parallelism challenge: fixed communication and coordination costs can offset distributed-compute gains. NVIDIA Groq 3 LPX addresses this with compiler-scheduled execution across 256 LP30 local processing units, 128 GB of collective SRAM, and 96 112-Gbps C2C links per chip, planning transfers before execution and overlapping communication with computation at 320-byte-vector granularity. Artificial Analysis measured 3,431 output tokens per second at 100K context on Gemma 4 31B, and 3,382 tokens per second at 10K context. On SPEED-Bench coding tasks, the same system reached a median 4,767 tokens per second and a P80 of 5,520. The post describes pairing LPX with Vera Rubin NVL72 through prefill-decode, attention-FFN, or external-drafter speculative-decoding configurations for projected large-model agentic workloads.

Takeaways

1. 1

   Artificial Analysis measured 3,431 output tokens per second on Gemma 4 31B at 100K context and 3,382 output tokens per second at 10K context.

2. 2

   Groq 3 LPX uses point-to-point C2C links, compiler-planned routing, and LPU-level scheduling to minimize transfer initialization and contention.

3. 3

   On SPEED-Bench coding tasks, the system achieved a median of 4,767 output tokens per second and a P80 of 5,520 tokens per second.