---
title: "Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding"
description: "Alibaba released the model weights for Qwen3.8-Flash-Next as a preview of the upcoming Qwen4 architecture, enabling developers to experiment with a multimodal mixture-of-experts model with a 125B-parameter main model, 51B N-gram embeddings, and 6B parameters activated per token. It supports a native 262,144-token context window, extensible to 1M tokens with YaRN. To reduce long-context attention compute and KV-cache growth, its hybrid design uses Gated DeltaNet in three of every four layers and Qwen Sparse Attention in the remaining layer, where micro-block importance guides retrieval. On NVIDIA GB300 NVL72, it delivers more than 16K tokens per second per GPU and more than 200 tokens per second per user; QSA achieved up to 7.6x prefill and 4.9x decoding speedups versus full attention. Fine-tuning is available through NVIDIA NeMo AutoModel with SFT or LoRA, and reinforcement learning through NVIDIA NeMo RL."
---

# Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Michelle Horton · Aug 26, 2026

**Type:** Announcement

## Summary

Alibaba released the model weights for Qwen3.8-Flash-Next as a preview of the upcoming Qwen4 architecture, enabling developers to experiment with a multimodal mixture-of-experts model with a 125B-parameter main model, 51B N-gram embeddings, and 6B parameters activated per token. It supports a native 262,144-token context window, extensible to 1M tokens with YaRN. To reduce long-context attention compute and KV-cache growth, its hybrid design uses Gated DeltaNet in three of every four layers and Qwen Sparse Attention in the remaining layer, where micro-block importance guides retrieval. On NVIDIA GB300 NVL72, it delivers more than 16K tokens per second per GPU and more than 200 tokens per second per user; QSA achieved up to 7.6x prefill and 4.9x decoding speedups versus full attention. Fine-tuning is available through NVIDIA NeMo AutoModel with SFT or LoRA, and reinforcement learning through NVIDIA NeMo RL.

## Context

Qwen3.8-Flash-Next targets high-volume, context-intensive applications such as agentic coding, document processing, and tool-driven workflows. As context grows, attention compute and KV-cache memory become bottlenecks; the model addresses these long-context inference constraints while supporting experimentation on NVIDIA rack-scale and local hardware.

## Approach / What changed

The model combines Gated DeltaNet and Qwen Sparse Attention in a hybrid architecture: three out of every four layers use GDN to compress historical context into a fixed-size recurrent state, while the remaining QSA layer retrieves relevant full-context regions through micro-block importance estimates. NVIDIA provides functional support through SGLang, vLLM, and TensorRT LLM, plus validation on GB300 NVL72 and post-training recipes through NeMo AutoModel and NeMo RL.

## Takeaways

- Three out of every four layers use Gated DeltaNet to maintain a fixed-size recurrent state, preventing KV-cache growth as sequences lengthen; Qwen Sparse Attention handles precise retrieval in the remaining layer.
- At a 1M-token context, QSA's attention kernel delivered reported speedups of up to 7.6x during prefill and 4.9x during decoding versus full attention.
- The model can be fine-tuned with full SFT or memory-efficient LoRA through NVIDIA NeMo AutoModel, then post-trained with NeMo RL; supported inference options include SGLang, vLLM, and TokenSpeed.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Deployment](https://yomu.fyi/topic/deployment), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance)

- Source: [NVIDIA](https://developer.nvidia.com/blog/experiment-with-qwen3-8-flash-next-on-nvidia-gb300-nvl72-for-agentic-coding)
- Source URL: https://developer.nvidia.com/blog/experiment-with-qwen3-8-flash-next-on-nvidia-gb300-nvl72-for-agentic-coding
- Ingested by Yomu: 2026-08-29T01:39:07.907Z

[Read original post](https://developer.nvidia.com/blog/experiment-with-qwen3-8-flash-next-on-nvidia-gb300-nvl72-for-agentic-coding)