Tricks from OpenAI gpt-oss YOU 🫵 can use with transformers

Summary

To support OpenAI's GPT-OSS series of models, the transformers library introduced several performance upgrades that apply across supported architectures. The release integrates zero-build custom kernels downloadable directly from the Hub, reducing external dependency bloat and compilation friction for operations like Liger RMSNorm and MegaBlocks MoE. Native support for MXFP4 quantization groups vector elements into 32-value blocks with shared scales, allowing GPT-OSS 20B to fit in roughly 16 GB of VRAM and GPT-OSS 120B in roughly 80 GB. In addition, transformers incorporates Flash Attention 3 with attention sinks, continuous batching via the generate_batch API for experimentation, and automatic memory pre-allocation to speed up model loading on GPUs.

Takeaways

1. 1

   Pre-built custom kernels can be downloaded directly from the Hub by passing use_kernels=True, bypassing local compilation and dependency bloat for kernels like Liger RMSNorm and MegaBlocks MoE.

2. 2

   MXFP4 uses an E2M1 4-bit layout with blockwise scaling across 32 elements, reducing VRAM footprint so that GPT-OSS 20B requires roughly 16 GB and GPT-OSS 120B requires roughly 80 GB.

3. 3

   Model loading times are accelerated automatically by reading device maps and pre-allocating large GPU memory blocks before copying weights, preventing repeated small memory allocation calls.