Get your VLM running in 3 simple steps on Intel CPUs

Summary

Deploying Vision Language Models locally presents computational challenges despite providing data privacy and independence from external servers. Optimum Intel and OpenVINO address this by optimizing lightweight models like SmolVLM2-256M for low-resource hardware, including Intel CPUs and GPUs. The process involves exporting the model to the OpenVINO Intermediate Representation format, applying 8-bit weight-only or static quantization, and executing inference with the transformed model. In single-image CPU benchmarks on an Intel Core Ultra 7 265K processor, OpenVINO with 8-bit weight-only quantization reduced time to first token from 5.150 seconds under PyTorch down to 0.247 seconds. The optimized configuration also raised decoding throughput from 0.722 to 63.928 tokens per second.

Context

Running Vision Language Models on local devices is computationally demanding, but local execution offers enhanced privacy, speed, and reliability without reliance on internet connections or external servers.

Approach / What changed

Using Optimum Intel and OpenVINO, SmolVLM2-256M is converted to the OpenVINO IR format, quantized using either 8-bit Weight Only Quantization (WOQ) or mixed static quantization with a calibration dataset, and executed on Intel CPUs or GPUs.

Takeaways

• Converting SmolVLM2-256M to OpenVINO IR cut time to first token on an Intel Core Ultra 7 CPU from 5.150s to 0.420s and boosted throughput from 0.722 to 47.237 tokens/s.
• Applying 8-bit Weight Only Quantization further improved performance on OpenVINO, achieving a 0.247s time to first token and 63.928 tokens/s decoding throughput.
• Static quantization requires a calibration step using representative data samples and can be applied specifically to vision embeddings while retaining weight-only quantization for other components.

Related reading

• Hugging Face · Sep 11, 2025

  Tricks from OpenAI gpt-oss YOU 🫵 can use with transformers

  To support OpenAI's GPT-OSS series of models, the transformers library introduced several performance upgrades that apply across supported architectures. The release integrates zero-build custom kernels downloadable directly from the Hub, reducing external dependency bloat and compilation friction for operations like Liger RMSNorm and MegaBlocks MoE. Native support for MXFP4 quantization groups vector elements into 32-value blocks with shared scales, allowing GPT-OSS 20B to fit in roughly 16 GB of VRAM and GPT-OSS 120B in roughly 80 GB. In addition, transformers incorporates Flash Attention 3 with attention sinks, continuous batching via the generate_batch API for experimentation, and automatic memory pre-allocation to speed up model loading on GPUs.

  Aritra Roy Gosthipaty, Sergio Paniego, Vaibhav Srivastav, Pedro Cuenca, Arthur Zucker, Nathan Habib, Cyril Vallez

• NVIDIA Developer Blog · Jul 10, 2026

  AI Model Co-Design: Hardware-Friendly LLM Design