Get your VLM running in 3 simple steps on Intel CPUs

Summary

Deploying Vision Language Models locally presents computational challenges despite providing data privacy and independence from external servers. Optimum Intel and OpenVINO address this by optimizing lightweight models like SmolVLM2-256M for low-resource hardware, including Intel CPUs and GPUs. The process involves exporting the model to the OpenVINO Intermediate Representation format, applying 8-bit weight-only or static quantization, and executing inference with the transformed model. In single-image CPU benchmarks on an Intel Core Ultra 7 265K processor, OpenVINO with 8-bit weight-only quantization reduced time to first token from 5.150 seconds under PyTorch down to 0.247 seconds. The optimized configuration also raised decoding throughput from 0.722 to 63.928 tokens per second.

Takeaways

1. 1

   Converting SmolVLM2-256M to OpenVINO IR cut time to first token on an Intel Core Ultra 7 CPU from 5.150s to 0.420s and boosted throughput from 0.722 to 47.237 tokens/s.

2. 2

   Applying 8-bit Weight Only Quantization further improved performance on OpenVINO, achieving a 0.247s time to first token and 63.928 tokens/s decoding throughput.

3. 3

   Static quantization requires a calibration step using representative data samples and can be applied specifically to vision embeddings while retaining weight-only quantization for other components.