Unlocking Peak Performance on Qualcomm NPU with LiteRT

Summary

Google introduces the LiteRT Qualcomm AI Engine Direct (QNN) Accelerator, developed with Qualcomm to replace the previous TFLite QNN delegate and simplify NPU deployment across supported Qualcomm SoCs. It integrates vendor compilers and runtimes behind a unified workflow, supports ahead-of-time and on-device compilation, and can fall back to GPU or CPU when needed. Across 72 canonical vision, audio, and NLP models, the accelerator supports 90 LiteRT ops, enables full NPU delegation for 64, and reports up to 100x CPU and 10x GPU speedups. On Snapdragon 8 Elite Gen 5, FastVLM-0.5B uses int8 weights, int16 activations, and specialized transformer kernels to reach 0.12 second time-to-first-token, over 11,000 prefill tokens per second, and over 100 decode tokens per second.

Takeaways

1. 1

   The accelerator supports 90 LiteRT ops and fully delegates 64 of 72 benchmarked vision, audio, and NLP models to the NPU.

2. 2

   Across the benchmark, NPU acceleration reaches up to 100x the CPU performance and 10x the GPU performance; on Snapdragon 8 Elite Gen 5, more than 56 models run under 5ms on NPU.

3. 3

   FastVLM-0.5B uses int8 weight and int16 activation quantization plus specialized attention kernels, achieving 0.12-second TTFT, over 11,000 prefill tokens per second, and over 100 decode tokens per second.