MediaTek NPU and LiteRT: Powering the next generation of on-device AI

Summary

MediaTek’s diverse NPU hardware and specialized compilation requirements have made on-device AI deployment difficult, especially for sophisticated generative models. The LiteRT NeuroPilot Accelerator, described as a ground-up successor to the TFLite NeuroPilot delegate, provides a unified API over MediaTek NPUs and supports both offline Ahead-of-Time (AOT) and on-device compilation. Its workflow can compile models for target SoCs, package Android assets and runtimes for Google Play’s Play for On-device AI, and invoke NPU inference through LiteRT with GPU or CPU fallback options. A new C++ API and Native Hardware Buffer Interoperability enable zero-copy transfers from AHardwareBuffer and OpenGL/OpenCL buffers, supporting real-time camera and video pipelines. On supported MediaTek chipsets, listed Gemma and Qwen models receive NPU-specific optimizations; the post reports up to 12x CPU and 10x GPU acceleration, including over 1600 tokens/sec prefill and 28 tokens/sec decode for Gemma 3n E2B.

Takeaways

1. 1

   AOT compilation is recommended for larger models because it reduces on-device initialization time; on-device compilation supports platform-agnostic distribution but adds first-run cost, taking over a minute for a model such as Gemma 3 270M.

2. 2

   LiteRT can package model assets and required runtime libraries into an Android AI Pack, which Play for On-device AI analyzes and delivers to compatible devices.

3. 3

   Native Hardware Buffer Interoperability supports zero-copy transfers from AHardwareBuffer and OpenGL/OpenCL buffers, avoiding CPU conversions in camera and video pipelines.