Blazing fast on-device GenAI with LiteRT-LM

Summary

LiteRT-LM is presented as an on-device runtime for deploying Gemma 4 across Android, iOS, and the web despite constrained memory, limited compute, and varied hardware. It combines LiteRT inference, quantization, XNNPACK and MLDrift kernels, CPU/GPU/NPU backends, transfer-avoiding pipelines, Multi-Token Prediction (MTP), and session management. Reported decode rates reach 52 tokens/sec on Android GPU, 56 on iOS Metal, and up to 76 on WebGPU; MTP provides up to a 2.2x speedup, while a ~2.58GB E2B model uses 607MB of physical memory on Apple mobile CPUs. The runtime also supports save/restore of KV caches, native vision and audio loading, Thinking Mode, constrained decoding, and function calling. New Swift and JavaScript APIs extend the integration surface, including WASM/WebGPU browser inference designed for serverless, secure, privacy-preserving applications.

Takeaways

1. 1

   LiteRT-LM reports decode speeds of 52 tokens/sec on Android GPU, 56 tokens/sec on iOS Metal, and up to 76 tokens/sec with WebGPU on a MacBook Pro.

2. 2

   Native Multi-Token Prediction support delivers up to a 2.2x decoding speedup by keeping the drafter and primary model on the same hardware IP and sharing local memory.

3. 3

   A ~2.58GB Gemma 4 E2B model runs with a 607MB physical memory footprint on Apple mobile CPUs using XNNPACK weight caching.