---
title: "Blazing fast on-device GenAI with LiteRT-LM"
description: "LiteRT-LM is presented as an on-device runtime for deploying Gemma 4 across Android, iOS, and the web despite constrained memory, limited compute, and varied hardware. It combines LiteRT inference, quantization, XNNPACK and MLDrift kernels, CPU/GPU/NPU backends, transfer-avoiding pipelines, Multi-Token Prediction (MTP), and session management. Reported decode rates reach 52 tokens/sec on Android GPU, 56 on iOS Metal, and up to 76 on WebGPU; MTP provides up to a 2.2x speedup, while a ~2.58GB E2B model uses 607MB of physical memory on Apple mobile CPUs. The runtime also supports save/restore of KV caches, native vision and audio loading, Thinking Mode, constrained decoding, and function calling. New Swift and JavaScript APIs extend the integration surface, including WASM/WebGPU browser inference designed for serverless, secure, privacy-preserving applications."
---

# Blazing fast on-device GenAI with LiteRT-LM

[Google](https://yomu.fyi/company/google) · Tenghui Zhu, Yu-hui Chen, Ram Iyengar · May 19, 2026

**Type:** Announcement

## Summary

LiteRT-LM is presented as an on-device runtime for deploying Gemma 4 across Android, iOS, and the web despite constrained memory, limited compute, and varied hardware. It combines LiteRT inference, quantization, XNNPACK and MLDrift kernels, CPU/GPU/NPU backends, transfer-avoiding pipelines, Multi-Token Prediction (MTP), and session management. Reported decode rates reach 52 tokens/sec on Android GPU, 56 on iOS Metal, and up to 76 on WebGPU; MTP provides up to a 2.2x speedup, while a ~2.58GB E2B model uses 607MB of physical memory on Apple mobile CPUs. The runtime also supports save/restore of KV caches, native vision and audio loading, Thinking Mode, constrained decoding, and function calling. New Swift and JavaScript APIs extend the integration surface, including WASM/WebGPU browser inference designed for serverless, secure, privacy-preserving applications.

## Context

Deploying Gemma 4 at the edge requires handling restricted memory, limited compute, fragmented hardware, and the differing capabilities of Android, iOS, and web platforms. The source also identifies latency, data-transfer overhead, long-context continuity, and memory footprint as challenges for on-device AI applications.

## Approach / What changed

LiteRT-LM combines LiteRT inference with quantization, XNNPACK and MLDrift kernels, CPU/GPU/NPU acceleration, transfer-avoiding pipelines, Multi-Token Prediction, session save and restore, dynamic loading of vision and audio encoders, Thinking Mode, constrained decoding, and function calling. It adds Swift and JavaScript APIs for Apple platforms and the web, with WASM and WebGPU support.

## Takeaways

- LiteRT-LM reports decode speeds of 52 tokens/sec on Android GPU, 56 tokens/sec on iOS Metal, and up to 76 tokens/sec with WebGPU on a MacBook Pro.
- Native Multi-Token Prediction support delivers up to a 2.2x decoding speedup by keeping the drafter and primary model on the same hardware IP and sharing local memory.
- A ~2.58GB Gemma 4 E2B model runs with a 607MB physical memory footprint on Apple mobile CPUs using XNNPACK weight caching.

**Tags:** [Inference](https://yomu.fyi/topic/inference), [LiteRT-LM](https://yomu.fyi/topic/litert-lm), [WebGPU](https://yomu.fyi/topic/webgpu)

- Source: [Google](https://developers.googleblog.com/blazing-fast-on-device-genai-with-litert-lm)
- Source URL: https://developers.googleblog.com/blazing-fast-on-device-genai-with-litert-lm
- Ingested by Yomu: 2026-09-09T14:29:58.688Z

[Read original post](https://developers.googleblog.com/blazing-fast-on-device-genai-with-litert-lm)