Loading…
Topic
LiteRT
4 posts about LiteRT. Every summary links to the original.
- 4 posts
- 1 company
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Mastering Edge AI on Raspberry Pi with LiteRT and Gemma
Running edge AI on a Raspberry Pi 5 is presented as a way to build autonomous, low-latency systems that operate offline, with Reachy Mini serving as the demonstration. The setup combines LiteRT, LiteRT-LM, Gemma models, and LiteRT’s WebGPU (Vulkan) backend via ML Drift, splitting workloads between the CPU and VideoCore VII GPU. On CPU, Gemma 4 E2B reportedly reaches 99 tokens/sec prefill, 9 tokens/sec decode, and 1,432 MB peak memory; its tokenizer supports about 27.3 characters/sec in the voice demo. The pipeline runs YOLO object detection on the GPU, Moonshine speech recognition, Gemma reasoning, and text-to-speech on the CPU, while LiteRT CLI covers conversion, quantization, benchmarking, inference, and model execution from the LiteRT Hugging Face Community.
LiteRT.js, Google's high performance Web AI Inference
Google announces LiteRT.js, a JavaScript binding of LiteRT that runs .tflite AI and ML models directly in web browsers. It targets local inference for JavaScript and TypeScript applications, with privacy benefits, zero server costs, and low latency, while providing smoother deployment for existing .tflite models. Through WebAssembly, it exposes LiteRT’s optimized runtime across XNNPACK CPU, ML Drift with WebGPU, and experimental WebNN for NPUs; LiteRT Torch supports one-step PyTorch conversion, and AI Edge Quantizer enables layer-specific quantization. The initial release includes the @litertjs/core npm package, demos, and examples covering YOLO26, depth estimation, and Real-ESRGAN upscaling. In controlled benchmarks on a 2024 Apple MacBook Pro with M4, it outperformed other web runtimes by up to 3x, while WebGPU or WebNN delivered 5–60x speedups over CPU execution; results may vary with hardware, thermal conditions, and browser drivers.
MediaTek NPU and LiteRT: Powering the next generation of on-device AI
MediaTek’s diverse NPU hardware and specialized compilation requirements have made on-device AI deployment difficult, especially for sophisticated generative models. The LiteRT NeuroPilot Accelerator, described as a ground-up successor to the TFLite NeuroPilot delegate, provides a unified API over MediaTek NPUs and supports both offline Ahead-of-Time (AOT) and on-device compilation. Its workflow can compile models for target SoCs, package Android assets and runtimes for Google Play’s Play for On-device AI, and invoke NPU inference through LiteRT with GPU or CPU fallback options. A new C++ API and Native Hardware Buffer Interoperability enable zero-copy transfers from AHardwareBuffer and OpenGL/OpenCL buffers, supporting real-time camera and video pipelines. On supported MediaTek chipsets, listed Gemma and Qwen models receive NPU-specific optimizations; the post reports up to 12x CPU and 10x GPU acceleration, including over 1600 tokens/sec prefill and 28 tokens/sec decode for Gemma 3n E2B.
Unlocking Peak Performance on Qualcomm NPU with LiteRT
Google introduces the LiteRT Qualcomm AI Engine Direct (QNN) Accelerator, developed with Qualcomm to replace the previous TFLite QNN delegate and simplify NPU deployment across supported Qualcomm SoCs. It integrates vendor compilers and runtimes behind a unified workflow, supports ahead-of-time and on-device compilation, and can fall back to GPU or CPU when needed. Across 72 canonical vision, audio, and NLP models, the accelerator supports 90 LiteRT ops, enables full NPU delegation for 64, and reports up to 100x CPU and 10x GPU speedups. On Snapdragon 8 Elite Gen 5, FastVLM-0.5B uses int8 weights, int16 activations, and specialized transformer kernels to reach 0.12 second time-to-first-token, over 11,000 prefill tokens per second, and over 100 decode tokens per second.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.