Mastering Edge AI on Raspberry Pi with LiteRT and Gemma

Summary

Running edge AI on a Raspberry Pi 5 is presented as a way to build autonomous, low-latency systems that operate offline, with Reachy Mini serving as the demonstration. The setup combines LiteRT, LiteRT-LM, Gemma models, and LiteRT’s WebGPU (Vulkan) backend via ML Drift, splitting workloads between the CPU and VideoCore VII GPU. On CPU, Gemma 4 E2B reportedly reaches 99 tokens/sec prefill, 9 tokens/sec decode, and 1,432 MB peak memory; its tokenizer supports about 27.3 characters/sec in the voice demo. The pipeline runs YOLO object detection on the GPU, Moonshine speech recognition, Gemma reasoning, and text-to-speech on the CPU, while LiteRT CLI covers conversion, quantization, benchmarking, inference, and model execution from the LiteRT Hugging Face Community.

Takeaways

1. 1

   Gemma 4 E2B on Raspberry Pi 5 is reported at 99 tokens/sec for prefill, 9 tokens/sec for decode, and 1,432 MB peak memory.

2. 2

   The Reachy Mini architecture runs Ultralytics YOLO object detection on the VideoCore VII GPU while Moonshine, Gemma reasoning, and text-to-speech run on the CPU.

3. 3

   LiteRT CLI consolidates model conversion, quantization, benchmarking, and inference, and can run compatible models from the LiteRT Hugging Face Community.