Native Model Authoring for Edge with LiteRT Tensor API

Summary

The LiteRT Tensor API is a new lightweight authoring layer that allows developers to handcraft models in C++ with full control over graph structure and operations. The same graph runs through LiteRT on CPU, GPU, or NPU, or directly on CPU via XNNPACK, without rewriting for each hardware target. The API enables direct weight ingestion from model checkpoints like Hugging Face .safetensors files and provides tensor-centric buffer interop to keep intermediate data on accelerators with zero copies. A reference implementation of Meta’s Segment Anything Model 2 (SAM 2) authored entirely with the API achieved over 30 frames per second at 512x512 on WebGPU in the browser. The API also provides structured primitives for AI coding agents to generate, rewire, and tune model graphs automatically.

Takeaways

1. 1

   The LiteRT Tensor API enables handcrafting models in C++ with full control over graph structure and operations, while maintaining cross-platform portability across CPU, GPU, and NPU.

2. 2

   The API supports direct weight ingestion from model checkpoints (e.g., .safetensors) without conversion scripts, and allows tensor-centric buffer interop to keep data on accelerators with zero copies.

3. 3

   A coding agent (Antigravity) used the Tensor API to optimize the SAM 2 model, more than tripling its frame rate by rewriting per-frame decisions as GPU-friendly operations, each change checked against a reference.

FAQ