DatabricksDaya Khudia, Leo Li, Lesheng JinExplainer
Production inference systems often use generic GPU kernels even though operation shapes combine static model parameters with request-time dimensions such as token counts. Proteus addresses this mismatch by having agents propose specialized kernels, then applying controlled-reference validation, static checks, isolated repeated timing, hidden tests, and consistency checks before retaining measured winners. Its knowledge layer uses actionable, scoped lessons retrieved through hierarchical tags and hybrid keyword-semantic search, while background jobs handle deeper distillation instead of burdening each attempt. On NVIDIA B200 GPUs with a Triton backend, a Gated DeltaNet packed-decode case for Qwen 3.5 122B produced shape-specific wins, including 1.6× for a Batch=4, Key=128, Value=128 path and 0.018 ms latency for Candidate 030. Across the reported kernels, Proteus achieved 1.8–5.2× speedups over the best available in vLLM, with validation and context management identified as the main difficulties.
Production inference systems commonly rely on generic kernels despite runtime operation shapes varying with request-specific factors such as token counts. The work also addresses reward-hacking risks in agentic optimization, including unfair timing, visible-test overfitting, stale compiled state, and benchmark-specific shortcuts.
Proteus has agents generate candidate GPU kernels, checks them against a controlled reference implementation, applies static and consistency checks, benchmarks only verified candidates with isolated repeated timing, and iteratively retains measured winners. Its knowledge layer supplies concise, actionable lessons through hierarchical tag filtering and hybrid keyword-semantic search, with deeper memory reorganization handled asynchronously.
Proteus generated Qwen 3.5 122B kernels reported as 1.8–5.2× faster than the best available kernels in vLLM.
The Gated DeltaNet packed-decode case produced a 1.6× speedup for a Batch=4, Key=128, Value=128 serving shape, while Candidate 030 reached 0.018 ms latency.
The harness treats validation as the limiting factor: candidates are checked against a controlled reference, timed symmetrically, retested, and evaluated on hidden inputs to reduce reward hacking.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
AI Town is a pixel-art simulation framework where autonomous agents maintain memories, relationships, and goals across time, creating observable environments for testing how models behave in long-context interactive settings.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
