Loading post…
NVIDIAElizabeth GoodmanBenchmark
TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor
Summary
Edge agentic workloads require repeated tool-calling turns, growing conversation histories, and valid function calls on constrained devices rather than single-prompt responses. On one NVIDIA Jetson AGX Thor Developer Kit, TensorRT Edge-LLM ran Qwen3.6-27B in SingleStream mode and completed the benchmark’s 1,007-turn performance workload in 24 minutes and 36 seconds, versus 2 hours and 37 minutes for the llama.cpp reference run. It reported 52.33 tokens per second, 247.12 ms median time to first token, and 87.94% BFCL overall accuracy. The implementation combines NVFP4 weights and activations, FP8 KV cache, reusable prompt prefixes with recurrent-state restoration, and tree-based multi-token prediction; roughly 96% of prompt tokens used hot cache, while tree-based MTP could achieve about 40% more decoding performance than linear MTP with three draft steps.
Context
MLPerf Inference v6.1 Edge Agentic measures an OpenAI-compatible model endpoint in performance and accuracy phases. The performance phase replays 20 recorded software-engineering agent conversations containing 1,007 generated turns, with input length reaching approximately 23.5K tokens. The accuracy phase uses BFCL v4 prompts to test function selection, argument validity, and whether a tool should be called.
Approach / What changed
The submission used Qwen3.6-27B with NVFP4 weights and activations, FP8 KV cache, and TensorRT Edge-LLM on one Jetson AGX Thor in MAXN power mode. The runtime reused cached prompt prefixes, recurrent state, and partial KV-page state, prefilling only new conversation suffixes. Tree-based MTP verified multiple candidate continuations through a 16-node tree, using eight draft steps and top-two candidates at each depth.
Takeaways
TensorRT Edge-LLM completed all 1,007 performance-workload turns in 24 minutes and 36 seconds, compared with 2 hours and 37 minutes for the llama.cpp reference submission on Jetson AGX Thor.
KV cache and recurrent-state reuse served about 96% of prompt tokens from hot cache, reducing prefilling to approximately 0.5 million of the workload’s 13.6 million prompt tokens.
The MLPerf server used eight draft steps, top-two candidates at each depth, and a 16-node verification tree; tree-based MTP could provide about 40% more decoding performance than linear MTP with three draft steps.
FAQ
How does KV cache reuse reduce work across agent turns?
Each new request repeats most of the prior conversation. TensorRT Edge-LLM restores reusable attention KV pages, recurrent state, and partial KV-page state, then prefills only the new suffix; about 96% of prompt tokens were served with hot cache on this workload.
What is the role of tree-based multi-token prediction?
A draft model proposes several candidate continuations in a tree, and the target model verifies them in one forward pass. The configuration used eight draft steps, top-two candidates at each depth, and a 16-node verification tree, which could add about 40% decoding performance over linear MTP with three draft steps.
How does the benchmark evaluate agentic accuracy?
Its accuracy phase uses Berkeley Function Calling Leaderboard v4 prompts with single-turn operation and reasoning disabled. It checks whether the model selects the correct function, generates valid arguments, and avoids a tool call when none is required.
Source and freshness
- Source
- developer.nvidia.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.