---
title: "TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor"
description: "Edge agentic workloads require repeated tool-calling turns, growing conversation histories, and valid function calls on constrained devices rather than single-prompt responses. On one NVIDIA Jetson AGX Thor Developer Kit, TensorRT Edge-LLM ran Qwen3.6-27B in SingleStream mode and completed the benchmark’s 1,007-turn performance workload in 24 minutes and 36 seconds, versus 2 hours and 37 minutes for the llama.cpp reference run. It reported 52.33 tokens per second, 247.12 ms median time to first token, and 87.94% BFCL overall accuracy. The implementation combines NVFP4 weights and activations, FP8 KV cache, reusable prompt prefixes with recurrent-state restoration, and tree-based multi-token prediction; roughly 96% of prompt tokens used hot cache, while tree-based MTP could achieve about 40% more decoding performance than linear MTP with three draft steps."
---

# TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Elizabeth Goodman · Sep 16, 2026

**Type:** Benchmark

## Summary

Edge agentic workloads require repeated tool-calling turns, growing conversation histories, and valid function calls on constrained devices rather than single-prompt responses. On one NVIDIA Jetson AGX Thor Developer Kit, TensorRT Edge-LLM ran Qwen3.6-27B in SingleStream mode and completed the benchmark’s 1,007-turn performance workload in 24 minutes and 36 seconds, versus 2 hours and 37 minutes for the llama.cpp reference run. It reported 52.33 tokens per second, 247.12 ms median time to first token, and 87.94% BFCL overall accuracy. The implementation combines NVFP4 weights and activations, FP8 KV cache, reusable prompt prefixes with recurrent-state restoration, and tree-based multi-token prediction; roughly 96% of prompt tokens used hot cache, while tree-based MTP could achieve about 40% more decoding performance than linear MTP with three draft steps.

## Context

MLPerf Inference v6.1 Edge Agentic measures an OpenAI-compatible model endpoint in performance and accuracy phases. The performance phase replays 20 recorded software-engineering agent conversations containing 1,007 generated turns, with input length reaching approximately 23.5K tokens. The accuracy phase uses BFCL v4 prompts to test function selection, argument validity, and whether a tool should be called.

## Approach / What changed

The submission used Qwen3.6-27B with NVFP4 weights and activations, FP8 KV cache, and TensorRT Edge-LLM on one Jetson AGX Thor in MAXN power mode. The runtime reused cached prompt prefixes, recurrent state, and partial KV-page state, prefilling only new conversation suffixes. Tree-based MTP verified multiple candidate continuations through a 16-node tree, using eight draft steps and top-two candidates at each depth.

## Takeaways

- TensorRT Edge-LLM completed all 1,007 performance-workload turns in 24 minutes and 36 seconds, compared with 2 hours and 37 minutes for the llama.cpp reference submission on Jetson AGX Thor.
- KV cache and recurrent-state reuse served about 96% of prompt tokens from hot cache, reducing prefilling to approximately 0.5 million of the workload’s 13.6 million prompt tokens.
- The MLPerf server used eight draft steps, top-two candidates at each depth, and a 16-node verification tree; tree-based MTP could provide about 40% more decoding performance than linear MTP with three draft steps.

## FAQ

### How does KV cache reuse reduce work across agent turns?

Each new request repeats most of the prior conversation. TensorRT Edge-LLM restores reusable attention KV pages, recurrent state, and partial KV-page state, then prefills only the new suffix; about 96% of prompt tokens were served with hot cache on this workload.

### What is the role of tree-based multi-token prediction?

A draft model proposes several candidate continuations in a tree, and the target model verifies them in one forward pass. The configuration used eight draft steps, top-two candidates at each depth, and a 16-node verification tree, which could add about 40% decoding performance over linear MTP with three draft steps.

### How does the benchmark evaluate agentic accuracy?

Its accuracy phase uses Berkeley Function Calling Leaderboard v4 prompts with single-turn operation and reasoning disabled. It checks whether the model selects the correct function, generates valid arguments, and avoids a tool call when none is required.

**Tags:** [Inference](https://yomu.fyi/topic/inference), [Jetson](https://yomu.fyi/topic/jetson), [MLPerf](https://yomu.fyi/topic/mlperf), [NVIDIA](https://yomu.fyi/topic/nvidia), [Quantization](https://yomu.fyi/topic/quantization)

- Source: [NVIDIA](https://developer.nvidia.com/blog/tensorrt-edge-llm-completes-the-mlperf-edge-agentic-benchmark-6-4x-faster-on-jetson-agx-thor)
- Source URL: https://developer.nvidia.com/blog/tensorrt-edge-llm-completes-the-mlperf-edge-agentic-benchmark-6-4x-faster-on-jetson-agx-thor
- Ingested by Yomu: 2026-09-17T00:01:24.387Z

[Read original post](https://developer.nvidia.com/blog/tensorrt-edge-llm-completes-the-mlperf-edge-agentic-benchmark-6-4x-faster-on-jetson-agx-thor)