Topic

MLPerf

1 post about MLPerf. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

NVIDIA
TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor
Edge agentic workloads require repeated tool-calling turns, growing conversation histories, and valid function calls on constrained devices rather than single-prompt responses. On one NVIDIA Jetson AGX Thor Developer Kit, TensorRT Edge-LLM ran Qwen3.6-27B in SingleStream mode and completed the benchmark’s 1,007-turn performance workload in 24 minutes and 36 seconds, versus 2 hours and 37 minutes for the llama.cpp reference run. It reported 52.33 tokens per second, 247.12 ms median time to first token, and 87.94% BFCL overall accuracy. The implementation combines NVFP4 weights and activations, FP8 KV cache, reusable prompt prefixes with recurrent-state restoration, and tree-based multi-token prediction; roughly 96% of prompt tokens used hot cache, while tree-based MTP could achieve about 40% more decoding performance than linear MTP with three draft steps.

The best engineering reads, once a week.

Topic

1 post about MLPerf. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

NVIDIA
TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor
Edge agentic workloads require repeated tool-calling turns, growing conversation histories, and valid function calls on constrained devices rather than single-prompt responses. On one NVIDIA Jetson AGX Thor Developer Kit, TensorRT Edge-LLM ran Qwen3.6-27B in SingleStream mode and completed the benchmark’s 1,007-turn performance workload in 24 minutes and 36 seconds, versus 2 hours and 37 minutes for the llama.cpp reference run. It reported 52.33 tokens per second, 247.12 ms median time to first token, and 87.94% BFCL overall accuracy. The implementation combines NVFP4 weights and activations, FP8 KV cache, reusable prompt prefixes with recurrent-state restoration, and tree-based multi-token prediction; roughly 96% of prompt tokens used hot cache, while tree-based MTP could achieve about 40% more decoding performance than linear MTP with three draft steps.

The best engineering reads, once a week.