Topic

Jetson

2 posts about Jetson. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

NVIDIA
TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor
Edge agentic workloads require repeated tool-calling turns, growing conversation histories, and valid function calls on constrained devices rather than single-prompt responses. On one NVIDIA Jetson AGX Thor Developer Kit, TensorRT Edge-LLM ran Qwen3.6-27B in SingleStream mode and completed the benchmark’s 1,007-turn performance workload in 24 minutes and 36 seconds, versus 2 hours and 37 minutes for the llama.cpp reference run. It reported 52.33 tokens per second, 247.12 ms median time to first token, and 87.94% BFCL overall accuracy. The implementation combines NVFP4 weights and activations, FP8 KV cache, reusable prompt prefixes with recurrent-state restoration, and tree-based multi-token prediction; roughly 96% of prompt tokens used hot cache, while tree-based MTP could achieve about 40% more decoding performance than linear MTP with three draft steps.

NVIDIA
Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson
Reasoning and agentic models have typically required data-center inference, creating network dependencies, higher costs, and potential data exposure for edge applications. The post explains how to choose between dense Qwen3.8-27B, which activates all 27 billion parameters, and mixture-of-experts Nemotron 3.5 Lightning, which has 30 billion total parameters but activates 3 billion per token. Deployment uses vLLM on Jetson with NVFP4 quantization and speculative decoding, where a draft model proposes tokens and the main model verifies them; the examples include DSpark for Nemotron and DFlash2 for Qwen. Reported tests show up to 6.28x decode-throughput speedup over BF16, while workload-specific throughput still varies. Validation with representative prompts is recommended before deployment, and custom quantization or speculator training is reserved for cases where accuracy or acceptance rates reveal a clear gap.

The best engineering reads, once a week.

Topic

2 posts about Jetson. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

NVIDIA
TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor
Edge agentic workloads require repeated tool-calling turns, growing conversation histories, and valid function calls on constrained devices rather than single-prompt responses. On one NVIDIA Jetson AGX Thor Developer Kit, TensorRT Edge-LLM ran Qwen3.6-27B in SingleStream mode and completed the benchmark’s 1,007-turn performance workload in 24 minutes and 36 seconds, versus 2 hours and 37 minutes for the llama.cpp reference run. It reported 52.33 tokens per second, 247.12 ms median time to first token, and 87.94% BFCL overall accuracy. The implementation combines NVFP4 weights and activations, FP8 KV cache, reusable prompt prefixes with recurrent-state restoration, and tree-based multi-token prediction; roughly 96% of prompt tokens used hot cache, while tree-based MTP could achieve about 40% more decoding performance than linear MTP with three draft steps.

NVIDIA
Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson
Reasoning and agentic models have typically required data-center inference, creating network dependencies, higher costs, and potential data exposure for edge applications. The post explains how to choose between dense Qwen3.8-27B, which activates all 27 billion parameters, and mixture-of-experts Nemotron 3.5 Lightning, which has 30 billion total parameters but activates 3 billion per token. Deployment uses vLLM on Jetson with NVFP4 quantization and speculative decoding, where a draft model proposes tokens and the main model verifies them; the examples include DSpark for Nemotron and DFlash2 for Qwen. Reported tests show up to 6.28x decode-throughput speedup over BF16, while workload-specific throughput still varies. Validation with representative prompts is recommended before deployment, and custom quantization or speculator training is reserved for cases where accuracy or acceptance rates reveal a clear gap.

The best engineering reads, once a week.