Accelerating Qwen3-8B Agent on Intel® Core™ Ultra with Depth-Pruned Draft Models

Summary

Local agentic AI workflows generate multi-step reasoning traces that increase token counts and strain real-time inference on personal computers. To accelerate execution, researchers implemented speculative decoding using OpenVINO.GenAI on an Intel Lunar Lake integrated GPU, pairing a 4-bit Qwen3-8B target model with a smaller Qwen3-0.6B draft model. By measuring angular distance across layers, the team pruned six of the draft model's twenty-eight layers to minimize draft latency and fine-tuned it on synthetic data generated from 500,000 prompts. This depth-pruned draft elevated speculative decoding throughput from a 1.3x speedup to a 1.4x speedup over the standalone baseline. Finally, deploying the optimized model alongside the Hugging Face smolagents framework allowed local agents to perform tool invocation and code generation tasks efficiently.

Context

Agentic workflows rely on reasoning models like Qwen3-8B that generate intermediate thinking steps and tool invocations, which rapidly expand token counts. High token consumption makes inference speed critical for maintaining responsiveness when running AI agents locally on client hardware like Intel Core Ultra.

Approach / What changed

Using OpenVINO.GenAI on an Intel Lunar Lake integrated GPU, researchers paired a 4-bit Qwen3-8B target model with a Qwen3-0.6B draft model for speculative decoding. They measured angular distance between layers to prune 6 of the 28 layers from the draft model, reducing draft latency. The pruned draft was then fine-tuned on 500,000 synthetic responses generated by Qwen3-8B from the BAAI/Infinity-Instruct dataset, and the system was integrated with Hugging Face smolagents for agent workflows.

Takeaways

• Speculative decoding pairing a 4-bit Qwen3-8B target with an int8 Qwen3-0.6B draft delivered a 1.3x generation speedup over the standalone 4-bit baseline on Intel Core Ultra integrated graphics.
• Pruning 6 of 28 layers from the Qwen3-0.6B draft via angular distance and fine-tuning on 500k synthetic prompt completions increased the speculative decoding speedup to approximately 1.4x.
• Integrating the depth-pruned speculative decoding pipeline with smolagents enabled a local agent to execute tool-calling workflows, including web search and slide generation via python-pptx.

Related reading

• huggingface.co · Apr 20, 2021

  Scaling-up BERT Inference on CPU (Part 1)

  Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.

  Morgan Funtowicz

• Jetbrains · Aug 24, 2026

  Junie Can Now Run Entirely on Your Mac – No Credits, No Cloud

  JetBrains has introduced Junie Local, a feature that runs the Junie coding agent entirely on local hardware without cloud dependencies, subscriptions, or credit meters. Users execute a single command inside Junie to download a 4-bit quantized Qwen3.6-27B model, requiring an Apple M5 Mac with 64 GB of RAM. The team prioritized prefill throughput over pure generation speed by utilizing 8-bit arithmetic instructions on the M5 Neural Accelerator, which increased prefill throughput by approximately 40%. Performance optimizations also include KV-cache reuse, speculative decoding to double generation speed, and disabling reasoning to preserve performance. In JetBrains' evaluations, the local configuration performed comparably to cloud-based Sonnet 4.5 on everyday tasks while ensuring source code, diffs, and prompts never leave the local environment.