# Accelerating Qwen3-8B Agent on Intel® Core™ Ultra with Depth-Pruned Draft Models

[Hugging Face](https://yomu.fyi/company/hugging-face) · Igor Margulis, Ofir Zafrir, Shira Guskin, Guy Boudoukh, Pedro Cuenca · Sep 29, 2025

**Type:** Problem & solution

## Summary

Local agentic AI workflows generate multi-step reasoning traces that increase token counts and strain real-time inference on personal computers. To accelerate execution, researchers implemented speculative decoding using OpenVINO.GenAI on an Intel Lunar Lake integrated GPU, pairing a 4-bit Qwen3-8B target model with a smaller Qwen3-0.6B draft model. By measuring angular distance across layers, the team pruned six of the draft model's twenty-eight layers to minimize draft latency and fine-tuned it on synthetic data generated from 500,000 prompts. This depth-pruned draft elevated speculative decoding throughput from a 1.3x speedup to a 1.4x speedup over the standalone baseline. Finally, deploying the optimized model alongside the Hugging Face smolagents framework allowed local agents to perform tool invocation and code generation tasks efficiently.

## Context

Agentic workflows rely on reasoning models like Qwen3-8B that generate intermediate thinking steps and tool invocations, which rapidly expand token counts. High token consumption makes inference speed critical for maintaining responsiveness when running AI agents locally on client hardware like Intel Core Ultra.

## Approach / What changed

Using OpenVINO.GenAI on an Intel Lunar Lake integrated GPU, researchers paired a 4-bit Qwen3-8B target model with a Qwen3-0.6B draft model for speculative decoding. They measured angular distance between layers to prune 6 of the 28 layers from the draft model, reducing draft latency. The pruned draft was then fine-tuned on 500,000 synthetic responses generated by Qwen3-8B from the BAAI/Infinity-Instruct dataset, and the system was integrated with Hugging Face smolagents for agent workflows.

## Takeaways

- Speculative decoding pairing a 4-bit Qwen3-8B target with an int8 Qwen3-0.6B draft delivered a 1.3x generation speedup over the standalone 4-bit baseline on Intel Core Ultra integrated graphics.
- Pruning 6 of 28 layers from the Qwen3-0.6B draft via angular distance and fine-tuning on 500k synthetic prompt completions increased the speculative decoding speedup to approximately 1.4x.
- Integrating the depth-pruned speculative decoding pipeline with smolagents enabled a local agent to execute tool-calling workflows, including web search and slide generation via python-pptx.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/intel-qwen3-agent)