---
title: "Reducing Qwen training time by 6x on AI Runtime"
description: "The post describes systematic performance optimization to reduce fine-tuning time for the Qwen 3.8 27B model on AI Runtime from 7 hours 44 minutes to 1 hour 17 minutes, a 6x improvement. The baseline showed 98% GPU utilization, but torch traces revealed the CPU was stalling the GPU with too many small operations. Installing manually fused kernels such as flash-linear-attention and causal-conv1d reduced the number of GPU operations by 15x and improved training speed by 3.7x. Further gains came from switching from 2D padded batches to 1D packed tensors with correct position_ids, which reduced the number of batches by 36% and eliminated dynamic shape recompilation. The post emphasizes measuring what matters beyond GPU utilization and leveraging pre-built community kernels and efficient batching."
---

# Reducing Qwen training time by 6x on AI Runtime

[Databricks](https://yomu.fyi/company/databricks) · Bruce Fontaine, Pardis Beikzadeh · Oct 15, 2026

**Type:** Tutorial

## Summary

The post describes systematic performance optimization to reduce fine-tuning time for the Qwen 3.8 27B model on AI Runtime from 7 hours 44 minutes to 1 hour 17 minutes, a 6x improvement. The baseline showed 98% GPU utilization, but torch traces revealed the CPU was stalling the GPU with too many small operations. Installing manually fused kernels such as flash-linear-attention and causal-conv1d reduced the number of GPU operations by 15x and improved training speed by 3.7x. Further gains came from switching from 2D padded batches to 1D packed tensors with correct position\_ids, which reduced the number of batches by 36% and eliminated dynamic shape recompilation. The post emphasizes measuring what matters beyond GPU utilization and leveraging pre-built community kernels and efficient batching.

## Context

Fine-tuning a 27B parameter model on AI Runtime initially took 7h44m with 98% GPU utilization, but torch traces revealed the CPU was bottlenecked by too many small GPU operations. The post provides a step-by-step optimization guide using fused kernels and efficient batching.

## Approach / What changed

Install flash-linear-attention and causal-conv1d Python packages to fuse many small GPU operations, reducing operation count by 15x. Then replace 2D padded batching with 1D packed tensors using correct position\_ids to reduce batch count by 36% and avoid dynamic shape recompilation.

## Takeaways

- GPU utilization alone is misleading; torch traces reveal CPU-side stalls from too many small GPU operations.
- Manually fused kernels like flash-linear-attention and causal-conv1d can reduce GPU operations by 15x and improve training speed by 3.7x.
- Packing sequences into 1D tensors with correct position\_ids reduces padding overhead and batch count, yielding a 1.6x speedup.

## FAQ

### Why does GPU utilization at 98% not reflect actual performance?

Standard GPU utilization only counts the percentage of time the GPU is active, not how efficiently it uses its cores. The CPU can stall the GPU by enqueuing too many small operations, causing underutilization even when utilization appears high.

### How do fused kernels improve training speed?

Fused kernels combine multiple small GPU operations into one, reducing costly transfers between GPU memory and on-die memory and allowing more efficient scheduling. This reduced the number of operations by 15x and improved training speed by 3.7x.

### What is the trade-off of using 1D packed tensors?

It makes the input pipeline more complex and risks introducing errors that affect model quality, but it reduces batch count by 36% and eliminates dynamic shape recompilation, yielding a 1.6x speedup.

**Tags:** [AI](https://yomu.fyi/topic/ai), [GPU](https://yomu.fyi/topic/gpu), [LLMs](https://yomu.fyi/topic/llm), [Performance](https://yomu.fyi/topic/performance), [Qwen](https://yomu.fyi/topic/qwen)

- Source: [Databricks](https://www.databricks.com/blog/reducing-qwen-training-time-6x-ai-runtime)
- Source URL: https://www.databricks.com/blog/reducing-qwen-training-time-6x-ai-runtime
- Ingested by Yomu: 2026-10-15T00:00:20.430Z

[Read original post](https://www.databricks.com/blog/reducing-qwen-training-time-6x-ai-runtime)