---
title: "Beyond LoRA: Can you beat the most popular fine-tuning technique?"
description: "Parameter-efficient fine-tuning (PEFT) reduces the memory required to adapt open models, enabling fine-tuning with a fraction of the memory and, in some cases, quantized models. LoRA is dominant because it freezes base weights and trains a small set of added parameters, but its popularity may also reflect ecosystem momentum and uneven comparisons. The post presents shared-condition benchmarks for language-model math reasoning and image generation, using identical models, datasets, code, hardware, and measures including test performance, VRAM, forgetting or drift, runtime, and checkpoint size. The results show that LoRA performs well but is not consistently best: other techniques can outperform it on one or more axes, especially in the image-generation benchmark. The recommended choice is therefore to compare techniques against a use case’s needs and constraints, including quantization support, layer coverage, adapter merging, and downstream serving compatibility."
---

# Beyond LoRA: Can you beat the most popular fine-tuning technique?

[Hugging Face](https://yomu.fyi/company/hugging-face) · Benjamin Bossan, Sayak Paul, Marian Tietz, Kashif Rasul · Jun 18, 2026

**Type:** Explainer

## Summary

Parameter-efficient fine-tuning (PEFT) reduces the memory required to adapt open models, enabling fine-tuning with a fraction of the memory and, in some cases, quantized models. LoRA is dominant because it freezes base weights and trains a small set of added parameters, but its popularity may also reflect ecosystem momentum and uneven comparisons. The post presents shared-condition benchmarks for language-model math reasoning and image generation, using identical models, datasets, code, hardware, and measures including test performance, VRAM, forgetting or drift, runtime, and checkpoint size. The results show that LoRA performs well but is not consistently best: other techniques can outperform it on one or more axes, especially in the image-generation benchmark. The recommended choice is therefore to compare techniques against a use case’s needs and constraints, including quantization support, layer coverage, adapter merging, and downstream serving compatibility.

## Context

Fine-tuning open models is memory-hungry, while quantized models cannot be fine-tuned directly. Although LoRA dominates PEFT usage, its popularity may be reinforced by visibility, tutorials, examples, and downstream support, making it difficult to determine whether it is actually the best technique for a particular use case from paper results alone.

## Approach / What changed

The PEFT library provides multiple techniques through a unified API. Its benchmarks compare techniques under the same base model, dataset, training and evaluation code, and hardware across language-model math reasoning and image generation, tracking test performance, VRAM usage, forgetting or drift, runtime, and checkpoint size. The library also supports converting some non-LoRA adapters into LoRA checkpoints for downstream serving.

## Takeaways

- LoRA freezes the base model weights and trains a small set of added parameters, producing smaller memory requirements and checkpoints than full fine-tuning.
- The benchmarks found that LoRA works well but can be beaten by other PEFT methods on test performance, memory usage, or other measured axes; image generation showed particularly strong cases for alternatives.
- PEFT supports changing techniques by replacing the configuration, while non-LoRA adapters can sometimes be converted to LoRA for serving systems that support only LoRA.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance)

- Source: [Hugging Face](https://huggingface.co/blog/peft-beyond-lora)
- Source URL: https://huggingface.co/blog/peft-beyond-lora
- Ingested by Yomu: 2026-08-29T01:50:44.209Z

[Read original post](https://huggingface.co/blog/peft-beyond-lora)