Beyond LoRA: Can you beat the most popular fine-tuning technique?

Summary

Parameter-efficient fine-tuning (PEFT) reduces the memory required to adapt open models, enabling fine-tuning with a fraction of the memory and, in some cases, quantized models. LoRA is dominant because it freezes base weights and trains a small set of added parameters, but its popularity may also reflect ecosystem momentum and uneven comparisons. The post presents shared-condition benchmarks for language-model math reasoning and image generation, using identical models, datasets, code, hardware, and measures including test performance, VRAM, forgetting or drift, runtime, and checkpoint size. The results show that LoRA performs well but is not consistently best: other techniques can outperform it on one or more axes, especially in the image-generation benchmark. The recommended choice is therefore to compare techniques against a use case’s needs and constraints, including quantization support, layer coverage, adapter merging, and downstream serving compatibility.

Takeaways

1. 1

   LoRA freezes the base model weights and trains a small set of added parameters, producing smaller memory requirements and checkpoints than full fine-tuning.

2. 2

   The benchmarks found that LoRA works well but can be beaten by other PEFT methods on test performance, memory usage, or other measured axes; image generation showed particularly strong cases for alternatives.

3. 3

   PEFT supports changing techniques by replacing the configuration, while non-LoRA adapters can sometimes be converted to LoRA for serving systems that support only LoRA.