LoRA (Low-Rank Adaptation)
LoRA is a parameter-efficient fine-tuning method that freezes a model's original weights and injects small trainable low-rank matrices into its layers. Only these compact matrices are trained, drastically cutting memory and storage while approaching the quality of full fine-tuning for many adaptation tasks.
Definition
LoRA (Low-Rank Adaptation)LoRA, introduced by Hu et al., is based on the observation that the weight updates needed to adapt a large model often have low intrinsic rank. Instead of updating a full weight matrix, LoRA learns two much smaller matrices whose product approximates the update, leaving the base weights untouched.
Because the base model is frozen, a single deployment can swap between many lightweight LoRA adapters, and each adapter is small enough to store and share easily. Quantized variants such as QLoRA push memory requirements low enough to fine-tune large models on a single GPU.
Frequently asked questions
What is LoRA (Low-Rank Adaptation)?
LoRA is a parameter-efficient fine-tuning method that freezes a model's original weights and injects small trainable low-rank matrices into its layers. Only these compact matrices are trained, drastically cutting memory and storage while approaching the quality of full fine-tuning for many adaptation tasks.
Sources
Related terms
← Full AI & prompt engineering glossary
Put LoRA (Low-Rank Adaptation) to work with Prompeteer, the Agentic Contextual AI Platform →