LoRA (Low-Rank Adaptation) is the most widely used PEFT technique — it freezes a pretrained weight matrix entirely, and adds a small, trainable, low-rank update alongside it, dramatically reducing the number of parameters that need training.
The Core Formula
\(\mathbf{W}\) is the original pretrained weight matrix (shape \(d\times k\)), kept completely frozen. Instead of learning a full \(d\times k\) update matrix \(\Delta\mathbf{W}\) directly, LoRA decomposes it into two much smaller matrices: \(\mathbf{B}\) (shape \(d\times r\)) and \(\mathbf{A}\) (shape \(r\times k\)), where the rank \(r\) is small — commonly 4, 8, or 16 — far smaller than \(d\) or \(k\).
The Parameter Savings, Precisely
For a typical LLM attention projection with \(d=k=4096\) and \(r=8\): the full update would need \(4096 \times 4096 \approx 16.8\) million parameters; LoRA's decomposition needs only \(8 \times (4096+4096) = 65{,}536\) parameters — a reduction of roughly 256x for this single matrix.
Why Low Rank Works
This connects directly back to Eigenvalues and matrix rank concepts from Linear Algebra — a low-rank matrix \(\mathbf{B}\mathbf{A}\) can only represent updates within a limited, \(r\)-dimensional subspace, rather than arbitrary full-rank changes. LoRA's empirical success is direct evidence for exactly the "low intrinsic dimensionality of adaptation" idea introduced in PEFT — most useful task adaptations genuinely fit within this constrained, low-rank space.
Initialization — Starting at Zero Change
\(\mathbf{A}\) is typically initialized randomly, while \(\mathbf{B}\) is initialized to exactly zero — making \(\Delta\mathbf{W} = \mathbf{B}\mathbf{A} = \mathbf{0}\) at the very start of training. This means the model behaves identically to the original pretrained model before any LoRA training has happened, and gradually diverges as training proceeds — a safe, well-behaved starting point.
Code
import torch
import torch.nn as nn
class LoRALinear(nn.Module):
def __init__(self, original_linear, r=8, alpha=16):
super().__init__()
self.original = original_linear
for param in self.original.parameters():
param.requires_grad = False # freeze the pretrained weight entirely
d_in, d_out = original_linear.in_features, original_linear.out_features
self.A = nn.Parameter(torch.randn(r, d_in) * 0.01)
self.B = nn.Parameter(torch.zeros(d_out, r)) # B starts at zero -- no initial change
self.scale = alpha / r
def forward(self, x):
original_output = self.original(x)
lora_update = (x @ self.A.T @ self.B.T) * self.scale
return original_output + lora_update # W'x = Wx + BAxCommon Mistakes
- Choosing \(r\) far larger than needed — this reduces the parameter efficiency benefit without necessarily improving performance, since most tasks' useful adaptation genuinely fits in a low-rank space.
- Initializing both \(A\) and \(B\) randomly instead of zeroing \(B\) — this would make the model behave differently from the pretrained model even before training starts, an unnecessary and potentially destabilizing change.
Interview Relevance
Q: "Explain the LoRA formula and why decomposing the weight update into two low-rank matrices saves so many parameters." LoRA freezes the original weight \(\mathbf{W}\) and represents the task-specific update as \(\mathbf{B}\mathbf{A}\), where \(\mathbf{B}\) is \(d\times r\) and \(\mathbf{A}\) is \(r\times k\), with \(r\) much smaller than \(d\) or \(k\). Since the number of trainable parameters is \(r(d+k)\) instead of the full \(d\times k\), and \(r\) is typically tiny (4–16) relative to \(d\) and \(k\) (often thousands), the parameter savings can be several hundred-fold for large matrices.
Practice Question
For a weight matrix of shape \(2048\times2048\) with LoRA rank \(r=16\), how many trainable parameters does the LoRA update introduce, compared to the full matrix's parameter count?
