How LoRA Works: Low Rank Adaptation Mechanics

The Core Insight: Low Rank Adaptation (LoRA) is based on a mathematical observation: when fine tuning a pre trained model, the optimal weight updates often lie in a low rank subspace. Instead of learning a full dense update matrix ΔW with dimensions m by n (which could be millions of parameters), LoRA decomposes this into two much smaller matrices. The Mathematical Trick: For each weight matrix W in the model (typically attention projections), LoRA introduces two new matrices A and B. Matrix A has dimensions m by r, and matrix B has dimensions r by n, where r is the rank and is much smaller than both m and n. The update becomes ΔW = A × B. Consider a concrete example: an attention weight matrix might be 4096 by 4096, containing roughly 16.8 million parameters. With LoRA at rank r = 8, you instead learn matrix A of size 4096 by 8 and matrix B of size 8 by 4096. That's only 32,768 plus 32,768 equals 65,536 parameters, a reduction from 16.8 million to 65 thousand, which is 256 times fewer parameters. Training Mechanics: During training, the forward pass computes W times x plus A times (B times x) for each adapted layer. The base weight W stays completely frozen, so gradients only flow through A and B. This means optimizer states (momentum, variance for Adam) are stored only for these small matrices. For a 7B parameter model with LoRA adapters totaling 50M parameters, optimizer states might be 200 MB instead of 84 GB. Which Layers Get Adapted: Practitioners typically target specific linear projections in the Transformer architecture. Common choices include the query, key, value, and output projections in multi head attention. Some implementations extend to feed forward layers: the up projection, gate projection, and down projection. Databricks experiments found that targeting all linear layers with rank 8 yielded better results than using rank 16 on attention only, suggesting that layer coverage matters more than rank size up to a point. Deployment Options: At inference time, you have two choices. You can merge the adapters by computing W prime = W + A times B offline, saving a single merged checkpoint per task. Inference then uses this merged weight with no extra computation. Alternatively, you can keep adapters separate and apply them dynamically: compute W times x plus A times (B times x) at runtime. The separate approach adds a small overhead from extra matrix multiplies (typically 3 to 7% latency increase), but enables loading different adapters without duplicating the base model in GPU memory.