Trade-offs: Full Fine Tuning vs LoRA vs QLoRA

The Fundamental Choice: When adapting a large language model to your task, you're choosing between three main approaches, each with concrete implications for quality, cost, and operational complexity. The decision framework hinges on your read write ratio for model updates, budget constraints, and quality requirements. Quality and Expressiveness: Full fine tuning updates every parameter, so in principle it can reach the absolute best specialization for your task. If you're training a model for medical diagnosis where 1% accuracy improvement translates to lives saved, and you have the budget, full fine tuning is worth considering. However, narrow task data often causes catastrophic forgetting: the model loses general capabilities. A customer support model trained only on refund queries might forget how to handle shipping questions. LoRA constrains updates to a low rank subspace. This typically gets you 95 to 98% of full fine tuning quality while preserving general capabilities much better. For assistant like systems that handle mixed workloads, this is often preferable. QLoRA adds 4 bit quantization, which introduces small numerical noise. Most tasks see 97 to 99% of full precision performance, but tasks requiring exact numerical outputs (code generation, arithmetic reasoning) can degrade by 5 to 10%. Adapter Size vs Capacity: Within LoRA and QLoRA, you control the rank r. Lower rank means fewer parameters and lower memory cost, but less expressiveness. Consider a production platform serving 1000 tenants: With rank 4 on attention only, each adapter might be 20 MB. You can fit 50 adapters in 1 GB of GPU memory for hot cache, serving high QPS multi tenant workloads with minimal latency. But performance on complex reasoning tasks may plateau. With rank 64 targeting all linear layers, adapters grow to 200 MB each. You fit only 5 in 1 GB, requiring more disk I/O for cold adapters, but you get near full fine tuning quality. The Databricks finding is instructive: going from rank 8 to rank 16 didn't help on their 3B model, but expanding from attention only to all linear layers did. This suggests layer coverage matters more than rank up to a threshold. Deployment Patterns: Merged adapters (computing W + A×B offline) give you the simplest serving: load one checkpoint, get baseline latency. This works for 5 to 10 high traffic specializations, such as major product surfaces. Inference latency matches the base model: 50 to 150 ms p50, 200 to 400 ms p99 for short generations on A100. Separate adapters enable multi tenancy but add complexity. You need adapter routing, cache management with Least Recently Used (LRU) eviction, and handling of cold start when an adapter isn't in GPU memory. The latency penalty is 3 to 7% from extra matrix multiplies if adapters are hot, or 20 to 50 ms if you must load from disk. This pattern shines when you serve hundreds of tasks, each at moderate QPS: aggregate 5000 QPS across 500 tenants is easier than maintaining 500 separate model deployments. When to Choose What: Use full fine tuning when you have a single mission critical task, the quality difference of 1 to 5% directly impacts business metrics, you can tolerate the cost of $500+ per training run, and you're willing to manage separate model artifacts and risk catastrophic forgetting. Use standard LoRA (16 bit or 8 bit) when you need to adapt to dozens of tasks, quality within 2 to 5% of full fine tuning is acceptable, you want to preserve general model capabilities, and you have access to multi GPU infrastructure for training but want lower serving costs. Use QLoRA when you have limited GPU budget (single cards), you need to experiment rapidly across many variants, quality within 3 to 10% is acceptable, or you're fine tuning models above 30B parameters where even standard LoRA would require multiple GPUs. For most production ML teams, QLoRA is the default starting point: fast iteration, low cost, and quality sufficient for 80% of use cases.