What is PEFT and how does LoRA fit into it?

PEFT (Parameter-Efficient Fine-Tuning) is the general strategy of training a small number of new parameters instead of the model's full weight set. LoRA (Low-Rank Adaptation) is the most widely used PEFT technique: it freezes the original weights and injects small trainable low-rank matrices alongside them, typically training well under 1% of the total parameter count.

What is QLoRA and how is it different from LoRA?

QLoRA is LoRA applied to a 4-bit quantized base model instead of a full-precision one — the frozen base weights are quantized (see the Quantization lesson) to cut memory further, while the small LoRA adapter matrices themselves stay in higher precision for training stability. It's what makes fine-tuning a 7B-70B model feasible on a single consumer GPU.

When should I use LoRA/PEFT instead of the hosted fine-tuning API?

The hosted fine-tuning API (see the Fine-Tuning lesson) is simpler when you're happy fine-tuning a provider's model and paying per-job. LoRA/PEFT matters when you need an open-weights model you can self-host, want to keep separate lightweight adapters for many different tasks against one shared base model, or need to fine-tune on hardware you control rather than send data to a third party.

Why does training only 0.18% of parameters still work?

The frozen base model already encodes general language ability from pretraining; fine-tuning for a specific task or format is mostly about steering that existing capability, not adding new knowledge from scratch. A small low-rank update to the attention weights is often enough steering — this is an empirical finding, not something provable in advance for a given task.

What does catastrophic forgetting mean and does LoRA avoid it?

Catastrophic forgetting is when fine-tuning on a narrow task degrades the model's general abilities, because full fine-tuning updates every weight and can overwrite previously-learned behavior. LoRA reduces this risk structurally — the base weights never change, only a small additive delta does — but doesn't eliminate it: an aggressively trained adapter can still push outputs toward the fine-tuning task even on unrelated prompts.

LoRA, QLoRA, and PEFT | SkillByExample

Fine-tuning every one of a 7B-parameter model’s weights needs enough GPU memory to hold gradients and optimizer state for all 7 billion of them — far more than the weights alone. LoRA fine-tunes a few hundred thousand new parameters instead, bolted onto a frozen model, and gets most of the practical benefit for a fraction of the memory and a checkpoint that’s two orders of magnitude smaller.

Attaching a LoRA Adapter

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("distilgpt2")
total_full = sum(p.numel() for p in model.parameters())
print(f"Full model parameters: {total_full:,}")

config = LoraConfig(
    r=8,                          # rank of the low-rank update matrices
    lora_alpha=16,                # scaling factor for the update
    target_modules=["c_attn"],    # which layers get an adapter
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
peft_model = get_peft_model(model, config)
peft_model.print_trainable_parameters()

$ python lora_attach.py
Full model parameters: 81,912,576
trainable params: 147,456 || all params: 82,060,032 || trainable%: 0.1797

147,456 trainable parameters out of 82 million — 0.18%. Every original weight in distilgpt2 is frozen (requires_grad=False); LoRA adds new, small, trainable matrices next to the attention projections and only those get gradients during training.

Training Updates Only the Adapter, and the Loss Really Drops

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

tok = AutoTokenizer.from_pretrained("distilgpt2")
tok.pad_token = tok.eos_token
model = AutoModelForCausalLM.from_pretrained("distilgpt2")

config = LoraConfig(r=8, lora_alpha=16, target_modules=["c_attn"], lora_dropout=0.05,
                     bias="none", task_type="CAUSAL_LM")
peft_model = get_peft_model(model, config)

# Teach the model to always answer a greeting in one fixed format
examples = [
    "Q: Hi there\nA: [GREETING] Hello! How can I help?",
    "Q: Hello\nA: [GREETING] Hello! How can I help?",
    "Q: Hey\nA: [GREETING] Hello! How can I help?",
    "Q: Good morning\nA: [GREETING] Hello! How can I help?",
]

optimizer = torch.optim.AdamW(peft_model.parameters(), lr=1e-3)
peft_model.train()

losses = []
for step in range(20):
    total_loss = 0.0
    for ex in examples:
        ids = tok(ex, return_tensors="pt")
        out = peft_model(**ids, labels=ids["input_ids"])
        out.loss.backward()
        total_loss += out.loss.item()
    optimizer.step()
    optimizer.zero_grad()
    avg = total_loss / len(examples)
    losses.append(avg)
    if step % 4 == 0 or step == 19:
        print(f"step {step:2}: loss={avg:.4f}")

print(f"\nLoss went from {losses[0]:.4f} to {losses[-1]:.4f}")

$ python lora_train.py
step  0: loss=4.3832
step  4: loss=3.8969
step  8: loss=3.1515
step 12: loss=2.3736
step 16: loss=1.4581
step 19: loss=1.0434

Loss went from 4.3832 to 1.0434

20 steps over 4 tiny examples, and the loss drops by more than 4x — with 147K trainable parameters, not 82 million. This toy example trains on 4 examples for 20 steps purely to make the loss curve visible quickly; a real task needs far more varied examples to generalize rather than memorize, exactly as the Fine-Tuning lesson’s data-preparation guidance covers.

The Checkpoint Is Genuinely Smaller

import os
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("distilgpt2")
config = LoraConfig(r=8, lora_alpha=16, target_modules=["c_attn"], lora_dropout=0.05,
                     bias="none", task_type="CAUSAL_LM")
peft_model = get_peft_model(model, config)

peft_model.save_pretrained("lora_adapter")
model.save_pretrained("full_model")

def dir_size(path):
    return sum(os.path.getsize(os.path.join(root, f))
               for root, _, files in os.walk(path) for f in files)

adapter_size = dir_size("lora_adapter")
full_size = dir_size("full_model")
print(f"LoRA adapter on disk: {adapter_size:,} bytes = {adapter_size/1024:.1f} KB")
print(f"Full model on disk:   {full_size:,} bytes = {full_size/1024**2:.1f} MB")
print(f"Adapter is {full_size/adapter_size:.0f}x smaller")

$ python lora_size.py
LoRA adapter on disk: 597,643 bytes = 583.6 KB
Full model on disk:   328,250,559 bytes = 313.0 MB
Adapter is 549x smaller

584KB instead of 313MB for the same fine-tuning result, on this tiny model — the ratio only gets more favorable at larger scale, since a LoRA adapter’s size depends on the rank and target layers, not the base model’s total parameter count. This is what makes it practical to keep a separate adapter per customer, per task, or per style, all sharing one loaded base model.

Where QLoRA Fits

QLoRA is this exact recipe with one change: the frozen base model is loaded in 4-bit quantized precision (see Quantization for what that costs in accuracy) instead of full precision, and only the small LoRA matrices train in higher precision. A 7B model that needs ~28GB in FP32 needs roughly 3.5-5GB in 4-bit — combined with LoRA’s <1% trainable parameters, that’s the difference between “needs a multi-GPU server” and “runs on one consumer GPU.” Loading and quantizing a full 7B model isn’t reproducible in this environment’s CPU-only setup, but the mechanism is exactly the two lessons combined: freeze and quantize the base (Quantization), train a small low-rank delta on top (this lesson).

Practice

1. Add a second target module (the MLP's `c_fc` layer) and see how much the trainable parameter count changes.

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("distilgpt2")
config = LoraConfig(r=8, lora_alpha=16, target_modules=["c_attn", "c_fc"],
                     lora_dropout=0.05, bias="none", task_type="CAUSAL_LM")
peft_model = get_peft_model(model, config)
peft_model.print_trainable_parameters()

trainable params: 331,776 || all params: 82,244,352 || trainable%: 0.4034

Adding c_fc roughly doubles the trainable count (147K to 332K) — the MLP layer is wider than the attention projection, so its low-rank adapter matrices are bigger too. Still under 1% of the full model.

2. Compare rank 4 vs rank 32 and confirm trainable parameters scale linearly with rank.

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

for r in [4, 32]:
    model = AutoModelForCausalLM.from_pretrained("distilgpt2")
    config = LoraConfig(r=r, lora_alpha=16, target_modules=["c_attn"],
                         lora_dropout=0.05, bias="none", task_type="CAUSAL_LM")
    peft_model = get_peft_model(model, config)
    trainable = sum(p.numel() for p in peft_model.parameters() if p.requires_grad)
    print(f"r={r}: {trainable:,} trainable parameters")

r=4: 73,728 trainable parameters
r=32: 589,824 trainable parameters

Exactly 8x more parameters for 8x the rank (4 to 32) — LoRA’s trainable parameter count is rank x (input_dim + output_dim) per target module, so it scales linearly with rank by construction. Higher rank means more capacity to adapt, and proportionally more to train.

3. Freeze the LoRA adapter (set requires_grad=False on it too) and confirm nothing trains.

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("distilgpt2")
config = LoraConfig(r=8, lora_alpha=16, target_modules=["c_attn"],
                     lora_dropout=0.05, bias="none", task_type="CAUSAL_LM")
peft_model = get_peft_model(model, config)

for p in peft_model.parameters():
    p.requires_grad = False

trainable = sum(p.numel() for p in peft_model.parameters() if p.requires_grad)
print(f"Trainable parameters after freezing everything: {trainable:,}")

Trainable parameters after freezing everything: 0

Zero — confirming there’s nothing magic about LoRA parameters being trainable by default, it’s the same requires_grad mechanism as any PyTorch parameter. get_peft_model just sets it to True on the new adapter matrices and False on everything else when it builds the model.

Attaching a LoRA Adapter

Training Updates Only the Adapter, and the Loss Really Drops

The Checkpoint Is Genuinely Smaller

Where QLoRA Fits

Practice

Frequently Asked Questions