Skip to main content
LLM Engineering · lesson 21 of 27

Beginner

8

Intermediate

11

Advanced

8

Also part of LLM Engineering

4
Open the LLM Engineering page →
LLM Engineering advanced Lesson 21 of 27

Quantization

Measure the real memory and quality trade-off of FP32, FP16, and INT8 precision, and where GPTQ, AWQ, and GGUF fit in production.

By SkillByExample · Published

A 7-billion-parameter model trained and stored at full 32-bit precision needs roughly 28GB of VRAM just to hold its weights — more than any consumer GPU has. The model’s knowledge isn’t 28GB of information; a large fraction of that is precision the model doesn’t actually need to produce good output. Quantization is spending less of that memory on precision.

Real Model Size at FP32 vs FP16

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("distilgpt2")

def model_size_bytes(m):
    return sum(p.numel() * p.element_size() for p in m.parameters())

fp32_bytes = model_size_bytes(model)
print(f"FP32: {fp32_bytes:,} bytes = {fp32_bytes / 1024**2:.1f} MB")

model_fp16 = AutoModelForCausalLM.from_pretrained("distilgpt2", dtype=torch.float16)
fp16_bytes = model_size_bytes(model_fp16)
print(f"FP16: {fp16_bytes:,} bytes = {fp16_bytes / 1024**2:.1f} MB")

print(f"Reduction: {(1 - fp16_bytes / fp32_bytes) * 100:.1f}%")

w = model.transformer.h[0].mlp.c_fc.weight
w16 = model_fp16.transformer.h[0].mlp.c_fc.weight
print(f"\nSame weight, dtype {w.dtype}: {w[0, :3].tolist()}")
print(f"Same weight, dtype {w16.dtype}: {w16[0, :3].tolist()}")
$ python size_fp16.py
FP32: 327,650,304 bytes = 312.5 MB
FP16: 163,825,152 bytes = 156.2 MB
Reduction: 50.0%

Same weight, dtype torch.float32: [0.12394002079963684, 0.10340151190757751, -0.00964804645627737]
Same weight, dtype torch.float16: [0.12396240234375, 0.1033935546875, -0.00965118408203125]

Exactly half the memory, as expected (32 bits down to 16 bits per weight), and the individual values barely moved — 0.12394... became 0.12396.... FP16 is still floating-point; it just has less mantissa precision, and 82M-parameter distilgpt2’s weights mostly fit comfortably in that reduced range.

Manual INT8 Quantization Has Real, Measurable Error

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("distilgpt2")
w = model.transformer.h[0].mlp.c_fc.weight.detach()

def quantize_int8(tensor):
    scale = tensor.abs().max() / 127.0
    q = torch.round(tensor / scale).clamp(-127, 127).to(torch.int8)
    return q, scale

def dequantize(q, scale):
    return q.float() * scale

q, scale = quantize_int8(w)
reconstructed = dequantize(q, scale)

print(f"Original dtype/size:     {w.dtype}, {w.numel() * w.element_size():,} bytes")
print(f"Quantized dtype/size:    {q.dtype}, {q.numel() * q.element_size():,} bytes (+4 bytes for the scale)")
print(f"Scale factor:            {scale.item():.6f}")
print()
print(f"Original values:         {w[0, :5].tolist()}")
print(f"Int8 codes:              {q[0, :5].tolist()}")
print(f"Dequantized values:      {reconstructed[0, :5].tolist()}")
print()
mae = (w - reconstructed).abs().mean().item()
max_err = (w - reconstructed).abs().max().item()
print(f"Mean absolute error:     {mae:.6f}")
print(f"Max absolute error:      {max_err:.6f}")
print(f"Original value range:    [{w.min().item():.4f}, {w.max().item():.4f}]")
$ python manual_int8.py
Original dtype/size:     torch.float32, 9,437,184 bytes
Quantized dtype/size:    torch.int8, 2,359,296 bytes (+4 bytes for the scale)
Scale factor:            0.041376

Original values:         [0.12394002079963684, 0.10340151190757751, -0.00964804645627737, 0.08984390646219254, -0.10196784138679504]
Int8 codes:              [3, 2, 0, 2, -2]
Dequantized values:      [0.12412916123867035, 0.0827527716755867, 0.0, 0.0827527716755867, -0.0827527716755867]
Mean absolute error:     0.010344
Max absolute error:      0.020688
Original value range:    [-2.3516, 5.2548]

This is the simplest possible quantization scheme: one scale factor for the entire tensor (symmetric, per-tensor quantization). The size drops 4x (32-bit float to 8-bit int) but the error is visible — 0.10340 became 0.08275 after round-tripping through an 8-bit integer, a >20% relative error on that individual value. Production INT8/INT4 methods (GPTQ, AWQ) don’t use one scale for the whole tensor; they use per-channel or even per-group scales, and they choose which weights to protect using calibration data, which is why they achieve much smaller quality loss than this naive version.

The Quality Trade-off Is Real But Small at FP16

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilgpt2")
model_fp32 = AutoModelForCausalLM.from_pretrained("distilgpt2")
model_fp16 = AutoModelForCausalLM.from_pretrained("distilgpt2", dtype=torch.float16)

prompt = "The capital of France is"
ids = tok(prompt, return_tensors="pt")

with torch.no_grad():
    logits32 = model_fp32(**ids).logits[0, -1]
    logits16 = model_fp16(**ids).logits[0, -1].float()

top5_32 = torch.topk(torch.softmax(logits32, dim=-1), 5)
top5_16 = torch.topk(torch.softmax(logits16, dim=-1), 5)

print("FP32 top 5:")
for p, i in zip(top5_32.values, top5_32.indices):
    print(f"  {tok.decode([i]):>10} {p.item():.4f}")

print("FP16 top 5:")
for p, i in zip(top5_16.values, top5_16.indices):
    print(f"  {tok.decode([i]):>10} {p.item():.4f}")

same_top1 = top5_32.indices[0].item() == top5_16.indices[0].item()
print(f"\nSame top-1 prediction: {same_top1}")
$ python quality_check.py
FP32 top 5:
         the 0.1177
        home 0.0823
         now 0.0499
           a 0.0493
       being 0.0356
FP16 top 5:
         the 0.1182
        home 0.0812
         now 0.0508
           a 0.0493
       being 0.0349

Same top-1 prediction: True

The FP16 probabilities shift by less than a percentage point and the top-1 prediction is identical — this is why FP16/BF16 is close to a free lunch and is the default for serving most production models. The bigger memory savings (and bigger risk to quality) come from INT8 and INT4, which is where GPTQ, AWQ, and llama.cpp’s GGUF format come in: real algorithms for choosing scales carefully enough that the error from the previous section stays small even at 4 bits per weight.

Practice

1. Measure the quantized model's size relative to FP32 as a percentage, and compare it to the theoretical 4x number for INT8. 
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("distilgpt2")

def total_bytes_as_int8(m):
    return sum(p.numel() * 1 for p in m.parameters())  # 1 byte per int8 weight

def total_bytes_fp32(m):
    return sum(p.numel() * p.element_size() for p in m.parameters())

fp32 = total_bytes_fp32(model)
int8 = total_bytes_as_int8(model)
print(f"FP32: {fp32 / 1024**2:.1f} MB")
print(f"INT8 (weights only): {int8 / 1024**2:.1f} MB")
print(f"Actual reduction: {(1 - int8/fp32)*100:.1f}%  (theoretical max for 32->8 bit: 75%)")
FP32: 312.5 MB
INT8 (weights only): 78.1 MB
Actual reduction: 75.0%  (theoretical max for 32->8 bit: 75%)

Exactly the theoretical number — 4 bytes per weight down to 1 byte is a 75% reduction, with no overhead beyond the per-tensor scale factors (a handful of floats, negligible against millions of weights).

2. Quantize a different weight tensor (the attention projection instead of the MLP) and compare its error. 
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("distilgpt2")
w = model.transformer.h[0].attn.c_attn.weight.detach()

scale = w.abs().max() / 127.0
q = torch.round(w / scale).clamp(-127, 127).to(torch.int8)
reconstructed = q.float() * scale

mae = (w - reconstructed).abs().mean().item()
print(f"attn.c_attn value range: [{w.min().item():.4f}, {w.max().item():.4f}]")
print(f"Mean absolute error: {mae:.6f}")
attn.c_attn value range: [-2.1074, 2.3830]
Mean absolute error: 0.004691

A narrower value range (-2.11 to 2.38, vs. the MLP layer’s -2.35 to 5.25) produces a smaller scale factor and less error per weight — the same 127-level integer grid covers less numeric range, so each level represents a finer step (roughly half the error of the MLP layer’s 0.010344). This is exactly why per-channel/per-tensor scaling matters: different layers have different value ranges, and one global scale for an entire model wastes precision on the layers with narrow ranges.

3. Confirm INT4 (16 levels instead of 256) has visibly more reconstruction error than INT8 on the same tensor. 
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("distilgpt2")
w = model.transformer.h[0].mlp.c_fc.weight.detach()

for bits, levels in [(8, 127), (4, 7)]:
    scale = w.abs().max() / levels
    q = torch.round(w / scale).clamp(-levels, levels)
    reconstructed = q * scale
    mae = (w - reconstructed).abs().mean().item()
    print(f"INT{bits}: mean absolute error = {mae:.6f}")
INT8: mean absolute error = 0.010344
INT4: mean absolute error = 0.102348

Roughly 10x more error at 4 bits than 8 bits, even though the bit width only halved — going from 254 representable levels down to 14 shrinks the grid far faster than linearly, so error grows much faster than linearly too. This is exactly why production INT4 methods (GPTQ, AWQ) don’t use this naive uniform scheme — they need much smarter per-group scaling to keep 4-bit models usable at all.

Frequently Asked Questions

What is quantization?
Storing a model's weights using fewer bits per number than they were trained with — typically FP32 (32-bit floats) down to FP16/BF16 (16-bit), INT8 (8-bit integers), or INT4 (4-bit) — to cut memory footprint and often increase inference speed, at the cost of some numerical precision.
Does quantization always hurt output quality?
FP16/BF16 (halving precision, still floating-point) is nearly lossless for most models — the measured example here changed logit values by a fraction of a percent and kept the identical top prediction. Going to INT8 or INT4 (integer quantization, a much bigger precision cut) introduces real reconstruction error, and quality loss becomes noticeable if done naively; production INT8/INT4 quantization methods use careful per-channel scaling specifically to control this.
What is the difference between GPTQ, AWQ, and GGUF?
GPTQ and AWQ are post-training quantization *algorithms* — GPTQ minimizes per-layer reconstruction error using calibration data, AWQ specifically protects the weight channels that activations show are most important. GGUF is a *file format* (used by llama.cpp/Ollama) for storing quantized weights efficiently for CPU/edge inference, independent of which algorithm produced them.
Why quantize instead of just using a smaller model?
A quantized larger model often outperforms a full-precision smaller model at the same memory footprint — quantization mostly costs precision per weight, not knowledge or capability, so a 7B model quantized to 4-bit (~3.5GB) frequently beats a native 1-3B model at similar memory cost.
How much VRAM does a 7B-parameter model need at each precision?
Roughly: FP32 = 28GB, FP16/BF16 = 14GB, INT8 = 7GB, INT4 = 3.5GB — one byte per billion parameters per 8 bits of precision, plus some overhead for activations and the KV cache during inference. This is why 4-bit quantization is what makes 7B-70B models runnable on consumer GPUs at all.