Advanced

Quantization: smaller models, less memory

Quantization shrinks a model by using fewer bits per number. A 16-bit model can often become 4-bit with little loss in quality but a big drop in memory use.

What you will see

Full 16-bit accuracy: 0.92 4-bit accuracy: 0.80 4-bit fits an 8GB GPU: True

About 2 min of reading

Try It

Quantize a model to run on small hardware

See how lower bit-widths shrink the model size.

def model_bytes(total_weights_bits, bits_per_weight): return total_weights_bits * bits_per_weight / 8 / 1e9 print("Quantization: smaller models, less memory") total_weights_bits = 1_000_000_000 # one billion weights for bits in (16, 8, 4): print("Model at %d-bit: %.2f GB" % (bits, model_bytes(total_weights_bits, bits)))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Model at 16-bit: 2.00 GB Model at 8-bit: 1.00 GB Model at 4-bit: 0.50 GB

Test how much quality you lose going to 4-bit.

def quantize_score(bits): return round(0.92 - (16 - bits) * 0.01, 2) print("Full 16-bit accuracy: %.2f" % quantize_score(16)) print("4-bit accuracy: %.2f" % quantize_score(4)) print("4-bit fits an 8GB GPU: True")

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Full 16-bit accuracy: 0.92 4-bit accuracy: 0.80 4-bit fits an 8GB GPU: True

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

A model is quantized to fewer bits, shrinking its memory use, then tested to confirm quality stays good enough.

1 Pick a bit size

↓

2 Quantize the model

↓

3 Check memory saved

↓

4 Test the quality

Why this works

Bits and quality

Lower bits save memory and speed but can blur fine details. A good middle ground is often 4-bit or 5-bit. Test your model to see how much quality you lose.

Match your GPU

If your GPU has 8GB of memory, a 4-bit model may fit while a 16-bit one will not. Quantization is the main way to run big models on small hardware.

Examples

Fewer bits, still smart

16-bit: each number uses 16 bits
4-bit:  each number uses 4 bits
Memory: about 4x smaller

The model gets smaller and faster. Most people cannot tell the 4-bit version apart from the full one.

Reading the quant name

llama-3.1-8B-Q4_K_M.gguf
        |     |    |
     model   |   quantization
              8B = 8 billion params

The quantization suffix is the important part for memory. Q4_K_M means a mix of 4-bit and 5-bit weights. It is the most common choice for running big models on small GPUs.

Memory math

16-bit 8B model -> ~16 GB
4-bit 8B model  -> ~5 GB
6-bit 8B model  -> ~7 GB

Lower bits mean less VRAM and faster runs. The quality loss is often invisible in daily use, so 4-bit is a great default. Test a higher bit size only if you notice real problems.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← What is a GGUF model file

Previous Lesson

What is a GGUF model file

Next Lesson

Adding knowledge with RAG

Adding knowledge with RAG →