Master

Optimising speed and memory

Once a model runs, you can make it faster and lighter with techniques like KV cache tuning, continuous batching, and better backends.

What you will see

KV cache VRAM usage: 16-bit, 1000 units: 16.00 GB 8-bit, 1000 units: 8.00 GB Lower KV cache precision frees VRAM: True

About 1 min of reading

Try It

Pick a backend and tune the KV cache

Compare backend latency across hardware.

def backend_latency(backend, device): if device == "nvidia" and backend == "vllm": return 1.0 if backend == "vllm": return 3.0 return 2.0 print() print("Optimising speed and memory") for backend in ("llama.cpp", "vllm"): print("%s on NVIDIA: %.2fs" % (backend, backend_latency(backend, "nvidia"))) print("%s on CPU: %.2fs" % (backend, backend_latency(backend, "cpu")))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Optimising speed and memory llama.cpp on NVIDIA: 2.00s llama.cpp on CPU: 2.00s vllm on NVIDIA: 1.00s vllm on CPU: 3.00s

Show how lower KV cache precision frees VRAM.

def kv_cache_vram(bits, capacity): return bits * capacity / 1000 # GB print() print("KV cache VRAM usage:") print("16-bit, 1000 units: %.2f GB" % kv_cache_vram(16, 1000)) print("8-bit, 1000 units: %.2f GB" % kv_cache_vram(8, 1000)) print("Lower KV cache precision frees VRAM: True")

Before you run it, say out loud what it will print. Then open the answer.

What it prints

KV cache VRAM usage: 16-bit, 1000 units: 16.00 GB 8-bit, 1000 units: 8.00 GB Lower KV cache precision frees VRAM: True

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

A model is served through a fast backend, with KV cache and batching tuned, so it runs more requests per second on the same hardware.

1 Pick a backend

↓

2 Tune KV cache

↓

3 Batch requests

↓

4 Measure throughput

Why this works

Backend choice

Different runners speed up models differently. llama.cpp works everywhere; vLLM is very fast on NVIDIA GPUs. Pick the backend that matches your hardware.

KV cache tuning

The KV cache stores earlier context. Tuning its precision and size can free a lot of VRAM, letting you handle longer prompts or more users at once.

Examples

Run it lean

Use a fast backend (llama.cpp, vLLM)
Enable KV cache quantization
Batch requests together

These tweaks can double or triple throughput on the same hardware without changing the model.

KV cache budget

KV cache grows with prompt length
Long prompt -> more cache -> less room for new tokens
Truncate or quantize to free space

The KV cache is the biggest hidden memory user. Quantizing it to 8-bit can free a lot of VRAM while costing little speed, letting you serve more users per second.

Continuous batching

Classic batching -> wait for all requests, slow
Continuous batching -> new input slots every step
   Throughput up 2-3x

Continuous batching lets the backend fill empty slots as requests finish, instead of waiting for a whole batch. vLLM uses PagedAttention to make this efficient and safe with long contexts.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Choosing hardware for local models

Previous Lesson

Choosing hardware for local models

Next Lesson

Deploying a private model safely

Deploying a private model safely →