System memory matters
If a model is bigger than your GPU, it spills into system RAM and slows down a lot. Having plenty of RAM keeps fallbacks from being painfully slow.
Choosing hardware for local models
Hardware decides what you can run. GPU memory is the main limit, so choosing the right GPU and RAM matters more than raw speed.
What you will see
Choosing hardware for local models
Model 8GB on 12GB GPU (dedicated): 16.00s
Model 16GB on 12GB GPU (spills to RAM): 320.00s
Model 16GB on CPU only: 800.00s
A dedicated GPU with enough VRAM is the best upgrade: True
About 1 min of reading
Try It
Choose hardware based on GPU memory
1
Compare inference speed on a dedicated GPU versus spilled to RAM versus CPU only.
def gpu_inference_time(params_gb, vram_gb, has_gpu):
if has_gpu and params_gb <= vram_gb:
return round(params_gb * 2, 2)
if has_gpu:
return round(params_gb * 20, 2)
return round(params_gb * 50, 2)
print()
print("Choosing hardware for local models")
has_gpu = True
vram_gb = 12
params_gb = 8
print("Model 8GB on 12GB GPU (dedicated): %.2fs" % gpu_inference_time(params_gb, vram_gb, has_gpu))
params_gb = 16
print("Model 16GB on 12GB GPU (spills to RAM): %.2fs" % gpu_inference_time(params_gb, vram_gb, has_gpu))
has_gpu = False
print("Model 16GB on CPU only: %.2fs" % gpu_inference_time(params_gb, vram_gb, has_gpu))
print("A dedicated GPU with enough VRAM is the best upgrade: True")
Before you run it, say out loud what it will print. Then open the answer.
What it prints
Choosing hardware for local models
Model 8GB on 12GB GPU (dedicated): 16.00s
Model 16GB on 12GB GPU (spills to RAM): 320.00s
Model 16GB on CPU only: 800.00s
A dedicated GPU with enough VRAM is the best upgrade: True
Data Flow
Hardware is chosen around GPU VRAM, then matched with enough system memory and a fast runner for good speed.
1
Count required VRAM
↓
2
Pick a GPU
↓
3
Add system RAM
↓
4
Choose a runner
Why this works
When you have a choice
GPU vs CPU
A GPU runs models far faster than a CPU. If you want usable speed, a dedicated GPU with enough VRAM is the best single upgrade you can make.
Examples
VRAM is king
Model needs ~2 bytes per parameter
8B model -> ~16GB at 16-bit
8B model -> ~5GB at 4-bitYour GPU's VRAM sets the biggest model you can run. More VRAM means bigger or higher-quality models.
Apple unified memory
NVIDIA GPU -> fast, capped by VRAM
Apple M chip -> CPU and GPU share RAM
64GB unified -> run big models slowlyApple's unified memory lets the model use system RAM as VRAM, so you can run larger models than a consumer NVIDIA card. They are slower but very flexible for a single machine.
The minimum viable setup
GPU: 8GB VRAM minimum
RAM: 16GB system
Disk: SSD with 20GB free
CPU: 4 cores is enough for fallbackYou do not need top-tier hardware to start. A 8GB GPU handles 3B-7B models at 4-bit. Upgrade VRAM first, then RAM, then only bother with a faster CPU if you run fallbacks often.
Words this lesson uses
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.
← Finetuning a local model
Previous Lesson
Finetuning a local model