GGUF is a file format for local models. It packs a model so it can run efficiently on a normal computer, including on a GPU if you have one.
About 1 min of reading
GGUF files come in many sizes. A 4GB file runs fast on any laptop. A 40GB file is smarter but needs a strong GPU. Match the size to your hardware.
The same model can be saved as different GGUF files with different sizes. Smaller means faster and less memory. Bigger means higher quality. Choose based on your machine.
llama3.2.gguf -> the brain
mistral.gguf -> another brain
phi3.gguf -> a small fast brainEach GGUF file is a different model. You pick one to load based on your computer's power and what you want to do.
Q4_K_M -> good balance, most popular
Q8_K -> near full quality, large
Q2_K -> tiny, quality dropsGGUF files bundle the model with a quantization level. The suffix tells you how many bits are used. Most people start with a Q4_K_M file for a good mix of speed and quality.
GGUF files are shared on places like Hugging Face. Each release names the model and the quantization, for example llama-3.2-3B-Instructor-Q4_K_M.gguf.
Read the author and the size before downloading. A file that is too big for your machine will load, but it will be painfully slow.
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.