Quantization
Storing a model's numbers in fewer bits, trading a little accuracy for a much smaller and faster file. It is why a model that needs 13GB can run on a laptop in 4GB.
- See also: LLM
- See also: Local model
- Used in: Local LLMs
Storing a model's numbers in fewer bits, trading a little accuracy for a much smaller and faster file. It is why a model that needs 13GB can run on a laptop in 4GB.