Quantization Explained
Learn how lower-precision weights reduce model size and what quality you may trade away.
The basic idea
Quantization represents model values with fewer bits than the original training representation. A lower-precision number needs less memory and can often be processed faster, especially on hardware designed for compact arithmetic. The model’s learned behavior is approximated rather than preserved exactly. Quantization is therefore an engineering trade-off between memory, speed, output quality, and compatibility with the chosen inference runtime.
Weights and activations
Weights are the parameters stored in the model file, while activations are intermediate values created during inference. Weight-only quantization is common because it greatly shrinks storage while keeping computation manageable. Quantizing activations can save additional memory but may be more sensitive to calibration and hardware support. Understanding which parts are quantized helps explain why two files with similar sizes can have different performance.
from pathlib import Path
for path in sorted(Path("models").glob("*.gguf")):
size_gb = path.stat().st_size / 1024**3
print(f"{path.name:32} {size_gb:.2f} GiB")Bit widths and naming
Labels such as eight-bit, six-bit, four-bit, or two-bit describe an approximate precision budget, but naming conventions vary between formats and tools. Some schemes use group-wise scales, mixed precision, or higher precision for sensitive layers. Do not compare labels as if they were universal quality grades. Inspect the format documentation and measure the exact file on the exact runtime you plan to use.
Quality trade-offs
Quantization can slightly change factual accuracy, instruction following, multilingual behavior, coding ability, or formatting reliability. The impact is often small for casual use and significant for narrow tasks that depend on exact tokens or long reasoning chains. Perplexity can provide a useful signal, but application-level tests are more informative. Include difficult examples, structured outputs, and safety cases in the comparison.
Memory and context
The model file is only one part of the memory budget. Runtime overhead, temporary buffers, tokenizer state, and the key-value cache grow with context length and concurrent requests. A quantized model may fit at startup but fail under a long prompt or multiple users. Measure peak memory during realistic requests, and leave headroom for the operating system and other processes.
Calibration and advanced methods
Some quantization methods use representative calibration data to estimate which values are most important, while others apply fixed or learned scales. The best method depends on architecture, runtime kernels, and workload. Calibration data should resemble real traffic without containing secrets. Preserve the calibration recipe and evaluation results, because a quantized artifact without its provenance is difficult to trust or reproduce.
Selecting a quantized model
Start with the highest-precision artifact that fits your latency and memory target, then test lower settings only if they provide a meaningful benefit. Compare answer quality, tokens per second, time to first token, peak memory, and failure rates. Keep the original model or a trusted baseline available. A smaller file is valuable only when it improves the complete product experience.
Worked example: fitting a model into a developer laptop
Suppose a full-precision artifact needs more memory than a laptop has available after the operating system and application are running. A four-bit version may fit, but the usable budget also includes runtime buffers and the key-value cache. Choose a fixed prompt set containing code, factual questions, JSON extraction, and a long context. Compare the full and quantized artifacts on the same runtime rather than judging quality from file size alone.
Code walkthrough
The Python snippet measures artifact size with filesystem metadata and converts bytes into GiB. It answers only the storage question; it does not tell you peak RAM, load time, or answer quality. Extend the experiment by launching each artifact, recording startup time and process memory, then replaying fixtures and saving outputs with the quantization label. A small CSV manifest makes it possible to inspect regressions without relying on memory or screenshots.
Trade-offs to measure
Higher precision usually protects fragile behaviors such as exact formatting, rare languages, and long-context retrieval, but consumes more memory and may reduce concurrency. Aggressive quantization can make a model affordable on edge hardware while causing subtle omissions that a casual chat test misses. Consider mixed-precision or a smaller base model when four-bit quality is unacceptable. Compatibility with kernels and accelerators can matter as much as the nominal bit width.
Practical exercise
Create three artifacts at different precisions and evaluate each on 20 fixed prompts. Track exact-match extraction, JSON validity, grounded answer quality, tokens per second, time to first token, load time, and peak memory. Mark failures by type instead of using one overall score. Select the lowest-precision artifact that stays within your quality threshold and document why the threshold is appropriate for the product.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.