Representing weights or activations in lower precision (for example 8-bit or 4-bit) to shrink memory and speed inference—common for local and high-throughput deployment.
Quantization trades numerical precision for footprint and throughput. Techniques vary from post-training quantization to training-aware schemes.
Practical takeaway
Expect accuracy shifts; always benchmark on your tasks after quantizing. Pairs naturally with local inference and cost-sensitive LLM serving—often via Ollama, vLLM, or similar stacks.