Glossary · AI
What is
Quantization?
Reducing the numerical precision of model weights to make inference cheaper and faster.
By Anish· Founder · Vedwix
·Definition
Quantization converts model weights from 32-bit or 16-bit floats to 8-bit or 4-bit integers. The model becomes much smaller and faster, with a small quality penalty. Common formats include GGUF, GPTQ, AWQ, and bitsandbytes. Quantization is essential for serving LLMs at scale or on-device.
Example
A 7B-parameter model runs at 6 GB in FP16, ~3.5 GB in 4-bit quantization — small enough for a laptop.
How Vedwix uses Quantization in client work
We quantize fine-tuned models for production serving — typically Q5 or Q6 GGUF for balance.
Building with Quantization?
We ship this.
If you're building with Quantization in production, we can help — from architecture review to full implementation.
Brief us