Model Quantization: Q2 to FP16 Complete Guide | Avondale.AI Technical Deep-Dive

Model Quantization: The Complete Technical Guide

Understanding Q2 through FP32 -- Bits, Blocks, and Trade-offs

📄 Technical Guide ⏱️ 30 min read 📂 Core AI Concepts

Quantization is the process of reducing the precision of neural network weights from high-precision floating-point (FP32/FP16) to lower-bit integer representations (INT8, INT4, etc.). This reduces model file size, VRAM requirements, and inference latency -- but at the cost of model quality. This guide covers every common quantization level, their real-world trade-offs, hardware requirements, and command-line tools for verification.

QuantizationGGUFGPTQAWQEXL2INT4INT8FP16VRAM

🎯 Bottom Line: For most users running local LLMs, Q4_K_M is the sweet spot -- it preserves 95-98% of model quality while cutting VRAM requirements by 60-75% compared to FP16. Q8 is excellent when you have the VRAM and want near-lossless quality. Q2 and Q3 are emergency modes for running models on hardware that would otherwise be incapable. Never use Q2 for production workloads unless you have no other option.

Key Concepts

What Is a Quant?

A "quant" refers to the number of bits used to represent each weight in a neural network. The original training is done in FP32 (32-bit floating point) or BF16 (16-bit bfloat). Quantization compresses these weights to fewer bits:

FP32 -- 32 bits per weight (4 bytes). The training default. No quantization.
FP16 / BF16 -- 16 bits per weight (2 bytes). Half precision. Standard for inference on GPU.
INT8 / Q8 -- 8 bits per weight (1 byte). 50% reduction from FP16. Near-lossless.
INT5 / Q5 -- 5 bits per weight. Good quality retention, moderate compression.
INT4 / Q4 -- 4 bits per weight (0.5 bytes). 75% reduction from FP16. The most popular quantization level.
INT3 / Q3 -- 3 bits per weight. Noticeable quality loss. Use only when VRAM-constrained.
INT2 / Q2 -- 2 bits per weight. Severe quality degradation. Last resort.
INT1 / Q1 -- 1 bit per weight. Experimental. Binary weights. Not practical for LLMs.

What Does the "K" Mean?

The K-suffix (Q4_K_S, Q4_K_M, Q5_K_M, etc.) refers to the k-quants method developed by Kawrakow for llama.cpp/GGUF. It uses a block-based quantization approach where weights are grouped into blocks and each block gets its own scaling factor:

_S (Small): Smaller blocks, slightly less accurate, smaller file size.
_M (Medium): Medium block size. The recommended default for most use cases.
_L (Large): Larger blocks with more accurate scaling. Best quality within that bit level, but larger file size.

Block Quantization Explained: Instead of mapping all weights with a single scale, k-quants divide weights into blocks (typically 32 or 64 weights per block). Each block gets its own scale factor and (for _M and _L variants) a per-block min/max. This preserves more information than naive linear quantization because different layers and weight distributions get appropriate scaling. This is why Q4_K_M outperforms older Q4_0/Q4_1 formats.

Files and Formats

Quantized models are distributed in several container formats. The format determines which inference engine you use:

GGUF (GPT-Generated Unified Format) -- The modern standard for llama.cpp, Ollama, LM Studio, and most CPU/GPU hybrid inference. Replaced the older GGML format. Supports all k-quant levels.
GPTQ -- GPU-optimized quantization format used with AutoGPTQ, vLLM, and Text Generation WebUI. Primarily INT4. Requires a GPU.
AWQ (Activation-aware Weight Quantization) -- GPU-optimized, similar to GPTQ but uses activation statistics for better weight selection. INT4 primarily. Used with vLLM and HuggingFace transformers.
EXL2 (ExLlamaV2) -- Variable-bit quantization. Allows mixing different bit rates within the same model (e.g., some layers at 4-bit, others at 8-bit). GPU-only. Excellent quality-to-size ratio.
FP16/BF16 safetensors -- Unquantized (or "full precision") HuggingFace format. Maximum quality, maximum VRAM.

Terminology You Will See

BPW (Bits Per Weight): The effective average bits per weight including metadata. For example, Q4_K_M might have a BPW of 4.8 because block scales add overhead.
Perplexity: A measure of how well a model predicts text. Lower is better. Quantization increases perplexity slightly. Used to compare quant quality.
Calibration data: A dataset used during quantization to determine optimal scaling factors. GPTQ and AWQ require calibration; GGUF k-quants do not (they use the weight statistics directly).
Group size: The number of weights sharing a single scale factor. Common values: 32, 64, 128. Smaller groups = better quality, larger file size.
Imatrix (Importance Matrix): An optional enhancement for GGUF quantization that uses calibration data to weight which weights matter most. Improves quality at low bit levels (Q2-Q4).

Quantization Formats Compared

Each format has different tooling, hardware requirements, and quality characteristics:

Format	Engine	Hardware	Bit Levels	Best For
GGUF	llama.cpp, Ollama, LM Studio, KoboldCpp	CPU, GPU (CUDA/Metal/Vulkan)	Q2_K through Q8_0, FP16	General-purpose. CPU inference. Mixed CPU+GPU. Most flexible.
GPTQ	AutoGPTQ, vLLM, Text Gen WebUI	GPU only (CUDA)	INT4 (primary), INT3, INT8	Fast GPU inference. Server deployments with vLLM.
AWQ	vLLM, HuggingFace transformers	GPU only (CUDA)	INT4 (primary)	Best INT4 quality on GPU. Production serving with vLLM.
EXL2	ExLlamaV2, TabbyAPI	GPU only (CUDA)	Variable (2.5-8.0 bpw)	Maximum quality at target file size. Fine-grained control.
FP16/BF16	All engines	GPU (recommended)	16-bit (no quantization)	Maximum quality. When VRAM is not a constraint.
FP32	PyTorch, HuggingFace	GPU or CPU	32-bit (training precision)	Training, fine-tuning, research. Never needed for inference.

GGUF vs GPTQ vs AWQ -- Which to Choose? If you are running locally with Ollama or llama.cpp, use GGUF -- it is the only format supported. If you are deploying on a GPU server with vLLM for maximum throughput, use AWQ or GPTQ (AWQ generally has slightly better quality). If you want the absolute best quality at a specific file size on GPU, EXL2 lets you dial in an exact bpw target.

Quantization Levels Explained

Here is every common quantization level with real-world characteristics. File sizes and VRAM are approximate for a 7B parameter model (FP16 baseline ~14GB):

Level	Bits/Weight	~Size (7B)	~VRAM (7B)	Quality	Use When
FP32	32.0	~28 GB	~30 GB	100% (baseline)	Never for inference. Training only.
FP16 / BF16	16.0	~14 GB	~15 GB	100% (no loss)	You have ample VRAM. Maximum quality inference.
Q8_0	8.5	~7.5 GB	~8.5 GB	~99% (near-lossless)	You have enough VRAM. Want near-perfect quality with 50% reduction.
Q6_K	6.6	~5.8 GB	~7 GB	~98%	Quality priority with some VRAM savings over Q8.
Q5_K_M	5.7	~5.0 GB	~6 GB	~97%	Good balance. Slightly better than Q4, moderate VRAM.
Q4_K_M	4.8	~4.2 GB	~5.5 GB	~96% (RECOMMENDED)	The sweet spot. Best quality-to-size ratio for most users.
Q4_K_S	4.5	~3.9 GB	~5 GB	~95%	Slightly smaller than Q4_K_M. Marginal quality loss.
Q4_0	4.5	~3.9 GB	~5 GB	~94%	Legacy. Use Q4_K_M instead unless you need max speed.
Q3_K_M	3.9	~3.4 GB	~4.5 GB	~92%	VRAM-constrained. Noticeable quality drop in complex reasoning.
Q3_K_S	3.4	~3.0 GB	~4 GB	~90%	Tight VRAM. Expect degraded output quality.
Q2_K	2.6	~2.3 GB	~3.5 GB	~85% (significant loss)	Emergency only. Model will produce lower-quality, more repetitive text.
IQ2_XXS	2.1	~1.8 GB	~3 GB	~80% (severe loss)	Extreme edge cases. Requires imatrix. Quality is notably degraded.
IQ1_S	1.6	~1.4 GB	~2.5 GB	~70% (experimental)	Experimental. Barely usable. Research purposes only.

About "I" Quants (I-Quants / Importance-Matrix Quants): IQ2, IQ3, IQ4 variants use an importance matrix (imatrix) during quantization. The imatrix is computed by running calibration data through the model to determine which weights are most critical. This allows lower-bit quantization to preserve more quality than standard k-quants at the same bit level. IQ3_XXS can outperform Q3_K_S despite having fewer effective bits. The trade-off: you need a good imatrix file, and imatrix computation takes time.

Quality -- What Do You Actually Lose?

Quality loss from quantization is not uniform across all tasks. Here is what degrades first:

Q8 and above: Effectively indistinguishable from FP16. No perceptible quality loss for any task.
Q5-Q6: Near-lossless. Minor degradation only visible in very long context generation or rare edge cases.
Q4: Small quality drop. Most users will not notice in casual conversation. May show slight degradation in: complex multi-step reasoning, code generation edge cases, mathematical precision, and subtle nuance in creative writing.
Q3: Noticeable degradation. More repetitive outputs. Reduced instruction following accuracy. Code quality drops measurably. Hallucination rate increases.
Q2: Significant quality loss. Model becomes noticeably less coherent on complex tasks. Increased repetition. Reduced factual accuracy. Fine-tuned behaviors (chat formatting, safety) may degrade. Usable for simple Q&A but unreliable for complex tasks.
Q1 / IQ1: Severe degradation. Model can barely follow instructions. Output is often garbled or repetitive. Research novelty only.

Hardware Trade-offs: GPU, RAM, and CPU

VRAM Calculation Formula

To estimate VRAM needed for a model:

# VRAM formula (approximate):
VRAM_GB = (Parameters_B x Bits_Per_Weight / 8) + Context_Overhead

# Example: 7B model at Q4_K_M (4.8 bpw) with 4K context:
VRAM = (7 x 4.8 / 8) + 1.0 = 4.2 + 1.0 = ~5.2 GB

# Example: 70B model at Q4_K_M with 4K context:
VRAM = (70 x 4.8 / 8) + 2.0 = 42 + 2.0 = ~44 GB

# Context overhead grows with context length:
#   2K context:  ~0.5-1.0 GB
#   4K context:  ~1.0-2.0 GB
#   8K context:  ~2.0-4.0 GB
#   16K context: ~4.0-8.0 GB
#   32K context: ~8.0-16.0 GB
                        

Real-World VRAM Requirements by Model Size

Minimum VRAM to load the model with a 4K context window (using GGUF with GPU offload):

Model Size	Q2_K	Q3_K_M	Q4_K_M	Q5_K_M	Q6_K	Q8_0	FP16
1.5B	~1.3 GB	~1.6 GB	~1.8 GB	~2.0 GB	~2.3 GB	~2.7 GB	~4.5 GB
3B	~2.3 GB	~2.8 GB	~3.2 GB	~3.6 GB	~4.1 GB	~4.8 GB	~7.5 GB
7B	~3.5 GB	~4.5 GB	~5.5 GB	~6.0 GB	~7.0 GB	~8.5 GB	~15 GB
8B	~4.0 GB	~5.0 GB	~6.0 GB	~6.8 GB	~7.8 GB	~9.5 GB	~17 GB
13B	~6.0 GB	~7.5 GB	~9.0 GB	~10.5 GB	~12.0 GB	~14.5 GB	~27 GB
14B	~6.5 GB	~8.0 GB	~9.5 GB	~11.0 GB	~12.5 GB	~15.5 GB	~29 GB
32B	~12.5 GB	~15.5 GB	~19 GB	~22 GB	~25 GB	~31 GB	~63 GB
70B	~25 GB	~31 GB	~38 GB	~44 GB	~50 GB	~62 GB	~140 GB
120B	~42 GB	~52 GB	~63 GB	~73 GB	~83 GB	~103 GB	~240 GB

VRAM vs RAM -- What Happens When You Exceed VRAM? With GGUF/llama.cpp, if a model does not fit entirely in GPU VRAM, layers are offloaded to CPU RAM. This is called "partial GPU offload." The model still works, but inference speed drops dramatically for the CPU-handled layers. A 70B model at Q4 on a 24GB GPU might put 60% of layers on GPU and 40% on CPU -- it runs, but at 5-15 tokens/sec instead of 40-80 tokens/sec. With Ollama, this happens automatically. With llama.cpp, use the -ngl (number of GPU layers) flag to control offloading.

CPU-Only Inference

When no GPU is available, GGUF models run entirely on CPU. Performance depends heavily on:

RAM speed: DDR4-3200 vs DDR5-5600 can mean 2x difference. Memory bandwidth is the bottleneck, not CPU cores.
RAM capacity: You need RAM = model size + 2GB overhead. A Q4 7B model needs ~6GB RAM.
CPU cores: llama.cpp uses all available cores. More threads = faster, up to the point where memory bandwidth saturates (typically 6-8 threads for DDR4, 8-12 for DDR5).
AVX2/AVX-512: Modern CPUs with AVX-512 get a significant speedup. Check with lscpu | grep avx.

Typical CPU-only speeds (7B Q4_K_M):

Modern CPU + DDR5: 15-25 tokens/sec
Modern CPU + DDR4: 8-15 tokens/sec
Older CPU (pre-AVX2): 3-8 tokens/sec

Inference Speed by Quantization

Lower bit quantization is not just smaller -- it is also faster. Less data to move from memory to compute:

Quant	Relative Speed (GPU)	Relative Speed (CPU)	Notes
FP16	1.0x (baseline)	1.0x (baseline)	Full precision. Slowest, largest.
Q8_0	~1.3x	~1.5x	Good speedup with minimal quality loss.
Q6_K	~1.5x	~1.7x	Excellent balance.
Q5_K_M	~1.7x	~2.0x	Strong speedup, good quality.
Q4_K_M	~2.0x	~2.3x	Best speed/quality ratio.
Q3_K_M	~2.2x	~2.5x	Faster but quality degrades.
Q2_K	~2.5x	~2.8x	Fastest but significant quality loss.

Choosing the Right Quantization

Decision Framework

Step 1: Check your VRAM. Look at the VRAM table above. Find the largest model you can fit at Q4_K_M. This is your starting point.

Step 2: Consider your use case.

Use Case	Recommended Quant	Why
Casual chat / general Q&A	Q4_K_M	Quality is more than good enough. Fast and small.
Code generation	Q5_K_M or Q6_K	Code requires precision. Q4 can introduce subtle syntax errors on edge cases.
Mathematical reasoning	Q6_K or Q8_0	Math degrades fastest under quantization. Higher precision needed.
Creative writing / storytelling	Q4_K_M or Q5_K_M	Creative tasks are more forgiving of quantization. Quality difference is subtle.
Production / enterprise deployment	Q8_0 or FP16	When quality is critical and you cannot afford any degradation. Use AWQ with vLLM for GPU.
RAG / document analysis	Q4_K_M	Retrieval quality matters more than generation precision. Q4 is sufficient.
Running on a server with 80GB+ GPU	FP16 or BF16	No reason to quantize if you have the VRAM. Maximum quality.
Running on 8GB GPU	Q4_K_M (7B model)	Fits comfortably. Good speed and quality.
Running on 4GB GPU	Q3_K_M (7B) or Q4_K_M (3B)	Tight VRAM. Choose smaller model at Q4 over larger at Q3 when possible.
CPU-only (16GB RAM)	Q4_K_M (7B)	Fits in RAM. CPU inference at 10-20 tok/sec.
CPU-only (8GB RAM)	Q3_K_S (7B) or Q4_K_M (3B)	Limited RAM. Prefer smaller model at higher quant.
Edge device / Raspberry Pi	Q2_K (1.5B or 3B model)	Extreme constraints. Use smallest model at lowest viable quant.

🎯 The Golden Rule: Always prefer a smaller model at Q4_K_M over a larger model at Q2_K. A 7B model at Q4 will almost always outperform a 13B model at Q2 in real-world quality. The larger model's capacity advantage is destroyed by extreme quantization. When VRAM is tight, reduce model size first, then reduce quantization level.

Common Mistakes

Using Q2 because "it fits." A Q2 70B model is almost always worse than a Q4 32B model that fits the same VRAM. Downsize the model, not the quantization.
Using FP16 when Q8 would do. Q8 is near-lossless. If you are VRAM-constrained, drop to Q8_0 first. The quality difference is negligible.
Mixing format and engine. GPTQ models do not work in Ollama. GGUF models do not work in vLLM. Match the format to your engine.
Ignoring context overhead. A model that "just fits" in VRAM will OOM when you start generating with a long context. Always leave 1-2GB headroom for context.
Assuming higher bit is always better. Q4_K_M with an imatrix can outperform Q5_K_S without one. The quantization method matters, not just the bit count.

Command-Line Verification

How to check what quantization a model is actually using. This is critical when downloading models -- filenames can be misleading, and you need to verify before running.

Method 1: Inspect GGUF Metadata (llama.cpp)

# Using llama.cpp's main binary to inspect a GGUF file:
$ ./llama-server --model model.gguf --n-gpu-layers 0 --port 0 2>&1 | head -20

# Output will show:
#   "model type" = "7B"
#   "model size" = "7.17 GiB"
#   "model type" and general architecture

# More detailed metadata inspection using Python:
$ python3 -c "
from llama_cpp import Llama
llm = Llama(model_path='model.gguf', n_ctx=1, verbose=True)
print(llm.model_params)
"
                        

Method 2: Read GGUF File Header Directly

# The GGUF format stores metadata as key-value pairs in the header.
# You can read them with this Python snippet (no dependencies needed):

$ python3 <<'EOF'
import struct

def read_gguf_metadata(filepath):
    with open(filepath, 'rb') as f:
        magic = f.read(4)
        if magic != b'GGUF':
            print("Not a GGUF file")
            return

        version = struct.unpack('# Key fields to look for in output:
#   general.name          = "model-name-Q4_K_M"
#   general.architecture  = "llama"
#   general.file_type     = 15    (15 = Q4_K_M, see enum below)
#   llama.context_length  = 4096
#   llama.embedding_length = 4096
#   llama.block_count     = 32
                        

GGUF File Type Enum Reference

The general.file_type field in GGUF metadata maps to these values:

Value	Quantization Type	Bits Per Weight
0	FP32	32.0
1	FP16	16.0
2	Q4_0	4.5
3	Q4_1	5.0
6	Q5_0	5.5
7	Q5_1	6.0
8	Q8_0	8.5
9	Q8_1	9.0
10	Q2_K	2.6
11	Q3_K_S	3.4
12	Q3_K_M	3.9
13	Q3_K_L	4.3
14	Q4_K_S	4.5
15	Q4_K_M	4.8
16	Q5_K_S	5.5
17	Q5_K_M	5.7
18	Q6_K	6.6

Method 3: Check with Ollama

# List installed models and their sizes:
$ ollama list

NAME                    ID            SIZE      MODIFIED
llama3:8b              365c0bf3be01  4.7 GB    2 weeks ago
llama3:8b-q8_0         a3f5c65f1234  7.5 GB    2 weeks ago
qwen2.5:7b             2b7e4f1a9c33  4.5 GB    1 week ago

# Show detailed model info including quantization:
$ ollama show llama3:8b

Model
  architecture        llama
  parameters          8.0B
  context length      8192
  embedding length    4096
  quantization        Q4_K_M

# Show the modelfile (includes quantization in parameters):
$ ollama show llama3:8b --modelfile

# Modelfile
FROM /usr/share/ollama/.ollama/models/...
PARAMETER num_ctx 8192
TEMPLATE {{ .System }}...
# The FROM line shows the actual GGUF path
# Check the GGUF file directly for full metadata

# Check running model status (shows what is loaded in VRAM):
$ ollama ps

NAME       ID           SIZE     VRAM     CPU
llama3:8b  365c0bf3be01 5.2 GB   100%     0%
# VRAM 100% = fully GPU offloaded
# VRAM 60% = partial offload, 40% on CPU
                        

Method 4: Check HuggingFace Model Config (GPTQ/AWQ)

# For GPTQ models, check the config.json:
$ python3 -c "
import json
with open('config.json') as f:
    cfg = json.load(f)
quant = cfg.get('quantization_config', {})
print(f'Method: {quant.get(\"quant_method\", \"none\")}')
print(f'Bits: {quant.get(\"bits\", \"unknown\")}')
print(f'Group size: {quant.get(\"group_size\", \"unknown\")}')
print(f'Damp percent: {quant.get(\"damp_percent\", \"unknown\")}')
"

# Typical output for a GPTQ model:
#   Method: gptq
#   Bits: 4
#   Group size: 128
#   Damp percent: 0.01

# For AWQ models:
#   Method: awq
#   Bits: 4
#   Group size: 128
#   Zero point: True

# Quick check from command line:
$ cat config.json | python3 -m json.tool | grep -A5 quantization
                        

Method 5: Verify File Size Against Expected Quant

# Quick sanity check -- compare file size to expected size for the quant level:
$ ls -lh model.gguf
-rw-r--r-- 1 steve steve 4.1G Jun 27 10:00 model.gguf

# For a 7B model:
#   Q2_K:    ~2.3 GB
#   Q3_K_M:  ~3.4 GB
#   Q4_K_M:  ~4.2 GB  <-- this file is 4.1GB, likely Q4_K_M
#   Q5_K_M:  ~5.0 GB
#   Q6_K:    ~5.8 GB
#   Q8_0:    ~7.5 GB
#   FP16:    ~14 GB

# More precise -- calculate BPW from file size:
$ python3 -c "
import os
size_bytes = os.path.getsize('model.gguf')
size_gb = size_bytes / (1024**3)
# Approximate params (adjust for your model):
params_b = 7.0
bpw = (size_gb * 8) / params_b
print(f'File size: {size_gb:.2f} GB')
print(f'Effective BPW: {bpw:.1f}')
if bpw > 15: q = 'FP16'
elif bpw > 8: q = 'Q8_0'
elif bpw > 6: q = 'Q6_K'
elif bpw > 5.3: q = 'Q5_K_M'
elif bpw > 4.3: q = 'Q4_K_M'
elif bpw > 3.3: q = 'Q3_K_M'
elif bpw > 2.3: q = 'Q2_K'
else: q = 'Unknown (very low bit)'
print(f'Likely quantization: {q}')
"
                        

Converting and Quantizing Models

Quantizing to GGUF with llama.cpp

# Step 1: Clone and build llama.cpp
$ git clone https://github.com/ggerganov/llama.cpp
$ cd llama.cpp
$ make GGML_CUDA=1  # Add CUDA for GPU support

# Step 2: Convert HuggingFace model to GGUF (FP16):
$ python3 convert_hf_to_gguf.py /path/to/hf-model/ --outfile model-fp16.gguf

# Step 3: Quantize to target level:
$ ./llama-quantize model-fp16.gguf model-q4_k_m.gguf Q4_K_M
$ ./llama-quantize model-fp16.gguf model-q8_0.gguf Q8_0
$ ./llama-quantize model-fp16.gguf model-q5_k_m.gguf Q5_K_M
$ ./llama-quantize model-fp16.gguf model-q2_k.gguf Q2_K

# Available quantization types:
#   Q2_K  Q3_K_S  Q3_K_M  Q3_K_L  Q4_K_S  Q4_K_M
#   Q5_K_S  Q5_K_M  Q6_K  Q8_0  IQ3_XXS  IQ3_S  IQ3_M
#   IQ4_NL  IQ4_XS  F16  F32

# Step 4 (Optional): Generate imatrix for better low-bit quants:
$ ./llama-imatrix -m model-fp16.gguf -f training-data.txt -o model.imatrix --chunks 200

# Step 5: Quantize WITH imatrix (for IQ quants):
$ ./llama-quantize model-fp16.gguf model-iq3_XXS.gguf IQ3_XXS --imatrix model.imatrix
                        

Quantizing to GPTQ with AutoGPTQ

# Install AutoGPTQ:
$ pip install auto-gptq optimum

# Quantize a HuggingFace model to INT4 GPTQ:
$ python3 <<'EOF'
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer

model_id = "meta-llama/Llama-3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Calibration data (use ~128-1024 samples for best results)
calibration = ["Sample text 1...", "Sample text 2...", ...]

quant_config = BaseQuantizeConfig(
    bits=4,              # 3, 4, or 8
    group_size=128,      # 32, 64, 128 (smaller = better quality, larger file)
    desc_act=False       # True = slightly better quality, slower quantization
)

model = AutoGPTQForCausalLM.from_pretrained(model_id, quant_config)
model.quantize(calibration)
model.save_quantized("./Llama-3-8B-GPTQ-INT4", use_safetensors=True)
tokenizer.save_pretrained("./Llama-3-8B-GPTQ-INT4")
EOF
                        

Quantizing to AWQ

# Install AWQ:
$ pip install autoawq

# Quantize to INT4 AWQ:
$ python3 <<'EOF'
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_id = "meta-llama/Llama-3-8B"
quant_path = "Llama-3-8B-AWQ-INT4"

quant_config = {
    "zero_point": True,     # Better quality with zero points
    "q_group_size": 128,    # 64, 128, or 256
    "w_bit": 4,             # 4 is standard; 8 also supported
    "version": "GEMM"       # GEMM for GPU, GEMV for large models
}

model = AutoAWQForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Calibration data
calibration = ["Sample text 1...", "Sample text 2...", ...]

model.quantize(tokenizer, quant_config=quant_config,
               calib_data=calibration)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
EOF
                        

Quantizing to EXL2 (Variable Bit)

# Install ExLlamaV2:
$ pip install exllamav2

# Quantize to EXL2 with target average BPW:
$ python3 -m exllamav2.generator convert \
    -i /path/to/hf-model/ \
    -o /path/to/exl2-output/ \
    -c /path/to/calibration-data/ \
    -b 4.0              # Target bits per weight (2.5-8.0)

# Common EXL2 targets:
#   -b 8.0  = near FP16 quality, large file
#   -b 6.0  = excellent quality, moderate size
#   -b 5.0  = good balance
#   -b 4.0  = equivalent to Q4 quality, but often better
#   -b 3.0  = aggressive, for tight VRAM
#   -b 2.5  = extreme, lowest practical
                        

Importing into Ollama

# Create a Modelfile for a custom GGUF:
$ cat > Modelfile <<'EOF'
FROM ./model-q4_k_m.gguf

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096

TEMPLATE """{{ .System }}
USER: {{ .Prompt }}
ASSISTANT:"""
EOF

# Import into Ollama:
$ ollama create mymodel-q4 -f Modelfile

# Run it:
$ ollama run mymodel-q4
                        

References & Citation

Source: Avondale.AI Technical Documentation Library
Category: Core AI Concepts
Type: Technical Reference Guide
License: CC BY 4.0

Model Quantization: The Complete Technical Guide

Model Quantization: The Complete Technical Guide

Key Concepts

What Is a Quant?

What Does the "K" Mean?

Files and Formats

Terminology You Will See

Quantization Formats Compared

Quantization Levels Explained

Quality -- What Do You Actually Lose?

Hardware Trade-offs: GPU, RAM, and CPU

VRAM Calculation Formula

Real-World VRAM Requirements by Model Size

CPU-Only Inference

Inference Speed by Quantization

Choosing the Right Quantization

Decision Framework

Common Mistakes

Command-Line Verification

Method 1: Inspect GGUF Metadata (llama.cpp)

Method 2: Read GGUF File Header Directly

GGUF File Type Enum Reference

Method 3: Check with Ollama

Method 4: Check HuggingFace Model Config (GPTQ/AWQ)

Method 5: Verify File Size Against Expected Quant

Converting and Quantizing Models

Quantizing to GGUF with llama.cpp

Quantizing to GPTQ with AutoGPTQ

Quantizing to AWQ

Quantizing to EXL2 (Variable Bit)

Importing into Ollama

References & Citation

Further Reading