David
/
Foundry
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Foundry/docs/quantize-model.md

3.7 KiB
Raw Blame History

quantize_model

CLI tool for offline model quantization. Converts safetensors models to INT8, INT4, or ternary format.

Source: examples/quantize_model.c Build: make quantize-model

Overview

Loads a safetensors model, quantizes all weight tensors to the target dtype, and saves a new safetensors file. Automatically skips norm layers, biases, embeddings, and small tensors. Reports compression ratio when done.

Usage

./examples/quantize_model \
  --input model.safetensors \
  --output model-quantized.safetensors \
  --dtype int8|int4|ternary \
  [--granularity per_tensor|per_channel|per_group] \
  [--group-size 128] \
  [--threshold 0.0]

CLI Options

Flag Default Description
--input PATH (required) Input safetensors model
--output PATH (required) Output quantized safetensors
--dtype TYPE (required) int8, int4, or ternary
--granularity MODE per_tensor per_tensor, per_channel, or per_group
--group-size N 128 Group size for per-group quantization
--threshold F 0.0 Ternary threshold (0 = auto absmean)

Examples

INT8 quantization (4x compression)

make USE_BLAS=1 quantize-model

./examples/quantize_model \
  --input ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
  --output ../models/base/RWKV-X-0.2B-q8.safetensors \
  --dtype int8

INT4 per-group quantization (8x compression)

./examples/quantize_model \
  --input ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
  --output ../models/base/RWKV-X-0.2B-q4.safetensors \
  --dtype int4 \
  --granularity per_group \
  --group-size 64

Ternary quantization (16x compression)

./examples/quantize_model \
  --input ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
  --output ../models/base/RWKV-X-0.2B-ternary.safetensors \
  --dtype ternary

Ternary with manual threshold

./examples/quantize_model \
  --input ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
  --output ../models/base/RWKV-X-0.2B-ternary-t05.safetensors \
  --dtype ternary \
  --threshold 0.05

Output

Quantizing model:
  Input:       ../models/base/RWKV-X-0.2B-64k-Base.safetensors (836.8 MB)
  Output:      /tmp/rwkvx-ternary.safetensors
  Dtype:       TERNARY
  Granularity: per_tensor

  [========================================] 100% (443/443) rwkv.ln_out.weight

  Input:  836.8 MB
  Output: 53.1 MB
  Ratio:  15.8x compression

Done.

Skip Logic

The following tensors are passed through without quantization (preserving original F32):

Pattern Example Why
Norm layers ln_0.weight, rmsnorm.weight Norms are sensitive to precision
Biases layer.0.bias Small tensors, negligible savings
Embeddings embed.weight, wte.weight Critical for token representation
1D tensors Any 1D param Typically norms/biases
Small tensors < 1024 elements Not worth the overhead
Non-float I8, I32, etc. Already quantized or integer params

All skip behavior is configurable via the C export API (fq_export_config).

Output Format

The quantized safetensors file contains:

  • Quantized weight tensors stored as I8 (INT8) or U8 (INT4/ternary packed)
  • Scale tensors (name.__scale) as F32 for reconstruction
  • Shape tensors (name.__shape) as I64 storing original dimensions
  • Passthrough tensors in original dtype (norms, biases, embeddings)
  • Metadata with quantization parameters (quantization.dtype, quantization.granularity, etc.)

See Also