141 lines
3.7 KiB
Markdown
141 lines
3.7 KiB
Markdown
# quantize_model
|
|
|
|
CLI tool for offline model quantization. Converts safetensors models to INT8, INT4, or ternary format.
|
|
|
|
**Source:** `examples/quantize_model.c`
|
|
**Build:** `make quantize-model`
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
Loads a safetensors model, quantizes all weight tensors to the target dtype, and saves a new safetensors file. Automatically skips norm layers, biases, embeddings, and small tensors. Reports compression ratio when done.
|
|
|
|
---
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
./examples/quantize_model \
|
|
--input model.safetensors \
|
|
--output model-quantized.safetensors \
|
|
--dtype int8|int4|ternary \
|
|
[--granularity per_tensor|per_channel|per_group] \
|
|
[--group-size 128] \
|
|
[--threshold 0.0]
|
|
```
|
|
|
|
---
|
|
|
|
## CLI Options
|
|
|
|
| Flag | Default | Description |
|
|
|------|---------|-------------|
|
|
| `--input PATH` | (required) | Input safetensors model |
|
|
| `--output PATH` | (required) | Output quantized safetensors |
|
|
| `--dtype TYPE` | (required) | `int8`, `int4`, or `ternary` |
|
|
| `--granularity MODE` | `per_tensor` | `per_tensor`, `per_channel`, or `per_group` |
|
|
| `--group-size N` | `128` | Group size for per-group quantization |
|
|
| `--threshold F` | `0.0` | Ternary threshold (0 = auto absmean) |
|
|
|
|
---
|
|
|
|
## Examples
|
|
|
|
### INT8 quantization (4x compression)
|
|
|
|
```bash
|
|
make USE_BLAS=1 quantize-model
|
|
|
|
./examples/quantize_model \
|
|
--input ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
|
|
--output ../models/base/RWKV-X-0.2B-q8.safetensors \
|
|
--dtype int8
|
|
```
|
|
|
|
### INT4 per-group quantization (8x compression)
|
|
|
|
```bash
|
|
./examples/quantize_model \
|
|
--input ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
|
|
--output ../models/base/RWKV-X-0.2B-q4.safetensors \
|
|
--dtype int4 \
|
|
--granularity per_group \
|
|
--group-size 64
|
|
```
|
|
|
|
### Ternary quantization (16x compression)
|
|
|
|
```bash
|
|
./examples/quantize_model \
|
|
--input ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
|
|
--output ../models/base/RWKV-X-0.2B-ternary.safetensors \
|
|
--dtype ternary
|
|
```
|
|
|
|
### Ternary with manual threshold
|
|
|
|
```bash
|
|
./examples/quantize_model \
|
|
--input ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
|
|
--output ../models/base/RWKV-X-0.2B-ternary-t05.safetensors \
|
|
--dtype ternary \
|
|
--threshold 0.05
|
|
```
|
|
|
|
---
|
|
|
|
## Output
|
|
|
|
```
|
|
Quantizing model:
|
|
Input: ../models/base/RWKV-X-0.2B-64k-Base.safetensors (836.8 MB)
|
|
Output: /tmp/rwkvx-ternary.safetensors
|
|
Dtype: TERNARY
|
|
Granularity: per_tensor
|
|
|
|
[========================================] 100% (443/443) rwkv.ln_out.weight
|
|
|
|
Input: 836.8 MB
|
|
Output: 53.1 MB
|
|
Ratio: 15.8x compression
|
|
|
|
Done.
|
|
```
|
|
|
|
---
|
|
|
|
## Skip Logic
|
|
|
|
The following tensors are passed through without quantization (preserving original F32):
|
|
|
|
| Pattern | Example | Why |
|
|
|---------|---------|-----|
|
|
| Norm layers | `ln_0.weight`, `rmsnorm.weight` | Norms are sensitive to precision |
|
|
| Biases | `layer.0.bias` | Small tensors, negligible savings |
|
|
| Embeddings | `embed.weight`, `wte.weight` | Critical for token representation |
|
|
| 1D tensors | Any 1D param | Typically norms/biases |
|
|
| Small tensors | < 1024 elements | Not worth the overhead |
|
|
| Non-float | I8, I32, etc. | Already quantized or integer params |
|
|
|
|
All skip behavior is configurable via the C export API (`fq_export_config`).
|
|
|
|
---
|
|
|
|
## Output Format
|
|
|
|
The quantized safetensors file contains:
|
|
|
|
- **Quantized weight tensors** stored as `I8` (INT8) or `U8` (INT4/ternary packed)
|
|
- **Scale tensors** (`name.__scale`) as `F32` for reconstruction
|
|
- **Shape tensors** (`name.__shape`) as `I64` storing original dimensions
|
|
- **Passthrough tensors** in original dtype (norms, biases, embeddings)
|
|
- **Metadata** with quantization parameters (`quantization.dtype`, `quantization.granularity`, etc.)
|
|
|
|
---
|
|
|
|
## See Also
|
|
|
|
- [quantization](quantization.md) - Quantization module API and ternary format details
|
|
- [safetensors](safetensors.md) - Safetensors file format
|