2.8 KiB

Raw Blame History

quantization/

C implementation of model quantization for efficient inference and training.

Python counterpart

bitsandbytes - https://github.com/TimDettmers/bitsandbytes

Why bitsandbytes?

bitsandbytes enables:

8-bit optimizers - Train with quantized optimizer states (saves memory)
8-bit inference - Run models with int8 weights
4-bit inference - QLoRA with 4-bit base model
NF4 format - Information-theoretically optimal 4-bit format

Alternatives (AutoGPTQ, AWQ) focus on specific quantization methods. bitsandbytes is most general for training.

Scope

This module implements:

INT8 quantization - Absmax and zero-point quantization
INT4 quantization - 4-bit weights (NF4, FP4)
Mixed precision - FP16 compute with quantized storage
Dequantization - Restore full precision for compute
Quantized matmul - Efficient int8/int4 matrix multiply

Getting Started

Dependencies

tensor/ module (tensor operations)
CUDA for GPU quantized ops

Key Files to Study

From bitsandbytes:

csrc/kernels.cu - CUDA kernels
bitsandbytes/functional.py - Quantization functions

From llama.cpp:

ggml-quants.c - Quantization formats (Q4_0, Q4_1, Q8_0, etc.)

Core API

// Quantize to INT8
fq_quantized* fq_quantize_int8(tm_tensor* t, fq_method method);
tm_tensor* fq_dequantize_int8(fq_quantized* q);

// Quantize to INT4 (NF4 format)
fq_quantized* fq_quantize_nf4(tm_tensor* t);
tm_tensor* fq_dequantize_nf4(fq_quantized* q);

// Quantized matrix multiply
tm_tensor* fq_matmul_int8(tm_tensor* a, fq_quantized* b);
tm_tensor* fq_matmul_nf4(tm_tensor* a, fq_quantized* b);

// Quantization methods
typedef enum {
    FQ_ABSMAX,      // Scale by max absolute value
    FQ_ZEROPOINT,   // Scale and shift
    FQ_NF4,         // Normal Float 4-bit
    FQ_FP4,         // Float Point 4-bit
} fq_method;

// Block-wise quantization (better accuracy)
fq_quantized* fq_quantize_blockwise(tm_tensor* t, int block_size, fq_method method);

Quantization Formats

INT8 (absmax):

scale = max(abs(tensor)) / 127
quantized = round(tensor / scale)
dequantized = quantized * scale

NF4 (4-bit normal float):

16 values optimally placed for normal distribution
Used in QLoRA

Block-wise:

Quantize in blocks of 64-256 elements
Each block has its own scale
Better accuracy than per-tensor

Implementation Order

INT8 absmax quantization
INT8 dequantization
Quantized matmul (CPU)
NF4 quantization
Block-wise quantization
CUDA kernels

2.8 KiB Raw Blame History