Pantheon/docs/architecture/gguf.md

GGUF Format Support

Pantheon's GGUF parser (src/formats/gguf.c) provides zero-copy model loading via mmap. Both GGUF v2 and v3 are supported.

File Layout

┌──────────────────────────┐
│  Magic: "GGUF" (4 bytes) │
│  Version: uint32         │  v2 or v3
│  Tensor count: uint64    │
│  Metadata count: uint64  │
├──────────────────────────┤
│  Metadata entries         │  typed key-value pairs
│  (variable length)        │
├──────────────────────────┤
│  Tensor info entries      │  name, dims, quant type, offset
│  (variable length)        │
├──────────────────────────┤
│  Padding                  │  aligned to 32 bytes (GGUF_DEFAULT_ALIGNMENT)
├──────────────────────────┤
│  Tensor data (binary)     │  mmap'd, accessed via offset
└──────────────────────────┘

Alignment

The tensor data section starts at a 32-byte aligned offset from the file start. This is the GGUF spec default (GGUF_DEFAULT_ALIGNMENT = 32). Files can override this via the general.alignment metadata key.

v2 vs v3

Both versions use the same binary layout. The parser accepts either. v3 added big-endian support (Pantheon assumes little-endian). The version field is stored in pn_gguf_file.version.

Metadata Types

13 value types supported:

Type	ID	Size
UINT8	0	1
INT8	1	1
UINT16	2	2
INT16	3	2
UINT32	4	4
INT32	5	4
FLOAT32	6	4
BOOL	7	1
STRING	8	8 + len
ARRAY	9	variable
UINT64	10	8
INT64	11	8
FLOAT64	12	8

Strings are length-prefixed (uint64 length + raw bytes). Arrays contain a type tag + uint64 count + elements.

Quantization Types

Implemented (full dequant + fused matvec)

Type	ID	Bits/Weight	Block Size	Bytes/Block	Notes
F32	0	32	1	4	Passthrough
F16	1	16	1	2	IEEE 754 half
Q4_0	2	4	32	18	scale + 4-bit nibbles
Q8_0	8	8	32	34	scale + int8 values
Q4_K	12	4.5	256	144	8 sub-blocks, 6-bit scales
Q5_K	13	5.5	256	176	5-bit values (4 low + 1 high)
Q6_K	14	6.5	256	210	6-bit values (4 low + 2 high)

Format-only (defined but no dequant kernel)

Q4_1, Q5_0, Q5_1, Q8_1, Q2_K, Q3_K, Q8_K

Mixed Quantization (Q4_K_M, Q5_K_M)

GGUF files labeled "Q4_K_M" use Q4_K for most layers with Q6_K for attention output and FFN down projections. This is handled transparently — each tensor has its own quant type, and pn_qmv dispatches per-row based on the weight's type field.

Key Metadata Keys

Model Architecture

Key	Type	Description
general.architecture	string	"llama", "qwen2", "phi", "mistral"
general.name	string	Human-readable model name
general.alignment	uint32	Data section alignment (default 32)

Model Dimensions

Prefixed by architecture name (llama.*, qwen2.*, etc.):

Key	Type	Description
{arch}.embedding_length	uint32	Hidden size (n_embd)
{arch}.attention.head_count	uint32	Query heads
{arch}.attention.head_count_kv	uint32	KV heads (for GQA)
{arch}.block_count	uint32	Number of layers
{arch}.feed_forward_length	uint32	FFN intermediate size
{arch}.context_length	uint32	Max context length
{arch}.rope.freq_base	float32	RoPE theta (default 10000)
{arch}.attention.layer_norm_rms_epsilon	float32	RMSNorm epsilon
{arch}.attention.sliding_window	uint32	Sliding window size (Mistral)

Tokenizer

Key	Type	Description
tokenizer.ggml.model	string	"gpt2" (BPE) or "llama" (SentencePiece)
tokenizer.ggml.tokens	string[]	Vocabulary tokens
tokenizer.ggml.merges	string[]	BPE merge rules (BPE only)
tokenizer.ggml.scores	float[]	Unigram scores (SentencePiece only)
tokenizer.ggml.bos_token_id	uint32	BOS token ID
tokenizer.ggml.eos_token_id	uint32	EOS token ID

Tensor Naming Convention

GGUF tensors follow the pattern blk.{layer}.{component}.weight:

token_embd.weight                    embedding table
blk.0.attn_norm.weight               pre-attention RMSNorm
blk.0.attn_q.weight                  query projection
blk.0.attn_k.weight                  key projection
blk.0.attn_v.weight                  value projection
blk.0.attn_output.weight             output projection
blk.0.attn_q.bias                    query bias (Qwen2 only)
blk.0.ffn_norm.weight                pre-FFN RMSNorm
blk.0.ffn_gate.weight                SwiGLU gate projection
blk.0.ffn_up.weight                  SwiGLU up projection
blk.0.ffn_down.weight                down projection
output_norm.weight                   final RMSNorm
output.weight                        LM head (may be tied to token_embd)

Quantized Weight Loading

Weights are loaded as pn_qweight structs that point directly into mmap'd data:

typedef struct {
    const void   *data;       /* raw quantized blocks in mmap */
    pn_gguf_quant type;       /* quant format */
    int           rows;       /* output dimension */
    int           cols;       /* input dimension */
    size_t        row_stride; /* bytes per row */
} pn_qweight;

GGUF stores dimensions innermost-first. The loader reverses them to C outermost-first: dims[n_dims-1] = rows, dims[0] = cols. row_stride is computed from cols and the quant format's bytes-per-block.

Matrix-vector multiply (pn_qmv) iterates rows, dequantizing one block (32 or 256 elements) at a time and accumulating the dot product. No full dequantization ever occurs for projection weights.