18 KiB

Raw Permalink Blame History

Supported Models

Models verified to load and generate correctly with Pantheon, with the config each one needs and a runnable command. This list grows as architecture support lands and new GGUFs are tested.

Hardware in the Tested column is what the row was actually measured on — M4 CPU is Apple Silicon, K80 GPU is a Tesla K80 (GK210, sm_37) via the CUDA backend, TT p150a is a Tenstorrent Blackhole p150a via the USE_TT=1 backend.

Tenstorrent: Ornith-1.5-9B is the only supported model. Other rows marked TT p150a ran on the card during development; they are not supported there. To run Ornith 1.5 on a card, see the README's "Ornith 1.5 on a Tenstorrent card".

Tesla K80: Qwen3.6-35B-A3B and Nex-N2-mini (Q3_K_M, one model per card) are the supported models, run with ./k80_qwen36.sh / ./k80_nexn2.sh (README "Qwen3.6 and Nex-N2-mini on a Tesla K80"). Other K80 GPU rows ran on the card; they are not supported there.

Llama Architecture

Model	Quant	Params	Layers	Heads (Q/KV)	Vocab	Context	Tested	Notes
SmolLM2-135M-Instruct	Q8_0	135M	30	9/3	49,152	8,192	M4 CPU	81.3 tok/s decode, 42ms load. Pre-tokenizer falls back with a warning — see Known Issues.
MiniCPM5-1B-SFT	Q8_0	1B	24	16/2	130,560	131,072	TT p150a	Ran on the card (not supported there) at 4.37 tok/s (see `../architecture/accel-foundry.md` §7 for its open correctness defect). Orchestrator. arch="llama" GGUF, head_dim=128 (decoupled: n_embd/n_head=96), rope_theta=5e6, rms_eps=1e-6. No new forward — config-only. ctx capped to 4096 for KV.

Source (SmolLM2): bartowski/SmolLM2-135M-Instruct-GGUF on HuggingFace Source (MiniCPM5-1B): openbmb/MiniCPM5-1B-GGUF (MiniCPM5-1B-Q8_0.gguf, ~1.15 GB, Apache-2.0). Loads into PN_ARCH_LLAMA — config-only, no new forward pass.

Best config: make USE_BLAS=1 (no threads needed, model is small)

huggingface-cli download bartowski/SmolLM2-135M-Instruct-GGUF SmolLM2-135M-Instruct-Q8_0.gguf --local-dir models/base/

./pantheon --model models/base/SmolLM2-135M-Instruct-Q8_0.gguf \
  --prompt "Hello, my name is" --max-tokens 32

Qwen2 Architecture

Model	Quant	Params	Layers	Heads (Q/KV)	Vocab	Context	Tested	Notes
Qwen2.5-0.5B-Instruct	Q8_0	494M	24	14/2	151,936	32,768	M4 CPU	Has attention biases, tied embeddings

Source: Qwen/Qwen2.5-0.5B-Instruct-GGUF on HuggingFace

Best config: make USE_BLAS=1 (threads may help at this size)

huggingface-cli download Qwen/Qwen2.5-0.5B-Instruct-GGUF qwen2.5-0.5b-instruct-q8_0.gguf --local-dir models/base/

./pantheon --model models/base/qwen2.5-0.5b-instruct-q8_0.gguf \
  --prompt "Hello, how are you?" --max-tokens 32

Mistral Architecture

Model	Quant	Params	Layers	Heads (Q/KV)	Vocab	Context	Sliding Window	GGUF Ver	Tested	Notes
Mistral-7B-Instruct-v0.3	Q4_K_M	7.2B	32	32/8	32,768	32,768	4,096	v3	M4 CPU	BPE tokenizer, coherent output
Mistral-7B-Instruct-v0.1	Q4_K_M	7.2B	32	32/8	32,000	32,768	—	v2	M4 CPU	SentencePiece tokenizer, GGUF v2

Source: bartowski/Mistral-7B-Instruct-v0.3-GGUF, TheBloke/Mistral-7B-Instruct-v0.1-GGUF on HuggingFace

Best config: make USE_BLAS=1 USE_THREADS=1 (7B model needs threads)

# v0.3 (BPE, GGUF v3)
huggingface-cli download bartowski/Mistral-7B-Instruct-v0.3-GGUF \
  Mistral-7B-Instruct-v0.3-Q4_K_M.gguf --local-dir models/base/

./pantheon --model models/base/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf \
  --prompt "Hello, how are you?" --max-tokens 32

# v0.1 (SentencePiece, GGUF v2)
huggingface-cli download TheBloke/Mistral-7B-Instruct-v0.1-GGUF \
  mistral-7b-instruct-v0.1.Q4_K_M.gguf --local-dir models/base/

./pantheon --model models/base/mistral-7b-instruct-v0.1.Q4_K_M.gguf \
  --prompt "Hello" --max-tokens 32

Phi-3 Architecture

Model	Quant	Params	Layers	Heads (Q/KV)	Vocab	Context	Tested	Notes
Phi-3-mini-4k-instruct	Q4_K_M	3.8B	32	32/32	32,064	4,096	M4 CPU	MHA, NeoX RoPE, fused QKV + gate/up

Source: microsoft/Phi-3-mini-4k-instruct-gguf on HuggingFace

Best config: make USE_BLAS=1 (threads help at this size)

huggingface-cli download microsoft/Phi-3-mini-4k-instruct-gguf \
  Phi-3-mini-4k-instruct-Q4_K_M.gguf --local-dir models/base/

./pantheon --model models/base/Phi-3-mini-4k-instruct-Q4_K_M.gguf \
  --prompt "The capital of France is" --max-tokens 32

Mixtral Architecture (MoE)

Model	Quant	Params	Layers	Heads (Q/KV)	Vocab	Experts (total/used)	Tested	Notes
Mixtral_7Bx2_MoE	Q4_K_M	~13B	32	32/8	32,000	2/2	M4 CPU	Merged expert tensors, SentencePiece tokenizer

Source: TheBloke/Mixtral_7Bx2_MoE-GGUF on HuggingFace

Best config: make USE_BLAS=1 USE_THREADS=1 (large model needs threads)

Architecture notes:

MoE extends Llama: attention path is identical, only FFN differs
Mixtral GGUFs use general.architecture = "llama", detected via llama.expert_count metadata
Per-layer FFN: router → softmax → top-k expert selection → SwiGLU per expert → weighted sum
Supports merged (ffn_gate_exps.weight 3D) and split (ffn_gate.E.weight per expert) tensor formats

huggingface-cli download TheBloke/Mixtral_7Bx2_MoE-GGUF \
  mixtral_7bx2_moe.Q4_K_M.gguf --local-dir models/base/

./pantheon --model models/base/mixtral_7bx2_moe.Q4_K_M.gguf \
  --prompt "Hello" --max-tokens 32

RWKV-X Architecture (Safetensors)

Model	Format	Params	Layers (RWKV/Attn)	Heads	Embd	Vocab	Tested	Notes
RWKV-X-0.2B-64k-Base	safetensors	0.2B	12/4	12	768	65,536	M4 CPU	FP32 and INT8 verified
RWKV-X-3.6B-64k-Base	safetensors	3.6B	32/8	40	2,560	65,536	M4 CPU	17.2 tok/s INT8, 2.6 tok/s FP32, 8.9 GB RSS

Source: howard-hou/RWKV-X on HuggingFace

Best config:

0.2B: make USE_BLAS=1 (no threads needed). Add --quantize for INT8 (~3x faster decode).
3.6B: make USE_BLAS=1 USE_THREADS=1. Add --quantize for INT8 (~6.4x faster decode).

Requires: RWKV World tokenizer vocab file (--vocab flag)

# Download models
huggingface-cli download howard-hou/RWKV-X \
  RWKV-X-0.2B-64k-Base.safetensors --local-dir models/base/

huggingface-cli download howard-hou/RWKV-X \
  RWKV-X-3.6B-64k-Base.safetensors --local-dir models/base/

# 0.2B FP32 inference
./pantheon --model models/base/RWKV-X-0.2B-64k-Base.safetensors \
  --vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
  --prompt "Hello world" --max-tokens 32

# 3.6B INT8 inference (recommended)
./pantheon --model models/base/RWKV-X-3.6B-64k-Base.safetensors \
  --vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
  --prompt "The capital of France is" --max-tokens 64 --quantize

Architecture notes:

Hybrid model: RWKV-7 recurrent blocks (WKV-7 kernel) + sparse attention blocks
0.2B: 12 RWKV + 4 attention blocks, interleaved [R,R,R,A,R,R,R,A,...]
3.6B: 32 RWKV + 8 attention blocks, same interleaving pattern
Dual state: fixed-size recurrent state per RWKV block + KV cache per attention block
Trie-based World tokenizer (65K vocab, UTF-8 byte-level)
Uses safetensors format (not GGUF)

RWKV-7 Architecture (Converted from .pth)

Model	Source Format	Params	Layers	Heads	Embd	Vocab	Tested	Notes
RWKV-x070-World-0.1B-v2.8	.pth → safetensors	0.1B	12	12	768	65,536	M4 CPU	BF16, 399 tensors, converted via `pantheon-convert`

Source: BlinkDL/rwkv-7-world on HuggingFace (original .pth format)

Conversion: Requires pantheon-convert to convert from PyTorch .pth to safetensors first. The .pth reader is a sandboxed pickle interpreter in pure C — no Python needed.

Best config: make USE_BLAS=1 (small model, no threads needed)

Requires: RWKV World tokenizer vocab file (--vocab flag)

# Download .pth model
huggingface-cli download BlinkDL/rwkv-7-world \
  RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth \
  --local-dir models/base/

# Convert to safetensors (no Python needed)
./pantheon-convert models/base/RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth \
  -o models/base/RWKV-7-0.1B.safetensors

# Run inference
./pantheon --model models/base/RWKV-7-0.1B.safetensors \
  --vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
  --prompt "Hello" --max-tokens 32

Architecture notes:

Pure RWKV-7 (no sparse attention blocks, unlike RWKV-X hybrid)
WKV-7 kernel with Generalized Delta Rule
ChannelMix FFN with squared ReLU
BFloat16 weights preserved through conversion (no precision loss)
Uses same inference engine as RWKV-X (shared pn_rwkvx_* code path)

qwen35moe Architecture (Hybrid Gated-Delta SSM + Shared-Expert MoE)

Model	Quant	Params	Layers (SSM/Attn)	Heads (Q/KV)	Head dim	Embd	Experts (total/used+shared)	Vocab	Tested	Notes
Nex-N2-mini	Q3_K_M	35B-A3B	30/10 (interval 4)	16/2	256	2,048	256/8 + 1	248,320	M4 CPU	Oracle-validated vs llama.cpp (layer-by-layer, `temp 0`). Reasoning model (`<think>`). load 376 ms, ~1.0 tok/s decode, 9.5 GB RSS — CPU reference path.
Nex-N2-mini	Q3_K_M	35B-A3B	30/10 (interval 4)	16/2	256	2,048	256/8 + 1	248,320	K80 GPU (2-die)	Supported on K80 (`./k80_nexn2.sh`). Serve, 2026-10-20: ~21 tok/s decode at 1k context, prefill 107-119 tok/s up to 8k, 74 at 14k, 53 at 30k. Earlier single-stream CLI: ~13.7 tok/s decode — Q3_K_M split 20/20 across the two dies of one K80 (small-BAR, both dies mapped at once). GPU output bit-matches the CPU forward (argmax + every per-token logit, ulp 0) with `QWEN35_FUSE=0x37 QWEN35_FLASH=0`, measured by the author; default flash prefill is argmax-exact only. ~68× the CPU path; ~15–20 with kernel fusion (Kepler bit-exact ceiling). See `../architecture/qwen35moe.md`.

| Qwen3.6-35B-A3B | Q3_K_M | 35B-A3B | 30/10 (interval 4) | 16/2 | 256 | 2,048 | 256/8 + 1 | 248,320 | K80 GPU (2-die) | Supported on K80 (./k80_qwen36.sh). Serve, 2026-10-20: ~21 tok/s decode at 1k context, prefill 116-121 tok/s up to 8k, 79 at 14k, 60 at 30k. |

Source: nex-agi/Nex-N2-mini (GGUF: bartowski/nex-agi_Nex-N2-mini-GGUF), Qwen/Qwen3.6-35B-A3B (GGUF: bartowski/Qwen_Qwen3.6-35B-A3B-GGUF)

Best config (M4, oracle): make USE_BLAS=1 (Accelerate) — the correctness reference, not a speed target.

Best config (Tesla K80, the speed path): make USE_CUDA=1; run the 2-die GPU decode with QWEN35_GPU_SPLIT_LAYER=20 QWEN35_GPU_DIE_A=0 QWEN35_GPU_DIE_B=1 ./pantheon --model <gguf> --prompt <text>. ~13.7 tok/s single-stream; bit-matching the oracle under QWEN35_FUSE=0x37 QWEN35_FLASH=0. Multi-card K80 hosts may need platform setup (PCI BAR sizing, NVIDIA driver options); Foundry's barswap/k80_quirk/ documents it.

Architecture notes:

Hybrid (Qwen3-Next lineage): 30 Gated-Delta-Net SSM layers + 10 sparse full-attention layers, attention when (i+1) % 4 == 0
Shared-expert MoE every layer: softmax + top-8/256 routed experts (SwiGLU, norm_topk_prob) plus an always-on shared expert with a sigmoid gate
SSM block: causal conv1d (k=4) → SiLU → L2-norm q/k → GQA head-broadcast → read-after-write delta-rule recurrence (scalar GDA decay) → gated RMSNorm
Attention block: GQA 16/2, head_dim 256, QK-RMSNorm, partial NEOX RoPE (first 64 of 256 dims), sigmoid output gate
Dual state per sequence: fixed recurrent SSM state (30 layers) + KV cache (10 layers), à la RWKV-X
Tokenizer: Qwen BPE, tokenizer.ggml.pre = "qwen35" pre-tokenizer; ChatML with <think> reasoning prefill
MTP/nextn trap: GGUF must carry nextn_predict_layers=0; the loader never reads blk.N.nextn.*
The A8 gibberish bug was the SSM GQA head-broadcast tiling (vh = r*n_k_heads + kh, interleaved like ggml_repeat); see ../architecture/qwen35moe.md

# Interactive (applies ChatML + <think>); --temperature 0 for oracle-style greedy
./pantheon --model models/nex-agi_Nex-N2-mini-Q3_K_M.gguf --chat --temperature 0

# One-shot (chat template applied automatically; --raw to send raw text)
./pantheon --model models/nex-agi_Nex-N2-mini-Q3_K_M.gguf \
  --temperature 0 --max-tokens 90 --prompt "How much does the ball cost?"

qwen35 Dense Architecture (Bonsai / Ornith)

A GGUF with general.architecture = "qwen35" and a dense FFN (blk.0.ffn_gate.weight, no merged expert tensors) loads as PN_ARCH_BONSAI (src/models/bonsai_forward.c): the same Gated-DeltaNet + sparse full-attention hybrid as qwen35moe, with a dense SwiGLU FFN in place of the MoE. A qwen35 GGUF with routed experts loads as qwen35moe instead.

Model	Quant	Params	Layers (SSM/Attn)	Tested	Notes
Ternary-Bonsai-27B	Q2_0 (g128 ternary, GGUF type 142)	27B	48/16	CPU	Oracle-equivalent vs PrismML's llama.cpp fork.
Ornith-1.5-9B	Q8_0	9B	24/8	TT p150a	The supported Tenstorrent model. `./install.sh && ./run_ornith.sh`. Serve: 42.9 tok/s decode, 1,630-token prompt in 2.43 s (670 tok/s), MTP speculative decoding on. Tool calls use the JSON dialect (HTTP API).
Ornith-1.0-9B	Q8_0	9B	—	K80 GPU, TT p150a	K80: 9.0 tok/s decode. TT (ran, not supported): output byte-identical to the CPU forward, 1.57 tok/s. Tool calls use the JSON dialect (HTTP API).

Backends: CPU; CUDA (single die, falls back to CPU if the upload fails); Tenstorrent (used only when no other accelerator took the model).

Pixtral Vision Encoder (library only)

C API only: pn_attach_vision, pn_describe / pn_describe_stream / pn_describe_embeds, pn_vision_encode_rgb, and a continuous-batch decoder (pn_batch_decoder_*) for bulk OCR (include/pantheon.h). The CLI and pantheon serve do not accept images; HTTP message content must be a string.
Decoding goes through the Llama-family path (pn_llama_prefill / pn_llama_forward_batch).
Exercised by tests/bench_vision.c.

Supported Quant Types

Type	Bits/Weight	Block Size	Status
F32	32	—	Supported
F16	16	—	Supported
Q8_0	8	32	Supported
Q4_0	4	32	Supported
Q3_K	3.4	256	Supported
Q4_K	4.5	256	Supported
Q5_K	5.5	256	Supported
Q6_K	6.5	256	Supported
BF16	16	—	Supported
Q2_0 (g128 ternary, type 142)	2.1	128	Supported (Bonsai)

Q3_K_M, Q4_K_M and Q5_K_M GGUFs mix the K-quant tiers (e.g. Q3_K for most layers with Q4_K/Q6_K for attention) — all supported. Q3_K was added for the Nex-N2-mini bring-up.

Hardware Tested

Hardware	OS	Build Command	Notes
Apple M4	macOS 15	`make USE_BLAS=1`	Accelerate BLAS, NEON SIMD
Apple M4	macOS 15	`make USE_BLAS=1 USE_THREADS=1`	Add threads for 1B+ models
Tesla K80 ×8 (16 GK210 dies, sm_37)	Ubuntu 24.04	`make USE_CUDA=1`	CUDA 11.4 driver. N2-mini 2-die GPU decode ~13.7 tok/s.
Tenstorrent Blackhole p150a	Linux	`make USE_TT=1`	See `../architecture/accel-foundry.md` §7.

Not Yet Tested (Should Work)

These models use supported architectures and quant types but haven't been verified yet.

Llama family:

Llama-3.2-1B-Instruct (Q4_K_M, Q8_0)
Llama-3.2-3B-Instruct (Q4_K_M)
SmolLM2-360M-Instruct (Q8_0)
TinyLlama-1.1B (Q4_0, Q8_0)

Qwen2 family:

Qwen2.5-1.5B-Instruct (Q4_K_M, Q8_0)
Qwen2.5-3B-Instruct (Q4_K_M)
Qwen2.5-7B-Instruct (Q4_K_M)
Qwen2.5-Coder-1.5B-Instruct (Q8_0)

Mistral family:

Mistral-7B-Instruct-v0.2 (Q4_K_M, Q8_0)

Phi-3 family:

Phi-3-mini-128k-instruct (Q4_K_M, Q8_0) — same arch, longer context
Phi-3-small-8k-instruct (Q4_K_M) — may have different head config
Phi-3-medium-4k-instruct (Q4_K_M) — larger model

Mixtral family:

Mixtral-8x7B-Instruct-v0.1 (Q4_K_M) — full 8 experts, top-2 routing, ~28 GB

Known Issues

Unsupported BPE pre-tokenizers fall back with a one-time warning. Recognised tokenizer.ggml.pre values are qwen35, qwen2, gpt-2, default, llama-bpe, llama3, llama-v3 (or unset). Anything else (e.g. SmolLM2's) uses the qwen35/GPT-2 pre-tokenizer and prints once: bpe: tokenizer.ggml.pre is unsupported; falling back to the qwen35/GPT-2 pre-tokenizer (token IDs may not match the reference for this model). Prompt token IDs can then differ from the reference tokenizer.

Not Yet Tested (RWKV-X)

No untested RWKV-X models remaining. Both 0.2B and 3.6B verified.

Not Yet Supported

Architecture	Status	Blocker
Phi-3/4 (other variants)	Phase 3.3+	Tested mini-4k only

18 KiB Raw Permalink Blame History Unescape Escape

Supported Models

Llama Architecture

Qwen2 Architecture

Mistral Architecture

Phi-3 Architecture

Mixtral Architecture (MoE)

RWKV-X Architecture (Safetensors)

RWKV-7 Architecture (Converted from .pth)

qwen35moe Architecture (Hybrid Gated-Delta SSM + Shared-Expert MoE)

qwen35 Dense Architecture (Bonsai / Ornith)

Pixtral Vision Encoder (library only)

Supported Quant Types

Hardware Tested

Not Yet Tested (Should Work)

Known Issues

Not Yet Tested (RWKV-X)

Not Yet Supported

18 KiB

Raw Permalink Blame History