Pantheon/docs/README.md

Pantheon Documentation

Pantheon is a pure-C11 inference engine and OpenAI-compatible model server, built on Foundry (a git submodule). Beyond Foundry it needs libc, libm, pthreads and libdl; BLAS and the Metal backend's Apple frameworks are optional.

⭐ Multi-Model Smart Routing — the flagship feature

Pantheon is a memory-aware, multi-model server. A single running instance hosts several models across several GPUs at once and routes every request intelligently — so a tiny host (e.g. 2 cores / 8 GB RAM) can drive a fleet of cards and run models on hardware they were never meant for. Tying multiple instances together is a separate future gateway; one instance is fully self-routing and needs nothing in front of it.

How routing works (all inside one pantheon serve process):

• One process binds one port. Each --model is an independent registry slot with its own weights, state, and die placement. A request picks its model by the model field in the JSON body, matched to each model's --name (one model loaded ⇒ the field is ignored).

• Replicas = the same --name on different dies. mr_acquire (src/serve/model_registry.c) load-balances across replicas by least-busy (fewest in-flight refs); with batching off each replica runs on its own die, so requests execute concurrently: two concurrent requests to a model replicated on dies 2 and 3 keep both dies busy at once and finish in about one request's wall-clock.

• Memory-aware admission (Memory-Aware Layer) gates every load and turn against live host-RAM + VRAM pressure, keeping the instance in a small host footprint.

• Optional per-model concurrency: QWEN35_BATCH_SLOTS=1 adds up to 8 batched slots on a single die (off by default — see reference/env-flags.md).

Launch it — one instance, an orchestrator plus a worker replicated across two dies:

./pantheon serve \
  --model /models/orchestrator.gguf --name orch   --dies 0,1 \
  --model /models/worker.gguf       --name worker --dies 2 \
  --model /models/worker.gguf       --name worker --dies 3 \
  --port 8201

Clients POST /v1/chat/completions with "model":"orch" or "model":"worker"; worker requests fan out across dies 2 and 3 by least-busy, concurrently. No router needed.

Known gap (roadmap): replica selection is least-busy but not yet session-sticky — under concurrent load a multi-turn session can hop replicas and miss its prefix cache. Session affinity across replicas is designed but not yet shipped.

⭐ Reference Library (flags · CLI · HTTP API)

The authoritative, code-audited reference (2026-07-12) — every environment flag, CLI flag, and HTTP endpoint, with values, defaults, effects, source file:line, and the parsing gotchas that bite (e.g. QWEN35_BATCH_SLOTS is an enable flag keyed on env[0]=='1', so =8 silently disables batching; --dies indices are relative to CUDA_VISIBLE_DEVICES, not physical devices).

• Environment Flags — all QWEN35_* / PN_* / FC_* + distributed vars (~40 flags)

• CLI Flags — pantheon serve global + per-model flags, --dies semantics, multi-model binding

• HTTP API — /v1/models, /v1/chat/completions (SSE), /admin/models/*, /admin/dies

• Supported Models — every verified model with its quant, config, measured throughput and a runnable command

Architecture — serving-era (code-audited 2026-07-12)

• Serve Layer — request lifecycle, model registry + multi-card load-balancing, the batch scheduler, streaming/SSE, the lock table

• Qwen3.5-MoE Model — the qwen35moe arch (e.g. Nex-N2-mini), 2-die split/placement, KV cache, decode slots, async step machine (supersedes the legacy qwen35moe.md below)

• Core Engine — pn_load/pn_load_on, the forward-step async seam, admission helpers

• Memory-Aware Layer — pressure levels (OK/WATCH/TIGHT/CRITICAL), pre-flight admission, the reactive ladder (why TIGHT is the mini's normal resting state)

• Accelerator / Die Manager + Foundry CUDA — die abstraction, CUDA↔die mapping under CUDA_VISIBLE_DEVICES, per-die VRAM budgeting, foundry primitives (streams/events/graphs), and §7 the second backend: Tenstorrent Blackhole p150a (make USE_TT=1; tt-kmd + SFPI only — no tt-metal, TT-NN, Python or C++) — what it serves today, its runtime flags, and the hazards that cost hardware

Guides

• Quickstart — Build, download a model, generate text

• Install — Per-platform build and run (CPU, macOS, CUDA, Tenstorrent), troubleshooting

• Chat Templates & Interactive Chat — Multi-turn chat, template detection, C API

• PyTorch .pth Converter — Convert .pth to safetensors in pure C, no Python

Architecture

• Engine Overview — How model loading, dispatch, and generation work, plus the vision-language encode/decode seam and continuous-batch VLM decoder for bulk OCR

• GGUF Format — GGUF parser, quant types, v2/v3 support

• Llama Forward Pass — Shared forward pass for Llama, Qwen2, Mistral, Mixtral + Phi-3 differences, batched multi-sequence decode, flash prefill, and the opt-in GPU paths

• Qwen3.5-MoE Forward Pass (legacy) — the original forward-pass writeup. ⚠ Predates the A–D serving stack; superseded by architecture/models-qwen35moe.md (the 2026-07-12 code audit). Kept for any unique detail; prefer the new doc.

• RWKV-X Forward Pass — Hybrid recurrent + sparse attention, WKV-7 kernel, safetensors loading

Serving & multi-tenant

The serving architecture itself is documented in architecture/serve.md; what follows is a map of the behaviour it implements.

• Multi-card concurrency — the host-memory + concurrency stack that lets a small host (8 GB / 2 cores) drive up to 8 cards: the die is the atomic unit, at most one model per die, reliability first, and capacity scales by adding servers rather than packing models.

• Single vs multi-tenant — one card serves either one fast user or K slower ones; the two knobs are multi-model placement and QWEN35_BATCH_SLOTS batching. See reference/env-flags.md.

• Multi-model serving — registry, model-field routing, runtime load/unload, die placement.

• Fleet bring-up — load models one at a time through /admin/models/load, gated and settle-waited. examples/deploy.sh is a working template.

API Reference

• include/pantheon.h — Public C API (opaque types, generation, sampling, vision-language describe, and the continuous-batch VLM decoder)

• src/core/engine.h — Internal types: config, architecture registry, state, pn_load_opts

• src/formats/gguf.h — GGUF file format types and parser

• src/models/llama.h — Llama architecture weight structs and forward pass

• src/models/phi3.h — Phi-3 weight structs (fused QKV, fused gate/up)

• src/models/rwkvx.h — RWKV-X adapter (safetensors loading, trie tokenizer, WKV-7 forward)

• src/core/chat.h — Chat template detection and formatting

• src/core/qweight.h — Quantized weight matrix (zero-copy mmap)

• src/accel/simd.h — SIMD kernels (NEON, AVX2, scalar)

• src/memory/arena.h — Bump allocator for inference scratch memory

Contributing

• ../CONTRIBUTING.md — build flags, code style, testing bar, and the licensing terms contributions are made under