Pantheon

An inference engine that assumes you don't have enough.

Not enough VRAM, not enough host RAM, not enough cores, not enough context, and not enough budget for current-generation GPUs. Pantheon is a memory-aware inference engine and OpenAI-compatible model server built for that case: a small host, older or mixed cards, several models per process.

It is pure C11 on top of Foundry, its companion tensor library (included as a git submodule). Beyond Foundry it needs only libc, libm, pthreads and libdl; BLAS (Accelerate on macOS, OpenBLAS on Linux) is optional, and the optional Metal backend links Apple's frameworks. Each backend builds to a single binary.

---

What "memory-aware" actually means

A common behavior when an engine runs out of memory is that the request fails. Pantheon treats running out as the normal operating condition and degrades on a ranked ladder, least-harmful first (src/core/mem/pn_ladder.h):

Pressure	What gives
WATCH	force Q4 KV cache — near-zero quality cost, 4–7× VRAM win
TIGHT	halve concurrency · clamp the prefill window · evict cold KV
CRITICAL	floor concurrency to one slot — a single request always still completes

Admission is checked before every load and every turn, against live host-RAM and VRAM pressure, so the engine refuses work it cannot finish instead of accepting it and falling over halfway. That layer (src/core/mem/) is about 2,100 lines of the codebase, which is a fair measure of how central it is.

It tells the client the truth. A prompt past the context cap is refused with a typed context_over_budget carrying the actual KV numbers, so the caller can compact and retry. Nothing is truncated silently: for an agent built on top, silent truncation would be silent corruption.

What that buys you

• Many models, one process, many GPUs. One port hosts N models across N devices, routed by the request's model field. Repeat a --name on another device and it becomes a replica, load-balanced least-busy. Load and unload at runtime without a restart. This is what lets a 2-core / 8 GB host drive several cards.
• Portable session state. Conversation state — recurrent SSM and attention KV — serializes to a versioned blob stamped with a model-identity tag the restore side validates. Hybrid state is hard to serialize because the recurrent half isn't position-addressable. It means session state can live on the user's machine instead of your disk.
• Reasoning that terminates. --reasoning-budget N forces </think> after N tokens, so a reasoning model that runs away still produces an answer instead of burning your context.
• Embeddable. C11, no C++ runtime, no exceptions, auditable dependency tree, links as a static library.

Old hardware

The CUDA backend targets sm_37 — Kepler, 2014. Measured by the author on one Tesla K80 (both GK210 dies) in a 2-core / 8 GB host: a 35B-A3B hybrid MoE served at 32K–128K context: Qwen3.6-35B-A3B and Nex-N2-mini (Q3_K_M) both decode at ~21 tok/s and prefill ~110-120 tok/s up to ~8k tokens (K80 quick start, supported models).

Correctness on that path, stated precisely:

• With the fused attention kernels off (QWEN35_FUSE=0x37) and flash prefill off (QWEN35_FLASH=0), the author measured the 2-die GPU forward's logits bit-identical to the CPU forward (argmax and every logit).
• The default flash prefill is argmax-exact, not bit-exact (~1e-6 relative drift; see env flags).
• The in-tree CUDA gates (tests/test_spec_qwen35, tests/test_users_qwen35, tests/test_prefix_cache_qwen35) check token-for-token identity between decode paths under QWEN35_FUSE=0x37. They need a model and a GPU and are not part of make test.

Kepler is the extreme case, not the point. The point is that hardware you already own should keep working.

---

Quick start

Prerequisites: a C11 compiler and make. On Linux, USE_BLAS=1 also needs OpenBLAS (apt install libopenblas-dev); on macOS it uses Accelerate. The model download below uses the Hugging Face CLI (pip install -U huggingface_hub); any way of fetching the .gguf works. Build on the machine you will run on: the default flags include -march=native, so a binary built on a newer CPU can fail with Illegal instruction on an older one. To build a binary you can copy between x86-64 machines, add PORTABLE=1 (-march=x86-64-v2, no AVX). Per-platform details: docs/guides/install.md.

git clone --recursive https://git.teamide.dev/David/Pantheon.git pantheon
cd pantheon && make USE_BLAS=1 pantheon
# already cloned without --recursive?  git submodule update --init

hf download bartowski/SmolLM2-135M-Instruct-GGUF \
  SmolLM2-135M-Instruct-Q8_0.gguf --local-dir models/

./pantheon --model models/SmolLM2-135M-Instruct-Q8_0.gguf \
  --prompt "Hello, my name is" --max-tokens 32

Interactive chat, with the model's own template detected automatically:

./pantheon --model models/SmolLM2-135M-Instruct-Q8_0.gguf --chat

Ornith 1.5 on a Tenstorrent card

Ornith 1.5 9B is the only model supported on Tenstorrent cards for now. On Linux with a Blackhole p150a:

git clone --recursive https://git.teamide.dev/David/Pantheon.git pantheon
cd pantheon && ./install.sh     # build pantheon-tt, set up card access (asks before sudo)
./run_ornith.sh                 # download the model (~10 GB, once), serve on 127.0.0.1:8080

run_ornith.sh fetches the files from dcherrera/Ornith-1.5-9B-Pantheon-TT and checks their sha256. The first start converts the model for the card (~10 GB more on disk); later starts skip that. It serves an OpenAI-compatible API, model id ornith. Options are in the script's header.

Qwen3.6 and Nex-N2-mini on a Tesla K80

On Linux with a Tesla K80 and the NVIDIA 470 driver. Supported K80 models: Qwen3.6-35B-A3B and Nex-N2-mini (both Q3_K_M), one model per card (both dies):

git clone --recursive https://git.teamide.dev/David/Pantheon.git pantheon
cd pantheon && ./install.sh --k80   # build pantheon-cuda (no CUDA toolkit needed)
./k80_qwen36.sh                     # or ./k80_nexn2.sh; downloads ~17 GB once, serves on 127.0.0.1

Measured on one K80 in a 2-core / 8 GB host (OpenAI streaming client, warm card, runs back to back):

prompt	Qwen3.6 first token	Nex-N2-mini first token	decode (both)
1k tokens	8 s (121 tok/s)	9 s (107 tok/s)	~21 tok/s
8k tokens	69 s (116 tok/s)	67 s (119 tok/s)	~18 tok/s
14k tokens	3.0 min (79 tok/s)	3.2 min (74 tok/s)	~15 tok/s
30k tokens	8.4 min (60 tok/s)	9.6 min (53 tok/s)	~10 tok/s

Good for: chat, writing, Q&A and background jobs — short prompts, long answers. Prefix caching is on: send a session_id, and send each earlier assistant turn back with its reasoning_content; later turns then only prefill what's new (a 4k-token conversation's second turn: ~38 s without the cache, 2-5 s with it). opencode does both on its own: point an @ai-sdk/openai-compatible provider at the server and its second turn (14k-token system prompt) measured 9 s instead of 170 s. One session id may carry several conversations (opencode's title request shares it); each keeps its own cache slot. Not good for: pasting long documents or codebases every request; prefill is the K80's weak spot. Options are in each script's header.

Serving

One model, no GPU required:

./pantheon serve --model models/SmolLM2-135M-Instruct-Q8_0.gguf --port 8080

That's a complete OpenAI-compatible server. Point any client at http://127.0.0.1:8080/v1 and it works.

Everything below is optional. --name defaults to the file name, and --dies is CUDA-only — omit it and the engine places the model itself. Repeat --model to serve several from one process:

./pantheon serve \
  --model models/big.gguf   --name chat  --ctx-cap 32768 \
  --model models/small.gguf --name draft \
  --port 8080

Clients choose with the model field in a normal OpenAI request body. On a multi-GPU box you can pin placement, and repeating a --name on another device makes it a replica that load-balances least-busy:

./pantheon serve \
  --model models/big.gguf   --name chat   --dies 0,1 \
  --model models/small.gguf --name worker --dies 2 \
  --model models/small.gguf --name worker --dies 3 \
  --port 8080

Two worker requests then occupy devices 2 and 3 concurrently, with nothing in front of the process doing the routing.

Bringing up more than a couple of models

On a constrained host, launching every model at once is how you OOM. The pattern that works is to start serve with one model and add the rest through POST /admin/models/load one at a time, each gated on free VRAM and host-RAM headroom, then settle-waited — visible in /v1/models — before starting the next. Bring-up becomes idempotent and restartable: after a failure you re-run and it picks up where it stopped, keeping what already loaded.

examples/deploy.sh is a working template that does exactly this, driven by a manifest (examples/models.conf):

chat  | /models/chat.gguf  | ctx_cap=32768 max_sessions=4 max_concurrent=2
draft | /models/draft.gguf |

./examples/deploy.sh --plan     # dry run — print the plan, change nothing
./examples/deploy.sh            # reconcile (idempotent)
./examples/deploy.sh --prune    # also unload anything not in the manifest

It needs bash and curl, nothing else — no jq, no Python, no nvidia-smi. GPU state comes from the engine's own /admin/dies, so the same script works unchanged on CPU, Metal and CUDA. Copy it and adapt the gates; every per-model option the serve command line accepts can be set per manifest line.

Endpoints	/v1/chat/completions (SSE), /v1/models, /admin/models/*, /admin/dies
Per-model caps	--ctx-cap, --max-sessions, --max-concurrent, --reasoning-budget
Scheduling	batch scheduler with a bounded queue, then an honest 429 + Retry-After
Caching	prefix cache with longest-common-prefix reuse across turns
Structured output	grammar-constrained decoding for tool calls and response_format
Ops	load/unload without restart; /v1/models stays live

A pantheon router subcommand fans across separate serve processes.

Supported models

Family	Covers
Llama	Llama, Qwen2, Mistral, Mixtral — one shared forward pass
Phi-3	fused QKV, NeoX RoPE
Qwen3.5 / 3.6 MoE	hybrid gated-delta SSM + attention, grouped experts, 2-die split
RWKV-X	hybrid recurrent + sparse attention, WKV-7 kernel
Pixtral	vision-language, continuous-batch decoder for bulk OCR

Formats: GGUF (v2/v3, K-quants and legacy) and safetensors, mmap'd zero-copy. Per-model quants, measured throughput and a runnable command: docs/reference/supported-models.md.

Backends

Backend	Flag	Notes
CPU	(default)	NEON, AVX2, scalar
BLAS	USE_BLAS=1	Accelerate on macOS, OpenBLAS on Linux
Threads	USE_THREADS=1	pthread pool
Metal	USE_METAL=1	Apple Silicon
CUDA	USE_CUDA=1	sm_37+, dlopened with kernels embedded as PTX — builds without nvcc. Supported on Tesla K80: Qwen3.6-35B-A3B and Nex-N2-mini
Tenstorrent	USE_TT=1	Blackhole p150a, Linux only; programs embedded, no TT software stack needed to build. Supported model: Ornith 1.5 9B only — other models may load on the card but are not supported

Because CUDA is dlopened, one binary runs on a machine with no NVIDIA driver at all and lights up the GPU path on a machine that has one.

make USE_BLAS=1 pantheon        # CLI
make USE_BLAS=1 libpantheon.a   # static library, to embed
make USE_BLAS=1 test            # test suites
make DEBUG=1 USE_BLAS=1 pantheon

The CLI binary is named for its backend

Each accelerator backend writes its own file, and pantheon is a symlink to whichever was built last:

Build	Binary	pantheon symlink
make	pantheon-cpu	→ pantheon-cpu
make USE_CUDA=1	pantheon-cuda	→ pantheon-cuda
make USE_TT=1	pantheon-tt	→ pantheon-tt
make USE_METAL=1	pantheon-metal	→ pantheon-metal

They used to be one file, so the last build silently owned the deployed path — which is how a CPU-only binary came to serve a 35B MoE with every GPU die idle. A build for one backend can no longer overwrite another's binary; at worst it re-points the symlink, and ln -sfn pantheon-cuda pantheon puts that back with no rebuild.

Ask a binary what it is rather than inferring it from its path:

./pantheon --backend                        # -> cuda|tt|metal|cpu (one token, for scripts)
./pantheon --version                        # -> Pantheon v0.4.0 / backend=cuda build=<commit>
curl -s localhost:8080/ | jq .backend       # a running serve
scripts/assert_backend.sh ./pantheon cuda   # deploy-time assertion; exit 1 on mismatch

pantheon serve prints backend=... build=... accelerator=... as its first line and the visible die count as its second, and it refuses to start when --dies names dies this binary has no backend for. To serve on the CPU deliberately, simply do not pass --dies.

Embedding

Pantheon is a library first and a CLI second. The public API is include/pantheon.h — opaque types, explicit ownership, error codes rather than exceptions, no global state. Link libpantheon.a from C or any language with a C FFI.

What Pantheon is not

Being clear about this saves everyone time:

• It is not built for speed on modern datacenter GPUs. Engines designed for that hardware will be faster there.
• Prefix caching, structured output, n-gram speculative decoding and continuous batching are table stakes, not differentiators. Everyone has them. They are here because an engine without them isn't serious, not because they're novel.
• It is not a distributed serving framework. One process is self-routing and bounded; you scale by adding servers, not by packing more into one.
• It is one person's project. Model coverage is narrower than that of larger projects.

Documentation

docs/ — start at docs/README.md.

• Quickstart · Chat templates · .pth converter
• Environment flags · CLI flags · HTTP API · Supported models
• Architecture: engine · serve layer · memory-aware layer · die manager · GGUF · Llama forward · Qwen3.5-MoE · RWKV-X

Built on Foundry

Pantheon uses Foundry for tensor compute, safetensors, tokenization, quantization, attention primitives and the CUDA layer. Same author, same license.

Foundry is a git submodule at foundry/ (https://git.teamide.dev/David/Foundry.git). The Makefile resolves it as FOUNDRY=/path, then ./foundry (the submodule, or a symlink), then ../foundry (a sibling checkout); if none exists it stops with a message saying so rather than a wall of missing headers.

License

AGPL-3.0 — see LICENSE.

Contributions come in under the CLA in CLA.md, with a DCO sign-off on every commit — see CONTRIBUTING.md. You keep your copyright; signing is one entry in .cla/signatures.md, added in your first pull request.

Copyright © 2026 David (Corey) Herrera and the Pantheon contributors. See NOTICE.