Pantheon/docs/reference/cli.md

Pantheon — CLI Reference

Entry point: cli/main.c. pantheon serve runs the OpenAI-compatible server (src/serve/serve_command.c), pantheon router the multi-instance dispatch layer (src/serve/router_command.c); anything else is the one-shot / interactive CLI.

---

pantheon (one-shot and chat)

pantheon --model <path> [--prompt <text> [--raw] | --chat [--system <text>]]
         [--max-tokens <n>] [--temperature <f>] [--top-p <f>] [--top-k <n>]
         [--config <path>] [--vocab <path>] [--quantize] [--bench]
pantheon --version | --backend | --help

Flag	Argument	Default	Effect
--model	<path>	—	GGUF or safetensors model. Without it, prints the version and checks Foundry links.
--prompt	<text>	"Hello"	One-shot generation. The model's chat template is applied unless --raw.
--raw	(flag)	off	One-shot --prompt without the chat template.
--chat	(flag)	off	Interactive multi-turn chat on stdin; template detected from the GGUF.
--system	<text>	unset	System prompt (chat mode only).
--max-tokens	<n>	128	Tokens to generate.
--temperature	<f>	0.8	Sampling temperature.
--top-p	<f>	0.9	Nucleus sampling threshold.
--top-k	<n>	40	Top-k sampling.
--config	<path>	unset	HuggingFace config.json (safetensors models).
--vocab	<path>	unset	RWKV World tokenizer vocab file.
--quantize	(flag)	off	INT8-quantize weights at load time.
--bench	(flag)	off	Report tok/s and memory.
--version	(flag)	—	Pantheon v<ver> then backend=<b> build=<commit>.
--backend	(flag)	—	Prints only the compiled backend token (cuda, tt, metal, cpu).
-h / --help	(flag)	—	Usage.

---

pantheon serve

pantheon serve --model <path> [--name <id>] [--dies <a[,b,...]>]
               [--ctx-cap <n>] [--reasoning-budget <n>]
               [--max-sessions <n>] [--max-concurrent <n>]
               [--model <path> ...]
               [--host <addr>] [--port <n>]
               [--config <path>] [--vocab <path>]
               [--quantize]
               [--served-name <id>]

--model is repeatable. The per-model flags --name and --dies bind to the most recently seen --model. There is no positional association; flags are matched by order.

Global flags

Flag	Argument	Default	Effect
--model	<path> (GGUF)	— required, at least one	Load and serve the model at this path. Repeatable; each occurrence starts a new model slot. Max MR_MAX_MODELS=16.
--host	<addr>	127.0.0.1	TCP bind address. Must be a dotted-IPv4 literal (inet_pton).
--port	<n>	8080	TCP port. Parsed with atoi.
--config	<path>	unset	HuggingFace config.json for safetensors models.
--vocab	<path>	unset	RWKV World tokenizer vocab file.
--quantize	(flag)	off	INT8-quantize weights at load time.
--served-name	<id>	unset	Back-compat alias: sets the first model's served name when it has no --name. Silently ignored if --name already set for model 0.
-h / --help	(flag)	—	Print usage and exit 0.

Per-model flags

These flags bind to the model specified by the preceding --model. If used before any --model the server exits with an error.

Flag	Argument	Default	Effect
--name	<id>	basename of path	Served model id (the model field clients send in requests). Used for routing in multi-model mode.
--dies	<a[,b,...]>	none (auto-place)	Pin to accelerator dies (CUDA or Tenstorrent builds). CUDA ordinals are relative to CUDA_VISIBLE_DEVICES, not physical device numbers. Max PN_MODEL_MAX_DIES=8. N dies split the layers N ways (VRAM headroom, not speed). A --dies naming dies this binary has no backend for refuses to start; omit it to serve on the CPU.
--ctx-cap	<n>	0 = engine default	Per-session context cap in tokens (the KV window). A longer prompt is refused with context_over_budget; nothing is silently dropped. Max 2^20.
--reasoning-budget	<n>	0 = unbounded	After <n> tokens inside <think> without closing it, force </think> so the model answers.
--max-sessions	<n>	0 = default 8 (BS_MAX_USERS)	Cap on live prefix-cache sessions; a new session over the cap is refused 429/503.
--max-concurrent	<n>	0 = the resolved --max-sessions	Cap on simultaneous decode slots; surplus requests queue, then get 429/503.

Multi-model semantics

pantheon serve \
  --model /models/big.gguf   --name big   --dies 0,1 \
  --model /models/small.gguf --name small --dies 2

All models share the single --port (serve_command.c:333, one hs_listen call).

Each --model occupies an independent registry slot (model_registry.c:117–122). Per slot, on load:

• Always: pn_model (weights) + gen_lock (non-batched serializer).
• Non-batched path (QWEN35_BATCH_SLOTS unset): also gets a pn_state; gen_lock serializes single-user access — second concurrent request to the same model → 429.
• Batched path (QWEN35_BATCH_SLOTS=1): gets a pn_batch_sched instead (model_registry.c:272–282, bs_create); the gen_lock+single-pn_state serializer is bypassed. Up to BS_MAX_USERS=8 concurrent users share one model.

Requests route by the model field in the JSON body (openai_chat.c:616–629, route_model → mr_acquire). With exactly one loaded model, the field is ignored (back-compat).

GPU uploads are serialized (one at a time) via reg->upload_lock to prevent stacking the ~2.4 GB host transient.

--served-name affects only model 0 when --name was not also given for it.

Replicas (same name, different dies)

Repeating the same --name on different --dies creates a load-balanced replica pool (model_registry.c:117–119):

pantheon serve \
  --model /models/worker.gguf --name worker --dies 2 \
  --model /models/worker.gguf --name worker --dies 3

mr_acquire (model_registry.c:322–355) picks the least-busy same-name entry (fewest in-flight refs); each replica has its own die and gen_lock, so concurrent requests run truly in parallel across those dies.

Session affinity across replicas within a single serve process is not yet implemented — requests always go to the least-busy replica. See the "Multi-Model Smart Routing" section in docs/README.md.

Die index semantics

--dies 0,1 means CUDA ordinals 0 and 1 as seen by the process — after CUDA_VISIBLE_DEVICES filtering. If CUDA_VISIBLE_DEVICES=4,5,6,7 is set, --dies 0,1 maps to physical cards 4 and 5. The indices are NOT physical device numbers.

Startup sequence

1. pn_memwatch_start() — host RAM poll thread starts.
2. Signal handlers installed (SIGINT/SIGTERM → clean CUDA teardown; crash signals → _exit() so the kernel closes the nvidia fd and reclaims VRAM).
3. CUDA init (fc_init(), pn_dm_init_cuda()).
4. Model registry init.
5. For each --model: mr_load() (blocks; uploads weights to GPU).
6. TCP listen socket bound.
7. Accept loop: each connection spawns a detached thread.

Runtime behavior

• Cheap routes (GET /v1/models, OPTIONS) are never queued behind a generation.
• Per-model generation is serialized by gen_lock. A second request to the same model while one is generating → immediate 429.
• With QWEN35_BATCH_SLOTS=1, the scheduler (batch_sched) replaces the gen_lock; up to BS_MAX_USERS=8 concurrent users share one model.
• QWEN35_GPU_FULL_LOGITS being set causes immediate exit before any model loads.

---

pantheon router

pantheon router [--host <addr>] [--port <n>]

Flag	Default	Effect
--host	127.0.0.1	Bind address.
--port	8100	Port.

A host-side dispatch layer over several pantheon serve processes: no model loads, no GPU. Serve instances register themselves when PANTHEON_REGISTRY_URL points at the router, then heartbeat; instances whose heartbeat lapses are evicted. Requests go to the least-busy live instance of the requested model, and a session (X-Session-Id) sticks to the instance that served it so its prefix cache is reused. No authentication: bind it to a trusted interface.

Route	Purpose
POST /registry/register	A serve announces a model instance
POST /registry/heartbeat	Refresh liveness and load
DELETE /registry/deregister	Graceful removal
GET /registry/fleet	Model-level fleet view
GET /v1/models	Live models, deduplicated by model id
POST /v1/chat/completions, POST /v1/completions	Least-busy dispatch, SSE pass-through

---

pantheon-convert (cli/convert.c)

A separate binary (make pantheon-convert), not a subcommand: converts PyTorch .pth checkpoints to safetensors. See ../guides/pth-convert.md.