David
/
Pantheon
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Pantheon/docs/reference/http-api.md

815 lines
36 KiB
Markdown
Raw Blame History

# Pantheon — HTTP API Reference
The server speaks HTTP/1.1 over TCP. Every response carries
`Access-Control-Allow-Origin: *` and `Connection: close`. No authentication.
All bodies are JSON unless noted (SSE for streaming).
---
## Routes
| Method | Path | Handler | Audience |
|--------|------|---------|----------|
| `GET` | `/` | health | client / operator |
| `OPTIONS` | `*` | CORS preflight | client |
| `GET` | `/v1/models` | list models | client |
| `POST` | `/v1/chat/completions` | chat completion | client |
| `GET` | `/admin/models` | list live model instances | operator |
| `POST` | `/admin/models/load` | load a model | operator |
| `POST` | `/admin/models/unload` | unload a model | operator |
| `GET` | `/admin/dies` | die status | operator |
| `POST` | `/admin/session/export` | export a session's state as a blob | operator |
| `POST` | `/session/restore` | restore a session from a blob | client |
| `POST` | `/session/release` | free a session's slot | client |
| `GET` | `/session/estimate` | warm-restore vs re-prefill time estimate | client |
| `GET` | `/session/watch` | hold a session alive while connected | client |
Any other path → `404 {"error":{"message":"unknown route","type":"not_found","code":null}}`.
---
## GET /
Health check. Always 200.
**Response:**
```json
{"service":"pantheon","status":"ok","backend":"cuda",
"build":"5d818bfd1c9a","accelerator":true,"dies_visible":8}
```
| Field | Type | Meaning |
|-------|------|---------|
| `backend` | string | The accelerator backend this binary was COMPILED with: `cuda`, `tt`, `metal`, `cpu`. Compile-time constant, not a probe. |
| `build` | string | Commit this binary was built from (`-dirty` when the tree was not clean), or `unknown`. |
| `accelerator` | bool | False means every forward pass in this process runs on the host CPU, whatever the flags say. |
| `dies_visible` | int | Dies the die manager holds (`pn_dm_count()`). `0` on a CUDA build means no driver / no card / empty `CUDA_VISIBLE_DEVICES`. |
The cheapest route on the server is also the one that answers "what am I
running?" — see [`GET /v1/models`](#backend--build--what-this-binary-actually-is)
for why that question needed an endpoint at all.
---
## OPTIONS \*
CORS preflight. Always 200 with `{}` body.
---
## GET /v1/models
**Serve's own endpoint** (`openai_chat.c:oai_handle_models`, `serve_command.c:dispatch`):
same-name replica slots (multi-die LB copies) are deduplicated — clients see one
logical id per served name. The response does **not** include `live` or `free_slots`.
**Response — 200:**
```json
{
"object": "list",
"backend": "cuda",
"build": "5d818bfd1c9a",
"data": [
{
"id": "my-model",
"object": "model",
"created": 0,
"owned_by": "pantheon",
"max_sessions": 4,
"max_concurrent": 4,
"context_window": 32000,
"tool_choice_required_with_thinking": true,
"graphs": {"live": 30, "total": 40, "failed": 0,
"ineligible": 10, "pending": 0},
"attention": {"ondie": 10, "host": 0, "total": 10}
}
]
}
```
| Field | Type | Meaning |
|-------|------|---------|
| `backend` | string | **Top-level.** The accelerator backend this binary was COMPILED with: `cuda`, `tt`, `metal` or `cpu`. Compile-time constant, not a probe. |
| `build` | string | **Top-level.** The commit this binary was built from (`-dirty` suffixed when the tree was not clean), or `unknown`. |
| `id` | string | The routing key for `model` on a completion |
| `max_sessions` | int | RESIDENCY: how many conversations may hold KV at once |
| `max_concurrent` | int | CONCURRENCY: how many requests this model ACTUALLY decodes at once — the **served** value (`bs_get_load` capacity), not the configured cap. See below. |
| `context_window` | int | **Per-session context window in tokens** — see below. Omitted when unknown. |
| `tool_choice_required_with_thinking` | bool | `true` when `tool_choice: "required"` may be sent with thinking on: the think block is unconstrained and the turn ends in a parsed tool call (the think-gated grammar models ORNITH_JSON and QWEN_XML; `false` for MiniCPM's attribute dialect, which no grammar models, and for every model when `PANTHEON_TOOL_GRAMMAR=0`). |
| `graphs` | object | **CUDA-graph residency** — see below. Omitted when the model has no graph slots. |
| `attention` | object | **Attention-path residency** — see below. Omitted until the forward has reached an attention layer. |
### `backend` / `build` — what this binary actually is
Top-level on the response, not per-model, because it is a property of the
process: the accelerator backend it was **compiled** with (`cuda`, `tt`, `metal`,
`cpu`) and the commit it came from.
Compile-time constant, deliberately — not a runtime probe. A serve whose binary
has no CUDA compiled in cannot use a GPU whatever the driver reports, and the
completion text is identical either way, so nothing downstream can detect it
from the output.
The same values are on `GET /` (plus `dies_visible`), and out of band from
`pantheon --backend`, which prints one lowercase token for a deploy script to
assert against.
### `max_concurrent` — the SERVED number, not the configured cap
`max_concurrent` on `/v1/models` is what the model actually decodes at once
(`bs_get_load`'s capacity), which is **not** always what `--max-concurrent` asked
for. It reads `1` whenever there is no batch scheduler — no CUDA compiled in,
`QWEN35_BATCH_SLOTS` off, `bs_create` failed — and it drops below the cap when
the memory-pressure ladder clamps the slot count.
Same argument as `context_window` below: a client sizes its in-flight requests
from this number, so advertising a cap the serve cannot honour (e.g. `2` with no
scheduler, which turns every second concurrent chat into a `429`) is worse than
advertising nothing.
Operators still get the configured value: `/admin/models` carries **both**, as
`max_concurrent` (served) and `max_concurrent_configured` (asked for). Seeing
them differ is the signal.
### `context_window` — the advertised window
`context_window` is the window a client may actually plan a prompt against, in
tokens. It is the **effective** window (`pn_effective_ctx_window`): the operator's
`--ctx-cap` **after** the reactive memory ladder's runtime clamp — *not* the
configured cap, and not the model's theoretical maximum.
That distinction is the whole point of the field. Publishing the configured cap
while admission enforces a smaller one hands the client a prefill the serve then
rejects — on a K80 that rejection lands after minutes of cold-prefill TTFT, and it
lands mid-task.
Two properties clients depend on:
- **It is the same number a completion reports as `usage.kv_cap`.** The model list
is the client's INITIAL plan; `kv_cap` is the LIVE truth on every turn. A client
that reads both needs no polling and no extra endpoint to notice a mid-session
clamp.
- **It is additive and omitted when unknown.** A plain OpenAI-compatible client
ignores the extra key; a client that reads it never sees a `0` it has to
special-case. An absent field means *undetected*; a client should fall back to
a conservative window rather than guess.
Field name: Pantheon emits `context_window`. Clients that detect the window from
other servers' spellings (`context_length`, `max_model_len`, `n_ctx`, ...) need
to accept this one.
### `graphs` — CUDA-graph residency
Each layer of the hybrid stack can hold a captured CUDA graph that collapses its
~13-30 kernel launches into one replay. Across 40 layers those graphs are worth
about 1.8 ms/step. A capture that **fails** falls back to the direct kernel body
and produces **correct output** — so losing them costs only speed, silently.
This object is the only surface that reports it.
| Key | Meaning |
|-----|---------|
| `live` | Slots holding a captured graph right now |
| `total` | Slots the forward has actually reached (see below) |
| `failed` | Attempted and did not stick — **the only value that is a fault** |
| `ineligible` | Gated out; this model never captures. **Not a fault.** |
| `pending` | Eligible, warming up, no attempt yet |
| `gave_up` | Subset of `failed` whose bounded retry budget is spent (omitted when 0) |
**`ineligible` is normal.** The captured path requires the grouped MoE kernel,
which requires Q3_K expert tensors. A Q5_K_M model reports `live 0 / total 40 /
ineligible 40 / failed 0` — correct behaviour, zero exposure, and also zero of
the +4%. Read `failed`, not `live/total`, to decide whether something is wrong.
**`total` counts only exercised slots.** A serve decoding batched never touches
its single-token graph slots, so those are not reported as missing graphs.
A failed capture is retried on a bounded backoff and only stops after the budget
is spent, so a transient `failed` on one read that is `live` on the next is the
system self-healing, not a flapping metric.
### `attention` — on-die vs host-softmax residency
The same argument as `graphs`, only much sharper. Each full-attention layer
normally runs the on-die flash kernels. When it cannot it falls back to host-side
causal GQA softmax on the serve host's CPU — the exact wall this engine was built
to escape (it was 87.5% of an 8k prefill; moving attention on-die took that
prefill from 26 minutes to 3.82). The fallback returns the **same tokens**, logs
nothing per step, and on the depth bench costs **13.02 -> 1.22 tok/s** (K=2,
depth 4096) — and it worsens with context, because host softmax is O(context)
per layer per token. Without
this object a user simply waits, and nothing anywhere says why.
| Key | Meaning |
|-----|---------|
| `ondie` | Attention layers running the on-die flash kernels right now. |
| `host` | Layers on the host-softmax fallback — **any non-zero value is a fault** |
| `total` | Layers the forward has actually reached (same rule as `graphs.total`) |
| `gave_up` | Layers whose KV-ring allocation spent its retry budget (omitted when 0) |
| `impossible` | Subset of `gave_up`: the rings cannot fit this die at all (omitted when 0) |
`ondie + host == total`. Unlike `graphs`, there is no benign state here: `host >
0` always means this model is serving degraded, and the `reasons` histogram on
`/admin/models` says why. The two common causes are a per-slot KV-ring allocation
that did not fit (lower `--ctx-cap` or `--max-sessions`) and a session that has
run past the KV window, which wraps the ring.
A ring allocation that fails is retried on a bounded backoff, so a `host` reading
that clears on the next poll is the system self-healing. `gave_up` is the number
that should page someone.
No error cases (returns an empty `data` array when nothing is loaded).
**Router's `/v1/models`** (`registry.c:reg_models_json`) is a different shape: it
aggregates across live serve instances and adds `live` (count of live instances serving
that id) and `free_slots` (summed across live replicas). Do not conflate the two.
The router also emits `context_window`, aggregated the only honest way: the
**minimum** across that model_id's live replicas (a dispatch may land on any of
them), and **omitted entirely** if any live replica has not reported one. Serve
instances report their window on both `/registry/register` and every
`/registry/heartbeat`, because the ladder can move it at runtime.
---
## POST /v1/chat/completions
OpenAI-compatible chat completion. Supports streaming (SSE) and non-streaming.
Includes reasoning (`<think>…</think>` split) and tool-call support.
### Request headers
| Header | Required | Effect |
|--------|----------|--------|
| `Content-Type: application/json` | no | Not checked; the body is parsed as JSON regardless. |
| `X-Session-Id: <sid>` | no | Prefix-cache session key. A body key (`session_id`, `prompt_cache_key`, `promptCacheKey`) takes precedence. opencode sends this header on every request. |
### Request body
```json
{
"model": "my-model",
"messages": [
{"role": "system", "content": "..."},
{"role": "user", "content": "hello"}
],
"temperature": 0.7,
"top_p": 0.95,
"top_k": 40,
"max_tokens": 1024,
"stop": ["<|im_end|>"],
"stream": false,
"tools": [...],
"tool_choice": "auto",
"session_id": "abc123"
}
```
| Field | Type | Default | Notes |
|-------|------|---------|-------|
| `model` | string | — | Routing key. With one loaded model, any value (or absent) routes to it. With multiple models, must match a served name exactly; unknown → 404. |
| `messages` | array | *required* | Non-empty array of `{role, content}`. Roles: `system`, `user`, `assistant`, `tool`; a missing role is treated as `user`. `content` must be a string — any other type (e.g. an array of content parts) is read as empty. |
| `messages[].reasoning_content` | string | unset | Assistant turns only. The reasoning this serve returned for that turn; replayed into the prompt so a session can resume from its prefix cache. |
| `messages[].tool_calls` | array | unset | Assistant turns only. Rendered back into the prompt in the model's tool dialect. |
| `temperature` | float | `0.7` | `<= 0` is greedy argmax. |
| `top_p` | float | `0.95` | Nucleus sampling. |
| `top_k` | int | `40` | Top-K sampling. |
| `max_tokens` | int | `1024` | Max completion tokens. `0` or negative → `512`. Clamped to 1048576. |
| `max_completion_tokens` | int | `1024` | Alias; `max_tokens` wins if both are present. |
| `stop` | string or array | unset | Stop strings. |
| `stream` | bool | `false` | `true` → SSE (`text/event-stream`). |
| `stream_options.include_usage` | bool | `false` | Streaming only: send a terminal `usage` chunk before `[DONE]`. |
| `seed` | number | unset | Makes sampling reproducible. With `n` > 1, choice *i* uses `seed + i`. |
| `presence_penalty` | float | `0` | Clamped to [-2, 2]. |
| `frequency_penalty` | float | `0` | Clamped to [-2, 2]. |
| `logit_bias` | object | unset | `{"<token id>": bias}`; bias clamped to [-100, 100]. Non-numeric keys or values are skipped. |
| `logprobs` | bool | `false` | Return per-token log-probabilities (`choices[].logprobs.content[]`, with `token`, `logprob`, `bytes`). |
| `top_logprobs` | int | `0` | Alternatives per token; clamped to [0, 20]. |
| `n` | int | `1` | Number of completions. Each is a full separate decode (only the first uses the session's prefix cache). Capped at 8; a larger `n` returns 8 and sets `n_status: "capped_to_8"`. |
| `response_format` | object | `{"type":"text"}` | `json_object` constrains the answer to valid JSON. `json_schema` gets the same JSON constraint but the schema's shape is **not** enforced. See `response_format_status` below. |
| `enable_thinking` | bool | unset | Think-style templates: `true` opens a think block, `false` prefills a closed one. Absent → the `PANTHEON_NO_THINK` env decides. |
| `reasoning_budget` | int | per-model | Force `</think>` after N reasoning tokens. Overrides the model's `--reasoning-budget`; `0` or negative disables it. `max_reasoning_tokens` is an alias (`reasoning_budget` wins). |
| `tools` | array | unset | OpenAI-format tool definitions. Rendered into a system turn in the model's tool dialect (below). Ignored when `tool_choice` is `"none"`. |
| `tool_choice` | string or object | `"auto"` | `"none"` disables tools. `"required"` forces a call: the grammar is armed by default (`PANTHEON_TOOL_GRAMMAR=0` opts out). With thinking on the think block is unconstrained and the call is compelled from `</think>` (models advertising `tool_choice_required_with_thinking`). `{"type":"function","function":{"name":"X"}}` is enforced only when `PANTHEON_TOOL_GRAMMAR=1`; otherwise it behaves like `"auto"`. |
| `session_id` | string | unset | Prefix-cache session key (overrides `X-Session-Id`). Enables delta prefill on later turns of the same session. |
| `prompt_cache_key` / `promptCacheKey` | string | unset | Aliases for `session_id` (OpenAI's name; opencode's camelCase body field). Order: `session_id`, `prompt_cache_key`, `promptCacheKey`, then the `X-Session-Id` header. |
### Tool calling
Tools are rendered into a system turn (merged with a leading `system` message)
in the dialect the routed model emits. The dialect is chosen from the model name
and architecture (`tool_calling.c:tc_dialect_for_model`), not from the GGUF chat
template:
| Dialect | Used for | Call syntax |
|---------|----------|-------------|
| Qwen3-Coder XML | default | `<tool_call><function=NAME><parameter=KEY>VALUE</parameter></function></tool_call>` |
| MiniCPM XML | Llama-arch models whose template adds BOS (MiniCPM5) | `<function name="NAME"><param name="KEY">VALUE</param>…` |
| JSON | served names starting `ornith` | fenced or bare `{"function"\|"name": NAME, "arguments": {…}}` |
The model's output is parsed into OpenAI `tool_calls`, with arguments typed from
each tool's JSON schema. `PANTHEON_TOOL_GRAMMAR` decides which requests are
grammar-constrained:
| Value | Constrained requests |
|-------|----------------------|
| unset (default) | `tool_choice: "required"` only — after reasoning, the output is one well-formed call to an offered tool, arguments typed and `required` keys present. Everything else decodes **unconstrained**. |
| `1` | every request with tools: `"auto"` masks once a call opener appears; `"required"` and a named function compel a call. |
| `0` | none — `"required"` is unconstrained too. |
On the Tenstorrent serve a `"required"` turn keeps speculative decoding: drafts
are checked against the grammar and the output equals non-speculative
constrained decoding. On a `"required"` turn (every dialect, single-user and
batched) an end token is refused before the call opens and, inside the call,
allowed only where the call can still be closed; the serve then appends the
closing text. Any other call the model started but did not finish is left
out of `tool_calls` and reported in `tool_call_error` (`code`, optional
`function`, `parameter`, `message`) on the choice.
`tools` and `response_format` cannot both constrain one decode: when the request
carries tools, tools win and `response_format_status` is `"ignored_tools_present"`.
### Non-streaming response — 200
```json
{
"id": "chatcmpl-a1b2c3d4",
"object": "chat.completion",
"created": 1720000000,
"model": "my-model",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "Hello!",
"reasoning_content": "...",
"tool_calls": [...]
},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 12,
"completion_tokens": 7,
"total_tokens": 19,
"prompt_tokens_details": {"cached_tokens": 0},
"kv_used": 12,
"kv_cap": 32000
}
}
```
- `reasoning_content` — the text inside a `<think>…</think>` block; present only
when non-empty. Only think-style chat templates with thinking on produce it.
Non-think templates (plain ChatML such as SmolLM2, Mistral, Phi-3) return the
whole answer as `content`.
- `tool_calls` — present when a call was parsed; `finish_reason` is then
`"tool_calls"`. `content` is `null` for a pure tool call.
- `finish_reason` — `stop`, `length`, `tool_calls`, or `repetition_loop` (the
decode was stopped because the model was looping; outranks `tool_calls`).
- `logprobs` — on the choice, when requested.
- `prompt_tokens_details.cached_tokens` — prompt tokens reused from the prefix
cache rather than computed. Always present (`0` on a cold turn).
- `kv_used` — this session's KV occupancy in tokens after this turn.
- `kv_cap` — the **effective** per-session context window, the same number
`GET /v1/models` advertises as `context_window`, re-read every turn. If the
memory ladder moves the window mid-session, the next completion carries the
new value. Treat the model list as the initial plan and this as authoritative.
- `session_conflict: true` (inside `usage`) — two conversations are sharing one
session id, so neither can resume from cache.
- `response_format_status` (top-level, when the request sent `response_format`)
— `applied`, `schema_downgraded`, `ignored_tools_present`,
`ignored_unknown_type`, or `not_enforced` (a constraint was armed but the
output did not close a complete JSON object).
- `n_status` (top-level) — why fewer choices came back than `n` asked for:
`capped_to_8`, `partial_slot_unavailable`, `partial_generation_failed`.
- `usage` covers the whole request: the prompt is counted once, completion
tokens are summed across choices.
A prompt longer than the effective window is **refused** with
`context_over_budget` (below); nothing is silently dropped. The
`context_capped` field the writer supports is never set by the current
generation path.
### Streaming response — 200 (SSE)
`Content-Type: text/event-stream`. Each chunk is:
```
data: {"id":"chatcmpl-...", "object":"chat.completion.chunk", "created":...,
"model":"...", "choices":[{"index":0,"delta":{...},"finish_reason":null}]}
```
Delta fields emitted in order:
1. `{"role":"assistant"}` — first chunk of each choice.
2. `{"reasoning_content":"<fragment>"}` — inside a think block (think-style templates only).
3. `{"content":"<fragment>"}` — answer text. `logprobs` rides the chunk carrying those tokens.
4. Tool-call chunks (when a call was parsed):
- Opening: `{"tool_calls":[{"index":0,"id":"...","type":"function","function":{"name":"...","arguments":""}}]}`
- Arguments: `{"tool_calls":[{"index":0,"function":{"arguments":"..."}}]}`
5. Closing chunk: `{"delta":{},"finish_reason":"..."}`. Carries `tool_call_error`
on the choice and `response_format_status` top-level when applicable.
6. With `stream_options.include_usage`: a chunk with `"choices":[]` and `usage`.
7. Terminal sentinel: `data: [DONE]`
With `n` > 1 the choices stream one after another, told apart by `choices[].index`.
A client that disconnects aborts the turn and frees its slot.
Errors detected after the stream opened cannot change the status code:
`context_over_budget` is sent as a terminal event
`data: {"error":{"message":"...","type":"context_over_budget","code":null,"kv_used":N,"kv_cap":M}}`
and the stream ends; other generation failures close the choice with
`finish_reason: "stop"`.
### Error responses
| Status | `type` | Condition |
|--------|--------|-----------|
| 400 | `invalid_request_error` | Empty body, malformed JSON, body not an object, `messages` missing or empty |
| 404 | `model_not_found` | Several models loaded and `model` matches none |
| 422 | `context_over_budget` | Prompt exceeds the effective per-session window. Not transient: shorten the conversation and retry. Non-streaming: `kv_used`/`kv_cap` are in the message text; streaming: they are fields of the error object. |
| 429 | `server_busy` | Single-user path: a generation is already running on this model. Batched path: all slots busy and the queue full, or 30 s queue wait timed out. `Retry-After` when the scheduler supplies one. |
| 429 | `model_at_session_capacity` | A new `session_id` while every session slot (`max_sessions`) holds a live session. `Retry-After: 30`; the message estimates the re-prefill time. |
| 503 | `no_models_loaded` | No model loaded |
| 503 | `server_memory_constrained` | Host memory pressure: admission refused the prompt, or the queue is full under critical pressure |
| 503 | `server_shutting_down` | Server is draining for shutdown. `Retry-After: 1` when refused at the route. |
| 500 | `server_error` | Generation failed / out of memory |
Source: `openai_chat.c:oai_handle_chat`, `serve_generation.c:sg_run_completion`,
`serve_command.c:dispatch`.
---
## POST /admin/models/load
Operator-only. Load a model at runtime (Phase 7). Serialized against other
uploads via `reg->upload_lock` (one GPU upload in flight at a time).
### Request body
```json
{
"path": "/models/my-model.gguf",
"name": "my-model",
"dies": [0, 1],
"quantize": false,
"ctx_cap": 32768,
"max_sessions": 4,
"max_concurrent": 2,
"reasoning_budget": 1024
}
```
| Field | Type | Required | Notes |
|-------|------|----------|-------|
| `path` | string | yes | GGUF file path |
| `name` | string | no | Served id (defaults to file basename) |
| `dies` | int array | no | Device ordinals (same `CUDA_VISIBLE_DEVICES`-relative semantics as `--dies`). Max `PN_MODEL_MAX_DIES=8`. |
| `quantize` | bool | no | INT8-quantize at load time |
| `ctx_cap` | int | no | Per-session context cap in tokens; a prompt past it is refused with `context_over_budget`. `0`/absent = engine default. Mirrors `--ctx-cap`. |
| `max_sessions` | int | no | **Residency** — how many named sessions may hold KV at once. Over the cap a new session id gets `429 model_at_session_capacity`. `0`/absent = engine default. Mirrors `--max-sessions`. |
| `max_concurrent` | int | no | **Concurrency** — how many requests may decode simultaneously (batch-scheduler slots). Surplus queues, then `429`/`503`. Distinct from `max_sessions`. `0`/absent = the resolved `max_sessions`. Mirrors `--max-concurrent`. |
| `reasoning_budget` | int | no | Force `</think>` after N tokens of unclosed reasoning. `0`/absent = unbounded. Mirrors `--reasoning-budget`. |
Every per-model option available on the `serve` command line is settable here, so a model
added at runtime ends up configured identically to one launched with the process.
### Response — 200
```json
{
"id": "my-model",
"object": "model",
"path": "/models/my-model.gguf",
"dies": [0, 1],
"vram_bytes": 12345678,
"max_sessions": 4,
"max_concurrent": 4,
"context_window": 32000,
"context_window_configured": 32000,
"graphs": {
"live": 30, "total": 40, "failed": 0, "gave_up": 0,
"ineligible": 10, "pending": 0,
"captures": 30, "capture_failures": 0,
"batched_attn": {"live": 10, "total": 10, "failed": 0, "ineligible": 0, "pending": 0},
"batched_ssm": {"live": 20, "total": 30, "failed": 0, "ineligible": 10, "pending": 0}
},
"attention": {
"ondie": 10, "host": 0, "total": 10,
"gave_up": 0, "impossible": 0,
"host_engagements": 0, "ring_alloc_failures": 0
}
}
```
The operator surface carries **both** context numbers, because an operator's
question is not the client's: `context_window` is what sessions actually get right
now, `context_window_configured` is what `--ctx-cap` asked for. They are equal
unless the memory ladder is clamping this instant — and seeing them differ is
exactly the signal that memory pressure is costing context. (`/v1/models`
advertises only the effective one; a client has no use for a window it cannot
have.) Both are omitted when unknown.
Falls back to `{"status":"loaded"}` if the row cannot be found in the snapshot (race).
### Error responses
| Status | `type` | Condition |
|--------|--------|-----------|
| 400 | `invalid_request_error` | Missing `path`, malformed JSON, too many dies |
| 400 | `load_failed` | `dies` named on a serve with no accelerator backend compiled in, or with no dies visible. Not retryable — the fix is a different binary (`pantheon --backend`) or an omitted `dies`. |
| 409 | `load_failed` | Model already loaded / already unloading |
| 507 | `load_failed` | Admission refused (VRAM or host), die binding conflict, registry full |
| 404 | `load_failed` | File not found |
| 500 | `load_failed` | Other load failure |
Source: `model_admin.c:ma_handle_load`.
---
## POST /admin/models/unload
Operator-only. Unload a model by served name. Drains any in-flight generation
before freeing.
### Request body
```json
{"name": "my-model"}
```
| Field | Type | Required | Notes |
|-------|------|----------|-------|
| `name` | string | yes | Served model id to unload |
### Response — 200
```json
{"unloaded": "my-model"}
```
### Error responses
| Status | `type` | Condition |
|--------|--------|-----------|
| 400 | `invalid_request_error` | Missing `name`, malformed JSON |
| 404 | `unload_failed` | Name not loaded |
| 409 | `unload_failed` | Already unloading |
| 500 | `unload_failed` | Other failure |
Source: `model_admin.c:ma_handle_unload`.
---
## GET /admin/models
Operator-only. Every **live instance**, one row each. Unlike `/v1/models` these
are **not** deduplicated by name: two instances of one model on different dies
are two rows, because to an operator they are two different things.
The row is the same one `POST /admin/models/load` returns — this route exists so
its operator-only fields can be read at any time rather than only as a side
effect of loading a model.
### Response — 200
```json
{
"object": "list",
"data": [
{
"id": "my-model",
"object": "model",
"path": "/models/my-model.gguf",
"dies": [2, 3],
"vram_bytes": 12345678,
"max_sessions": 4,
"max_concurrent": 4,
"max_concurrent_configured": 4,
"context_window": 32000,
"context_window_configured": 32000,
"graphs": { "live": 30, "total": 40, "failed": 0, "gave_up": 0,
"ineligible": 10, "pending": 0,
"captures": 30, "capture_failures": 0,
"batched_attn": {"live": 10, "total": 10, "failed": 0,
"ineligible": 0, "pending": 0},
"batched_ssm": {"live": 20, "total": 30, "failed": 0,
"ineligible": 10, "pending": 0} },
"attention": { "ondie": 8, "host": 2, "total": 10,
"gave_up": 2, "impossible": 0,
"host_engagements": 2, "ring_alloc_failures": 10,
"reasons": {"ring_alloc_failed": 2} }
}
]
}
```
The operator row carries **both** concurrency numbers: `max_concurrent` is what
the model serves right now (`bs_get_load` capacity, the same value `/v1/models`
advertises) and `max_concurrent_configured` is what `--max-concurrent` asked for.
They differ when there is no batch scheduler at all (no CUDA compiled in,
`QWEN35_BATCH_SLOTS` off, `bs_create` failed) or when the memory-pressure ladder
clamped the slot count — the same split, for the same reason, as
`context_window` vs `context_window_configured`.
The operator `graphs` object extends the `/v1/models` one with `gave_up`, the
lifetime `captures` / `capture_failures` counters, and a per-class split:
`batched_attn` and `batched_ssm` (the multi-user decode path), `single_ssm` (the
one-user per-layer path) and `segment` (the per-die full-step graph). A class
the model has never exercised is omitted rather than reported as an empty
deficit. Field meanings: see `graphs` under `GET /v1/models`.
The operator `attention` object adds `gave_up` / `impossible`, the lifetime
`host_engagements` / `ring_alloc_failures` counters, and `reasons` — a
histogram, present only when `host > 0`, naming why each layer is off-die
(`ring_alloc_failed`, `dec_scratch_alloc_failed`, `kv_geometry_or_ring_wrap`,
`no_slot_decode_kernel`, `no_flash_kernels`, `no_batch_scratch`,
`flash_slots_off`). Field meanings: see `attention` under `GET /v1/models`.
No error cases (returns an empty `data` array when nothing is loaded).
Source: `model_admin.c:ma_handle_models`.
---
## GET /admin/dies
Operator-only. Snapshot of all probed accelerator dies, their VRAM budget/used,
and which model occupies them.
### Response — 200
```json
{
"object": "list",
"n_dies": 8,
"data": [
{
"die": 0,
"card": 0,
"model": "my-model",
"n_models": 1,
"vram_budget": 12884901888,
"vram_used": 11674869760
}
]
}
```
| Field | Type | Notes |
|-------|------|-------|
| `die` | int | Die manager ordinal (0-indexed across all probed dies) |
| `card` | int | Physical card index (`pn_dm_card_of(die)`) |
| `model` | string | Name of the model occupying this die, or empty string |
| `n_models` | int | Number of models on this die (typically 0 or 1) |
| `vram_budget` | int (bytes) | Die's total VRAM minus `PN_DIE_RESERVE_BYTES` |
| `vram_used` | int (bytes) | Bytes allocated by the model manager |
Source: `model_admin.c:ma_handle_dies`.
---
## Session routes
A session is a live prefix-cache / decode slot keyed by `session_id`. In each
route below the session id comes from the body's `session_id`, else the
`X-Session-Id` header (export reads the body only). `name` selects the model;
absent → the default model. Unknown model → `404 model_not_found`.
### POST /admin/session/export
Operator. Serialize a live session's state so it can be restored later.
Request: `{"name": "my-model", "session_id": "abc123"}` (`session_id` required).
Response — 200:
```json
{"session_id":"abc123","weights_version":123,"n_tok":4096,
"blob_bytes":31457280,"blob_b64":"<base64>","tokens":[...]}
```
`tokens` is the committed history the blob covers; it must be sent back on
restore.
| Status | `type` | Condition |
|--------|--------|-----------|
| 400 | `invalid_request_error` | Empty/malformed body, missing `session_id` |
| 400 | `session_export_unsupported` | The model's architecture has no session snapshot format |
| 400 | `session_caching_disabled` | The model has no session cache |
| 404 | `session_not_found` | No live session with that id |
| 429 | `session_busy` | A turn is in flight on that session; retry after it completes |
| 500 | `server_error` | Serialize / out-of-memory failure |
Source: `model_admin.c:ma_handle_session_export`.
### POST /session/restore
Install an exported session into a live slot.
Request:
```json
{"name":"my-model","session_id":"abc123","blob":"<blob_b64 from export>","tokens":[...]}
```
`session_id`, `blob` (base64) and `tokens` are required. The blob must come from
the same model architecture and the same weights.
Response — 200: `{"restored":true,"session_id":"abc123","n_tok":4096}`
| Status | `type` | Condition |
|--------|--------|-----------|
| 400 | `invalid_request_error` | Empty/malformed body, missing `session_id`/`blob`/`tokens` |
| 400 | `session_caching_disabled` | The model has no session cache |
| 400 | `bad_session_blob` | Not base64, corrupt, wrong architecture, or does not fit this model |
| 409 | `weights_version_mismatch` | Blob captured under different weights — replay the conversation instead |
| 429 | `model_at_session_capacity` | No free session slot |
| 500 | `server_error` | Restore failed |
Source: `openai_chat.c:oai_handle_session_restore`.
### POST /session/release
Free a session's slot now instead of waiting for the idle timeout (12 h default,
`PN_SESSION_IDLE_TTL_MS`).
Request: `{"name": "my-model", "session_id": "abc123"}`
Response — 200: `{"released":true,"session_id":"abc123"}`. Idempotent: an unknown
or already-freed session also returns 200.
Errors: `400 invalid_request_error` (empty/malformed body, no session id),
`404 model_not_found`.
Source: `openai_chat.c:oai_handle_session_release`.
### GET /session/estimate
`GET /session/estimate?tokens=N[&name=M][&blob_bytes=B][&upload_bps=R]`
Arithmetic estimate of restoring a session from a blob vs re-prefilling it.
`tokens` (> 0) is required. Without `blob_bytes`, the blob is estimated at
16 KiB per token; without `upload_bps` (bytes/s), 12.5 MB/s is assumed.
Response — 200:
```json
{"model":"my-model","tokens":4096,"prefill_tok_per_s":17.00,
"cold_reprefill_s":240.94,"warm_restore_s":5.391,"blob_bytes_est":67108864}
```
`prefill_tok_per_s` is the serve's measured prefill rate (a fixed fallback
before one is measured). An unknown `name` is not an error; `model` is then `""`.
Errors: `400 invalid_request_error` when `tokens` is missing or not positive.
Source: `openai_chat.c:oai_handle_session_estimate`.
### GET /session/watch
`GET /session/watch?session_id=S[&name=M]` (or `X-Session-Id`).
Liveness hold. On success the server answers `200` with
`Content-Type: text/event-stream`, a `: watch established` comment, and keeps the
connection open. While it is held the session is never reaped for idleness;
when it closes, the session is released after a grace period (45 s default,
`PN_SESSION_WATCH_GRACE_MS`) unless a new watch for it arrives first.
| Status | `type` | Condition |
|--------|--------|-----------|
| 400 | `invalid_request_error` | No session id |
| 400 | `session_caching_disabled` | The model has no session cache |
| 404 | `model_not_found` | Unknown model |
| 503 | `watch_unavailable` | Watch manager not running, or its table is full |
Source: `serve_command.c:handle_session_watch`, `session_watch.c`.
---
## Error body format
All error responses use:
```json
{"error":{"message":"<text>","type":"<code>","code":null}}
```
The `code` field is always `null`.
---
## Concurrency and connection model
- One detached pthread per accepted TCP connection. `accept()` never blocks
behind a generation.
- On `pthread_create` failure the connection is handled inline (no request
is dropped).
- SIGPIPE is ignored so a client disconnect mid-stream cannot kill the server.
- Generation serialization:
- **Legacy path** (`sched == NULL`): `gen_lock` trylock per model. Second
concurrent request to the same model → immediate 429.
- **Batched path** (`QWEN35_BATCH_SLOTS=1`): the C3 scheduler owns
concurrency. Up to 8 concurrent users; overflow waits in a per-model FIFO
queue (default depth 32, `BS_QUEUE_DEPTH`). 30-second queue wait timeout.
Reference in New Issue View Git Blame Copy Permalink