# Serve Layer Architecture Pantheon's serve layer implements an OpenAI-compatible HTTP inference server. Entry point: `src/serve/serve_command.c`. The serve layer sits above the engine (`src/core/engine.c`) and is reached via `pantheon serve`. --- ## Overview At startup `pn_serve_main` (`serve_command.c:206`) parses one or more `--model` flags, loads each model into an `mr_registry`, binds a TCP socket, and enters an infinite `accept()` loop. Every accepted connection is handed to a detached `pthread` (`spawn_connection`, `serve_command.c:185`); if `pthread_create` fails it falls back to handling the request inline so no connection is ever silently dropped. The connection thread calls `handle_connection` → `dispatch`. Dispatch is a straight O(route-count) `strcmp` chain (`serve_command.c:124`): | Method | Path | Handler | |--------|------|---------| | `OPTIONS` | any | CORS preflight: `hs_send_json 200 {}` | | `GET` | `/v1/models` | `oai_handle_models` | | `POST` | `/v1/chat/completions` | `oai_handle_chat` | | `POST` | `/admin/models/load` | `ma_handle_load` | | `POST` | `/admin/models/unload` | `ma_handle_unload` | | `GET` | `/admin/dies` | `ma_handle_dies` | | `GET` | `/` | health: `{"service":"pantheon","status":"ok"}` | `GET /v1/models` and admin routes never touch any generation lock; they can respond concurrently with an in-flight generation on any model. ### Request lifecycle (socket → SSE) ``` accept() └─ spawn detached pthread └─ hs_read_request() (http_server.c — reads headers + body) └─ dispatch() └─ oai_handle_chat() ├─ jr_parse(body) ├─ parse_chat_request() (extracts model, params, session_id) ├─ route_model() (mr_acquire → entry + refcount++) │ ├─ [batched path: QWEN35_BATCH_SLOTS=1] │ ├─ bs_slot_acquire_or_wait() (blocks up to 30 s) │ └─ sg_run_completion_batched() → SSE │ └─ [single-user path: sched == NULL] ├─ pthread_mutex_trylock(&entry->gen_lock) → 429 if busy └─ sg_run_completion() → SSE └─ mr_release(reg, entry) (refcount--) ``` --- ## Model Registry and Routing **Files:** `src/serve/model_registry.h`, `src/serve/model_registry.c` ### Data structures ```c /* model_registry.h:54 */ typedef struct { char name[MR_MAX_NAME]; /* served model id */ pn_model *model; pn_state *state; /* single-user state (sched==NULL) */ pthread_mutex_t gen_lock; /* single-user generation serializer */ pn_batch_sched *sched; /* C3: non-NULL when batching is on */ pn_prefix_cache *prefix_cache; /* PC1: session prefix cache */ pthread_mutex_t pc_lock; /* serializes prefix-cache ops */ int dies[PN_MODEL_MAX_DIES]; int n_dies; /* 0 = CPU-resident */ uint64_t vram_bytes; int refs; /* in-flight acquires */ bool unloading; bool used; } mr_entry; /* model_registry.h:86 */ typedef struct { mr_entry entries[MR_MAX_MODELS]; /* MR_MAX_MODELS = 16 */ int n; pthread_mutex_t lock; pthread_cond_t drained; /* signaled on every mr_release */ pthread_mutex_t upload_lock; /* A3: one GPU weight upload at a time */ } mr_registry; ``` ### Loading (`mr_load`, `model_registry.c:90`) Load is a four-phase operation designed to keep the registry serviceable throughout: 1. **Slot reservation** (under `reg->lock`): a slot is marked `used=true` with no model pointer yet — the name is taken but the entry is not yet acquirable (`require_ready` skips it in `find_locked`). 2. **Placement + VRAM admission** (lock-free): auto-placement via the die manager (`pn_dm_place`), or explicit dies from `--dies`. `pn_admit_model_load` gates against the per-die budget using a conservative 8 k context guess (`MR_ADMIT_CTX_GUESS = 8192`, `model_registry.c:36`). `pn_dm_bind` commits the reservation. 3. **Weight upload** (under `reg->upload_lock`): `pn_load_on` runs while the registry stays fully serviceable. `upload_lock` ensures at most one GPU upload is in flight at a time, preventing two concurrent 2.4 GB transients stacking on an 8 GB host (`model_registry.c:213`). 4. **Publish** (under `reg->lock`): model, state, locks, and die info are written atomically and `reg->n` is incremented. After publish, the batch scheduler (`bs_create`) and prefix cache (`pn_prefix_cache_new`) are created and published under the lock. Both are fail-soft: failure leaves the model in single-user mode or full-prefill mode respectively. ### Acquire / release refcounting ```c mr_entry *mr_acquire(mr_registry *reg, const char *name); /* model_registry.c:322 */ void mr_release(mr_registry *reg, mr_entry *entry); /* model_registry.c:358 */ ``` `mr_acquire` locks `reg->lock`, scans entries, increments `refs`, and returns the pointer. Every user of an entry MUST pair `mr_acquire` with `mr_release`. `mr_release` decrements `refs` and broadcasts `reg->drained`; `mr_unload` waits on that condvar until `refs == 0` before freeing. ### Multi-card load-balancing (`model_registry.c:329`) An operator may place one model id on several cards (`--dies 4,5` and `--dies 6,7` both under `--name my-model`). `mr_acquire` finds all ready entries that share the requested name and picks the one with the fewest in-flight `refs` — round-robin load balance at the reference granularity: ```c for (int i = 0; i < MR_MAX_MODELS; i++) { mr_entry *c = ®->entries[i]; ... if (!e || c->refs < e->refs) e = c; } ``` Empty/null name resolves to the first (lowest slot index) resident entry — the back-compat single-model shortcut (`model_registry.c:344`). ### Route model (`openai_chat.c:499`) ```c static mr_entry *route_model(int client_fd, mr_registry *reg, const char *name) { mr_entry *entry = (mr_count(reg) <= 1) ? mr_acquire(reg, NULL) : mr_acquire(reg, name); ... } ``` With exactly one resident model, every request routes to it regardless of the `model` field — preserving single-model server behavior. With more than one, an exact name match is required; an unknown name returns 404, no loaded model returns 503. ### `/v1/models` dedupe (`openai_chat.c:266`) `oai_handle_models` calls `mr_snapshot` to get all resident entries, then dedupes by name before serializing JSON — a model placed on two cards appears as one logical id: ```c bool seen = false; for (int j = 0; j < i; j++) if (strcmp(info[j].name, info[i].name) == 0) { seen = true; break; } if (seen) continue; ``` ### Unload (`mr_unload`, `model_registry.c:369`) Marks `unloading=true` (blocking new acquires), decrements `reg->n`, then blocks on `reg->drained` until `refs == 0`. Detaches all pointers under the lock and frees them outside: prefix cache → batch scheduler → die binding → state → model. --- ## Batched Decode (`batch_sched.c`) **Feature gate:** `QWEN35_BATCH_SLOTS=1` (env var, CUDA build only). `bs_enabled()` (`batch_sched.c:44`) caches the result. When enabled, `mr_load` calls `bs_create(model, BS_MAX_USERS)` where `BS_MAX_USERS = 8` (`batch_sched.h:173`). The scheduler owns 8 `pn_state*` instances (one per slot) and a background driver thread. ### Slot lifecycle ``` IDLE -(bs_slot_acquire/acquire_or_wait)-> PREFILL -(bs_slot_bind / bs_slot_bind_ex)-> BOUND -(driver first step)--------------> DECODE -(bs_slot_release)----------------> IDLE ``` Only the owner thread moves a slot back to IDLE. The driver marks `finished=true` on EOS/budget but never calls `bs_slot_release` — it always signals the owner to drain and release. ### GPU-ownership turnstile (`batch_sched.c:72`) The model's single-user forward machinery (C2 async phase machine, shared live SSM buffers, shared GPU KV ring) supports exactly one in-flight GPU forward at a time. Pre-C3 the per-model `gen_lock` enforced this. The batch scheduler uses a FIFO ticket turnstile instead — `gpu_ticket_next` / `gpu_ticket_serving` / `cv_gpu` — which prevents the free-running L3 driver from barging ahead of a waiting prefiller: ```c static void bs_gpu_acquire_locked(pn_batch_sched *sched) { /* batch_sched.c:72 */ uint64_t my = sched->gpu_ticket_next++; while (sched->gpu_ticket_serving != my) pthread_cond_wait(&sched->cv_gpu, &sched->mu); } ``` A turn owner never `cond_wait` (chunks and steps are bounded GPU work), so the turnstile is deadlock-free by construction. ### D1 overflow queue + D3 admission backpressure `bs_slot_acquire_or_wait` (`batch_sched.c:1475`, called from `openai_chat.c:577` with a 30 s timeout) replaces the immediate 429: - **Fast path:** slot free AND `pressure_level < BS_PRESSURE_HOLD_LEVEL` (3 = CRITICAL) → acquire immediately. - **Slow path:** enqueue (`n_queued++`) and block on `cv_idle`. - Queue full + pressure < CRITICAL → 429 + `Retry-After: 5`. - Queue full + CRITICAL → 503. - Timeout after `wait_ms` → 429 + `Retry-After: 2`. - Shutdown → 503. `BS_QUEUE_DEPTH_DEFAULT = 32` (`batch_sched.h:274`). Override with `BS_QUEUE_DEPTH`. `bs_notify_pressure` (`batch_sched.c:1589`) is called by the serve path before every `bs_slot_acquire_or_wait` (`openai_chat.c:569`) to push the live `pn_mem_state` into the scheduler, so the gate sees the current host state at decision time. ### D2 — continuous batching / dynamic-width step The driver compacts the active (BOUND/DECODE) slots into GPU indices `[0, n_active)` each step. Idle slots pay zero GPU compute. When `active_set_changed` is set (join, leave, or finish), `rebuild_gpu_map` (`batch_sched.c:237`) runs a greedy permutation algorithm that handles arbitrary reorderings — including cycles that require a scratch slot (`cap` index, `batch_sched.c:279`). This fixed the K≥5-concurrent-joiners bug where the old downward-only move refusal caused all active users to emit empty content. ### L3 — in-driver sampling (default on) `bs_slot_bind_ex` (`batch_sched.c:1294`) passes the request's sampling params, repeat-penalty window, and token budget to the driver. After each `decode_slots` step the driver samples every slot itself — sequentially, deterministically — and pushes the result to a per-slot output ring sized to the full budget: ```c uint32_t next = pn_sample_next_token(lo[g], sched->n_vocab, &sl->sp, sl->recent, sl->n_recent); sl->ring[sl->ring_w++] = next; /* batch_sched.c:545 */ ``` Owners drain the ring with `bs_slot_pop_token` and do detokenize/emitter/SSE entirely off the step's critical path. A slow or stalled client cannot stall the rest of the batch. The ring is never resized after bind (`ring_cap == budget`), so it can never overflow and the driver never blocks on a slow consumer. Stop-string detection stays owner-side (text level); a slot may over-generate past a stop string until its owner drains and releases — surplus ring tokens are discarded. Disable with `QWEN35_BS_DRIVER_SAMPLE=0` to restore legacy lockstep logits/token exchange. ### L2 — chunked prefill with yield windows Prefill runs in chunks of `QWEN35_BS_PREFILL_CHUNK` tokens (default 128). After each chunk (except the last) `bs_prefill_yield` (`batch_sched.c:1040`) makes the slot's host mirror authoritative via `gpu_state_to_host`, releases the GPU turn, and re-acquires it FIFO — giving the driver one decode step between chunks. This bounds a joiner's stall on concurrent decoders to one chunk instead of one full prompt. ### Single-user fallback When `QWEN35_BATCH_SLOTS` is not set (or the build has no CUDA), `entry->sched` is `NULL`. The serve path (`openai_chat.c:658`) falls back to: ```c if (pthread_mutex_trylock(&entry->gen_lock) != 0) { hs_send_error(client_fd, 429, "server_busy", "a generation is already in progress on this model; " "retry shortly"); ... } ``` A second concurrent request to the same model gets an immediate 429. Requests to different models run concurrently (each holds only its own entry's lock). --- ## Generation and Streaming (`serve_generation.c`) ### Prompt building `build_prompt` (`serve_generation.c:120`) tokenizes the full `messages[]` array into ChatML format using the model's tokenizer: `<|im_start|>role\ncontent<|im_end|>\n` for each turn, with `tool` messages wrapped in ``. Opens the assistant turn and prefills `` for reasoning models (`PN_CHAT_CHATML_THINK`). ### Admission Before any prefill, `pn_admit` checks live host + die memory and either passes, caps the context (sliding-window oldest-token drop), or denies. A denied admission returns `SG_ADMIT_DENIED`, which the serve path converts to a 503 (`openai_chat.c:683`). A capped admission sets `result->context_capped = 1` and disables prefix caching for that turn; the client sees `"context_capped": true` on the response. ### Single-user completion (`sg_run_completion`, `serve_generation.c:525`) ``` build_prompt → pn_admit → [prefix cache acquire] → prefill chunks → sample first token → decode loop (sg_forward_step per token) → [prefix cache commit] → return sg_result ``` `sg_forward_step` (`serve_generation.c:411`) uses an async non-blocking launch + event-poll when the model supports it (`pn_forward_step_begin/poll`), sleeping 50 µs → 500 µs between polls. This allows up to 8 per-card connection threads to run their cards simultaneously from a 2-core host without blocking each other. ### Batched completion (`sg_run_completion_batched`, `serve_generation.c:744`) Same flow as single-user but: - Prefill takes the GPU turn via `bs_prefill_begin` or `bs_prefill_resume`. - Chunked prefill with `bs_prefill_yield` windows (L2). - After the last chunk, `bs_slot_bind_ex` (L3) or `bs_slot_bind` (legacy) hands the slot to the driver. - L3 path: drain the ring with `bs_slot_pop_token` → detokenize → emitter → SSE, all off the critical path. - `bs_slot_finalize` (if caching) then `bs_slot_release`. ### Streaming (SSE) `run_streaming` / `run_streaming_batched` (`openai_chat.c:434,464`) call `hs_begin_sse` and register `stream_delta_cb` as the `delta_fn`. The callback is invoked by the emitter for each text fragment: ```c static void stream_delta_cb(const char *text, int is_reasoning, void *ud) { /* openai_chat.c:397 */ jw_raw(&jw, is_reasoning ? "\"reasoning_content\":" : "\"content\":"); jw_string(&jw, text); send_chunk(sc, jw.data, NULL); /* hs_send_sse */ } ``` A final chunk carries `finish_reason`; then `hs_end_sse` writes `data: [DONE]`. ### Reasoning / stop-sequence emitter The `emitter` struct (`serve_generation.c:179`) handles three channels: - **in_think**: text before `` → `reasoning_content`; switches to content channel on ``. - **content**: after ``, before a stop string or ``. - **in_tool**: raw text after `` → captured in `tool` channel for structured parsing by `tool_calling.c`. A `carry` buffer holds the longest suffix that could start any marker or stop string across a token boundary. ### Prefix cache (`src/core/prefix_cache.c`, used via `serve_generation.c:574`) Keyed by `session_id` (from body field `session_id` or header `X-Session-Id`). Single-user path: `pn_prefix_cache_acquire` returns the session's `pn_state` at the delta boundary; on commit, `pn_prefix_cache_commit` records the full token history. The `pc_lock` is not needed on the single-user path (`gen_lock` already serializes), so `pcx.pc_lock = NULL` (`openai_chat.c:674`). Batched path (PC-batched): session affinity is slot-resident. `bs_slot_acquire_biased` prefers the slot that already holds this session's warm state. `bs_prefill_resume` checks if the slot's committed history (`hist`) is an exact prefix of the new prompt and skips the already-committed tokens. `bs_slot_finalize` snapshots the GPU state to host and publishes the driver-maintained `ct` (committed-token log) as `hist` for the next turn. --- ## Concurrency Model | Lock | Scope | What it protects | |------|-------|-----------------| | `reg->lock` | `mr_registry` | slot table (used/refs/unloading/n); brief critical sections only | | `reg->drained` | condvar on `reg->lock` | unload drain wait | | `reg->upload_lock` | `mr_registry` | one GPU weight upload in flight at a time | | `entry->gen_lock` | `mr_entry` | single-user generation serializer (trylock → 429) | | `entry->pc_lock` | `mr_entry` | prefix cache ops on the single-user path (unused — gen_lock covers it; kept for the batched concurrent-slot case) | | `sched->mu` | `pn_batch_sched` | ALL slot state: status words, token/logits flags, rings, turnstile counters, D1 queue depth, D3 pressure level | | `sched->cv_gpu` (turnstile) | under `sched->mu` | one GPU forward in flight at a time (prefill chunk, bind, or decode step) | | `sched->cv_work` | under `sched->mu` | driver wake (gate may have opened) | | `sched->cv_idle` | under `sched->mu` | D1 waiter wake (slot freed or pressure dropped) | | `sl->cv` | per-slot, under `sched->mu` | driver → owner wakeup (logits/ring/finished) | Connection threads are otherwise independent: each owns its own `hs_request`, `chat_request`, `sg_result`, and `emitter`. The sampler scratch (`pn_sample_next_token`) is `_Thread_local`. The `accept()` loop is single-threaded but non-blocking per iteration. Heavy work (parsing, prefill, decode, SSE write) runs entirely in per-connection threads, so cheap routes (`/v1/models`, `/`) are never queued behind an in-flight generation. --- ## Admin API (`model_admin.c`) | Route | Body | Action | |-------|------|--------| | `POST /admin/models/load` | `{"path":"..","name":"..","dies":[..],"quantize":false}` | `mr_load` — may block ~55 s for a GPU upload | | `POST /admin/models/unload` | `{"name":".."}` | `mr_unload` — drains in-flight refs then frees | | `GET /admin/dies` | — | `pn_dm_get` snapshot of all dies (budget/used/model) | Error codes from `status_for_reason` (`model_admin.c:28`): admission/table-full → 507, duplicate/unloading → 409, not-found → 404, other → 500. A runtime `/admin/models/load` arriving while a startup upload is in flight blocks on `reg->upload_lock` and is refused promptly by the slot scan once the prior load finishes — never stacks two 2.4 GB transients. --- ## Router Registration (`serve_registry.c`) **Gating:** fully off unless `PANTHEON_REGISTRY_URL` is set. `rc_enabled()` is checked at `serve_registry_start` entry; with it unset the function returns immediately and the serve process is byte-for-byte unchanged (`serve_command.c:350`, `serve_registry.c:78`). When enabled, at startup `serve_registry_start` (`serve_registry.c:76`) iterates every loaded model (via `mr_snapshot`) and calls `rc_register` for each with: ``` model_id = info[i].name /* the shared served identity */ instance_id = host:port:model /* serve_registry.c:98 — unique per process */ endpoint = http://host:port capacity = batch-slot count (1 if single-user) ``` A background heartbeat thread then re-acquires each entry every `RW_HEARTBEAT_SECONDS` to read live load (`bs_get_load`) and POST it to the router. ### What this means for in-process replicas Two replicas of `ornith` on dies 2 and 3 in **the same process** share the same `host:port`, so they produce the same `instance_id = host:port:ornith` and **collapse to one instance** at the router. The router sees one `ornith` endpoint, not two. This is correct and intentional: **in-instance replica load-balancing is handled entirely by `mr_acquire`** (least-busy across same-name entries — see above). The router is the **multi-instance dispatch layer** for tying separate `pantheon serve` processes together. A single self-routing instance does not need it. > Cross-link: docs/README.md "Multi-Model Smart Routing" — "one instance is fully > self-routing and needs nothing in front of it." --- ## Environment Variables | Variable | Default | Effect | |----------|---------|--------| | `QWEN35_BATCH_SLOTS` | off | `1` enables multi-user batched decode (C3) | | `QWEN35_BS_DRIVER_SAMPLE` | `1` | `0` restores legacy owner-side lockstep sampling | | `QWEN35_BS_PREFILL_BATCH` | `1` | `0` disables batched chunk prefill (token-by-token) | | `QWEN35_BS_PREFILL_CHUNK` | `128` | chunk size in tokens for L2 yield windows | | `BS_QUEUE_DEPTH` | `32` | max D1 queue depth before 429 | | `QWEN35_BS_STATS` | off | `1` logs per-step timing to stderr | | `QWEN35_KV_Q4` | `1` | `0` disables Q4 KV quantization in admission math | | `QWEN35_GPU_FULL_LOGITS` | off | Refused at serve startup — forces 248k-vocab D2H copy | | `PN_ASYNC_POLL_US` | `500` | Max sleep between async forward-step event polls (µs) | | `PANTHEON_REGISTRY_URL` | unset | Set to enable router registration (`serve_registry.c:78`); unset = no-op | | `PANTHEON_NODE_ID` | `node0` | Node identifier sent in router registration (`serve_registry.c:83`) | | `PANTHEON_REGISTRY_TRANSPORT` | `local` | Transport hint sent to router on registration (`serve_registry.c:84`) | --- ## Known Sharp Edges - **`mr_unload` blocks its caller** until all in-flight `refs` drain (`model_registry.c:389`). A long generation on the model being unloaded will hold the admin response for its full duration. There is no timeout. - **Duplicate names are explicitly allowed** (`model_registry.c:119`). An operator placing the same model id on cards 4 and 5, then unloading by name, will unload whichever slot `find_locked` finds first — the other remains. There is no unload-by-slot-index API; the operator must call unload twice. - **`make_completion_id` uses `rand()`** without seeding (`openai_chat.c:87`). `rand()` is not seeded with `srand()`, so id generation is not cryptographically random and starts from the same sequence on every process restart. Collision risk is low (hex8 = 4B permutations) but completions ids are not guaranteed unique across restarts. - **`pc_lock` is allocated but never taken on the single-user path** (`openai_chat.c:674`, `pcx.pc_lock = NULL`). On the batched path each slot owns its own `hist`/`ct` structures guarded by `sched->mu`, so `pc_lock` is also unused there. The lock exists as a slot for a future concurrent-slot single-cache design. - **Stop-string detection fires owner-side** after the driver has already stepped tokens into the ring (L3 path). The driver steps until EOS/budget; the owner drains until a stop string is detected in the decoded text. Over-generation is bounded by `max_tokens` but the surplus ring tokens are silently discarded (`serve_generation.c:930-942`). This means `completion_tokens` in the usage field may be slightly lower than the number of steps the driver ran. - **GPU lazy init hangs if called too early** (`batch_sched.c:714`). GPU lane registration (`bs_lazy_init_gpu`) is deferred to the first `bs_slot_bind` to avoid a CUDA context timing issue right after a long weight upload. If the context is still unsettled when the first request arrives, the bind will hang or fail. The scheduler is fail-soft here (bind failure → -1 → request gets a 500), but a burst of concurrent first requests could all fail before the context settles. - **The `reg->lock` is brief but coarse** during `mr_load`'s slot reservation phase (`model_registry.c:123`). A concurrent load + multiple request threads all briefly contend on the same mutex. In practice this is negligible (the critical sections are O(MR_MAX_MODELS) array scans at most), but worth noting for any future lock-free refactor. - **Cross-replica session affinity is not implemented.** `mr_acquire` selects the least-busy replica by `refs` with no stickiness (`model_registry.c:329-341`). A multi-turn session with `session_id` can hop replicas between turns, missing its prefix cache on the new replica and paying a full reprefill. The single-die batched path has slot-resident affinity (`bs_slot_acquire_biased`), but that only applies within one replica. Cross-replica affinity is designed but not yet shipped. - **`QWEN35_GPU_FULL_LOGITS` is a hard startup guard** (`serve_command.c:279`). Setting it causes the server to refuse to start with an explicit error. It is a debug knob that forces the full 248k-vocab logit tensor D2H per step and caps throughput on the K80 host link (~0.4 GB/s). The guard is intentional and correct; do not remove it.