Pantheon/docs/reference/env-flags.md

Pantheon — Environment Flag Reference

Every getenv() call in src/ and foundry/ is listed here. Flags are grouped by prefix. All flags are process-constant unless noted: they are read once into a static cache on first call and never re-read.

---

QWEN35_* — Qwen3.5-MoE model flags

Core KV / context window

Flag	Values	Default	Parse	Effect	Source
QWEN35_KV_Q4	0 or anything	ON	env[0]=='0' disables	KV-cache format: Q4_0 (~4–7× smaller) vs fp32. Default changed to ON (B1). Only =0 disables.	qwen35moe_forward.c:51, engine.c:430
QWEN35_MAX_SEQ	integer	4096	atoi, clamped [16, 262144]	KV-window size per attention layer. Memory scales linearly. Ceiling PN_QWEN35MOE_MAX_SEQ_CAP=262144.	qwen35moe_forward.c:81, qwen35moe_forward_cuda.c:621, engine.c:454
QWEN35_KV_Q4_WIN	integer 0–256	128	atoi, clamped [0, 256]	fp16 recent-window exemption size inside the Q4 KV cache. 0 disables the window (pure Q4).	qwen35moe_kv_q4.h:54
QWEN35_KV_Q4_SINK	integer 0–8	4	atoi, clamped [0, 8]	fp16 attention-sink prefix kept exempt from Q4 quantization.	qwen35moe_kv_q4.h:67

GPU placement / split

Flag	Values	Default	Parse	Effect	Source
QWEN35_GPU_SPLIT_LAYER	integer	40 (GPU_DIE_SPLIT_LAYER)	atoi, must be >0; ignored when explicit 1-die placement	Layer boundary between die A and die B. Layers [0, split) → die A; [split, n_layer) → die B. Ignored when explicit single-die placement is authoritative.	qwen35moe_forward_cuda.c:702
QWEN35_GPU_DIE_A	integer (CUDA ordinal)	0	atoi	Die A CUDA device ordinal. Superseded by explicit --dies placement (warning printed).	qwen35moe_forward_cuda.c:732
QWEN35_GPU_DIE_B	integer (CUDA ordinal)	1	atoi	Die B CUDA device ordinal. Same supersession rule as QWEN35_GPU_DIE_A.	qwen35moe_forward_cuda.c:733
QWEN35_GPU_MAX_LAYERS	integer	unset (= all layers)	atoi, must be >0	Single-die only: upload/validate only the first N global layers. Ignored in 2-die mode. Also disables async scheduling (QWEN35_ASYNC_SCHED).	qwen35moe_forward_cuda.c:1407, 7164
QWEN35_GPU_BARSWAP	any	unset	presence check (== NULL → small-BAR)	Legacy barswap (single MMIO slot) vs small-BAR mode (all dies live). Default is small-BAR (no barswap). Set to any non-empty value to restore the old barswap 2-die path.	qwen35moe_forward_cuda.c:1416
QWEN35_DROP_WEIGHTS	0 or unset	unset (= drop)	env[0]=='0' opts out	After GPU upload, call madvise(MADV_DONTNEED) on the host weight mmap to free its page cache (~15 GB for a 35B MoE). On by default (qwen35 and llama CUDA paths); =0 keeps the pages for host-fallback debugging. Host-fallback matvec refaults clean pages from disk.	qwen35moe_forward_cuda.c:1935, llama_forward_cuda.c:211

Batched decode (C3 scheduler)

Flag	Values	Default	Parse	Effect	Source
QWEN35_BATCH_SLOTS	1 or anything	OFF	env[0]=='1' — only the character '1' enables	Enable multi-user batched decode. Any other value (including =8, =4, =on) silently keeps batching OFF.	batch_sched.c:51
QWEN35_BS_STATS	1 or anything	OFF	env[0]=='1'	Print per-step driver timing to stderr (M6 instrumentation).	batch_sched.c:110
QWEN35_BS_DRIVER_SAMPLE	0 or anything	ON	env[0]=='0' disables	In-driver sampling (default ON). Set =0 to restore legacy owner-side lockstep (each request thread samples its own token).	batch_sched.c:651
BS_QUEUE_DEPTH	integer >0	32	atoi	Per-model overflow queue depth (D1). Requests queue here when all slots are busy before getting 429.	batch_sched.c:725
QWEN35_BS_PREFILL_BATCH	0 or anything	ON	env[0]=='0' disables	Enable batched prefill (one GPU pass per chunk, ~2× vs token-by-token). =0 restores the sequential token loop.	serve_generation.c:471
QWEN35_BS_PREFILL_CHUNK	integer >=0	128	atoi; <0 → 0	Prefill chunk size in tokens. 0 = entire prompt in one pass (unbounded scratch).	serve_generation.c:482

Logits / sampling

Flag	Values	Default	Parse	Effect	Source
QWEN35_GPU_FULL_LOGITS	any	unset	presence check	Force full n_vocab (248K) D2H logits copy per token. Serve refuses to start if this is set — it cripples batched decode throughput (K×992 KB/step over host link). Debug/eval only.	serve_command.c:279, qwen35moe_forward_cuda.c:2747
QWEN35_GPU_TOPK	integer 0–256	40	atoi, clamped [0, FC_TOPK_MAX=256]	On-die top-K reduction before D2H. 0 or forced by QWEN35_GPU_FULL_LOGITS → full copy.	qwen35moe_forward_cuda.c:2751

CUDA-graph / fusion

Flag	Values	Default	Parse	Effect	Source
QWEN35_GRAPH	0 or anything	ON	env[0]=='0' disables	Master gate for CUDA-graph replay of SSM layers. Disabling also disables QWEN35_GRAPH_FULL and QWEN35_GRAPH_SLOTS.	qwen35moe_forward_cuda.c:1899
QWEN35_GRAPH_FULL	0 or anything	ON	env[0]=='0' disables	Full-step segment graphs (C1): capture every layer of a die segment (SSM+attention interleaved) into one graph, replay with one cuGraphLaunch. Requires QWEN35_GRAPH.	qwen35moe_forward_cuda.c:1916
QWEN35_GRAPH_SLOTS	0 or anything	ON	env[0]=='0' disables	Per-SSM-layer graphs on the batched-decode (slots) path (L9). Sub-gate of QWEN35_GRAPH. Attention layers stay direct.	qwen35moe_forward_cuda.c:1931
QWEN35_GRAPH_CAPTURE_FAIL_INJECT	int N	0 (off)	atoi(env)	Test only. Forces the first N capture attempts of EVERY graph slot to be treated as failures, so the bounded-retry path can be exercised on real hardware. N < 5 exercises recovery, N >= 5 exercises the permanent give-up. Never set on a serving process.	gpu_graph_capture_tracker.c:pn_graph_capture_inject_failures
QWEN35_FUSE	hex bitmask	0x3F (all on)	strtol(env, NULL, 0) (hex/decimal)	Fusion-site bitmask: bit0=ssm proj4, bit1=ssm out+res, bit2=attn qkv, bit3=attn norm+rope, bit4=moe pre, bit5=moe post. Set specific bits =0 to bisect regressions.	qwen35moe_forward_cuda.c:1887

Flash attention

Flag	Values	Default	Parse	Effect	Source
QWEN35_FLASH	0 or anything	ON	env[0]=='0' disables	Flash attention for batched prefill. NOT bit-exact vs host softmax (~1e-6 relative drift). =0 restores host-softmax path.	qwen35moe_forward_cuda.c:3257
QWEN35_FLASH_DECODE	0 or anything	ON	env[0]=='0' disables	Flash attention for single-token decode. =0 restores host-softmax decode path.	qwen35moe_forward_cuda.c:3406
QWEN35_FLASH_DECODE_KPS	integer >=16	PN_QWEN35MOE_DEC_KEYS_PER_SPLIT	atoi, floor 16	Keys per decode split (tune for multi-split combine path in tests).	qwen35moe_forward_cuda.c:3419
QWEN35_FLASH_SLOTS	0 or anything	ON	env[0]=='0' disables	Flash attention on the batched-decode (slots) path (L7). =0 → host-softmax slots path.	qwen35moe_forward_cuda.c:4917
QWEN35_ATTN_WINDOW	integer >0	0 (full)	strtol	StreamingLLM sliding-window attention recent-window size. 0/unset = full attention. Applies to all 10 global attention layers.	qwen35moe_forward_cuda.c:3272
QWEN35_ATTN_SINK	integer >=0	4	strtol	StreamingLLM attention-sink prefix size (used with QWEN35_ATTN_WINDOW).	qwen35moe_forward_cuda.c:3283

2-die pipeline / threading

Flag	Values	Default	Parse	Effect	Source
QWEN35_ASYNC_SCHED	0 or anything	ON	env[0]=='0' disables	Non-blocking async dispatch for single-token decode (C2). =0 restores fully synchronous path. Automatically disabled when QWEN35_GPU_MAX_LAYERS is set or LM head needs host fallback.	qwen35moe_forward_cuda.c:7074
QWEN35_ASYNC_KVMIRROR	any non-0	OFF	env[0] && env[0]!='0'	Defer host-KV-mirror stores in batched decode (L8). Experimental.	qwen35moe_forward_cuda.c:5017
QWEN35_TWO_THREAD	any non-0	OFF	env[0] && env[0]!='0'	Two-die two-thread decode (small-BAR, 2 dies, nslots≥1).	qwen35moe_forward_cuda.c:5965
QWEN35_DIE_PIPELINE	any non-0	OFF	env[0] && env[0]!='0'	Software-pipeline two batch groups across the die pair so both dies compute simultaneously (L5). Requires 2 dies + grouped MoE + nslots≥2.	qwen35moe_forward_cuda.c:5979
QWEN35_VOCAB_SPLIT	any non-0	OFF	env[0] && env[0]!='0'	Split LM head across 2 dies (T3). Only useful in 2-die non-barswap mode with peer access.	qwen35moe_forward_cuda.c:5991

SSM

Flag	Values	Default	Parse	Effect	Source
QWEN35_SSM_2PASS	any non-0	OFF	env[0] && env[0]!='0'	Route SSM recurrence to fused-reduction (2-pass) kernel: one LDS sweep, 3×S shared, slightly lossy by reassociation. Default OFF (bit-exact baseline).	foundry/cuda/fc_cuda.c:3375

Debug / profiling

Flag	Values	Default	Parse	Effect	Source
QWEN35_DEBUG	any	unset	presence check	Enable per-layer activation dumps to stderr. Also forces full-logits D2H (gpu_topk_k() returns 0).	qwen35moe_forward.c:106, qwen35moe_forward_cuda.c:991
QWEN35_PROF	any	unset	presence check	Per-token wall-clock profiler: accumulates matvec time by quant type and forward phase, printed per token. Adds fc_sync around timed ops.	qwen35moe_forward_cuda.c:794
QWEN35_SPROF	any	unset	presence check	Per-step batch-decode profiler: phase breakdown printed per step, cumulative summary every 10 steps.	qwen35moe_forward_cuda.c:930

---

PN_* — Pantheon server / memory flags

Flag	Values	Default	Parse	Effect	Source
PN_THREADS	integer	0 (auto)	atoi	CPU thread-pool size for USE_THREADS builds. 0 = auto-detect.	engine.c:772
PN_ASYNC_POLL_US	integer 50–20000	500	atol, clamped [50, 20000]	Async step poll interval in microseconds (exponential backoff up to this cap).	serve_generation.c:422
PN_HOST_RESERVE_BYTES	integer (bytes)	max(1 GiB, 15% × MemTotal)	strtoull	Override host RAM reserve held off-limits for admission. Values ≤0 or unparseable fall back to the formula.	pn_memwatch.c:62
PN_DIE_RESERVE_BYTES	integer (bytes)	1073741824 (1 GiB)	strtoull	Override per-die VRAM reserve subtracted from each die's reported total before scheduling.	pn_memwatch.c:72
PN_PSI_FULL_REJECT	float (percent)	15.0	strtod; values ≤0 ignored	PSI-full avg10 threshold (% wall time) at/above which admission rejects new requests as CRITICAL.	pn_memwatch.c:78
PN_LADDER_DISABLE	any non-0	unset (enabled)	env[0] && env[0]!='0'	Disable the reactive degradation ladder entirely. The plan mirrors the config at all pressure levels.	pn_ladder.c:21
PN_LOAD_SPIKE_BYTES	integer (bytes)	2516582400 (~2.4 GB)	strtoull; ≤0 falls back to default	Host anon RSS budget for GPU-upload transient (measured on a Tesla K80 host: ~2.4 GB for ~55 s). Admission checks this before starting a load.	pn_admission.c:280
PN_VLM_DEBUG	any	unset	presence check	Print top-3 logits + sampled token for the first 6 steps. Diagnoses empty VLM output.	engine.c:1686
PN_GPU_PREFILL	any	unset	presence check	(llama/Metal) Upload Q4_K decoder weights + run prefill GEMMs on the GPU. Off by default (measured net loss for bulk OCR vs CPU+GPU overlap).	llama.c:2068
PN_BATCH_GPU	any	unset	presence check	(llama/Metal, batch≤32) Upload Q4_K decoder weights for GPU-based batched GEMM decode. Experimental.	llama.c:2398

---

FC_* — foundry/cuda MoE kernel tuning

Flag	Values	Default	Parse	Effect	Source
FC_MOE_DOWN_WPP	1, 2, 4, or 8	2 (FC_MOE_DOWN_WPP_DEFAULT)	atoi; only exact power-of-2 values accepted	Warps-per-pair for the cooperative MoE down-projection CUDA kernel. Values outside {1,2,4,8} are silently ignored.	foundry/cuda/fc_cuda.c:2645
FC_MOE_DOWN_THREADS	64, 128, 256, or 512	128 (FC_MOE_DOWN_THREADS_DEFAULT)	atoi; must be multiple of 32×wpp	Thread count for the MoE down kernel. Only exact values {64,128,256,512} that are a multiple of 32×wpp accepted; others silently ignored.	foundry/cuda/fc_cuda.c:2652

---

Distributed training (foundry/distributed)

Flag	Values	Default	Parse	Effect	Source
WORLD_SIZE	integer	1	atoi	Distributed training world size. Used by fd_init_from_env() in the foundry trainer — not the inference server.	foundry/distributed/distributed.c:31
RANK	integer	0	atoi	This process's rank. Same scope as WORLD_SIZE.	foundry/distributed/distributed.c:32

---

Key parsing gotchas

1. QWEN35_BATCH_SLOTS is an enable flag, not a slot count. Only env[0]=='1' enables batching. QWEN35_BATCH_SLOTS=8 leaves batching OFF. (batch_sched.c:51)

2. QWEN35_KV_Q4 default is now ON. Only =0 disables. The old opt-in value =1 still works but is redundant. Any other value (e.g. =true) also keeps it ON. (qwen35moe_forward.c:51)

3. QWEN35_GPU_FULL_LOGITS is blocked at serve startup. The server refuses to start with this set — it is not silently ignored. (serve_command.c:279)

4. QWEN35_GPU_SPLIT_LAYER is ignored when 1-die placement is active. gpu_split_layer() returns n_layer (single-die path) when n_placed==1 regardless of the env. (qwen35moe_forward_cuda.c:703)

5. QWEN35_GPU_DIE_A/B are superseded by --dies / pn_load_on placement. A warning is printed but they are not honoured. (qwen35moe_forward_cuda.c:724)

6. FC_MOE_DOWN_WPP and FC_MOE_DOWN_THREADS silently reject invalid values and keep the compiled default. No warning is emitted. (foundry/cuda/fc_cuda.c:2645–2659)

7. QWEN35_FUSE parses via strtol(env, NULL, 0) — it accepts hex (0x3f) and decimal. (qwen35moe_forward_cuda.c:1888)

8. PN_LOAD_SPIKE_BYTES=0 falls back to the 2.4 GB default, not zero — a zero budget would silently disarm the host-RAM gate. (pn_admission.c:284)