# models.conf — deploy manifest for examples/deploy.sh
#
# One model per line:
#
#   name | /path/to/model.gguf | key=value key=value ...
#
# Lines are loaded top to bottom, one at a time. Put the model you least want
# to lose FIRST — on a constrained host, bring-up stops when memory runs out,
# and everything above the stopping point is already serving.
#
# Blank lines and #-comments are ignored. The options field may be empty.
#
# Recognised options (all optional; they mirror the serve command-line flags
# and POST /admin/models/load fields — see docs/reference/http-api.md):
#
#   dies=0,1            pin to GPU device ordinals (CUDA only; relative to
#                       CUDA_VISIBLE_DEVICES). Omit to let the engine place it.
#   ctx_cap=32768       per-session context cap in tokens
#   max_sessions=4      how many named sessions may hold KV at once
#   max_concurrent=2    how many requests may decode simultaneously
#   reasoning_budget=1024   force </think> after N tokens of unclosed reasoning
#   quantize=true       INT8-quantize at load time
#
# ---------------------------------------------------------------------------

# A chat model with a bounded window and two decode slots.
chat  | /models/chat.gguf  | ctx_cap=32768 max_sessions=4 max_concurrent=2

# A small model, engine-placed, all defaults.
draft | /models/draft.gguf |

# Two replicas of the same worker on different GPUs. Same name = replicas,
# load-balanced least-busy. Requires CUDA.
# worker | /models/worker.gguf | dies=2 ctx_cap=8192
# worker | /models/worker.gguf | dies=3 ctx_cap=8192