36 lines
1.6 KiB
Plaintext
36 lines
1.6 KiB
Plaintext
# models.conf — deploy manifest for examples/deploy.sh
|
|
#
|
|
# One model per line:
|
|
#
|
|
# name | /path/to/model.gguf | key=value key=value ...
|
|
#
|
|
# Lines are loaded top to bottom, one at a time. Put the model you least want
|
|
# to lose FIRST — on a constrained host, bring-up stops when memory runs out,
|
|
# and everything above the stopping point is already serving.
|
|
#
|
|
# Blank lines and #-comments are ignored. The options field may be empty.
|
|
#
|
|
# Recognised options (all optional; they mirror the serve command-line flags
|
|
# and POST /admin/models/load fields — see docs/reference/http-api.md):
|
|
#
|
|
# dies=0,1 pin to GPU device ordinals (CUDA only; relative to
|
|
# CUDA_VISIBLE_DEVICES). Omit to let the engine place it.
|
|
# ctx_cap=32768 per-session context cap in tokens
|
|
# max_sessions=4 how many named sessions may hold KV at once
|
|
# max_concurrent=2 how many requests may decode simultaneously
|
|
# reasoning_budget=1024 force </think> after N tokens of unclosed reasoning
|
|
# quantize=true INT8-quantize at load time
|
|
#
|
|
# ---------------------------------------------------------------------------
|
|
|
|
# A chat model with a bounded window and two decode slots.
|
|
chat | /models/chat.gguf | ctx_cap=32768 max_sessions=4 max_concurrent=2
|
|
|
|
# A small model, engine-placed, all defaults.
|
|
draft | /models/draft.gguf |
|
|
|
|
# Two replicas of the same worker on different GPUs. Same name = replicas,
|
|
# load-balanced least-busy. Requires CUDA.
|
|
# worker | /models/worker.gguf | dies=2 ctx_cap=8192
|
|
# worker | /models/worker.gguf | dies=3 ctx_cap=8192
|