David
/
Pantheon
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Initial public release

This commit is contained in:
David Herrera 2026-10-20 01:29:27 -05:00
commit 0547505bab
477 changed files with 169693 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

47
.cla/signatures.md Normal file
View File

@ -0,0 +1,47 @@
# Pantheon CLA signatures
This file is the signature roster for the Pantheon Contributor License
Agreement. Adding your entry here, in a pull request signed off by you, **is
your signature** — see [`CLA.md`](../CLA.md) for the agreement itself and
[`CLA-CORPORATE.md`](../CLA-CORPORATE.md) for organizations.
Do not edit or remove anyone else's entry. Entries are append-only; the git
history of this file is the record of who agreed, to what, and when.
The CI check in `.github/workflows/contributions.yml` matches the **`git-email`**
values below against the author address on every commit in a pull request. If
you commit under more than one address, list each on its own `git-email:` line.
---
## Individual signatures
```yaml
- name: David (Corey) Herrera
git-email: coreyh55@gmail.com
forge-username: dcherrera
commit-name: Zero # git author name on this project's history — same person
role: project owner and copyright holder
date: 2026-08-11
agreement: ICLA v1.0
statement: I have read the Pantheon ICLA and I agree to it.
```
<!-- Add your entry above this line, following the format exactly. -->
## Corporate signatures
Organizations sign by email (see `CLA-CORPORATE.md`); their entry is added here
after countersignature, with one `git-email:` line per authorized contributor
from Schedule A.
```yaml
# - organization: Example, Inc.
# agreement: CCLA v1.0
# signed: YYYY-MM-DD
# schedule-a-updated: YYYY-MM-DD
# contributors:
# - name: Contributor Name
# git-email: contributor@example.com
# forge-username: their-username
```

23
.github/PULL_REQUEST_TEMPLATE.md vendored Normal file
View File

@ -0,0 +1,23 @@
## What this changes
<!-- One logical change. A few lines — what, and why. Not an essay.
If this is over 1,000 reviewable lines or 15 files, say why it can't be split. -->
## Evidence
<!-- Forward pass or kernel: parity against the reference path (CPU, or an existing
oracle) — argmax-exact, on a named model and quant. Performance: the command, the
device, before/after in tok/s, and the noise floor. Untested? Say so. -->
## Checklist
- [ ] **I license this contribution under the CLA in [`CLA.md`](../CLA.md)** — my entry
is in [`.cla/signatures.md`](../.cla/signatures.md). I keep my copyright.
- [ ] Every commit signed off (`git commit -s`) — `./scripts/check-dco.sh`
- [ ] Builds clean: `make USE_BLAS=1 pantheon`, zero warnings under `-Wall -Wextra`
- [ ] `make USE_BLAS=1 test` passes, and new functionality has tests
- [ ] New files carry the SPDX pair and the `@file` header block
- [ ] Docs updated if the public API changed
<!-- Contributing on behalf of an employer that owns your work? Your organization
signs CLA-CORPORATE.md by email instead — see that file. -->

43
.github/workflows/contributions.yml vendored Normal file
View File

@ -0,0 +1,43 @@
name: contributions
# DCO sign-off + CLA roster check on every pull request.
# The same scripts run locally: scripts/check-dco.sh, scripts/check-cla.sh
#
# Path note: Gitea/Forgejo Actions read .gitea/workflows first and fall back to
# .github/workflows, so this one file serves both forges.
on:
pull_request:
branches: [main]
permissions:
contents: read
jobs:
dco-and-cla:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
ref: ${{ github.event.pull_request.head.sha }}
- name: Fetch base branch
env:
BASE: ${{ github.event.pull_request.base.ref }}
run: git fetch --no-tags origin "+refs/heads/$BASE:refs/remotes/origin/$BASE"
- name: DCO sign-off
env:
BASE: ${{ github.event.pull_request.base.ref }}
run: ./scripts/check-dco.sh "origin/$BASE..HEAD"
- name: CLA signature
env:
BASE: ${{ github.event.pull_request.base.ref }}
run: ./scripts/check-cla.sh "origin/$BASE..HEAD"
- name: File headers on added sources
env:
BASE: ${{ github.event.pull_request.base.ref }}
run: ./scripts/check-headers.sh "origin/$BASE..HEAD"

165
.gitignore vendored Normal file
View File

@ -0,0 +1,165 @@
# Binaries.
# `pantheon` is now a SYMLINK to the per-backend binary the last build produced
# (Makefile: PN_BIN / PN_COMPAT_BIN); each backend also writes its own file, so
# a TT build can no longer overwrite a deployed CUDA one. All of them are
# ignored, and the combination names too, so a `make USE_CUDA=1 USE_TT=1` tree
# stays clean.
pantheon
pantheon-cpu
pantheon-cuda
pantheon-tt
pantheon-metal
pantheon-cuda-tt
pantheon-cuda-metal
pantheon-tt-metal
pantheon-cuda-tt-metal
tests/test_backend_guard
tests/test_build
tests/test_graph_capture_concurrency
tests/test_attn_ring_retry
tests/test_graph_capture_retry
tests/test_gguf
tests/test_arena
tests/test_qmv
tests/test_simd
tests/test_chat
tests/test_chat_request
tests/test_tool_calling
tests/test_tool_call_defects
tests/test_tool_call_truncation
tests/test_empty_argument
tests/test_frame_capture
tests/test_tool_grammar
tests/test_tool_grammar_json
tests/test_grammar_spec_chooser
tests/test_grammar_mask_shortlist
tests/test_grammar_topn_sampler
tests/test_serve_emitter
tests/test_tool_call_stream
tests/test_repetition_loop_detector
tests/test_registry
tests/test_base64
tests/test_session_affinity
tests/test_session_watch
tests/test_serve_shutdown
tests/test_bs_load
tests/test_sampler_parity
tests/test_bpe_pretokenize
tests/test_rwkvx
# Debug symbols
*.dSYM/
# Object files
*.o
# macOS
.DS_Store
# TeamIDE: only what its PM reads from git is tracked -- project.yml and the
# tasks/plans/notes/agents markdown (TeamIDE reads the kanban via `git ls-tree`,
# so a blanket .teamide/ rule would hide it). Everything else under .teamide/ is
# machine- or session-specific: subrepos.json (per-user clone URLs), chat
# history, activity log, agent runs and reports, caches.
.teamide/*
!.teamide/project.yml
!.teamide/tasks/
!.teamide/plans/
!.teamide/notes/
!.teamide/agents/
# qwen35moe test binaries (sources tracked, binaries not)
tests/test_qwen35moe
tests/test_qwen35moe_metal_coverage
tests/test_qwen35moe_metal_coverage_fake
# qwen35moe golden-vector generator binary (source + generated goldens tracked)
tests/gen_golden_qwen35
# Accelerator-parity harness binaries (sources tracked, binaries not)
tests/test_chat_parity_gate
tests/test_prefix_projection
# qwen35moe GPU-vs-CPU parity harness binary (source tracked, binary not)
tests/test_kernel_parity
# 2-die barswap context-swap smoke test binary (source tracked, binary not)
tests/test_2die_swap
tests/bench_batch_decode
tests/test_batch_decode_correct
tests/test_batch_decoder
# quality-eval driver binary + run artifacts (source, prompts, corpus tracked)
tests/eval_ppl
evals/out/
# MemoryAware unit-test binary (source tracked, binary not)
tests/test_memwatch
tests/test_die_manager
tests/test_model_registry
tests/test_model_unload_reclaim
# A4 embed-row unit-test binary (source tracked, binary not)
tests/test_embed_row
# Serve-scheduler + PC-batched gate binaries (source tracked, binary not)
tests/test_serve_sched_qwen35
tests/test_serve_sched_prefix_cache
tests/test_prefix_cache_qwen35
tests/bench_serve_prefix_cache
tests/test_prefix_cache_sys
# Local agent/editor notes; deliberately untracked.
CLAUDE.md
# Compiled test binaries (source tracked, binary not)
tests/test_response_format_grammar
tests/test_sampling_modifiers
# Build artifacts (were in .git/info/exclude, which is local-only and never cloned)
/libpantheon.a
/build/
# Model weights — multi-GB GGUFs land here. Never track.
/models/
# Foundry: the public tree carries it as a git submodule at ./foundry; a private
# checkout may use a local symlink there instead, which must never be tracked.
# The Makefile finds Foundry at FOUNDRY=<path>, else ./foundry, else ../foundry.
# Plans, specs and benchmark write-ups live outside this repo. Symlinked in
# for local convenience; never tracked or published here.
/plans
__pycache__/
tests/bench_sampler_temp_delta
tests/test_sparse_sampler_identity
tests/test_topn_row_exact
tests/test_grammar_arm_window
tests/test_gdn_layout
tests/test_tt_column_runs
tests/test_qwen35_ckpt_band
# Compiled test binaries. This used to be one line per test, so every NEW test
# was untracked until someone remembered to add it -- and a `git add tests/`
# then swept 2.8 MB of build artifacts into a commit. Pattern, not a roll-call.
tests/test_*
tests/bench_*
tests/gen_golden_*
!tests/*.c
!tests/*.h
# Parity-gate dumps and logs (L11 Mac CPU oracle): large, regenerable
tests/parity_out/
# Claude Code hooks are generated per machine by TeamIDE (they list local paths).
.claude/hooks/
# bench_ngram_accept --generate output (recorded token streams); make your own.
tests/data/ngram_accept_dumps/
# checkpoint-ceiling round captures (Mac side)
tests/serve_prefix_think_ckpt/out/
# toolcall stream probe captures
tests/serve_toolcall_stream/out/
# required-with-thinking proof captures
tests/serve_required_thinking/out/
tests/serve_required_thinking/probe_bin/

3
.gitmodules vendored Normal file
View File

@ -0,0 +1,3 @@
[submodule "foundry"]
path = foundry
url = https://git.teamide.dev/David/Foundry.git

150
CLA-CORPORATE.md Normal file
View File

@ -0,0 +1,150 @@
# Pantheon Corporate Contributor License Agreement
**Version 1.0**
Use this agreement if contributions to Pantheon are being made **by employees or
contractors on behalf of an organization** that owns the resulting intellectual
property. Individuals contributing on their own behalf should use
[`CLA.md`](./CLA.md) instead.
This agreement mirrors the Individual CLA. Its purpose is to make sure that the
entity that actually owns the work has granted the rights, so that neither of us
is relying on an individual employee's authority to license their employer's IP.
**The organization keeps its copyright.** This is a license, not an assignment.
---
## 1. Definitions
**"I"**, **"me"**, **"my"** — David (Corey) Herrera, the copyright holder of Pantheon,
and my successors and assigns.
**"You"**, **"your"** — the organization signing this agreement, together with
all entities that control, are controlled by, or are under common control with
it. "Control" means ownership of more than 50% of outstanding shares, or the
power to direct the entity's management.
**"Contribution"** — any original work of authorship, including any modification
or addition to an existing work, intentionally submitted to me for inclusion in
Pantheon by you or by anyone authorized to submit on your behalf, except
anything conspicuously marked **"Not a Contribution."**
## 2. Copyright license
You grant me a perpetual, worldwide, non-exclusive, no-charge, royalty-free,
irrevocable copyright license to reproduce, prepare derivative works of,
publicly display, publicly perform, sublicense, and distribute your Contribution
and derivative works of it.
## 3. Patent license
You grant me a perpetual, worldwide, non-exclusive, no-charge, royalty-free,
irrevocable (except as stated in this section) patent license to make, have
made, use, offer to sell, sell, import, and otherwise transfer your
Contribution. This license covers only those patent claims licensable by you
that are necessarily infringed by your Contribution alone, or by the combination
of your Contribution with Pantheon.
If any entity institutes patent litigation alleging that your Contribution, or
Pantheon itself, constitutes direct or contributory patent infringement, then
every patent license granted to that entity under this agreement terminates as
of the date the litigation is filed.
## 4. Outbound licensing — what I may do with your Contribution
The licenses in sections 2 and 3 include the right to license and sublicense
your Contribution to others under **any terms I choose**: the project's current
outbound license (AGPL-3.0-or-later), a different open-source license if the project's
outbound license changes, or **proprietary or closed-source terms**.
I may also **assign or transfer** these rights, in whole or in part, together
with the project or with the copyrights in it, to a successor, acquirer, or
foundation, who then holds them on these same terms. A sale, merger, or
relicensing of Pantheon does not require me to come back to you for permission.
I am not obligated to use your Contribution, and I may modify or remove it at
any time.
## 5. Your representations
You represent that:
1. **You own it and can license it.** Each Contribution is an original creation
owned by you, and the person signing below is authorized to bind you to this
agreement.
2. **Your people are authorized.** The individuals listed in Schedule A are
authorized to submit Contributions on your behalf. You will keep that list
current, and you remain responsible for Contributions submitted by anyone
you have authorized, whether or not they are listed.
3. **No hidden third-party code.** Contributions do not knowingly include
anyone else's code, except as identified under section 6.
4. **You'll tell me** if any of the above stops being true.
## 6. Third-party submissions
Work that is not your original creation may be submitted only if it is
identified separately in the pull request, retains its original copyright and
license notices, and is conspicuously marked
`Submitted on behalf of a third party: <author> — license: <SPDX id>`. I may
decline it. **GPL- or AGPL-only third-party code cannot be accepted**, because
it would prevent the project from being relicensed in the future.
## 7. No warranty, no obligation
Contributions are provided **"AS IS"**, without warranty or condition of any
kind, express or implied, to the maximum extent permitted by law. You are not
expected to provide support, maintenance, or updates.
## 8. Governing law
This agreement is governed by the laws of the State of Missouri, United States,
without regard to its conflict-of-law rules.
## 9. Entire agreement
This is the entire agreement between us about your Contributions, and it
supersedes any earlier understanding on the subject. If any part of it is held
unenforceable, the rest stays in force.
---
## How to sign
Corporate signatures are handled by email rather than by pull request, because
they need a signature from someone authorized to bind the organization.
1. Email **coreyh55@gmail.com** with the subject `Pantheon CCLA — <organization>`,
including the completed block below and Schedule A. A scanned signature, a
digital signature, or a reply from an authorized signer's corporate email
address all work.
2. Once countersigned, I add an entry for your organization to
[`.cla/signatures.md`](./.cla/signatures.md), listing each authorized
contributor's commit email so CI can verify their pull requests.
3. Your contributors still sign off every commit with the DCO (`git commit -s`)
— see [`CONTRIBUTING.md`](./CONTRIBUTING.md).
```
Organization: ____________________________________________
Mailing address: ____________________________________________
Point of contact: ____________________________________________
Contact email: ____________________________________________
Authorized signer
Name: ____________________________________________
Title: ____________________________________________
Signature: ____________________________________________
Date: ____________________________________________
Agreement: Pantheon CCLA v1.0
```
### Schedule A — authorized contributors
| Full name | Commit email | Forge username |
|-----------|--------------|----------------|
| | | |
To add or remove people later, email an updated Schedule A. Contributions from
someone not covered by a current Schedule A will be held until the list is
updated.

155
CLA.md Normal file
View File

@ -0,0 +1,155 @@
# Pantheon Individual Contributor License Agreement
**Version 1.0**
Thank you for contributing to Pantheon.
This agreement lets me include your contribution in Pantheon and distribute it
under the project's licenses — the one it ships under today (AGPL-3.0-or-later) and any license the
project moves to later.
**You keep your copyright.** This is a license, not an assignment. Your
contribution stays yours: you can use it, publish it, sell it, or license it to
anyone else, anywhere, forever, on any terms you like.
Sign once — see [How to sign](#how-to-sign) at the end. It covers everything you
contribute to Pantheon, before and after signing.
---
## 1. Definitions
**"I"**, **"me"**, **"my"** — David (Corey) Herrera, the copyright holder of Pantheon,
and my successors and assigns.
**"You"**, **"your"** — the individual who signs this agreement. If you are
signing on behalf of an organization, use
[`CLA-CORPORATE.md`](./CLA-CORPORATE.md) instead.
**"Contribution"** — any original work of authorship, including any modification
or addition to an existing work, that you intentionally submit to me for
inclusion in Pantheon. "Submit" means any form of electronic, verbal, or written
communication sent to me or to the project — pull requests, patches, issue
comments, mailing-list or chat messages — except anything you conspicuously mark
**"Not a Contribution."**
## 2. Copyright license
You grant me a perpetual, worldwide, non-exclusive, no-charge, royalty-free,
irrevocable copyright license to reproduce, prepare derivative works of,
publicly display, publicly perform, sublicense, and distribute your Contribution
and derivative works of it.
## 3. Patent license
You grant me a perpetual, worldwide, non-exclusive, no-charge, royalty-free,
irrevocable (except as stated in this section) patent license to make, have
made, use, offer to sell, sell, import, and otherwise transfer your
Contribution. This license covers only those patent claims licensable by you
that are necessarily infringed by your Contribution alone, or by the combination
of your Contribution with Pantheon.
If any entity institutes patent litigation alleging that your Contribution, or
Pantheon itself, constitutes direct or contributory patent infringement, then
every patent license granted to that entity under this agreement terminates as
of the date the litigation is filed.
## 4. Outbound licensing — what I may do with your Contribution
The licenses in sections 2 and 3 include the right to license and sublicense
your Contribution to others under **any terms I choose**. Specifically:
- under the project's current outbound license — AGPL-3.0-or-later;
- under a different open-source license, if the project's outbound license
changes;
- under **proprietary or closed-source terms**, as part of a commercial license
or a product.
I may also **assign or transfer** the rights you grant here, in whole or in
part, together with the project or with the copyrights in it — to a successor,
an acquirer, or a foundation. Whoever holds them then holds them on these same
terms. A sale, merger, or relicensing of Pantheon does not require me to come
back to you for permission.
I am not obligated to use your Contribution, and I may modify or remove it at
any time.
## 5. Your representations
You represent that:
1. **It's yours.** Each Contribution is your original creation and you are
legally entitled to grant the licenses above.
2. **Your employer is handled.** If your employer has rights to intellectual
property you create, you have either received permission to contribute on
your own behalf, or your employer has waived those rights, or your employer
has signed the Corporate CLA.
3. **No hidden third-party code.** Your Contribution does not knowingly include
anyone else's code, except as identified under section 6.
4. **You'll tell me** if any of the above stops being true.
## 6. Third-party submissions
You may submit work that is not your original creation only if you:
- identify it separately in the pull request,
- keep its original copyright and license notices intact, and
- mark it conspicuously as
`Submitted on behalf of a third party: <author> — license: <SPDX id>`.
I may decline it. Whether it can be accepted at all depends on its license
permitting commercial distribution alongside the rest of the tree — **GPL- or
AGPL-only third-party code cannot be accepted**, because it would prevent the
project from being relicensed in the future.
## 7. No warranty, no obligation
Your Contribution is provided **"AS IS"**, without warranty or condition of any
kind, express or implied, to the maximum extent permitted by law. You are not
expected to provide support, maintenance, or updates for it.
## 8. Governing law
This agreement is governed by the laws of the State of Missouri, United States,
without regard to its conflict-of-law rules.
## 9. Entire agreement
This is the entire agreement between us about your Contributions, and it
supersedes any earlier understanding on the subject. If any part of it is held
unenforceable, the rest stays in force.
---
## How to sign
**1. Add yourself to [`.cla/signatures.md`](./.cla/signatures.md)** as part of
your first pull request:
```yaml
- name: Your Full Name
git-email: you@example.com
forge-username: your-username
date: YYYY-MM-DD
agreement: ICLA v1.0
statement: I have read the Pantheon ICLA and I agree to it.
```
Use the **same email address you commit with** — the CI check matches on it. If
you commit under more than one address, list each on its own `git-email:` line.
**2. Sign off every commit** with the DCO (`git commit -s`) — see
[`CONTRIBUTING.md`](./CONTRIBUTING.md).
That pull request, carrying your entry and signed off by you, **is your
signature**. The git history is the record: who agreed, to what version, and
when. Nothing to print, scan, or email.
If you would rather not sign, open an issue describing your change instead of a
pull request. An idea costs nothing to accept.
---
*Questions about this agreement: coreyh55@gmail.com. If something here is
ambiguous, ask rather than guessing — I would rather clarify it than have
someone rely on a reading I did not intend.*

177
CONTRIBUTING.md Normal file
View File

@ -0,0 +1,177 @@
# Contributing to Pantheon
Thanks for your interest in Pantheon. This document covers the basics.
## Licensing of contributions — read this first
Pantheon ships under **AGPL-3.0** (see [`LICENSE`](./LICENSE)).
> **Contributions are made under the Contributor License Agreement in
> [`CLA.md`](./CLA.md)**, which grants me a broad copyright and patent license,
> including the right to sublicense your contribution under any terms and to
> transfer that right along with the project.
**You keep your copyright.** The CLA is a license, not an assignment — nothing
is transferred away from you. Your contribution stays yours to use, publish,
sell, or license to anyone else, anywhere, forever.
Signing takes one commit: add your entry to
[`.cla/signatures.md`](./.cla/signatures.md) in your first pull request. Nothing
to print, scan, or email. Organizations whose employees contribute on company
time sign [`CLA-CORPORATE.md`](./CLA-CORPORATE.md) by email instead.
If you would rather not sign, open an issue describing the change instead of a
pull request. An idea costs nothing to accept.
### Sign your commits (DCO)
Every commit must carry a `Signed-off-by` line certifying the
[Developer Certificate of Origin](https://developercertificate.org/) — that you
wrote the code, or otherwise have the right to submit it:
```bash
git commit -s -m "your message"
```
which appends:
```
Signed-off-by: Your Name <your.email@example.com>
```
Use your real name and a working email address — the same address as your CLA
roster entry, since that is what CI matches on. If you are contributing work your
employer owns, confirm you actually have the right to do so before you sign off.
Both checks run in CI on every pull request, and you can run them yourself first:
```bash
./scripts/check-dco.sh # every commit signed off, matching its author
./scripts/check-cla.sh # every author on the CLA roster
./scripts/check-headers.sh # every source file you ADD has SPDX + the @file block
```
### License header on new files
Every source file starts with:
```c
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
```
Add it to new files you create. Do not edit the copyright line on existing files
— you keep your copyright in what you contribute without needing a line in the
header, and a per-file list of names becomes unmaintainable fast. `git log` and
`.cla/signatures.md` are the authoritative record of who wrote what.
## Getting Started
Pantheon builds on Foundry, a git submodule at `./foundry`. Clone with
`git clone --recursive https://git.teamide.dev/David/Pantheon.git`, or run
`git submodule update --init` in an existing clone. The Makefile uses, in order,
`FOUNDRY=<path>`, then `./foundry` (the submodule or a symlink), then `../foundry`
(a sibling checkout).
```bash
make USE_BLAS=1 pantheon # build the CLI
make USE_BLAS=1 test # build + run the test suites
make DEBUG=1 USE_BLAS=1 pantheon
make clean
```
| Flag | What | When |
|------|------|------|
| `USE_BLAS=1` | SGEMV/SGEMM via Accelerate or OpenBLAS | always |
| `USE_THREADS=1` | pthread pool for parallel ops | multi-core inference |
| `USE_METAL=1` | Metal GPU compute | macOS / Apple Silicon |
| `USE_CUDA=1` | CUDA via `dlopen` (cuBLAS + embedded PTX) | Linux + NVIDIA |
## Code Style
- **C11** (`-std=c11`); no dependencies beyond Foundry, libc, libm, pthreads and libdl
(BLAS and accelerator frameworks are opt-in build flags)
- **`pn_` prefix** on all Pantheon symbols
- Errors: return `NULL` or an error code — no `exit()` in library code
- Opaque types in the public API (`include/pantheon.h`); concrete structs stay
internal
- Callers free returned objects via `pn_free()`
- `snprintf`, never `sprintf`
- No warnings under `-Wall -Wextra`
Broader conventions live in `coding-best-practices.md`.
## Testing
Tests use Foundry's `lib/test.h` framework and live in `tests/`. Add a matching
target in the `Makefile`.
**If you change a forward pass or a kernel, show numerical parity.** The bar is
argmax-exact against the reference path (CPU, or an existing oracle) on a real
model — not "it produces plausible text." Include how you measured it in the PR.
## Pull Requests
Code is reviewed against
[`docs/coding-best-practices.md`](docs/coding-best-practices.md), cited by rule
number. This section covers the PR itself.
**One logical change per PR.**
**Size.** Reviewable lines = added + deleted, excluding generated files, pure
renames, and vendored imports.
| Reviewable lines | |
|---|---|
| ≤ 400 | normal |
| 400–1,000 | expect a request to split |
| > 1,000, or > 15 files | state in one line why it can't be split |
**Exempt from the size limits — but must be alone in their own PR:**
1. **Generated sources.** State the regeneration command, toolchain version, and
target arch. Reviewed by rebuilding, not by reading.
2. **Pure moves and renames.** No content change in the same commit.
3. **Mechanical sweeps.** Include the command that reproduces the diff.
4. **Vendored or backend imports.**
**Evidence.**
- **Performance claim** — the command, the device, before/after in tok/s, and the
noise floor. A multiplier on its own is not a result.
- **Forward pass or kernel** — parity against the reference path, argmax-exact or
ulp-0, naming the model and quantization.
- **A test you call green** — show it can go red (rule 8). Delete the `.o` *and*
the test binary first, or you may be running the old one.
- **Untested** — say so plainly rather than leaving it implied.
**Keep it short.** Title is `scope: imperative summary`, lowercase scope, ≤72
characters. The body says what changed and why in a few lines. Commit messages
the same: a subject line, and a body only when the reason isn't visible in the
diff. Long rationale belongs in a spec file, not a commit message. No tool,
assistant, or generator attribution anywhere — title, body, or comments.
**Review comments carry a severity prefix:**
- `blocker:` — must change before merge
- `question:` — answer before merge
- `nit:` — optional, never blocks
Unprefixed reads as `nit:`. The author does not resolve a `blocker:` thread — the
reviewer does. Approval goes stale when new commits land, unless they are a
rebase or a fixup.
**Merge bar:** builds clean with zero warnings, tests pass, DCO and CLA checks
green, and the code meets the practices doc.
## Reporting Bugs
Open an issue with:
- what you expected, and what actually happened
- minimal reproduction steps
- the model and quantization involved (many bugs are format-specific)
- your platform, compiler version, and GPU + driver if relevant
Bugs that only appear at a particular context depth, batch size, or concurrency
level are common here — please say which, since it is usually the whole clue.

661
LICENSE Normal file
View File

@ -0,0 +1,661 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.

202
LICENSES/Apache-2.0.txt Normal file
View File

@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

1286
Makefile Normal file
View File

File diff suppressed because it is too large Load Diff

54
NOTICE Normal file
View File

@ -0,0 +1,54 @@
Pantheon
Copyright (C) 2026 David (Corey) Herrera and the Pantheon contributors.
This program is free software: you can redistribute it and/or modify it under
the terms of the GNU Affero General Public License as published by the Free
Software Foundation, either version 3 of the License, or (at your option) any
later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License along
with this program. If not, see <https://www.gnu.org/licenses/>.
The full license text is in LICENSE.
SPDX-License-Identifier: AGPL-3.0-or-later
Contributions
-------------
Contributions are accepted under the Contributor License Agreement in CLA.md
(CLA-CORPORATE.md for organizations), with a Developer Certificate of Origin
sign-off on every commit. Contributors retain copyright in their contributions.
The signature roster is .cla/signatures.md.
Third-party code
----------------
Pantheon has no external dependencies. It links Foundry
(https://git.teamide.dev/David/Foundry.git), which is under the same AGPL-3.0
terms and the same copyright holder.
src/accel/tt/pn_tt_device.c uses the ARC telemetry tag numbers and names
defined by the TelemetryTag enumeration of tt-umd
(https://github.com/tenstorrent/tt-umd), (c) 2025 Tenstorrent Inc.,
licensed under the Apache License, Version 2.0. Only those numbers and names
are used; no tt-umd source code is included. A copy of the Apache License 2.0
is in LICENSES/Apache-2.0.txt.
Model weights executed by Pantheon carry their own licenses, entirely separate
from this one and entirely between you and whoever published them.
Trademarks
----------
No right is granted to use the "Pantheon" name or any associated logo to name or
promote a derivative product. Factual statements that your product uses Pantheon
are fine.
Contact
-------
coreyh55@gmail.com

344
README.md Normal file
View File

@ -0,0 +1,344 @@
# Pantheon
**An inference engine that assumes you don't have enough.**
Not enough VRAM, not enough host RAM, not enough cores, not enough context, and
not enough budget for current-generation GPUs. Pantheon is a memory-aware inference
engine and OpenAI-compatible model server built for that case: a small host, older or
mixed cards, several models per process.
It is pure C11 on top of [Foundry](https://git.teamide.dev/David/Foundry.git), its
companion tensor library (included as a git submodule). Beyond Foundry it needs only
libc, libm, pthreads and libdl; BLAS (Accelerate on macOS, OpenBLAS on Linux) is
optional, and the optional Metal backend links Apple's frameworks. Each backend builds
to a single binary.
---
## What "memory-aware" actually means
A common behavior when an engine runs out of memory is that the request fails. Pantheon
treats running out as the *normal operating condition* and degrades on a ranked ladder,
least-harmful first (`src/core/mem/pn_ladder.h`):
| Pressure | What gives |
|---|---|
| **WATCH** | force Q4 KV cache — near-zero quality cost, 4–7× VRAM win |
| **TIGHT** | halve concurrency · clamp the prefill window · evict cold KV |
| **CRITICAL** | floor concurrency to one slot — a single request always still completes |
Admission is checked before every load *and* every turn, against live host-RAM and VRAM
pressure, so the engine refuses work it cannot finish instead of accepting it and
falling over halfway. That layer (`src/core/mem/`) is about 2,100 lines of the codebase,
which is a fair measure of how central it is.
**It tells the client the truth.** A prompt past the context cap is refused with a typed
`context_over_budget` carrying the actual KV numbers, so the caller can compact and
retry. Nothing is truncated silently: for an agent built on top, silent truncation would
be silent corruption.
## What that buys you
- **Many models, one process, many GPUs.** One port hosts N models across N devices,
routed by the request's `model` field. Repeat a `--name` on another device and it
becomes a replica, load-balanced least-busy. Load and unload at runtime without a
restart. This is what lets a 2-core / 8 GB host drive several cards.
- **Portable session state.** Conversation state — recurrent SSM *and* attention KV —
serializes to a versioned blob stamped with a model-identity tag the restore side
validates. Hybrid state is hard to serialize because the recurrent half isn't
position-addressable. It means session state can live on the user's machine instead
of your disk.
- **Reasoning that terminates.** `--reasoning-budget N` forces `</think>` after N tokens,
so a reasoning model that runs away still produces an answer instead of burning your
context.
- **Embeddable.** C11, no C++ runtime, no exceptions, auditable dependency tree, links as
a static library.
## Old hardware
The CUDA backend targets **`sm_37` — Kepler, 2014**. Measured by the author on one Tesla
K80 (both GK210 dies) in a 2-core / 8 GB host: a **35B-A3B hybrid MoE served at 32K–128K
context**: Qwen3.6-35B-A3B and Nex-N2-mini (Q3_K_M) both decode at ~21 tok/s and prefill
~110-120 tok/s up to ~8k tokens ([K80 quick start](#qwen36-and-nex-n2-mini-on-a-tesla-k80),
[supported models](docs/reference/supported-models.md)).
Correctness on that path, stated precisely:
- With the fused attention kernels off (`QWEN35_FUSE=0x37`) and flash prefill off
(`QWEN35_FLASH=0`), the author measured the 2-die GPU forward's logits **bit-identical
to the CPU forward** (argmax and every logit).
- The default flash prefill is argmax-exact, **not** bit-exact (~1e-6 relative drift; see
[env flags](docs/reference/env-flags.md)).
- The in-tree CUDA gates (`tests/test_spec_qwen35`, `tests/test_users_qwen35`,
`tests/test_prefix_cache_qwen35`) check token-for-token identity between decode paths
under `QWEN35_FUSE=0x37`. They need a model and a GPU and are not part of `make test`.
Kepler is the extreme case, not the point. The point is that hardware you already own
should keep working.
---
## Quick start
Prerequisites: a C11 compiler and `make`. On Linux, `USE_BLAS=1` also needs OpenBLAS
(`apt install libopenblas-dev`); on macOS it uses Accelerate. The model download below
uses the Hugging Face CLI (`pip install -U huggingface_hub`); any way of fetching the
`.gguf` works. Build on the machine you will run on: the default flags include
`-march=native`, so a binary built on a newer CPU can fail with `Illegal instruction` on
an older one. To build a binary you can copy between x86-64 machines, add `PORTABLE=1`
(`-march=x86-64-v2`, no AVX). Per-platform details: [`docs/guides/install.md`](docs/guides/install.md).
```bash
git clone --recursive https://git.teamide.dev/David/Pantheon.git pantheon
cd pantheon && make USE_BLAS=1 pantheon
# already cloned without --recursive? git submodule update --init
hf download bartowski/SmolLM2-135M-Instruct-GGUF \
SmolLM2-135M-Instruct-Q8_0.gguf --local-dir models/
./pantheon --model models/SmolLM2-135M-Instruct-Q8_0.gguf \
--prompt "Hello, my name is" --max-tokens 32
```
Interactive chat, with the model's own template detected automatically:
```bash
./pantheon --model models/SmolLM2-135M-Instruct-Q8_0.gguf --chat
```
### Ornith 1.5 on a Tenstorrent card
Ornith 1.5 9B is the only model supported on Tenstorrent cards for now. On Linux with a
Blackhole p150a:
```bash
git clone --recursive https://git.teamide.dev/David/Pantheon.git pantheon
cd pantheon && ./install.sh # build pantheon-tt, set up card access (asks before sudo)
./run_ornith.sh # download the model (~10 GB, once), serve on 127.0.0.1:8080
```
`run_ornith.sh` fetches the files from
[dcherrera/Ornith-1.5-9B-Pantheon-TT](https://huggingface.co/dcherrera/Ornith-1.5-9B-Pantheon-TT)
and checks their sha256. The first start converts the model for the card (~10 GB more on
disk); later starts skip that. It serves an OpenAI-compatible API, model id `ornith`. Options
are in the script's header.
### Qwen3.6 and Nex-N2-mini on a Tesla K80
On Linux with a Tesla K80 and the NVIDIA 470 driver. **Supported K80 models: Qwen3.6-35B-A3B
and Nex-N2-mini (both Q3_K_M), one model per card** (both dies):
```bash
git clone --recursive https://git.teamide.dev/David/Pantheon.git pantheon
cd pantheon && ./install.sh --k80 # build pantheon-cuda (no CUDA toolkit needed)
./k80_qwen36.sh # or ./k80_nexn2.sh; downloads ~17 GB once, serves on 127.0.0.1
```
Measured on one K80 in a 2-core / 8 GB host (OpenAI streaming client, warm card, runs back to back):
| prompt | Qwen3.6 first token | Nex-N2-mini first token | decode (both) |
|---|---|---|---|
| 1k tokens | 8 s (121 tok/s) | 9 s (107 tok/s) | ~21 tok/s |
| 8k tokens | 69 s (116 tok/s) | 67 s (119 tok/s) | ~18 tok/s |
| 14k tokens | 3.0 min (79 tok/s) | 3.2 min (74 tok/s) | ~15 tok/s |
| 30k tokens | 8.4 min (60 tok/s) | 9.6 min (53 tok/s) | ~10 tok/s |
**Good for:** chat, writing, Q&A and background jobs — short prompts, long answers. Prefix
caching is on: send a `session_id`, and send each earlier assistant turn back with its
`reasoning_content`; later turns then only prefill what's new (a 4k-token conversation's second
turn: ~38 s without the cache, 2-5 s with it). opencode does both on its own: point an
`@ai-sdk/openai-compatible` provider at the server and its second turn (14k-token system prompt)
measured 9 s instead of 170 s. One session id may carry several conversations (opencode's title
request shares it); each keeps its own cache slot.
**Not good for:** pasting long documents or codebases every request; prefill is the K80's weak
spot. Options are in each script's header.
## Serving
One model, no GPU required:
```bash
./pantheon serve --model models/SmolLM2-135M-Instruct-Q8_0.gguf --port 8080
```
That's a complete OpenAI-compatible server. Point any client at
`http://127.0.0.1:8080/v1` and it works.
Everything below is optional. `--name` defaults to the file name, and `--dies` is
CUDA-only — omit it and the engine places the model itself. Repeat `--model` to serve
several from one process:
```bash
./pantheon serve \
--model models/big.gguf --name chat --ctx-cap 32768 \
--model models/small.gguf --name draft \
--port 8080
```
Clients choose with the `model` field in a normal OpenAI request body. On a multi-GPU
box you can pin placement, and repeating a `--name` on another device makes it a
**replica** that load-balances least-busy:
```bash
./pantheon serve \
--model models/big.gguf --name chat --dies 0,1 \
--model models/small.gguf --name worker --dies 2 \
--model models/small.gguf --name worker --dies 3 \
--port 8080
```
Two `worker` requests then occupy devices 2 and 3 concurrently, with nothing in front of
the process doing the routing.
### Bringing up more than a couple of models
On a constrained host, launching every model at once is how you OOM. The pattern that
works is to start `serve` with one model and add the rest through
[`POST /admin/models/load`](docs/reference/http-api.md) **one at a time**, each gated on
free VRAM *and* host-RAM headroom, then settle-waited — visible in `/v1/models` — before
starting the next. Bring-up becomes idempotent and restartable: after a failure you
re-run and it picks up where it stopped, keeping what already loaded.
[`examples/deploy.sh`](examples/deploy.sh) is a working template that does exactly this,
driven by a manifest ([`examples/models.conf`](examples/models.conf)):
```
chat | /models/chat.gguf | ctx_cap=32768 max_sessions=4 max_concurrent=2
draft | /models/draft.gguf |
```
```bash
./examples/deploy.sh --plan # dry run — print the plan, change nothing
./examples/deploy.sh # reconcile (idempotent)
./examples/deploy.sh --prune # also unload anything not in the manifest
```
It needs **bash and curl, nothing else** — no jq, no Python, no `nvidia-smi`. GPU state
comes from the engine's own `/admin/dies`, so the same script works unchanged on CPU,
Metal and CUDA. Copy it and adapt the gates; every per-model option the `serve` command
line accepts can be set per manifest line.
| | |
|---|---|
| **Endpoints** | `/v1/chat/completions` (SSE), `/v1/models`, `/admin/models/*`, `/admin/dies` |
| **Per-model caps** | `--ctx-cap`, `--max-sessions`, `--max-concurrent`, `--reasoning-budget` |
| **Scheduling** | batch scheduler with a bounded queue, then an honest `429` + `Retry-After` |
| **Caching** | prefix cache with longest-common-prefix reuse across turns |
| **Structured output** | grammar-constrained decoding for tool calls and `response_format` |
| **Ops** | load/unload without restart; `/v1/models` stays live |
A `pantheon router` subcommand fans across separate serve processes.
## Supported models
| Family | Covers |
|---|---|
| **Llama** | Llama, Qwen2, Mistral, Mixtral — one shared forward pass |
| **Phi-3** | fused QKV, NeoX RoPE |
| **Qwen3.5 / 3.6 MoE** | hybrid gated-delta SSM + attention, grouped experts, 2-die split |
| **RWKV-X** | hybrid recurrent + sparse attention, WKV-7 kernel |
| **Pixtral** | vision-language, continuous-batch decoder for bulk OCR |
**Formats:** GGUF (v2/v3, K-quants and legacy) and safetensors, mmap'd zero-copy.
Per-model quants, measured throughput and a runnable command:
[`docs/reference/supported-models.md`](docs/reference/supported-models.md).
## Backends
| Backend | Flag | Notes |
|---|---|---|
| CPU | *(default)* | NEON, AVX2, scalar |
| BLAS | `USE_BLAS=1` | Accelerate on macOS, OpenBLAS on Linux |
| Threads | `USE_THREADS=1` | pthread pool |
| Metal | `USE_METAL=1` | Apple Silicon |
| CUDA | `USE_CUDA=1` | `sm_37`+, `dlopen`ed with kernels embedded as PTX — builds without nvcc. **Supported on Tesla K80: Qwen3.6-35B-A3B and Nex-N2-mini** |
| Tenstorrent | `USE_TT=1` | Blackhole p150a, Linux only; programs embedded, no TT software stack needed to build. **Supported model: Ornith 1.5 9B only** — other models may load on the card but are not supported |
Because CUDA is `dlopen`ed, one binary runs on a machine with no NVIDIA driver at all and
lights up the GPU path on a machine that has one.
```bash
make USE_BLAS=1 pantheon # CLI
make USE_BLAS=1 libpantheon.a # static library, to embed
make USE_BLAS=1 test # test suites
make DEBUG=1 USE_BLAS=1 pantheon
```
### The CLI binary is named for its backend
Each accelerator backend writes its **own** file, and `pantheon` is a symlink to whichever
was built last:
| Build | Binary | `pantheon` symlink |
|---|---|---|
| `make` | `pantheon-cpu` | → `pantheon-cpu` |
| `make USE_CUDA=1` | `pantheon-cuda` | → `pantheon-cuda` |
| `make USE_TT=1` | `pantheon-tt` | → `pantheon-tt` |
| `make USE_METAL=1` | `pantheon-metal` | → `pantheon-metal` |
They used to be one file, so the last build silently owned the deployed path — which is
how a CPU-only binary came to serve a 35B MoE with every GPU die idle. A build for one
backend can no longer overwrite another's binary; at worst it re-points the symlink, and
`ln -sfn pantheon-cuda pantheon` puts that back with no rebuild.
Ask a binary what it is rather than inferring it from its path:
```bash
./pantheon --backend # -> cuda|tt|metal|cpu (one token, for scripts)
./pantheon --version # -> Pantheon v0.4.0 / backend=cuda build=<commit>
curl -s localhost:8080/ | jq .backend # a running serve
scripts/assert_backend.sh ./pantheon cuda # deploy-time assertion; exit 1 on mismatch
```
`pantheon serve` prints `backend=... build=... accelerator=...` as its first line and the
visible die count as its second, and it **refuses to start** when `--dies` names dies this
binary has no backend for. To serve on the CPU deliberately, simply do not pass `--dies`.
## Embedding
Pantheon is a library first and a CLI second. The public API is
[`include/pantheon.h`](include/pantheon.h) — opaque types, explicit ownership, error codes
rather than exceptions, no global state. Link `libpantheon.a` from C or any language with
a C FFI.
## What Pantheon is not
Being clear about this saves everyone time:
- **It is not built for speed on modern datacenter GPUs.** Engines designed for that
hardware will be faster there.
- **Prefix caching, structured output, n-gram speculative decoding and continuous
batching are table stakes**, not differentiators. Everyone has them. They are here
because an engine without them isn't serious, not because they're novel.
- **It is not a distributed serving framework.** One process is self-routing and bounded;
you scale by adding servers, not by packing more into one.
- **It is one person's project.** Model coverage is narrower than that of larger projects.
## Documentation
[`docs/`](docs/) — start at [`docs/README.md`](docs/README.md).
- [Quickstart](docs/guides/quickstart.md) · [Chat templates](docs/guides/chat.md) · [.pth converter](docs/guides/pth-convert.md)
- [Environment flags](docs/reference/env-flags.md) · [CLI flags](docs/reference/cli.md) · [HTTP API](docs/reference/http-api.md) · [Supported models](docs/reference/supported-models.md)
- Architecture: [engine](docs/architecture/engine.md) · [serve layer](docs/architecture/serve.md) · [memory-aware layer](docs/architecture/memory-aware.md) · [die manager](docs/architecture/accel-foundry.md) · [GGUF](docs/architecture/gguf.md) · [Llama forward](docs/architecture/llama-forward.md) · [Qwen3.5-MoE](docs/architecture/models-qwen35moe.md) · [RWKV-X](docs/architecture/rwkvx.md)
## Built on Foundry
Pantheon uses **Foundry** for tensor compute, safetensors, tokenization, quantization,
attention primitives and the CUDA layer. Same author, same license.
Foundry is a git submodule at `foundry/`
([`https://git.teamide.dev/David/Foundry.git`](https://git.teamide.dev/David/Foundry.git)).
The Makefile resolves it as `FOUNDRY=/path`, then `./foundry` (the submodule, or a
symlink), then `../foundry` (a sibling checkout); if none exists it stops with a message
saying so rather than a wall of missing headers.
## License
**AGPL-3.0** — see [`LICENSE`](LICENSE).
Contributions come in under the CLA in [`CLA.md`](CLA.md), with a DCO sign-off on every
commit — see [`CONTRIBUTING.md`](CONTRIBUTING.md). You keep your copyright; signing is one
entry in [`.cla/signatures.md`](.cla/signatures.md), added in your first pull request.
Copyright © 2026 David (Corey) Herrera and the Pantheon contributors. See [`NOTICE`](NOTICE).

56
SECURITY.md Normal file
View File

@ -0,0 +1,56 @@
# Security Policy
## Reporting a Vulnerability
**Do not open a public issue for a security vulnerability.**
Email **coreyh55@gmail.com** with:
- a description of the vulnerability
- steps to reproduce
- potential impact
- a suggested fix, if you have one
You will get an acknowledgement within 48 hours. Critical issues are aimed at a
fix within 7 days. If you want credit in the fix commit, say so and give me the
name to use.
## Scope
Pantheon runs untrusted model files and serves untrusted network input, so the
security-relevant surface is larger than the codebase's size suggests:
- **`src/formats/`** — GGUF parsing, quantized-tensor dequantization, and
memory-mapped I/O. A malformed model file reaching a bad offset or a bogus
tensor shape is the highest-value bug class here.
- **`src/core/`** — tokenizer and BPE pre-tokenization; chat-template parsing.
Both consume attacker-controlled text.
- **`src/serve/`** — the HTTP/SSE server: request parsing, JSON handling,
session and slot management, sampling parameter bounds.
- **`src/memory/`** — KV cache and session state, including the paths that
export and restore session blobs.
Out of scope: the behavior of the models themselves. Prompt injection,
jailbreaks, and model output content are properties of the weights you load, not
of this engine.
## Supported Versions
| Version | Supported |
|---------|-----------|
| main | Yes |
Older tags are not patched. Fixes land on `main`.
## Best Practices
When contributing to Pantheon:
- `snprintf`, never `sprintf`
- validate every offset and length read out of a model file before using it —
treat the file as hostile input, not as data you wrote
- check every `malloc`/`calloc` return for NULL
- bounds-check array indices derived from file or network data
- watch for integer overflow in size calculations, especially
`count * element_size` on values taken from a header
- no `exit()` in library code — return an error and let the caller decide

141
cli/convert.c Normal file
View File

@ -0,0 +1,141 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/*
* pantheon-convert: Convert PyTorch .pth files to safetensors format
*
* Usage: pantheon-convert <input.pth> -o <output.safetensors> [--inspect]
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "pth/pth.h"
#include "safetensors/safetensors.h"
static void print_usage(const char *prog) {
fprintf(stderr, "Usage: %s <input.pth> -o <output.safetensors> [--inspect]\n", prog);
fprintf(stderr, "\nOptions:\n");
fprintf(stderr, " --inspect Print tensor table and exit (no conversion)\n");
}
int main(int argc, char **argv) {
const char *input = NULL;
const char *output = NULL;
int inspect = 0;
for (int i = 1; i < argc; i++) {
if (strcmp(argv[i], "-o") == 0 && i + 1 < argc) {
output = argv[++i];
} else if (strcmp(argv[i], "--inspect") == 0) {
inspect = 1;
} else if (argv[i][0] != '-') {
input = argv[i];
} else {
fprintf(stderr, "Unknown option: %s\n", argv[i]);
print_usage(argv[0]);
return 1;
}
}
if (!input) {
fprintf(stderr, "Error: no input file specified\n");
print_usage(argv[0]);
return 1;
}
if (!inspect && !output) {
fprintf(stderr, "Error: no output file specified (use -o or --inspect)\n");
print_usage(argv[0]);
return 1;
}
/* Open .pth file */
printf("Opening %s...\n", input);
pt_file *pth = pt_open(input);
if (!pth) {
fprintf(stderr, "Error: failed to open %s\n", input);
return 1;
}
int n = pt_tensor_count(pth);
printf("Found %d tensors\n\n", n);
/* Print tensor table */
if (inspect || !output) {
printf("%-50s %-8s %-20s %12s\n", "Name", "Dtype", "Shape", "Size");
printf("%-50s %-8s %-20s %12s\n",
"--------------------------------------------------",
"--------", "--------------------", "------------");
size_t total_bytes = 0;
for (int i = 0; i < n; i++) {
const pt_tensor *t = pt_tensor_at(pth, i);
if (!t) continue;
char shape_str[128] = {0};
int off = 0;
off += snprintf(shape_str + off, sizeof(shape_str) - off, "[");
for (int d = 0; d < t->ndim; d++) {
if (d > 0) off += snprintf(shape_str + off, sizeof(shape_str) - off, ", ");
off += snprintf(shape_str + off, sizeof(shape_str) - off, "%lld",
(long long)t->shape[d]);
}
snprintf(shape_str + off, sizeof(shape_str) - off, "]");
char size_str[32];
if (t->data_size >= 1024 * 1024)
snprintf(size_str, sizeof(size_str), "%.1f MB", (double)t->data_size / (1024 * 1024));
else if (t->data_size >= 1024)
snprintf(size_str, sizeof(size_str), "%.1f KB", (double)t->data_size / 1024);
else
snprintf(size_str, sizeof(size_str), "%zu B", t->data_size);
printf("%-50s %-8s %-20s %12s\n",
t->name, fs_dtype_name(t->dtype), shape_str, size_str);
total_bytes += t->data_size;
}
printf("\nTotal: %d tensors, %.1f MB\n", n, (double)total_bytes / (1024 * 1024));
if (inspect) {
pt_close(pth);
return 0;
}
}
/* Convert to safetensors */
printf("Converting to %s...\n", output);
fs_writer *w = fs_writer_new();
if (!w) {
fprintf(stderr, "Error: failed to create safetensors writer\n");
pt_close(pth);
return 1;
}
int skipped = 0;
for (int i = 0; i < n; i++) {
const pt_tensor *t = pt_tensor_at(pth, i);
if (!t || !t->data || t->data_size == 0) {
fprintf(stderr, " Warning: skipping tensor '%s' (no data)\n", t ? t->name : "?");
skipped++;
continue;
}
fs_writer_add_tensor(w, t->name, t->dtype, t->shape, t->ndim, t->data);
}
if (!fs_writer_save(w, output)) {
fprintf(stderr, "Error: failed to write %s\n", output);
fs_writer_free(w);
pt_close(pth);
return 1;
}
fs_writer_free(w);
pt_close(pth);
printf("Done! Converted %d tensors (%d skipped)\n", n - skipped, skipped);
return 0;
}

289
cli/main.c Normal file
View File

@ -0,0 +1,289 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "../include/pantheon.h"
#include "../src/accel/backend_identity.h"
#include "../src/core/bench.h"
#include "../src/core/sample_profiler.h"
#include "../src/core/engine.h"
#include "../src/core/chat.h"
#include "../src/serve/serve_command.h"
#include "../src/serve/router_command.h"
#include "tensor/tensor.h"
#define PN_VERSION "0.4.0"
static void print_usage(const char *prog) {
printf("Pantheon v%s — Pure C Model Hosting Engine\n\n", PN_VERSION);
printf("Usage: %s [options]\n\n", prog);
printf("Options:\n");
printf(" --model <path> Model file (GGUF or safetensors)\n");
printf(" --config <path> HuggingFace config.json (for safetensors models)\n");
printf(" --vocab <path> RWKV World tokenizer vocab file\n");
printf(" --prompt <text> Input prompt (one-shot mode; chat template applied unless --raw)\n");
printf(" --raw One-shot --prompt: send raw text, no chat template\n");
printf(" --chat Interactive chat mode\n");
printf(" --system <text> System prompt (chat mode only)\n");
printf(" --max-tokens <n> Maximum tokens to generate (default: 128)\n");
printf(" --temperature <f> Sampling temperature (default: 0.8)\n");
printf(" --top-p <f> Nucleus sampling threshold (default: 0.9)\n");
printf(" --top-k <n> Top-k sampling (default: 40)\n");
printf(" --quantize INT8 quantize weights at load time\n");
printf(" --bench Run benchmark (report tok/s and memory)\n");
printf(" --help Show this help\n");
printf(" --version Show version, compiled backend and build commit\n");
printf(" --backend Print ONLY the compiled backend token\n");
printf(" (cuda|tt|metal|cpu) and exit — the deploy-time\n");
printf(" assertion that this binary is the backend the\n");
printf(" fleet manifest expects\n");
}
static void stream_print_cb(const char *token, void *user_data) {
(void)user_data;
printf("%s", token);
fflush(stdout);
}
int main(int argc, char **argv) {
pn_sample_profiler_start(); /* PN_PROFILE=<file>: a SIGPROF sample profile, dumped at exit */
/* Subcommand dispatch. `pantheon serve ...` starts the OpenAI HTTP server;
* everything else is the original flag-driven CLI below. */
if (argc > 1 && strcmp(argv[1], "serve") == 0)
return pn_serve_main(argc - 1, argv + 1);
/* `pantheon router ...` runs the dispatch layer (registry + LB router).
* Pure host-side networking, no model load / CUDA. */
if (argc > 1 && strcmp(argv[1], "router") == 0)
return pn_router_main(argc - 1, argv + 1);
const char *model_path = NULL;
const char *config_path = NULL;
const char *vocab_path = NULL;
const char *prompt = NULL;
const char *system_prompt = NULL;
int max_tokens = 128;
int bench = 0;
int chat_mode = 0;
int quantize = 0;
int raw = 0; /* --raw: bypass the chat template in one-shot --prompt mode */
pn_sampling_params params = pn_sampling_defaults();
/* Parse arguments */
for (int i = 1; i < argc; i++) {
if (strcmp(argv[i], "--help") == 0 || strcmp(argv[i], "-h") == 0) {
print_usage(argv[0]);
return 0;
}
/* `--backend` prints ONE lowercase token and nothing else, so a deploy
* script can assert on it with a plain string compare:
* [ "$("$BIN" --backend)" = cuda ] || exit 1
* That is the whole contract — stdout is the token, exit status is 0.
* Keep it the first thing parsed and never let anything else reach
* stdout on this path. */
if (strcmp(argv[i], "--backend") == 0) {
printf("%s\n", pn_backend_token());
return 0;
}
if (strcmp(argv[i], "--version") == 0) {
/* First line byte-identical to what it always was — scripts parse
* it. The backend goes on a SECOND line rather than into that one. */
printf("Pantheon v%s\n", PN_VERSION);
printf("backend=%s build=%s\n", pn_backend_token(),
pn_backend_build_commit());
return 0;
}
if (strcmp(argv[i], "--model") == 0 && i + 1 < argc) {
model_path = argv[++i];
} else if (strcmp(argv[i], "--config") == 0 && i + 1 < argc) {
config_path = argv[++i];
} else if (strcmp(argv[i], "--vocab") == 0 && i + 1 < argc) {
vocab_path = argv[++i];
} else if (strcmp(argv[i], "--quantize") == 0) {
quantize = 1;
} else if (strcmp(argv[i], "--prompt") == 0 && i + 1 < argc) {
prompt = argv[++i];
} else if (strcmp(argv[i], "--chat") == 0) {
chat_mode = 1;
} else if (strcmp(argv[i], "--system") == 0 && i + 1 < argc) {
system_prompt = argv[++i];
} else if (strcmp(argv[i], "--max-tokens") == 0 && i + 1 < argc) {
max_tokens = atoi(argv[++i]);
} else if (strcmp(argv[i], "--temperature") == 0 && i + 1 < argc) {
params.temperature = (float)atof(argv[++i]);
} else if (strcmp(argv[i], "--top-p") == 0 && i + 1 < argc) {
params.top_p = (float)atof(argv[++i]);
} else if (strcmp(argv[i], "--top-k") == 0 && i + 1 < argc) {
params.top_k = atoi(argv[++i]);
} else if (strcmp(argv[i], "--bench") == 0) {
bench = 1;
} else if (strcmp(argv[i], "--raw") == 0) {
raw = 1;
} else {
fprintf(stderr, "Unknown option: %s\n", argv[i]);
print_usage(argv[0]);
return 1;
}
}
/* If no model specified, just print version and verify Foundry links */
if (!model_path) {
printf("Pantheon v%s\n", PN_VERSION);
printf("backend=%s build=%s\n", pn_backend_token(),
pn_backend_build_commit());
printf("Built on Foundry tensor library.\n\n");
/* Verify Foundry links correctly */
int64_t shape[] = {2, 3};
tm_tensor *t = tm_ones(shape, 2, TM_F32);
if (t) {
printf("Foundry link check: OK (created %lldx%lld tensor)\n",
(long long)t->shape[0], (long long)t->shape[1]);
tm_free(t);
} else {
printf("Foundry link check: FAILED\n");
return 1;
}
if (!prompt && !chat_mode) {
printf("\nRun with --model <path> --prompt <text> to generate text.\n");
printf("Run with --model <path> --chat for interactive chat.\n");
return 0;
}
}
/* Load model */
if (model_path) {
double t0 = pn_time_ms();
pn_load_opts opts = { .vocab_path = vocab_path, .config_path = config_path, .quantize = quantize };
pn_model *model = pn_load_ex(model_path, &opts);
if (!model) {
fprintf(stderr, "Failed to load model: %s\n", model_path);
return 1;
}
double load_ms = pn_time_ms() - t0;
if (bench) fprintf(stderr, " load: %.1f ms\n", load_ms);
pn_state *state = pn_state_new(model);
if (!state) {
fprintf(stderr, "Failed to create inference state\n");
pn_model_free(model);
return 1;
}
if (chat_mode) {
/* Interactive chat mode */
pn_chat_style style = (pn_chat_style)model->chat_style;
ft_tokenizer *tok = model->tokenizer;
printf("Chat mode (%s). Type /quit to exit, /clear to reset.\n",
pn_chat_style_name(style));
char line[4096];
int turn = 0;
while (1) {
printf("> ");
fflush(stdout);
if (!fgets(line, sizeof(line), stdin)) break;
/* Strip trailing newline */
size_t len = strlen(line);
if (len > 0 && line[len - 1] == '\n') line[len - 1] = '\0';
/* Handle commands */
if (strcmp(line, "/quit") == 0 || strcmp(line, "/exit") == 0) break;
if (strcmp(line, "/clear") == 0) {
pn_state_reset(state);
turn = 0;
printf("Conversation cleared.\n");
continue;
}
/* Skip empty lines */
if (line[0] == '\0') continue;
/* Format and generate */
pn_token_seq seq;
if (turn == 0)
seq = pn_chat_format_first(style, tok, system_prompt, line);
else
seq = pn_chat_format_turn(style, tok, line);
if (!seq.tokens || seq.n_tokens == 0) {
fprintf(stderr, "chat: tokenization failed\n");
free(seq.tokens);
continue;
}
/* The exact token ids this turn prefills with. A parity harness takes token
* ids, so without this the only way to re-run a chat prompt against the CPU
* oracle is to guess at the template's own tokenization. */
if (getenv("PN_PROMPT_TOKEN_DUMP")) {
fprintf(stderr, "PROMPT_TOKENS %d:", seq.n_tokens);
for (int t = 0; t < seq.n_tokens; t++)
fprintf(stderr, " %d", seq.tokens[t]);
fprintf(stderr, "\n");
}
pn_stream_tokens(model, state, seq.tokens, seq.n_tokens,
max_tokens, &params, stream_print_cb, NULL);
printf("\n");
free(seq.tokens);
turn++;
}
} else if (bench) {
/* Benchmark mode */
pn_bench_result result = {0};
result.memory_bytes = pn_memory_usage();
double gen_start = pn_time_ms();
char *output = pn_generate(model, state, prompt ? prompt : "Hello",
max_tokens, &params);
double gen_end = pn_time_ms();
if (output) {
printf("%s\n", output);
free(output);
}
result.n_decode = max_tokens;
result.decode_ms = gen_end - gen_start;
result.memory_bytes = pn_memory_usage();
pn_bench_report(&result);
} else {
/* One-shot generation. Apply the model's chat template (single
* turn) so output matches the reference; --raw or a model with no
* chat template falls back to raw completion. */
const char *one_shot = prompt ? prompt : "Hello";
pn_chat_style style = (pn_chat_style)model->chat_style;
if (!raw && style != PN_CHAT_NONE) {
pn_token_seq seq = pn_chat_format_first(style, model->tokenizer,
system_prompt, one_shot);
if (seq.tokens && seq.n_tokens > 0) {
pn_stream_tokens(model, state, seq.tokens, seq.n_tokens,
max_tokens, &params, stream_print_cb, NULL);
printf("\n");
free(seq.tokens);
} else {
fprintf(stderr, "one-shot: chat formatting failed\n");
free(seq.tokens);
}
} else {
char *output = pn_generate(model, state, one_shot,
max_tokens, &params);
if (output) {
printf("%s\n", output);
free(output);
}
}
}
pn_state_free(state);
pn_model_free(model);
}
return 0;
}

117
docs/README.md Normal file
View File

@ -0,0 +1,117 @@
# Pantheon Documentation
Pantheon is a pure-C11 inference engine and OpenAI-compatible model server, built on Foundry (a git submodule). Beyond Foundry it needs libc, libm, pthreads and libdl; BLAS and the Metal backend's Apple frameworks are optional.
## ⭐ Multi-Model Smart Routing — the flagship feature
Pantheon is a **memory-aware, multi-model server**. A **single running instance** hosts **several models across several GPUs at once** and routes every request intelligently — so a tiny host (e.g. 2 cores / 8 GB RAM) can drive a fleet of cards and run models on hardware they were never meant for. Tying *multiple* instances together is a separate future **gateway**; **one instance is fully self-routing** and needs nothing in front of it.
**How routing works (all inside one `pantheon serve` process):**
- One process binds **one port**. Each `--model` is an independent registry slot with its own weights, state, and die placement. A request picks its model by the **`model` field** in the JSON body, matched to each model's `--name` (one model loaded ⇒ the field is ignored).
- **Replicas = the same `--name` on different dies.** `mr_acquire` (`src/serve/model_registry.c`) load-balances across replicas by **least-busy** (fewest in-flight refs); with batching off each replica runs on its own die, so requests execute **concurrently**: two concurrent requests to a model replicated on dies 2 and 3 keep both dies busy at once and finish in about one request's wall-clock.
- **Memory-aware admission** ([Memory-Aware Layer](architecture/memory-aware.md)) gates every load and turn against live host-RAM + VRAM pressure, keeping the instance in a small host footprint.
- **Optional per-model concurrency:** `QWEN35_BATCH_SLOTS=1` adds up to 8 batched slots on a single die (off by default — see [reference/env-flags.md](reference/env-flags.md)).
**Launch it** — one instance, an orchestrator plus a worker replicated across two dies:
```bash
./pantheon serve \
--model /models/orchestrator.gguf --name orch --dies 0,1 \
--model /models/worker.gguf --name worker --dies 2 \
--model /models/worker.gguf --name worker --dies 3 \
--port 8201
```
Clients POST `/v1/chat/completions` with `"model":"orch"` or `"model":"worker"`; `worker`
requests fan out across dies 2 and 3 by least-busy, concurrently. No router needed.
> **Known gap (roadmap):** replica selection is least-busy but **not yet session-sticky** — under concurrent load a multi-turn session can hop replicas and miss its prefix cache. Session affinity across replicas is designed but not yet shipped.
## ⭐ Reference Library (flags · CLI · HTTP API)
The authoritative, code-audited reference (2026-07-12) — every environment flag, CLI flag, and HTTP endpoint, with values, defaults, effects, source `file:line`, and the **parsing gotchas that bite** (e.g. `QWEN35_BATCH_SLOTS` is an *enable* flag keyed on `env[0]=='1'`, so `=8` silently disables batching; `--dies` indices are relative to `CUDA_VISIBLE_DEVICES`, not physical devices).
- [**Environment Flags**](reference/env-flags.md) — all `QWEN35_*` / `PN_*` / `FC_*` + distributed vars (\~40 flags)
- [**CLI Flags**](reference/cli.md) — `pantheon serve` global + per-model flags, `--dies` semantics, multi-model binding
- [**HTTP API**](reference/http-api.md) — `/v1/models`, `/v1/chat/completions` (SSE), `/admin/models/*`, `/admin/dies`
- [**Supported Models**](reference/supported-models.md) — every verified model with its quant, config, measured throughput and a runnable command
## Architecture — serving-era (code-audited 2026-07-12)
- [Serve Layer](architecture/serve.md) — request lifecycle, model registry + multi-card load-balancing, the batch scheduler, streaming/SSE, the lock table
- [Qwen3.5-MoE Model](architecture/models-qwen35moe.md) — the qwen35moe arch (e.g. Nex-N2-mini), 2-die split/placement, KV cache, decode slots, async step machine *(supersedes the legacy `qwen35moe.md` below)*
- [Core Engine](architecture/core-engine.md) — `pn_load`/`pn_load_on`, the forward-step async seam, admission helpers
- [Memory-Aware Layer](architecture/memory-aware.md) — pressure levels (OK/WATCH/TIGHT/CRITICAL), pre-flight admission, the reactive ladder (why TIGHT is the mini's normal resting state)
- [Accelerator / Die Manager + Foundry CUDA](architecture/accel-foundry.md) — die abstraction, CUDA↔die mapping under `CUDA_VISIBLE_DEVICES`, per-die VRAM budgeting, foundry primitives (streams/events/graphs), and **§7 the second backend: Tenstorrent Blackhole p150a** (`make USE_TT=1`; tt-kmd + SFPI only — no tt-metal, TT-NN, Python or C++) — what it serves today, its runtime flags, and the hazards that cost hardware
## Guides
- [Quickstart](guides/quickstart.md) — Build, download a model, generate text
- [Install](guides/install.md) — Per-platform build and run (CPU, macOS, CUDA, Tenstorrent), troubleshooting
- [Chat Templates & Interactive Chat](guides/chat.md) — Multi-turn chat, template detection, C API
- [PyTorch .pth Converter](guides/pth-convert.md) — Convert .pth to safetensors in pure C, no Python
## Architecture
- [Engine Overview](architecture/engine.md) — How model loading, dispatch, and generation work, plus the vision-language encode/decode seam and continuous-batch VLM decoder for bulk OCR
- [GGUF Format](architecture/gguf.md) — GGUF parser, quant types, v2/v3 support
- [Llama Forward Pass](architecture/llama-forward.md) — Shared forward pass for Llama, Qwen2, Mistral, Mixtral + Phi-3 differences, batched multi-sequence decode, flash prefill, and the opt-in GPU paths
- [Qwen3.5-MoE Forward Pass (legacy)](architecture/qwen35moe.md) — the original forward-pass writeup. **⚠ Predates the A–D serving stack; superseded by **[**architecture/models-qwen35moe.md**](architecture/models-qwen35moe.md)** (the 2026-07-12 code audit). Kept for any unique detail; prefer the new doc.**
- [RWKV-X Forward Pass](architecture/rwkvx.md) — Hybrid recurrent + sparse attention, WKV-7 kernel, safetensors loading
## Serving & multi-tenant
The serving architecture itself is documented in [architecture/serve.md](architecture/serve.md); what follows is a map of the behaviour it implements.
- **Multi-card concurrency** — the host-memory + concurrency stack that lets a small host (8 GB / 2 cores) drive up to 8 cards: the die is the atomic unit, at most one model per die, reliability first, and capacity scales by adding servers rather than packing models.
- **Single vs multi-tenant** — one card serves either one fast user or K slower ones; the two knobs are multi-model placement and `QWEN35_BATCH_SLOTS` batching. See [reference/env-flags.md](reference/env-flags.md).
- **Multi-model serving** — registry, `model`-field routing, runtime load/unload, die placement.
- **Fleet bring-up** — load models one at a time through `/admin/models/load`, gated and settle-waited. [`examples/deploy.sh`](../examples/deploy.sh) is a working template.
## API Reference
- [`include/pantheon.h`](../include/pantheon.h) — Public C API (opaque types, generation, sampling, vision-language describe, and the continuous-batch VLM decoder)
- [`src/core/engine.h`](../src/core/engine.h) — Internal types: config, architecture registry, state, `pn_load_opts`
- [`src/formats/gguf.h`](../src/formats/gguf.h) — GGUF file format types and parser
- [`src/models/llama.h`](../src/models/llama.h) — Llama architecture weight structs and forward pass
- [`src/models/phi3.h`](../src/models/phi3.h) — Phi-3 weight structs (fused QKV, fused gate/up)
- [`src/models/rwkvx.h`](../src/models/rwkvx.h) — RWKV-X adapter (safetensors loading, trie tokenizer, WKV-7 forward)
- [`src/core/chat.h`](../src/core/chat.h) — Chat template detection and formatting
- [`src/core/qweight.h`](../src/core/qweight.h) — Quantized weight matrix (zero-copy mmap)
- [`src/accel/simd.h`](../src/accel/simd.h) — SIMD kernels (NEON, AVX2, scalar)
- [`src/memory/arena.h`](../src/memory/arena.h) — Bump allocator for inference scratch memory
## Contributing
- [`../CONTRIBUTING.md`](../CONTRIBUTING.md) — build flags, code style, testing bar, and the licensing terms contributions are made under

371
docs/architecture/accel-foundry.md Normal file
View File

@ -0,0 +1,371 @@
# Accelerator / Die-Manager Layer and Foundry CUDA Base
Covers `src/accel/die_manager.c|h` and `foundry/cuda/fc_cuda.c|h`.
---
## 1. Die abstraction
A **die** is one GK210 GPU — the unit of VRAM ownership in Pantheon. A K80
card physically packages **two GK210 dies** connected by an on-card PLX
switch. They appear to CUDA as **two separate devices** with independent
contexts, streams, and VRAM pools. The die manager hard-codes this pairing:
`die_id / 2 = card_id` (`pn_dm_card_of`, `die_manager.h:82`) and
`{2i, 2i+1}` are card-mates (`die_mate`, `die_manager.c:195`). The die
manager supports up to 16 dies (`PN_DM_MAX_DIES`, `die_manager.h:41`; mirroring
`PN_MEM_MAX_DIES`).
### K80 hardware constraints
Reflected in the kernel target (`fc_cuda.h:17`):
- Compute capability sm_37 — no FP16 math, no tensor cores.
- `__shfl_down()` (Kepler) rather than `__shfl_down_sync()` (Pascal+).
- VRAM bandwidth ~240 GB/s; K80 decode is **launch-bound** (~80 µs/launch,
`fc_cuda.h:562`), not compute-bound — the fused-layer and graph-capture
mechanisms exist specifically to reduce launch count.
---
## 2. CUDA device → die mapping under `CUDA_VISIBLE_DEVICES`
`pn_dm_init_cuda` (`die_manager.c:72`) enumerates CUDA ordinals `[0, n_dies)`
by calling `fc_cuda_init_device(i)` for each. Because `CUDA_VISIBLE_DEVICES`
re-numbers the visible devices starting at 0, die ordinal `i` inside Pantheon
always equals CUDA device `i` within whatever the variable permits. Dies with
ordinals not visible to the process are not reachable: `cuDeviceGet(i)` returns
error 101 when `i >= CUDA_VISIBLE_DEVICES` count — the benign probe that
terminates the enumeration loop (`die_manager.c:89`):
```c
fc_cuda_ctx *ctx = fc_cuda_init_device(i);
if (!ctx || !fc_cuda_ctx_available(ctx))
break; /* ordinals are contiguous — stop */
```
`fc_cuda_init_device` calls `p_cuDeviceGet(&ctx->device, device_id)`
(`fc_cuda.c:910`); on error it logs `cuDeviceGet(%d) failed: %d` and returns
NULL, which is the break condition above.
---
## 3. VRAM budgeting and admission at the die level
### Reserve
The die budget is `total_vram − PN_DIE_RESERVE_BYTES` (default **1 GiB**,
`pn_memwatch.h:42`; overridable via `PN_DIE_RESERVE_BYTES`).
`pn_dm_init_cuda` computes this per die (`die_manager.c:95`):
```c
uint64_t budget = die_total > reserve ? die_total - reserve : 0;
```
### External occupancy
VRAM already consumed by another process at init time is detected and credited
to `vram_used` so Pantheon doesn't over-commit (`die_manager.c:101`):
```c
uint64_t external = (die_total - die_free) > reserve
? (die_total - die_free) - reserve : 0;
g_dm.die[i].vram_used = external < budget ? external : budget;
```
### Two tallies
The die manager maintains two numbers per die (`die_manager.c:37`):
| Field | Meaning |
|---|---|
| `vram_budget` | Usable bytes (total − reserve). Set at init, never changes. |
| `vram_used` | Bytes committed by bindings + external occupancy. Mutated by bind/release. |
Free bytes at any moment = `vram_budget − vram_used` (`die_free_bytes`,
`die_manager.c:183`).
### memwatch feed
After computing each die's budget, `pn_dm_init_cuda` calls
`pn_memwatch_set_die(i, die_free, budget, fc_ctx_alloc_bytes(ctx))`
(`die_manager.c:105`) so `MemoryAware` admission sees the same per-die truth.
---
## 4. Bind / release lifecycle
### `pn_dm_place` (auto-placement, `die_manager.c:200`)
Does not reserve — only selects. Accepts `n_dies_wanted` of 1 or 2.
**2-die request:** Scans even ordinals, looking for a same-card pair `{a, a+1}`
both with `die_free_bytes ≥ per_die`. Prefers a fully-idle pair; among ties
picks the pair with the most aggregate free space (`die_manager.c:219`).
**1-die request:** Prefers a die whose card-mate is already occupied ("pack")
so free same-card pairs remain available for future 2-die models. Among
candidates picks the most-free die (`die_manager.c:244`).
### `pn_dm_bind` (`die_manager.c:270`)
Commits a placement under the global mutex:
1. Finds a free slot in `g_dm.binding[]` (cap `PN_DM_MAX_BINDINGS = 16`).
2. Re-checks `die_free_bytes(id) >= per_die` for every die in the set —
guards against TOCTOU between `pn_dm_place` and `pn_dm_bind`.
3. Writes the binding (name, die list, `bytes_per_die`).
4. Increments `g_dm.die[id].vram_used` by `per_die` for every die in the set.
The whole operation is atomic under the lock; on any failure nothing is
partially written (`goto refuse`, `die_manager.c:313`).
**Same-name multi-card rule** (`die_manager.c:283`): a model name may be bound
multiple times to DISJOINT die-sets (one entry per copy, for load-balanced
multi-card serving). The code **does not reject duplicate names** at bind time.
Overlap is caught by the per-die room check (step 2): if a second binding would
use the same die, `die_free_bytes` will be too small and the bind refuses.
`pn_dm_release` is first-match on name (`die_manager.c:326`), so N bindings
with the same name require N `pn_dm_release` calls to free all of them.
### `pn_dm_release` (`die_manager.c:318`)
Walks the binding table, finds the first matching name, subtracts
`bytes_per_die` from each die's `vram_used` (clamped to 0), zeros the slot,
and returns. Only releases one binding per call.
---
## 5. Foundry CUDA primitives
### 5.1 Driver loading (`fc_cuda.c:517`)
`fc_driver_load()` (called lazily by `fc_cuda_init_device`):
1. `dlopen("libcuda.so.1")` — falls back to unversioned `libcuda.so`.
2. Resolves all driver API symbols via `fc_load_cuda_symbols`
(`fc_cuda.c:414`). `_v2` variants are preferred for context/memory ops
(required by cuBLAS); a fall-back to the unversioned name is attempted for
each.
3. `cuInit(0)`.
4. `dlopen("libcublas.so.11")` (optional — custom PTX kernels work without it).
The resolved function-pointer table is **process-global** and shared by all
contexts (`fc_cuda.c:152`). The table includes optional symbols for peer copies,
graph capture, and events; missing ones leave their pointer NULL and the
corresponding feature self-disables gracefully.
### 5.2 Per-device context (`fc_cuda_init_device`, `fc_cuda.c:896`)
One `fc_cuda_ctx` per die. Sequence:
1. `p_cuDeviceGet(&ctx->device, device_id)` — binds the ordinal.
2. `p_cuCtxCreate(&ctx->context, 0, ctx->device)` — creates a CUDA context;
leaves it current.
3. `p_cuStreamCreate(&ctx->stream, 0)` — one stream per context.
4. `cublasCreate` + `cublasSetStream` on the stream (non-fatal if cuBLAS absent).
5. `fc_ctx_load_kernels` — `cuModuleLoadData` of all embedded PTX blobs into
this context's private module slots. Optional kernels (graph-capture variants,
8-wide tile variants, slot-attention kernels) are looked up tolerantly and
left NULL if absent.
6. `fc_ctx_leave()` — pops the new context off the thread's stack.
7. `fc_registry_add(ctx)` — registers in the live-context table for
crash-time teardown.
The driver function table is shared; module handles and kernel function handles
are **per-context** (copied into each `fc_cuda_ctx`'s module fields,
`fc_cuda.c:224`).
### 5.3 Context isolation (push/pop, `fc_cuda.c:577`)
Every device op calls `fc_ctx_enter(ctx)` → `cuCtxPushCurrent` before issuing
work and `fc_ctx_leave()` → `cuCtxPopCurrent` on return. This keeps concurrent
dies from bleeding into each other even when ops from different dies interleave
on the same host thread.
### 5.4 VRAM accounting (`ctx->alloc_bytes`)
Every `cuMemAlloc` incremented path (`fc_buffer_from_ptr_ctx` at
`fc_cuda.c:1156`, `fc_buffer_alloc_ctx` at `fc_cuda.c:1184`) adds `bytes` to
`ctx->alloc_bytes`. `fc_buffer_free` subtracts it (clamped, `fc_cuda.c:1220`).
This counter is the **predictive** side of admission; `fc_ctx_alloc_bytes(ctx)`
exposes it. `pn_dm_init_cuda` passes it to `pn_memwatch_set_die` so both
tallies start consistent.
### 5.5 Memory operations
| Function | Operation |
|---|---|
| `fc_buffer_from_ptr_ctx` | `cuMemAlloc` + `cuMemcpyHtoD` (blocking) |
| `fc_buffer_alloc_ctx` | `cuMemAlloc` only (output buffer) |
| `fc_buffer_free` | `cuMemFree` + decrement `alloc_bytes` |
| `fc_buffer_to_cpu_n` | `cuMemcpyDtoH` (blocking) |
| `fc_buffer_to_cpu_n_async` | `cuMemcpyDtoHAsync` on ctx stream |
| `fc_buffer_copy_d2d_ctx` | `cuMemcpyDtoDAsync` on ctx stream |
| `fc_buffer_copy_peer` | `cuMemcpyPeer` (blocking, cross-die) |
| `fc_buffer_copy_peer_async` | `cuMemcpyPeerAsync` on stream_ctx stream |
| `fc_buffer_offset_view` | Non-owning wrapper (struct + pointer offset only; no cuMemAlloc/Free) |
Peer copies require `fc_cuda_enable_peer(a, b)` first, which calls
`cuDeviceCanAccessPeer` and `cuCtxEnablePeerAccess` in both directions
(`fc_cuda.c:1400`). Error 704 (already-enabled) is treated as success.
### 5.6 Synchronization
`fc_sync_ctx(ctx)` (`fc_cuda.c:4468`) calls `cuStreamSynchronize` on
`ctx->stream` (falling back to `cuCtxSynchronize` if the stream is NULL).
This blocks the host thread until all work queued on the stream is done.
**Events** (`fc_event_*`, `fc_cuda.c:4391`): created with
`CU_EVENT_DISABLE_TIMING` (no timing overhead). `fc_event_record_ctx` queues
`cuEventRecord` on the ctx stream; `fc_event_query` calls `cuEventQuery` and
returns 0 (done), 1 (CUDA_ERROR_NOT_READY = still running), or -1 (error).
The async serve scheduler uses events to poll all 8 card streams without
blocking any host thread — the alternative, `fc_sync_ctx`, would spin/yield a
core per card and starve other cards' launch turns on a 2-core host
(`fc_cuda.c:4383`).
### 5.7 Graph capture (`fc_graph_*`, `fc_cuda.c:4296`)
Optional (symbols checked at load; all six must be non-NULL or `fc_graph_begin_ctx`
returns -1). `fc_graph_begin_ctx` calls `cuStreamBeginCapture(stream, mode=0
= GLOBAL)`. `fc_graph_end_ctx` calls `cuStreamEndCapture` then
`cuGraphInstantiate`. `fc_graph_launch` replays via `cuGraphLaunch`. Used by the
decode-attention block for K80 launch-bound relief (one graph replay replaces
hundreds of `cuLaunchKernel` calls).
### 5.8 Back-compat single-die path
`fc_init()` creates a process-default context on device 0 (`_fc_default_ctx`).
All non-`_ctx` functions (`fc_buffer_from_ptr`, `fc_sgemm`, `fc_sync`, …)
delegate to it. Multi-die callers use the explicit `fc_cuda_ctx*` variants.
---
## 6. Correctness notes — multi-card same-name bindings
`pn_dm_bind` intentionally **does not prevent duplicate names**
(`die_manager.c:283`): N copies of a model on N different card-pairs all
register under the same name to support load-balanced serving. This is correct
as long as:
1. **Disjoint die-sets:** enforced by the per-die `die_free_bytes < per_die`
check. If two bindings were attempted on overlapping dies, the second
would see reduced free space and refuse — provided the first binding's
`bytes_per_die` correctly covers the die's actual weight bytes.
2. **N releases required:** `pn_dm_release` is first-match; releasing a
multi-copy model requires exactly N calls (one per binding slot). A single
call frees only one copy's VRAM accounting. A caller that calls release once
and assumes the die is fully reclaimed will have a stale `vram_used`.
3. **`pn_dm_get` reports only the first bound model name** for a die
(`die_manager.c:167`); `n_models` counts all bindings touching that die.
Status consumers that read only `model_name` will not see co-resident
same-card models.
No gap in the overlap-rejection path: the per-die room check is atomic under
the global lock, so two concurrent binds on the same die cannot both succeed
if there is only room for one.
---
## 7. Second backend — Tenstorrent Blackhole p150a (`USE_TT=1`)
CUDA is not the only accelerator behind the die manager. `pn_dm_init_tt()`
(`die_manager.h:114`) enumerates a **Tenstorrent Blackhole p150a** instead of
K80 dies; the host driver is `src/accel/tt/pn_tt_{device,residency,oncard}.c`
over Foundry's `tt/` backend.
**Host:** any Linux machine with the card; the reference host is a small 2-core / 8 GB
box (no AVX) that also drives Tesla K80s. With tt-kmd the card appears as
`/dev/tenstorrent/<n>`.
### Build
```bash
make USE_TT=1 # Linux only; adds -DUSE_TT -DFT_TT_PHASE1B_NOC_ENABLED
# writes ./pantheon-tt, then points ./pantheon at it
```
This build writes **`pantheon-tt`**, not `pantheon` — the backends have separate
output names now, so a TT build on a host that also runs CUDA can no longer
overwrite a deployed `pantheon-cuda`. It does re-point the `pantheon` symlink,
and on a host that also serves CUDA that has to be put back:
`ln -sfn pantheon-cuda pantheon` (no rebuild needed). Confirm with
`./pantheon --backend` before deploying anything.
`FT_TT_PHASE1B_NOC_ENABLED` is deliberately **not** a second knob: without it
`ft_tt_mem_info()` cannot read a bank count and `pn_dm_init_tt()` refuses every
device, so a build could link the backend and then silently decline to see the
card. Device kernels are **not** built here — the kernel ELFs and Tensix programs
are pre-baked into `foundry/tt/ft_tt_kernels.h` / `ft_tt_programs.h`, the same
trick `cuda/` uses for PTX, so building needs no SFPI (SFPI is only needed to
rebuild those programs).
**Run-time dependency: a way to reach the card, nothing else.** The default
transport is tt-kmd (`/dev/tenstorrent/<n>`); `FT_TT_DRIVER=udrv` uses Foundry's
own userspace driver over PCI sysfs with no kernel module, which needs root or the
udev rule `foundry/tt/tools/60-foundry-blackhole.rules`. No tt-metal runtime, no
TT-NN, no Python, and no C++ anywhere — host or device. That boundary is the
point of the backend, not an incidental property. Foundry's TT layer was rewritten
from public documentation and measured behaviour; no tt-metal, TT-NN, tt-umd or
tt-llk source is compiled, linked or vendored. The vLLM/tt-metal stack cannot
run on the reference host at all: 2 cores / 8 GB and **no AVX**, so TT's prebuilt
wheels SIGILL.
### Runtime knobs
| Variable | Effect |
|---|---|
| `PN_TT_PROGRAM_DIR` | **override only.** Read Tensix programs from this directory instead of the ones embedded in the binary. Unset — the default — means embedded. An override that cannot be satisfied REFUSES; it never falls back to embedded |
| `PN_TT_DISPATCH` | dispatch-path selection |
| `PN_TT_ONCARD` | on-card op families (reduce / eltwise / top-k) |
| `PN_TT_CORE`, `PN_TT_CORE_{REDUCE,ELTWISE,SAMPLE}` | pin a residency to a Tensix index |
### What it can serve today
`pantheon serve` runs **MiniCPM5-1B** on the card over the OpenAI-compatible
endpoint, multi-turn, across idle gaps: **4.37 tok/s best, 4.14 sustained**
(2026-08-15). Eight op families are green on silicon.
Three limits to plan around:
1. **Everything measured is ONE core of 140. Multi-core is not built.** The
forward runs 169 matvecs on a single matmul core. The figure quoted
elsewhere as *8.94 decode tok/s* is **composed** from measured per-op costs
(3.745 ms/layer × 24 + 22.0 ms lm_head) — **no end-to-end forward pass has
been run to produce it.** The 4.37 tok/s above is the measured end-to-end
number.
2. **MoE is not built.** Routed-MoE models (qwen35moe, e.g. Nex-N2) do not run
on the card. The dense qwen35 hybrid (Gated-DeltaNet + attention, e.g.
Ornith) does, byte-identical to the CPU forward (`engine.c`, `PN_ARCH_BONSAI`
branch; Foundry `ft_tt_op_gdn`).
3. **Open correctness defect.** A chat-templated prompt — i.e. every request a
serve receives — costs **+22.1% perplexity** and disagrees with the CPU
oracle on **16.9%** of decode steps. Mechanism unidentified.
Weights are not GGUF on the card: `foundry/tt/convert/` turns a GGUF into a
tiled **FTTW** artifact (bf16 / Bfp8_b / Bfp4_b) and the loader is a byte copy.
**BFP4 is refuted on quality** (+40.2% perplexity) and is no faster than BFP8.
### Operational hazards — these cost hardware, not a retry
- **A wrong NoC coordinate hangs the chip and costs a physical power cycle.**
Coordinates come from ARC telemetry; never probe or guess one.
- **`PHASE1B=1` is mandatory for on-card builds**; off-hardware suites are
built without the NoC fence so they cannot reach the card even when wrong.
- **Never assert the Tensix BACKEND soft-reset bits** — it silently and
permanently corrupts that core's compute until a host reboot.
- **Retire by asking, never by reset** (`FT_TT_QUEUE_OP_STOP` / `stop_request`).
`assert_reset` on a blocked resident kernel is what wedges cores, and a
resident triad must never be parked.
- **Never more than 4 destinations per multicast** — 5 wedges reliably.
- **On-card runs are detached and polled, never held over an SSH session.**
- **Card→host DMA does not exist on Blackhole.** Host→card is 287 MB/s on the
reference host's x1 Gen2 link and readback is 2.3–4.7 MB/s, so on-card sampling is
mandatory rather than an optimisation. **A card reset drops the link to
Gen1** — retrain and verify afterwards, or every host-involved number is
silently halved.
- **On-card residencies are bounded** and exit on an idle spin limit — and a
serve idles by definition. `pn_tt_oncard_rearm()` is what recovers them.
Full detail: `foundry/tt/DEV_NOTES.md` (the authority on hazards),
`foundry/tt/README.md`, `foundry/docs/tenstorrent.md`.

180
docs/architecture/core-engine.md Normal file
View File

@ -0,0 +1,180 @@
# Core Engine API
Source: `src/core/engine.c`, `src/core/engine.h`.
Covers the loading surface (`pn_load`/`pn_load_on`), the model and state structs, the async forward-step seam, scratch estimation, and the admission helpers at the engine level. The memory-aware layer that gates admission is documented separately in `memory-aware.md`.
---
## Public API surface
### Loading
```c
// Thin wrappers — all paths converge on pn_load_on.
pn_model* pn_load(const char *path); // engine.c:412
pn_model* pn_load_ex(const char *path, const pn_load_opts *opts); // engine.c:549
pn_model* pn_load_on(const char *path, const pn_load_opts *opts, // engine.c:553
const int *dies, int n_dies);
```
`pn_load` and `pn_load_ex` are thin wrappers that call `pn_load_on` with `dies=NULL, n_dies=0`.
`pn_load_on` is the placement-aware entry point added for multi-model serving (Phase 7). When `dies` is non-NULL the die list is recorded on the model before any GPU upload (`model_record_dies` at engine.c:538), so the subsequent `pn_qwen35moe_gpu_upload` call reads the correct target dies via `pn_qwen35moe_model_set_dies`.
**Load opts:**
```c
typedef struct {
const char *vocab_path; // RWKV World tokenizer path
const char *config_path; // HF config.json for Qwen/LLaMA safetensors
bool quantize; // INT8-quantize at load time (RWKV-X only)
} pn_load_opts; // engine.h:116
```
### pn_model struct (engine.h:74)
```c
struct pn_model {
pn_config config; // parsed dimensions (stack)
void *weights; // arch-specific weight struct
pn_forward_fn forward; // forward pass function pointer
ft_tokenizer *tokenizer; // BPE/SP tokenizer (NULL for RWKV-X)
char *path;
int chat_style;
void *vision; // pn_vision_model* for VLMs
struct fw_pool *pool; // thread pool (USE_THREADS only)
int gpu_dies[PN_MODEL_MAX_DIES]; // explicit die placement (pn_load_on)
int n_gpu_dies; // 0 = none, env/defaults apply
};
```
`forward` is the architecture dispatch: a function pointer of type `tm_tensor* (*)(pn_model*, uint32_t token, pn_state*)`. The CPU and CUDA forwards are distinct pointers; `pn_admit_gpu_resident` identifies the CUDA path by comparing `model->forward == pn_qwen35moe_forward_cuda` (engine.c:441).
### pn_state struct (engine.h:89)
```c
struct pn_state {
void *kv_cache; // arch-specific: pn_llama_kv_cache / pn_qwen35moe_state / rx_state
int pos; // current sequence position
pn_config *config; // pointer to model config
struct pn_arena *arena; // bump-allocator scratch (NULL for RWKV-X and Qwen35MOE)
const float *embd_override; // if set, forward uses this embedding (image token)
};
```
`pn_state_new` allocates architecture-specific state:
- **Llama/Phi/Mistral/Mixtral** (engine.c:870): allocates `pn_llama_kv_cache` and an arena sized by `compute_arena_size`.
- **Qwen3.5-MoE** (engine.c:860): allocates `pn_qwen35moe_state` (dual state: SSM conv/ssm + attention KV). No arena.
- **RWKV-X** (engine.c:849): allocates `rx_state` (recurrent + KV + workspace). No arena.
---
## Loading paths
`pn_load_on` branches on file format:
### Safetensors
1. If `opts->config_path` is set: HF/ternary safetensors loader → `pn_llama_load_safetensors` + `pn_llama_load_hf_tokenizer`. Sets `model->forward = pn_llama_forward` (engine.c:579).
2. Otherwise: RWKV-X path → `pn_rwkvx_load` + optional vocab. Sets `model->forward = pn_rwkvx_forward` (engine.c:627).
### GGUF
Architecture detected from `general.architecture` metadata (engine.c:226). For Qwen3.5-MoE specifically (engine.c:694):
1. `pn_qwen35moe_load(gguf, &config)` — zero-copy mmap weights.
2. Explicit die placement applied to the arch model: `pn_qwen35moe_model_set_dies(qw, model->gpu_dies, model->n_gpu_dies)` (engine.c:704).
3. `fc_init()` — idempotent CUDA init; no-op without a GPU.
4. `pn_qwen35moe_gpu_upload(qw)` — uploads weights to VRAM; on success sets `model->forward = pn_qwen35moe_forward_cuda` (engine.c:716). Falls back silently to `pn_qwen35moe_forward` (CPU) if the upload fails or no GPU is present.
---
## Scratch (arena) estimation
`compute_arena_size` (engine.c:357) sizes the per-state bump allocator for transformer models:
- **Per layer**: `5*n_embd + 3*n_ff + 2*n_kv_head*head_dim` floats. MoE adds `n_experts + 2*n_experts_used + 4*n_embd + 2*n_ff` floats.
- **One-time**: `n_embd + n_head*max_seq + n_embd + n_vocab` floats.
- **max_seq cap**: `min(n_ctx, sliding_window if > 0, 4096)` — the scores buffer is bounded at 4096 (engine.c:374).
- **Alignment overhead**: 16 bytes × (n_layer×11+5) allocations, plus an additional 5 per layer for MoE.
- **Safety margin**: +25% (engine.c:405).
The arena resets between forward calls; nothing in it persists across tokens.
---
## Async forward-step seam (C2)
For the batched GPU serving path a non-blocking single-token step is exposed (engine.h:228):
```c
int pn_forward_step_begin(pn_model *model, uint32_t token, pn_state *state);
int pn_forward_step_poll(pn_model *model, pn_state *state, tm_tensor **logits_out);
```
**begin** (engine.c:1173): dispatches to `pn_qwen35moe_forward_step_begin` when `model->forward == pn_qwen35moe_forward_cuda`; returns 1 ("unsupported") for everything else — the caller uses `model->forward` blocking path instead.
**poll** (engine.c:1182): dispatches to `pn_qwen35moe_forward_step_poll`; returns 1 while running, 0 when done (writes `*logits_out`), -1 on error. Returns -1 for unsupported models.
The design: each connection thread owns one model slot and parks in a poll loop between `begin` and `done`. Because `begin` returns without blocking, the 2-core K80 host can keep multiple cards' steps in flight simultaneously. The per-model `gen_lock` in the serve layer guarantees one in-flight step per model at a time.
---
## Admission helpers (engine-level)
These live in `engine.c:426–534` and are the engine-level face of the memory-aware admission system (full detail in `memory-aware.md`).
### pn_admit (engine.c:481)
```c
pn_admit_status pn_admit(pn_model *model, int n_tokens, int *fit_tokens_out);
```
Called from the serve layer before any prefill work. Returns `PN_ADMIT_FIT`, `PN_ADMIT_CAPPED`, or `PN_ADMIT_DENIED`.
Steps:
1. Fail-open if `!pn_memwatch_running()` — returns `FIT` with no check (engine.c:487).
2. `pn_memwatch_read(&st)` — reads the live seqlock-protected snapshot.
3. Determines `kv_q4` (env `QWEN35_KV_Q4`), `gpu_resident` (`forward == pn_qwen35moe_forward_cuda`), `die_id` (tightest-free-VRAM die), and `max_window` (clamped by `QWEN35_MAX_SEQ` and `model->config.n_ctx`).
4. **Ladder rung 3** (engine.c:504): calls `pn_ladder_plan_from_state` and, if `ladder.prefill_window < max_window`, clamps `max_window` and logs `[ladder] TIGHT -> prefill window clamped 16384 -> 8192`.
5. Calls `pn_admit_request` with the adjusted `max_window` and the full `pn_mem_state`.
6. Maps `PN_ADMIT_CAP` → `PN_ADMIT_CAPPED` (sets `*fit_tokens_out`), `PN_ADMIT_REJECT` → `PN_ADMIT_DENIED`.
### Internal helpers (engine.c:427–479)
| Helper | Purpose |
|---|---|
| `pn_admit_kv_q4()` | Returns `true` unless `QWEN35_KV_Q4=0` |
| `pn_admit_gpu_resident(model)` | `true` iff `model->forward == pn_qwen35moe_forward_cuda` |
| `pn_admit_max_window(model)` | Reads `QWEN35_MAX_SEQ`, clamps to `PN_QWEN35MOE_MAX_SEQ_CAP` and `config.n_ctx` |
| `pn_admit_worst_die(st)` | Finds the die with the least free VRAM (the conservative scheduling target) |
---
## Sampler seam
```c
uint32_t pn_sample_next_token(float *logits, int n_vocab,
pn_sampling_params *params,
uint32_t *recent_tokens, int n_recent); // engine.c:1160
```
Public wrapper exposing the internal `sample_token` to the serve layer, so it does not re-implement sampling. Mutates `logits[]` in place. Not thread-safe (shared thread-local scratch buffer per vocab size).
---
## Architecture registry
`pn_get_forward_fn(pn_arch arch)` (engine.c:214) maps arch enum → forward function pointer. Llama, Qwen2, Mistral, Mixtral all share `pn_llama_forward`; only Phi and Qwen3.5-MoE have distinct implementations.
---
## State lifecycle
```
pn_state_new(model) → allocates kv_cache + arena (arch-specific)
pn_state_reset(state) → zeros kv tensors, resets pos to 0 (engine.c:923)
pn_state_free(state) → frees kv_cache, arena, struct
```
`pn_state_reset` for Llama/Phi explicitly zeros every K and V tensor in the cache (engine.c:936–943). For Qwen3.5-MoE and RWKV-X it delegates to arch-specific reset functions.

229
docs/architecture/engine.md Normal file
View File

@ -0,0 +1,229 @@
# Engine Architecture
The engine (`src/core/engine.c`) is Pantheon's central coordinator. It handles model loading, architecture dispatch, tokenization, sampling, and the generation loop.
## Model Loading Pipeline
Two entry points: `pn_load(path)` for GGUF models (legacy), and `pn_load_ex(path, opts)` for all formats including safetensors.
```
pn_load_ex(path, opts)
├── pn_is_safetensors(path)?
│ ├── yes → RWKV-X path:
│ │ ├── pn_rwkvx_load(path, config, quantize) load safetensors weights
│ │ ├── pn_rwkvx_load_tokenizer(weights, vocab) load trie tokenizer from vocab file
│ │ ├── forward = pn_rwkvx_forward set forward function
│ │ └── tokenizer = NULL (RWKV-X uses internal trie tokenizer)
│ │
│ └── no → GGUF path:
│ ├── pn_gguf_open(path) mmap file, parse header + metadata
│ ├── detect arch from metadata "general.architecture" → pn_arch enum
│ ├── pn_llama_load_qweights(gguf) load weights (architecture-specific)
│ ├── extract tokenizer from GGUF tokens + merges/scores arrays
│ ├── pn_get_forward_fn(arch) select forward pass function pointer
│ └── detect chat template style
│
└── create thread pool (if USE_THREADS=1)
```
`pn_load(path)` is equivalent to `pn_load_ex(path, NULL)`.
### Load Options
```c
typedef struct {
const char *vocab_path; /* RWKV World tokenizer (required for safetensors) */
bool quantize; /* INT8 quantize at load time */
} pn_load_opts;
```
### Format Detection
| Function | Detection Method |
|----------|-----------------|
| `pn_is_gguf(path)` | First 4 bytes = `"GGUF"` magic |
| `pn_is_safetensors(path)` | 8-byte LE header length < 10M, followed by `{` JSON |
### Architecture Detection
GGUF models: the `general.architecture` metadata key maps to:
| Value | Enum | Forward Pass |
|-------|------|-------------|
| `"llama"` | `PN_ARCH_LLAMA` | `pn_llama_forward` |
| `"qwen2"` | `PN_ARCH_QWEN2` | `pn_llama_forward` (shared) |
| `"phi"`, `"phi3"` | `PN_ARCH_PHI` | `pn_phi3_forward` (separate) |
| `"mistral"` | `PN_ARCH_MISTRAL` | `pn_llama_forward` (shared) |
| `"llama"` + `expert_count > 0` | `PN_ARCH_MIXTRAL` | `pn_llama_forward` (shared, MoE FFN) |
| unknown | `PN_ARCH_UNKNOWN` | falls back to Llama |
Safetensors models: architecture is always `PN_ARCH_RWKVX` (currently the only safetensors architecture).
Qwen2 and Mistral share the Llama forward pass — architecture differences (attention biases, sliding window, metadata key prefixes) are handled by config flags.
Phi-3 has its own forward pass (`src/models/phi3.c`) due to fused QKV weights, fused gate+up weights, and NeoX-style RoPE that differ significantly from the Llama code path.
RWKV-X has its own complete engine (`src/models/rwkvx.c`) — a hybrid recurrent + sparse attention architecture that is fundamentally different from transformer models. See [RWKV-X Architecture](rwkvx.md) for details.
### Config Parsing
Model dimensions are read from GGUF metadata using architecture-specific key prefixes:
```
llama.* → Llama, Mistral (Mistral GGUFs use llama.* keys)
qwen2.* → Qwen2/Qwen2.5
phi3.* → Phi-3 (read by phi3.c's own config reader)
```
Key parameters:
| Config Field | GGUF Key | Default |
|-------------|----------|---------|
| `n_vocab` | `{arch}.embedding_length` | - |
| `n_embd` | `{arch}.embedding_length` | - |
| `n_head` | `{arch}.attention.head_count` | - |
| `n_kv_head` | `{arch}.attention.head_count_kv` | `n_head` |
| `n_layer` | `{arch}.block_count` | - |
| `n_ff` | `{arch}.feed_forward_length` | - |
| `n_ctx` | `{arch}.context_length` | 2048 |
| `rope_theta` | `{arch}.rope.freq_base` | 10000.0 |
| `rms_norm_eps` | `{arch}.attention.layer_norm_rms_epsilon` | 1e-5 |
| `sliding_window` | `{arch}.attention.sliding_window` | 0 (disabled) |
## Tokenizers
Pantheon supports three tokenizer types, routing automatically based on model architecture.
### GGUF Tokenizer Extraction
For GGUF models, tokenizer data is extracted from metadata arrays:
- `tokenizer.ggml.tokens` — vocabulary (string array)
- `tokenizer.ggml.merges` — BPE merge rules (string array, BPE only)
- `tokenizer.ggml.scores` — unigram scores (float array, SentencePiece only)
- `tokenizer.ggml.bos_token_id` — beginning-of-sequence token
- `tokenizer.ggml.eos_token_id` — end-of-sequence token
- `tokenizer.ggml.model` — tokenizer type (`"gpt2"` = BPE, `"llama"` = SentencePiece)
### Tokenizer Types
| Type | Models | Encoding | BOS | Vocab |
|------|--------|----------|-----|-------|
| BPE | SmolLM2, Qwen2.5, Mistral v0.3 | GPT-2 byte encoding + BPE merges | No | Varies |
| SentencePiece | Mistral v0.1, Llama 1/2 | Leading space prepend + Viterbi unigram | Yes (auto) | Varies |
| RWKV World | RWKV-X | Trie-based greedy longest-match on UTF-8 bytes | No | 65,536 |
BPE and SentencePiece tokenizers use Foundry's `ft_tokenizer` and are stored in `pn_model->tokenizer`.
The RWKV World tokenizer is fundamentally different — it's a trie-based tokenizer that operates on raw UTF-8 bytes with greedy longest-match. It's stored inside the RWKV-X internal model struct (accessed via `pn_rwkvx_encode`/`pn_rwkvx_decode`), and `pn_model->tokenizer` is NULL for RWKV-X models. The engine's `pn_generate()`/`pn_stream()` functions detect `PN_ARCH_RWKVX` and route to the internal tokenizer automatically.
## Generation Loop
```
pn_generate(model, state, prompt, max_tokens, params)
├── encode prompt → token IDs
│ ├── BPE: gpt2_bytes_encode(prompt) → ft_encode()
│ └── SentencePiece: ft_encode_with_special(prompt, add_bos=true)
│
├── prefill: for each prompt token
│ └── logits = model->forward(model, token, state)
│
├── decode: for i in 0..max_tokens
│ ├── next_token = sample_token(logits, params)
│ ├── if next_token == EOS: break
│ └── logits = model->forward(model, next_token, state)
│
└── decode token IDs → output string
```
### Sampling
Applied in order:
1. **Temperature** — `logits[i] /= temperature` (0 = greedy argmax)
2. **Repetition penalty** — penalize tokens in the recent window (`repeat_last_n`)
3. **Top-k** — keep only the k highest-probability tokens
4. **Top-p (nucleus)** — keep tokens until cumulative probability exceeds p
5. **Softmax** — convert to probabilities
6. **Multinomial sampling** — weighted random selection
## Vision-Language and Bulk OCR
The vision-language path (`pn_describe` / `pn_describe_stream`) is split into a vision-encode seam and a decode seam so a bulk pipeline can stage work and run vision and decode on different units.
```
pn_describe_stream(image, prompt)
├── pn_vision_encode_rgb(image) → [n_img x n_embd] decoder-space embeds
└── pn_describe_embeds(embeds, prompt)
├── build ChatML vision turn:
│ <|vision_start|> + <|image_pad|> * n_img + <|vision_end|> + prompt
├── splice the supplied embeds at the image-pad positions
├── batched prefill (pn_llama_prefill)
└── decode loop → text
```
- `pn_vision_encode_rgb` runs just the CPU vision encoder for one image and returns the `[n_img x n_embd]` embeddings (caller owns them). Splitting it out lets a bulk caller encode ahead of decode, or substitute an external encoder (e.g. a Core ML GPU or hand-rolled Metal ViT) that produces the same `[n_img x n_embd]` buffer.
- `pn_describe_embeds` is the decode seam: it takes pre-supplied embeds instead of encoding, and is what `pn_describe_stream` now delegates to after its own encode. The portable single-image path is unchanged.
### Continuous-Batch VLM Decoder
For bulk OCR (many pages at once), the pipeline drives many sequences through the batched decode (`pn_llama_forward_batch`) at the higher aggregate throughput described in [Llama Forward Pass](llama-forward.md#batched-multi-sequence-decode).
- `pn_vlm_prefill_embeds` factors the shared prefill out of `pn_describe_embeds`: it prefills a state's KV cache from image embeds (the ChatML vision turn) and samples the first token, but does not run the decode loop. It leaves the state ready to hand to the batch decoder.
- `pn_batch_decoder` admits prefilled pages into up to `max_batch` slots. Each `step()` runs one `pn_llama_forward_batch` over all active slots with per-sequence sampling, EOS detection, and detokenization, and returns each page that finished this step as a `pn_page_out` (`page_id`, the now-recyclable `pn_state`, and the decoded text). The caller slots a fresh page into each freed slot (continuous batching). `flush()` drains all active slots for early stop.
```
for each page: pn_vision_encode_rgb → pn_vlm_prefill_embeds → pn_batch_decoder_admit
loop while active > 0:
n = pn_batch_decoder_step(out, max_out) # one M=B decode over all slots
for each finished page in out: emit text, recycle state, admit next page
```
The batched decoder is byte-identical to sequential `pn_describe_embeds`: two pages with different embeds decoded together reproduce each page's standalone greedy transcription and finish at their own lengths (verified by `test_batch_decoder`). If the model does not support batched decode (MoE, F32 legacy, ternary), `pn_batch_decoder_step` returns -1 and the caller falls back to the sequential path.
## Memory Model
```
pn_model
├── config model dimensions (stack allocated)
├── weights architecture-specific (heap, points into mmap or owned memory)
├── tokenizer BPE/SentencePiece tokenizer (heap, NULL for RWKV-X)
├── forward forward pass function pointer
├── path model file path (heap)
├── chat_style chat template enum
└── pool thread pool (optional, heap)
pn_state
├── kv_cache architecture-specific state:
│ GGUF models: pn_llama_kv_cache (KV tensors per layer)
│ RWKV-X: rx_state (recurrent state + KV cache + workspace)
├── pos current sequence position
├── config pointer to model's config
└── arena bump allocator for per-token scratch (NULL for RWKV-X)
```
For GGUF models, the arena allocator eliminates per-token `malloc`/`free` calls during inference. All intermediate buffers (projections, activations, attention scores) are allocated from the arena, which resets to zero after each token. This removes ~330 allocations per token in the forward pass.
For RWKV-X models, the engine uses its own pre-allocated workspace with ping-pong buffers. The arena is not used.
## Weight Loading Modes
The engine supports two weight storage modes:
### Quantized (Phase 2, default)
Weights stay in their original GGUF quantized format, pointing directly into mmap'd memory. Dequantization happens per-block during matrix-vector multiply (`pn_qmv`). Only embeddings and layer norms are dequantized to F32 on load.
Benefits: ~4x less RAM for Q4 models, zero-copy weight loading.
### F32 Legacy (Phase 1 fallback)
All weights dequantized to F32 `tm_tensor` on load. Used as fallback for unsupported quant types. Higher memory usage but simpler code path.
## Thread Pool
When built with `USE_THREADS=1`, a thread pool is created at model load. The main thread participates in work (no idle main thread). Currently used for:
- Parallel attention head computation — each head runs independently via `fw_for(pool, n_heads, fn, arg)`
Thread count is auto-detected from hardware.

151
docs/architecture/gguf.md Normal file
View File

@ -0,0 +1,151 @@
# GGUF Format Support
Pantheon's GGUF parser (`src/formats/gguf.c`) provides zero-copy model loading via mmap. Both GGUF v2 and v3 are supported.
## File Layout
```
┌──────────────────────────┐
│ Magic: "GGUF" (4 bytes) │
│ Version: uint32 │ v2 or v3
│ Tensor count: uint64 │
│ Metadata count: uint64 │
├──────────────────────────┤
│ Metadata entries │ typed key-value pairs
│ (variable length) │
├──────────────────────────┤
│ Tensor info entries │ name, dims, quant type, offset
│ (variable length) │
├──────────────────────────┤
│ Padding │ aligned to 32 bytes (GGUF_DEFAULT_ALIGNMENT)
├──────────────────────────┤
│ Tensor data (binary) │ mmap'd, accessed via offset
└──────────────────────────┘
```
### Alignment
The tensor data section starts at a 32-byte aligned offset from the file start. This is the GGUF spec default (`GGUF_DEFAULT_ALIGNMENT = 32`). Files can override this via the `general.alignment` metadata key.
### v2 vs v3
Both versions use the same binary layout. The parser accepts either. v3 added big-endian support (Pantheon assumes little-endian). The version field is stored in `pn_gguf_file.version`.
## Metadata Types
13 value types supported:
| Type | ID | Size |
|------|----|------|
| UINT8 | 0 | 1 |
| INT8 | 1 | 1 |
| UINT16 | 2 | 2 |
| INT16 | 3 | 2 |
| UINT32 | 4 | 4 |
| INT32 | 5 | 4 |
| FLOAT32 | 6 | 4 |
| BOOL | 7 | 1 |
| STRING | 8 | 8 + len |
| ARRAY | 9 | variable |
| UINT64 | 10 | 8 |
| INT64 | 11 | 8 |
| FLOAT64 | 12 | 8 |
Strings are length-prefixed (uint64 length + raw bytes). Arrays contain a type tag + uint64 count + elements.
## Quantization Types
### Implemented (full dequant + fused matvec)
| Type | ID | Bits/Weight | Block Size | Bytes/Block | Notes |
|------|----|-------------|------------|-------------|-------|
| F32 | 0 | 32 | 1 | 4 | Passthrough |
| F16 | 1 | 16 | 1 | 2 | IEEE 754 half |
| Q4_0 | 2 | 4 | 32 | 18 | scale + 4-bit nibbles |
| Q8_0 | 8 | 8 | 32 | 34 | scale + int8 values |
| Q4_K | 12 | 4.5 | 256 | 144 | 8 sub-blocks, 6-bit scales |
| Q5_K | 13 | 5.5 | 256 | 176 | 5-bit values (4 low + 1 high) |
| Q6_K | 14 | 6.5 | 256 | 210 | 6-bit values (4 low + 2 high) |
### Format-only (defined but no dequant kernel)
Q4_1, Q5_0, Q5_1, Q8_1, Q2_K, Q3_K, Q8_K
### Mixed Quantization (Q4_K_M, Q5_K_M)
GGUF files labeled "Q4_K_M" use Q4_K for most layers with Q6_K for attention output and FFN down projections. This is handled transparently — each tensor has its own quant type, and `pn_qmv` dispatches per-row based on the weight's type field.
## Key Metadata Keys
### Model Architecture
| Key | Type | Description |
|-----|------|-------------|
| `general.architecture` | string | `"llama"`, `"qwen2"`, `"phi"`, `"mistral"` |
| `general.name` | string | Human-readable model name |
| `general.alignment` | uint32 | Data section alignment (default 32) |
### Model Dimensions
Prefixed by architecture name (`llama.*`, `qwen2.*`, etc.):
| Key | Type | Description |
|-----|------|-------------|
| `{arch}.embedding_length` | uint32 | Hidden size (n_embd) |
| `{arch}.attention.head_count` | uint32 | Query heads |
| `{arch}.attention.head_count_kv` | uint32 | KV heads (for GQA) |
| `{arch}.block_count` | uint32 | Number of layers |
| `{arch}.feed_forward_length` | uint32 | FFN intermediate size |
| `{arch}.context_length` | uint32 | Max context length |
| `{arch}.rope.freq_base` | float32 | RoPE theta (default 10000) |
| `{arch}.attention.layer_norm_rms_epsilon` | float32 | RMSNorm epsilon |
| `{arch}.attention.sliding_window` | uint32 | Sliding window size (Mistral) |
### Tokenizer
| Key | Type | Description |
|-----|------|-------------|
| `tokenizer.ggml.model` | string | `"gpt2"` (BPE) or `"llama"` (SentencePiece) |
| `tokenizer.ggml.tokens` | string[] | Vocabulary tokens |
| `tokenizer.ggml.merges` | string[] | BPE merge rules (BPE only) |
| `tokenizer.ggml.scores` | float[] | Unigram scores (SentencePiece only) |
| `tokenizer.ggml.bos_token_id` | uint32 | BOS token ID |
| `tokenizer.ggml.eos_token_id` | uint32 | EOS token ID |
## Tensor Naming Convention
GGUF tensors follow the pattern `blk.{layer}.{component}.weight`:
```
token_embd.weight embedding table
blk.0.attn_norm.weight pre-attention RMSNorm
blk.0.attn_q.weight query projection
blk.0.attn_k.weight key projection
blk.0.attn_v.weight value projection
blk.0.attn_output.weight output projection
blk.0.attn_q.bias query bias (Qwen2 only)
blk.0.ffn_norm.weight pre-FFN RMSNorm
blk.0.ffn_gate.weight SwiGLU gate projection
blk.0.ffn_up.weight SwiGLU up projection
blk.0.ffn_down.weight down projection
output_norm.weight final RMSNorm
output.weight LM head (may be tied to token_embd)
```
## Quantized Weight Loading
Weights are loaded as `pn_qweight` structs that point directly into mmap'd data:
```c
typedef struct {
const void *data; /* raw quantized blocks in mmap */
pn_gguf_quant type; /* quant format */
int rows; /* output dimension */
int cols; /* input dimension */
size_t row_stride; /* bytes per row */
} pn_qweight;
```
GGUF stores dimensions innermost-first. The loader reverses them to C outermost-first: `dims[n_dims-1]` = rows, `dims[0]` = cols. `row_stride` is computed from cols and the quant format's bytes-per-block.
Matrix-vector multiply (`pn_qmv`) iterates rows, dequantizing one block (32 or 256 elements) at a time and accumulating the dot product. No full dequantization ever occurs for projection weights.

264
docs/architecture/llama-forward.md Normal file
View File

@ -0,0 +1,264 @@
# Llama Forward Pass
The Llama forward pass (`src/models/llama.c`) is shared by Llama, Qwen2, Mistral, and Mixtral architectures. Architecture-specific differences (attention biases, sliding window, MoE routing, metadata keys) are handled by config flags, not separate code paths.
Phi-3 has its own forward pass in `src/models/phi3.c` due to fused QKV/gate+up weights and NeoX-style RoPE. See the architecture differences table below.
## Forward Pass Flow
```
token (uint32)
│
├── embed: tok_embd[token] → hidden[n_embd]
│
├── for each layer (0..n_layer-1):
│ │
│ ├── RMSNorm(hidden, attn_norm)
│ │
│ ├── Attention:
│ │ ├── Q = Wq @ normed (+bias for Qwen2)
│ │ ├── K = Wk @ normed (+bias for Qwen2)
│ │ ├── V = Wv @ normed (+bias for Qwen2)
│ │ ├── RoPE(Q, K, pos, rope_theta)
│ │ ├── Write K,V to KV cache at pos
│ │ ├── GQA: for each query head
│ │ │ ├── scores = Q_head @ K_cache^T / sqrt(head_dim)
│ │ │ ├── causal mask (future tokens = -inf)
│ │ │ ├── softmax(scores)
│ │ │ └── output = scores @ V_cache
│ │ └── O = Wo @ concat(heads) (+bias for Qwen2)
│ │
│ ├── residual: hidden += O
│ │
│ ├── RMSNorm(hidden, ffn_norm)
│ │
│ ├── FFN (standard) or MoE FFN (Mixtral):
│ │ │
│ │ ├── Standard SwiGLU (n_experts == 0):
│ │ │ ├── gate = W_gate @ normed
│ │ │ ├── up = W_up @ normed
│ │ │ ├── act = silu(gate) * up
│ │ │ └── down = W_down @ act
│ │ │
│ │ └── MoE routing (n_experts > 0, Mixtral):
│ │ ├── gate_logits = W_router @ normed → [n_experts]
│ │ ├── probs = softmax(gate_logits)
│ │ ├── top_k = select top n_experts_used
│ │ ├── weights = renormalize(selected probs)
│ │ ├── for each selected expert e:
│ │ │ ├── gate_e = expert[e].W_gate @ normed
│ │ │ ├── up_e = expert[e].W_up @ normed
│ │ │ ├── act_e = silu(gate_e) * up_e
│ │ │ └── out_e = expert[e].W_down @ act_e
│ │ └── down = Σ (weight_e * out_e)
│ │
│ └── residual: hidden += down
│
├── RMSNorm(hidden, output_norm)
│
└── logits = W_output @ normed → float[n_vocab]
```
## Key Operations
### RMSNorm
```
rms = sqrt(mean(x^2) + eps)
out = (x / rms) * weight
```
Implemented as a single SIMD kernel (`pn_vrms_norm_f32`). Default `eps = 1e-5`, configurable per model via GGUF metadata.
### RoPE (Rotary Position Embeddings)
Applies learned rotation to Q and K vectors based on position:
```
for each pair (i, i+1) in head:
freq = 1.0 / (theta ^ (2i / head_dim))
angle = pos * freq
q[i], q[i+1] = rotate(q[i], q[i+1], angle)
```
`rope_theta` defaults to 10000 (Llama). Qwen2.5 uses 1,000,000 for extended context.
### Grouped Query Attention (GQA)
GQA reduces KV heads relative to query heads. Each KV head is shared across `n_head / n_kv_head` query heads:
| Model | Q Heads | KV Heads | Ratio |
|-------|---------|----------|-------|
| SmolLM2-135M | 9 | 3 | 3:1 |
| Qwen2.5-0.5B | 14 | 2 | 7:1 |
| Mistral-7B | 32 | 8 | 4:1 |
| Phi-3-mini | 32 | 32 | 1:1 (MHA) |
The forward pass maps query head `h` to KV head `h * n_kv_head / n_head`.
### SwiGLU FFN
```
gate = W_gate @ x
up = W_up @ x
out = W_down @ (silu(gate) * up)
```
The `silu(gate) * up` step is fused into a single SIMD kernel (`pn_vsilu_mul_f32`).
### Sliding Window (Mistral)
When `sliding_window > 0`, the KV cache operates as a circular buffer:
- Write position: `pos % max_seq` (wraps around)
- Readable length: `min(pos + 1, max_seq)` (prevents reading uninitialized slots)
- `max_seq = min(sliding_window, n_ctx)`
This limits memory to a fixed window size regardless of sequence length. Standard Llama/Qwen2 use `sliding_window = 0` (full context).
### Mixture of Experts (Mixtral)
When `n_experts > 0`, the FFN sublayer is replaced with expert routing:
```
normed = RMSNorm(hidden)
gate_logits = W_router @ normed → [n_experts]
probs = softmax(gate_logits)
top_idx, top_wt = top_k(probs, n_experts_used) → select best experts
top_wt = renormalize(top_wt) → weights sum to 1
ffn_out = 0
for each selected expert e:
gate = expert[e].W_gate @ normed
up = expert[e].W_up @ normed
hid = silu(gate) * up
out = expert[e].W_down @ hid
ffn_out += top_wt[e] * out
hidden += ffn_out
```
**Expert weight formats in GGUF:**
Merged (most common): 3D tensors sliced into per-expert views (zero-copy mmap)
- `blk.N.ffn_gate_exps.weight` `[n_experts, n_ff, n_embd]`
- `blk.N.ffn_up_exps.weight` `[n_experts, n_ff, n_embd]`
- `blk.N.ffn_down_exps.weight` `[n_experts, n_embd, n_ff]` (transposed)
Split (older): individual tensors per expert
- `blk.N.ffn_gate.E.weight` `[n_ff, n_embd]` for E=0..n_experts-1
The loader probes for merged format first, falls back to split.
## Architecture Differences
### Shared Llama Forward Pass (llama.c)
| Feature | Llama | Qwen2 | Mistral | Mixtral |
|---------|-------|-------|---------|---------|
| Attention biases | No | Yes (Q,K,V,O) | No | No |
| Sliding window | No | No | Yes (4096) | No |
| MoE experts | No | No | No | Yes (n_experts, top-k) |
| GGUF key prefix | `llama.*` | `qwen2.*` | `llama.*` | `llama.*` |
| GGUF arch string | `llama` | `qwen2` | `mistral` | `llama` (detected via `expert_count`) |
| RoPE theta | 10000 | 1000000 | 10000 | 10000 |
| RoPE style | Interleaved | Interleaved | Interleaved | Interleaved |
| Q/K/V weights | Separate | Separate | Separate | Separate |
| FFN weights | Separate | Separate | Separate | Per-expert (merged or split) |
| Tied embeddings | model-dependent | model-dependent | No | No |
Note: Mistral GGUFs use `llama.*` metadata keys (not `mistral.*`), but `general.architecture = "mistral"`. Mixtral GGUFs use `general.architecture = "llama"` and are detected via `llama.expert_count > 0`.
### Phi-3 (phi3.c — separate forward pass)
| Feature | Phi-3 |
|---------|-------|
| GGUF key prefix | `phi3.*` |
| Q/K/V weights | Fused `attn_qkv.weight` [3*n_embd, n_embd] |
| Gate/Up weights | Fused `ffn_up.weight` [2*n_ff, n_embd] |
| RoPE style | NeoX (split-half pairs: `head[i], head[i+half]`) |
| RoPE dim | Configurable via `rope.dimension_count` (96 for mini) |
| Attention | MHA (n_head == n_kv_head = 32) |
| Tied embeddings | No (separate output.weight) |
Phi-3 uses its own forward pass because the fused weight layout requires different matvec + split logic, and NeoX RoPE pairs elements differently from the interleaved style used by Llama/Qwen2/Mistral.
## Weight Storage
Two modes, selected automatically based on quant type support:
### Quantized (default)
All projection weights (`Wq`, `Wk`, `Wv`, `Wo`, `W_gate`, `W_up`, `W_down`, `W_output`) stay in mmap'd quantized form as `pn_qweight` structs. Matrix-vector multiply happens via `pn_qmv()` which dequantizes one block at a time.
Embeddings and norms are always F32 (owned memory), since they're accessed by index (embeddings) or element-wise (norms).
### F32 legacy
All weights dequantized to `tm_tensor` on load. Fallback for unsupported quant types. Uses standard `tm_matmul` for projections.
## Per-Token Memory
All intermediate buffers are allocated from a bump arena that resets after each token:
| Buffer | Size | Used For |
|--------|------|----------|
| `normed` | n_embd | RMSNorm output |
| `q`, `k`, `v` | n_embd | Q/K/V projections |
| `attn_out` | n_embd | Attention output |
| `ffn_gate`, `ffn_up` | n_ff | FFN intermediates |
| `ffn_act` | n_ff | SiLU activation |
| `ffn_out` | n_embd | FFN output |
| `logits_raw` | n_vocab | LM head output |
| `scores` | max_seq per head | Attention scores |
For MoE layers (Mixtral), additional per-layer buffers:
| Buffer | Size | Used For |
|--------|------|----------|
| `gate_logits` | n_experts | Router output |
| `ffn_out` | n_embd | Weighted expert accumulator |
| `exp_gate` | n_ff | Per-expert gate projection |
| `exp_up` | n_ff | Per-expert up projection |
| `exp_hid` | n_ff | Per-expert SiLU activation |
| `exp_out` | n_embd | Per-expert down projection |
Arena is sized at state creation: `sum of all buffers * sizeof(float) * 1.25` (25% safety margin). Typical sizes: 1.5 KB (SmolLM2-135M), 11 KB (Mistral-7B), 19 KB (Mixtral_7Bx2).
## Batched Multi-Sequence Decode
`pn_llama_forward_batch()` decodes one token for each of `B` independent sequences in a single pass, for bulk throughput (e.g. transcribing many pages at once). Each `states[b]` carries its own KV cache and position, so the sequences can have different lengths and each attends only to its own cache; row `b`'s input is `tokens[b]` or, for image-token positions, an `embd_rows[b]` override. It writes `[B, n_vocab]` row-major logits and advances every state's `pos` by one.
The batched path is restricted to the quantized, non-MoE Llama/Qwen weights. For MoE, F32 legacy, or ternary weights it returns without mutating any state so the caller can fall back to per-sequence `pn_llama_forward`. Results are bit-identical to running `B` separate sequential decodes (verified by `test_batch_decode_correct`: a `B=2` batch reproduces each sequence's standalone greedy stream with no cross-contamination).
### Fused AMX-panel M=B matmul (`pn_qmm`)
The projection and FFN matmuls in the batched path go through `pn_qmm` (`src/core/qweight.c`), a fused quantized `M=B` kernel: `out[b, r] = sum_c dequant(W[r, c]) * acts[b, c]`. Each weight row is dequantized once into a small L2-resident panel and multiplied against all `B` activation vectors, so the quantized weight is read and dequantized once per step while the matmul runs at AMX throughput (the panel GEMM lowers to AMX under BLAS). This avoids both the NEON-dot ceiling of the single-vector `pn_qmv` and the full-weight F32 copy to DRAM that a plain BLAS GEMM would need.
Because the per-token matmul FLOPs are irreducible, throughput is compute-bound and plateaus rather than scaling linearly with `B`. Measured on LightOnOCR 1B Q4_K, M4, against 38.8 tok/s single-stream `pn_qmv`:
| path | B=16 | B=32 |
|------|------|------|
| full-BLAS F32 copy | 49.7 | 46.5 (DRAM-bound, regresses) |
| NEON fused dot | 58.4 | - |
| AMX panel (`pn_qmm`) | 72.5 | 84.3 |
The AMX panel gives roughly 2.2x aggregate CPU throughput and settles near ~85 tok/s. `B=1` is intentionally suboptimal (the panel adds a scratch pass over the fused `pn_qmv`), so single sequences should use `pn_llama_forward`. `pn_qmm` falls back to a per-`b` `pn_qmv` loop above `PN_QMM_MAX_B` (64) or for ternary or GPU-resident weights.
## Flash (Blocked) Prefill Attention
Prefill over a long image-token sequence was the bulk-OCR bottleneck. The original per-query attention recomputed over the whole K/V cache once per query position, giving O(P^2) DRAM traffic across the ~1369 image-token queries of a page.
`prefill_flash_range` processes a block of queries (`PF_BQ=32`) together and streams K and V in blocks (`PF_BK=256`) that stay cache-resident and are reused across the whole query block. This cuts K/V reads from about `P` times to about `P/BQ`. The math is unchanged from the per-query path (full-precision softmax, with the scores tile kept L2-resident), and a per-query fallback is retained for the sliding-window wraparound case. It is parallelized over query heads.
Measured: prefill ~4.3 to ~3.2 s/page (~1.33x), bulk OCR ~6.3 to ~5.2 s/page steady state. It is pure CPU, so it helps regardless of the vision backend and without contending for the GPU (unlike GPU prefill below). Output stays correct and batched decode stays bit-identical.
## Optional GPU Paths (off by default)
Two Metal offload paths exist behind build flags and environment switches. Both are **off by default** because, on the current hardware, they lose to the CPU AMX path or contend with the vision GPU. The shipped app builds Pantheon without `USE_METAL`, so both are inert there.
- **GPU batched decode** (`PN_BATCH_GPU`, gated at runtime by `pn_qmm_set_gpu_decode`): uploads the Q4_K decoder weights once (`pn_qmv_gpu_upload`) and runs the `M=B` GEMM on the GPU (`fm_sgemm_q4_k`). Bit-identical to the CPU path, but not a win: `pn_qmm` stages, dispatches, syncs, and copies back per weight, so a step does ~168 CPU/GPU round-trips and the per-call sync latency dominates (~68 tok/s at `B=16` vs ~86 tok/s for the CPU AMX panel). Auto-batching cannot hide it because each GEMM's result is immediately consumed by a CPU op (bias, RoPE, attention). Beating AMX would need a GPU-resident decode (the whole layer chain on GPU, one sync per step); the flag is kept as the seam for that.
- **GPU F16 prefill** (`PN_GPU_PREFILL`): uploads the Q4_K decoder weights once and runs each prefill projection as an F16 MPS GEMM (`fm_gemm_q4_k_f16`) instead of the CPU dequant+AMX path. Also a net loss for bulk OCR, for two reasons: prefill is attention-bound, not GEMM-bound (the O(P^2) decoder attention over the image tokens dominates, so GPU-accelerating only the GEMMs is ~1.3x, not the ~5x a GEMM microbench suggests), and it contends with Core ML vision for the single GPU, breaking the CPU-compute / GPU-vision overlap that makes bulk fast (~6.3 s/page went to ~8-10 s/page). It is kept as an opt-in for the CPU-vision case, where the GPU is otherwise idle. Setting it also forces decode back onto the CPU AMX path (via the `pn_qmm_set_gpu_decode` gate) so the prefill upload does not drag decode onto the slower GPU route.
ABI note: `pn_qweight.gpu` is an unconditional struct field, present in every build (just `NULL` and unused without `USE_METAL`). It must not be wrapped in `#ifdef USE_METAL`: a conditional field makes the struct layout depend on the build flag, so a translation unit compiled the other way reads later fields at the wrong offset.

264
docs/architecture/memory-aware.md Normal file
View File

@ -0,0 +1,264 @@
# Memory-Aware Execution Layer
Sources: `src/core/mem/pn_memwatch.c`, `src/core/mem/pn_memwatch.h`, `src/core/mem/pn_mem_state.h`, `src/core/mem/pn_host_signals.h`, `src/core/mem/pn_host_signals.c`, `src/core/mem/pn_ladder.c`, `src/core/mem/pn_ladder.h`, `src/core/mem/pn_admission.c`, `src/core/mem/pn_admission.h`. Entry points wired in `src/core/engine.c`, `src/serve/serve_generation.c`, `src/serve/batch_sched.c`, `src/models/qwen35moe_forward_cuda.c`.
---
## Overview
The memory-aware layer tracks host RAM pressure (PSI + `/proc/meminfo`) and per-die VRAM in a continuously updated `pn_mem_state` snapshot, derives a four-level pressure enum, and exposes two behaviors to the rest of the engine:
1. **Pre-flight admission** (`pn_admit_request`, `pn_admit_model_load`): refuse or cap a request before any memory-growing work starts.
2. **Reactive ladder** (`pn_ladder_plan_for`): translate the current pressure level into concrete mitigation rungs (fewer slots, smaller context window, forced Q4 KV, KV eviction flag) that the serve and CUDA layers apply.
---
## Published state: `pn_mem_state` (pn_mem_state.h:46)
One immutable, value-type snapshot written by `pn_memwatch` and read by admission and the ladder. Seqlock-protected so readers never block the writer.
```c
typedef struct {
uint64_t host_total_bytes; // MemTotal / sysctl hw.memsize
uint64_t host_avail_bytes; // MemAvailable (allocatable without swap)
double host_psi_some_avg10; // /proc/pressure/memory "some avg10" %
double host_psi_full_avg10; // "full avg10" % — whole-box stall
uint64_t cgroup_current_bytes; // memory.current (0 if not in cgroup)
uint64_t cgroup_high_bytes; // memory.high (0 == "max"/unset)
uint64_t cgroup_max_bytes; // memory.max (0 == "max"/unset)
int n_dies;
uint64_t per_die_free[16]; // cuMemGetInfo free per die
uint64_t per_die_budget[16]; // usable budget (total - reserve) per die
uint64_t per_die_used[16]; // our own alloc tally
pn_mem_level level; // derived: worst-of host + any die
uint64_t seq; // monotonic snapshot counter
} pn_mem_state;
```
PSI and cgroup fields are Linux-only; on macOS they read 0 (treated as no pressure).
---
## pn_memwatch — the poll loop (pn_memwatch.c)
One background thread polls host signals every `PN_MEMWATCH_POLL_MS` (250 ms, pn_memwatch.h:46). The GPU side pushes die VRAM numbers via `pn_memwatch_set_die`.
**Seqlock protocol** (pn_memwatch.c:34):
- `g_seq` even = stable. Writer atomically increments to odd, copies state, increments back to even.
- Readers loop until `s1 == s2` (both even, copy stable). Never blocks the writer.
**Start / stop**:
```c
int pn_memwatch_start(void); // spawns poll thread; idempotent
void pn_memwatch_stop(void); // joins thread
bool pn_memwatch_running(void); // true once started
void pn_memwatch_poll_now(void);// force one immediate poll (no wait for next tick)
```
`pn_memwatch_start` runs one synchronous `poll_host_once` before creating the thread so the first `pn_memwatch_read` call returns real numbers (pn_memwatch.c:292).
**Die feed** (pn_memwatch.c:244):
```c
void pn_memwatch_set_die(int die_id, uint64_t free_bytes,
uint64_t budget_bytes, uint64_t used_bytes);
```
Called from the GPU side after each cuMemGetInfo query. Holds `g_write_lock`, merges into the state, re-derives the level, and publishes via the seqlock.
---
## Host signals (pn_host_signals.c)
`pn_host_signals_sample` populates a `pn_host_sample` with:
- **Linux**: `/proc/meminfo` (MemTotal, MemAvailable) + `/proc/pressure/memory` (some/full avg10) + cgroup-v2 `memory.{current,high,max}`.
- **macOS / other**: sysctl `hw.memsize` total + `host_statistics64` for available; PSI/cgroup fields stay 0.
All reads are cheap (~1 ms). The poll thread calls `pn_host_signals_sample` inside `poll_host_once` (pn_memwatch.c:217).
---
## Pressure levels (pn_mem_state.h:30)
```c
typedef enum {
PN_MEM_LEVEL_OK = 0, // ample headroom
PN_MEM_LEVEL_WATCH = 1, // headroom shrinking
PN_MEM_LEVEL_TIGHT = 2, // reserve nearly reached / PSI rising
PN_MEM_LEVEL_CRITICAL = 3, // at the reserve line / PSI stall — shed
} pn_mem_level;
```
### Host level derivation (pn_memwatch.c:105)
Reserve = `max(PN_HOST_RESERVE_MIN_BYTES=1 GiB, 15% × MemTotal)` (overridable via `PN_HOST_RESERVE_BYTES`).
| Condition | Level |
|---|---|
| `host_avail <= reserve` | CRITICAL |
| slack above reserve < reserve/4 | TIGHT |
| slack above reserve < reserve | WATCH |
| otherwise | OK |
**PSI escalation** (pn_memwatch.c:127):
- `full_avg10 >= PN_PSI_FULL_REJECT_DEFAULT` (15%, env `PN_PSI_FULL_REJECT`) → CRITICAL
- `full_avg10 > 0` → at least TIGHT (sub-threshold blip; see note below)
- `some_avg10 >= 10%` → at least TIGHT
- `some_avg10 > 0` → at least WATCH
**Important**: PSI avg10 decays over ~1 minute, so a transient spike (e.g. the ~90 s model load's dequant/staging burst) leaves `full_avg10` slightly above 0 for a full decay window after the box recovered. The PSI-full reject threshold (15%) is set high enough that this transient blip only escalates to TIGHT, not CRITICAL. Admission then still passes the byte-math check on a recovered box.
### Die level derivation (pn_memwatch.c:142)
Per die (die with `budget == 0` is skipped):
| free VRAM | Level |
|---|---|
| 0 | CRITICAL |
| < budget/16 | TIGHT |
| < budget/8 | WATCH |
| otherwise | OK |
Budget = total VRAM measured at boot − `PN_DIE_RESERVE_BYTES` (1 GiB default, env `PN_DIE_RESERVE_BYTES`).
`derive_level` returns `max(host_level, die_level)` (pn_memwatch.c:163).
### Why TIGHT is normal on the mini
The K80 test machine has 8 GiB of host RAM. The 15% reserve = ~1.2 GiB. The Qwen3.5-MoE model at steady state after load uses ~1.1 GiB RSS on host (measured 2026-07-09). Total host allocation sits well inside the TIGHT band (`slack < reserve`) at rest. This is expected and correct: TIGHT does not refuse requests on the GPU-resident path (PSI-gated, not byte-gated), and the ladder's rung-3 prefill-window clamp and rung-2 slot reduction only engage when the level is TIGHT and the box is actually under pressure. The admission gate at CRITICAL is the shed point for new concurrency (D3), not TIGHT.
---
## Admission control (pn_admission.c / pn_admission.h)
### Cost model
**Host activation peak** (`lut_activation_peak`, pn_admission.c:69): a 6-anchor LUT interpolated linearly. Anchors are conservative CPU-path measurements (the O(N²) attention scores dominate on CPU). On the GPU-resident path the O(N²) scores live on-die in bounded flash tiles — the host-RSS delta measured on a K80 host was ~0 at all tested lengths — so GPU-resident requests bypass this LUT entirely.
**KV per-token cost**: `PN_ADMIT_KV_BYTES_PER_TOKEN_Q4 = 5898` (~5.76 KiB) or `PN_ADMIT_KV_BYTES_PER_TOKEN_FP32 = 40960` (~40 KiB) (pn_admission.h:60).
**Total host cost** (pn_admission.c:98): `kv_per_tok * n_prompt + lut_activation_peak(n_prompt)`.
### pn_admit_request (pn_admission.c:153)
```c
void pn_admit_request(int n_prompt, bool kv_q4, bool gpu_resident,
int die_id, int max_window, const pn_mem_state *state,
pn_admit_result *out);
```
Decision order:
1. **PSI-full stall guard** (pn_admission.c:174): if `host_psi_full_avg10 >= reject_threshold` → REJECT ("sustained memory stall — retry").
2. **GPU-resident fast path** (pn_admission.c:193): if `gpu_resident`, host cost is declared 0. REJECT if `host_avail` is already inside the reserve; CAP if `n_prompt > max_window`; otherwise ADMIT at full context.
3. **CPU/host-fallback path** (pn_admission.c:208): check host budget (`host_avail − reserve`) and die free VRAM against the LUT cost. Binary-search for the largest fitting window if the full prompt doesn't fit; set CAP or REJECT.
### pn_admit_model_load (pn_admission.c:245)
Gates GPU weight uploads. Checks `weights_bytes + kv@max_ctx + activation_reserve <= die_free − die_reserve`. Used before uploading a model to a die.
### pn_admit_model_load_die (pn_admission.c:266)
Per-die variant for multi-die splits: checks `weights_slice + kv_slice + PN_ADMIT_DIE_SCRATCH_BYTES (1.5 GiB)` against one die's free VRAM. Does not add the host LUT (GPU-path scratch is bounded).
### pn_admit_load_spike (pn_admission.c:292)
Checks that `PN_ADMIT_LOAD_SPIKE_BYTES_DEFAULT` (2.4 GiB, dequant/staging burst measured on a K80 host during upload, env `PN_LOAD_SPIKE_BYTES`) fits within host headroom before a multi-minute weight upload starts.
### Engine-level wrapper: pn_admit (engine.c:481)
The serve layer calls `pn_admit(model, n_tokens, &fit_tokens_out)` which:
1. Fail-opens if `!pn_memwatch_running()`.
2. Reads the live state snapshot.
3. Derives `kv_q4`, `gpu_resident`, `die_id` (worst-free-VRAM die), `max_window`.
4. Consults the ladder (rung 3, see below) to possibly clamp `max_window` before calling `pn_admit_request`.
5. Maps the verdict to the public `pn_admit_status` enum.
Called from `sg_run_completion` (serve_generation.c:547) and `sg_run_completion_batched` (serve_generation.c:777) before any prefill starts.
### D3 — admission backpressure into the batch scheduler
When `sg_run_completion_batched` receives `PN_ADMIT_DENIED`, it calls `bs_notify_pressure(sched, (int)ms.level)` (serve_generation.c:786). This updates `sched->pressure_level` and broadcasts to `cv_idle` waiters. `bs_slot_acquire` (batch_sched.c:1493) then holds new requests in the queue if `pressure_level >= BS_PRESSURE_HOLD_LEVEL` (TIGHT/CRITICAL), preventing new work from being admitted until pressure clears. CRITICAL (level 3) triggers a fast-path 503 for queued waiters (batch_sched.c:1541). **D3 gates at CRITICAL, not TIGHT**: TIGHT alone does not block admission; only a DENIED verdict from `pn_admit` (which requires CRITICAL or PSI stall) pushes the pressure signal.
---
## Reactive degradation ladder (pn_ladder.c / pn_ladder.h)
The ladder is a **pure, stateless planner**: given a `pn_mem_level` and an operator config, it returns a `pn_ladder_plan` struct with absolute values (not deltas) the caller applies.
```c
typedef struct {
pn_mem_level level;
int max_slots; // rung 2: effective concurrency ceiling (>=1)
bool force_kv_q4; // rung 4a: force Q4 KV
bool kv_evict; // rung 4b: enable low-attention KV eviction (policy flag)
int prefill_window; // rung 3: max prefill tokens (0 = no clamp)
bool degraded; // true if any rung reduced a configured value
const char *note;
} pn_ladder_plan;
```
`pn_ladder_plan_for` (pn_ladder.c:51) and the convenience wrapper `pn_ladder_plan_from_state` are pure — no I/O, no allocation. At `PN_MEM_LEVEL_OK` the plan is byte-identical to the operator config.
Disable via `PN_LADDER_DISABLE` env var (`pn_ladder_disabled`, pn_ladder.c:19).
### Rung thresholds (pn_ladder.h:45)
| Rung | Name | Engages at | Action |
|---|---|---|---|
| 4a | force Q4 KV | WATCH | `force_kv_q4 = true` |
| 2 | shrink concurrency | TIGHT | `max_slots = cfg / 2`; CRITICAL → 1 |
| 3 | clamp prefill window | TIGHT | `prefill_window = cfg / 2`; CRITICAL → `cfg / 4` |
| 4b | KV eviction | TIGHT | `kv_evict = true` (policy flag; SnapKV selection not yet implemented) |
Rungs are reduce-only and monotonic: a higher pressure level never relaxes a rung a lower level engaged.
### Where the ladder is consumed
| Call site | What it reads | File:line |
|---|---|---|
| `pn_admit` (engine.c) | Rung 3 `prefill_window` to clamp `max_window` before admission | engine.c:504 |
| `pn_qwen35moe_gpu_ladder_max_slots` (qwen35moe_forward_cuda.c) | Rung 2 `max_slots` for the GPU decode-slot ceiling | qwen35moe_forward_cuda.c:4605 |
| `bs_new` (batch_sched.c) | Rung 2 ceiling to clamp `cap` at scheduler creation | batch_sched.c:629 |
| `pn_qwen35moe_decode_slot_bind_cuda` | Rung 2 ceiling to enforce the bind guard | qwen35moe_forward_cuda.c:4624 |
| `qwen35moe_kv_init` (qwen35moe_forward.c:142) | Rung 4a `force_kv_q4` at KV cache allocation time | qwen35moe_forward.c:145 |
**Rung 4a acts at KV-cache allocation time**, not reactively. The KV format (fp32 vs Q4) is fixed when the state is created; `force_kv_q4` only fires if the box is already at WATCH when `pn_qwen35moe_kv_init` runs (qwen35moe_forward.c:142). A live fp32 KV is never retroactively converted.
**Rung 4b (`kv_evict`)** is a policy flag only — it is set in the plan at TIGHT but no SnapKV-style row selection code exists yet. Nothing reads this flag today in a path that would evict rows from a live KV cache.
### The logged TIGHT clamp
The line `[ladder] TIGHT -> prefill window clamped 16384 -> 8192 (...)` is emitted by `pn_admit` at engine.c:507 whenever `ladder.prefill_window > 0 && ladder.prefill_window < max_window`. It means the incoming request's allowed context was halved (TIGHT divides by 2, pn_ladder.h:59) before admission checked it. This happens at every request on the mini while the server is at TIGHT.
---
## Concurrent-mutation hazard — segfault suspect
This section catalogs every place the ladder or memory layer could mutate or resize a buffer that an in-flight batched forward (`pn_qwen35moe_decode_slots_cuda` / the async begin/poll step) might be reading.
### 1. `gpu_attn_slot_ring_ensure` — KV ring reallocation (qwen35moe_forward_cuda.c:4946)
**CRITICAL.** Called inside `pn_qwen35moe_decode_slots_cuda` at the start of each attention-layer decode step. If `kv_slots_cap_seq != max_seq` or `kv_slots_cap_n < nslots` or `kv_q4 != last`, it frees and reallocates `k_slots_g`, `v_slots_g`, and the Q4 sink/win buffers (pn_qwen35moe_forward_cuda.c:4958–4989).
The trigger for `kv_slots_cap_n < nslots` is a new slot binding with a different `nslots` argument than the previous bind. The ladder (rung 2) can change the slot ceiling between `bs_new` time (where `cap` is clamped once at scheduler creation) and a subsequent `pn_qwen35moe_decode_slot_bind_cuda` call if the pressure level changed. However, the bind guard at qwen35moe_forward_cuda.c:4624 refuses any bind that would exceed the current ladder ceiling — so the reallocation path inside `gpu_attn_slot_ring_ensure` is only reached when `nslots` is *smaller* than expected (a bind with fewer slots, which can only shrink the allocation, not grow it). This is safe.
**The real hazard**: the async forward step (`pn_forward_step_begin/poll`) holds no lock during its multi-phase GPU operation (qwen35moe_forward_cuda.c:7053: "the per-model gen_lock guarantees one in-flight step per model"). If `gpu_attn_slot_ring_ensure` is somehow entered while an async step is in flight on the same model — e.g. a new connection calls `decode_slot_bind` (which calls `gpu_attn_slot_ring_ensure` via the bind path) before the previous step's poll completes — the reallocation frees buffers (`fc_buffer_free(d->k_slots_g)`) that a CUDA kernel may still be accessing on-device. Whether the serve layer's GPU-turn mechanism (`bs_gpu_acquire`) fully prevents this is the question (see item 3 below).
### 2. `conv_slots` / `ssm_slots` reallocation on slot-count change (qwen35moe_forward_cuda.c:4671)
Inside `pn_qwen35moe_decode_slot_bind_cuda`: if `d->slots_cap < n_slots`, the per-layer conv and SSM slot buffers are freed and reallocated (`fc_buffer_free(d->conv_slots)`, then `fc_buffer_alloc_ctx`). The graph is dropped (`d->slots_graph = NULL`) so the next step recaptures against the new allocation (comment at pn_qwen35moe_forward_cuda.c:4681: "L9 — the slot state buffers just moved"). If a decode step is in flight on the GPU and uses these buffers as kernel arguments via a captured CUDA graph, freeing them while the graph executes is a use-after-free at the CUDA level. The batch_sched comment at batch_sched.c:1144 says "the GPU turn is owned … this whole section is atomic w.r.t. every other forward on this model" — but `bs_slot_bind_common` acquires the GPU turn via `bs_gpu_acquire` (batch_sched.c:1207 calls `pn_qwen35moe_decode_slot_bind` while under the prefill turn, not the decode turn). Whether the `driver_thread`'s decode pass (`decode_slots` at batch_sched.c:480) and the prefill thread's bind path are mutually exclusive depends on the GPU-turn ticket system covering both.
### 3. `qw_defer` staging arena realloc (qwen35moe_forward_cuda.c:5057)
`qw_defer_reset` (called at the top of each decode step at qwen35moe_forward_cuda.c:6320) may `realloc(df->stage, need)` if the staging arena is too small. The comment at pn_qwen35moe_forward_cuda.c:5049 states explicitly: "an in-flight async DtoH writes into it — moving it would corrupt earlier layers' queued copies." The code handles this by pre-sizing up front so `qw_defer_stage` never reallocs mid-step. This is safe **within a single step** but not across steps: if a new step starts and `qw_defer_reset` reallocs `df->stage` while a prior step's async DtoH (from an earlier `begin`) has not been reaped by `poll`, those async writes still point at the old buffer address. The `as_*` phase machine (qwen35moe_forward_cuda.c:572) is supposed to prevent this, but verify that `poll` always fully drains async DtoH before `begin` is allowed to reenter.
### 4. Ladder-triggered max_slots change between scheduler creation and bind
`pn_qwen35moe_gpu_ladder_max_slots()` is called once at `bs_new` time (batch_sched.c:629) to clamp `cap`. It is called again inside `pn_qwen35moe_decode_slot_bind_cuda` (qwen35moe_forward_cuda.c:4629) for the bind guard. If the pressure level changed between these two calls — e.g. the box was OK when the scheduler was created (large cap) but is now TIGHT/CRITICAL (smaller ceiling) — the bind guard fires with `n_slots > slot_ceiling` and returns `-1` (engine.c style: the slot bind fails and the request fails cleanly). This is not a buffer corruption hazard, but it will produce a `[ladder] refusing N decode slots` error on requests that would have been fine at scheduler creation time.
### Summary table
| Location | Buffer mutated | Condition | Guarded by | Hazard for in-flight? |
|---|---|---|---|---|
| `gpu_attn_slot_ring_ensure` (line 4956) | `k_slots_g`, `v_slots_g`, sink/win bufs | `nslots` or `max_seq` or `q4` changed | GPU-turn ticket (prefill) | **Investigate**: does GPU-turn cover the decode step too? |
| `decode_slot_bind_cuda` (line 4671) | `conv_slots`, `ssm_slots` per SSM layer | `slots_cap < n_slots` | GPU-turn ticket (prefill) | **Investigate**: same question |
| `qw_defer_reset` (line 5057) | `df->stage` host staging arena | first step at a new nslots | pre-size contract within a step | Safe within step; verify async DtoH is always reaped before re-entry |
| `gpu_batch_host_ensure` (batch scratch) | `b->hidden` and friends | batch width grows | GPU-turn (prefill/decode) | `slots_graph_scratch` comment (line 271) notes graph invalidation on realloc |

281
docs/architecture/models-qwen35moe.md Normal file
View File

@ -0,0 +1,281 @@
# Qwen3.5-MoE Model Implementation
Pantheon implementation of the Nex-N2-mini architecture: a 40-layer hybrid Gated-Delta-Net SSM / sparse GQA attention stack with 256-expert MoE on every layer.
Source files:
- `src/models/qwen35moe.c` — loader, config parser, GGUF weight mapping
- `src/models/qwen35moe_internal.h` — config/block structs, dual-state types, cross-file kernel declarations
- `src/models/qwen35moe.h` — public API, `PN_QWEN35MOE_MAX_SEQ_CAP`
- `src/models/qwen35moe_forward.c` — CPU forward, dual-state lifecycle, KV ring init
- `src/models/qwen35moe_ssm.c` — Gated-Delta-Net SSM block kernel
- `src/models/qwen35moe_forward_cuda.c` — CUDA forward: 2-die split, GPU weight residency, decode slots, async step machine (7 348 lines)
---
## Architecture
### Dims (from `pn_qwen35moe_config`, `qwen35moe_internal.h:36–60`)
| Field | Value | GGUF key |
|---|---|---|
| `n_layer` | 40 | `qwen35moe.block_count` |
| `full_attention_interval` | 4 | `qwen35moe.full_attention_interval` |
| `n_embd` | 2048 | `qwen35moe.embedding_length` |
| `n_head` | 16 | `qwen35moe.attention.head_count` |
| `n_head_kv` | 2 | `qwen35moe.attention.head_count_kv` |
| `head_dim` | 256 | `qwen35moe.attention.key_length` |
| `rope_dim` | 64 | `qwen35moe.rope.dimension_count` (partial NEOX) |
| `n_expert` | 256 | `qwen35moe.expert_count` |
| `n_expert_used` | 8 | `qwen35moe.expert_used_count` |
| `expert_ff` | 512 | `qwen35moe.expert_feed_forward_length` |
| `shared_ff` | 512 | `qwen35moe.expert_shared_feed_forward_length` |
| `ssm_state` | 128 | `qwen35moe.ssm.state_size` |
| `ssm_n_group` | 16 | `qwen35moe.ssm.group_count` (K-heads, GQA key/query) |
| `ssm_dt_rank` | 32 | `qwen35moe.ssm.time_step_rank` (V-heads) |
| `ssm_d_inner` | 4096 | `qwen35moe.ssm.inner_size` |
| `ssm_conv` | 4 | `qwen35moe.ssm.conv_kernel` |
| `conv_dim` | 8192 | derived: read from `blk.0.attn_qkv.weight` outermost dim (`qwen35moe.c:163–167`) |
### Layer layout
Layer type is set by `qwen35moe_is_attn_layer()` (`qwen35moe.c:216–220`):
```c
return ((i + 1) % interval) == 0; /* interval default 4 */
```
With `full_attention_interval=4` and 40 layers, this yields:
- Layers 3, 7, 11, 15, 19, 23, 27, 31, 35, 39 → **GQA full-attention** (10 layers, `n_attn=10`)
- All other 30 layers → **Gated-Delta-Net SSM** (`n_ssm=30`)
`block_type[i]` and `block_index[i]` in `pn_qwen35moe_model` (`qwen35moe.c:59–60`) map any global layer index to its type and per-type slot. These are the only two arrays that must be consulted to dispatch a layer.
### Per-layer block structure
Every layer (SSM or attention) shares the same outer shell:
```
attn_norm → (SSM block | GQA attn block) → +residual
→ post_attention_norm → shared-MoE FFN → +residual
```
(`qwen35moe_forward.c:316–369`)
### MoE routing
Every layer carries an identical `pn_qwen35moe_moe` block (`qwen35moe_internal.h:75–84`):
1. **Router**: `ffn_gate_inp [n_expert, n_embd]` → logits → softmax → top-8 selection → `norm_topk_prob` renormalization → scaled by `expert_weights_scale` (default 1.0; key absent in current GGUFs, `qwen35moe.c:141–145`).
2. **Routed experts**: 256 experts, zero-copy sliced from merged 3-D GGUF tensors `blk.N.ffn_gate_exps.weight` / `ffn_up_exps.weight` / `ffn_down_exps.weight` (`qwen35moe.c:246–293`). Each expert is a SwiGLU: `silu(gate·x) * (up·x)` → `down`. `moe_out = Σ weight_e * SwiGLU_e(x)`.
3. **Shared expert** (always on): separate `ffn_gate_shexp` / `ffn_up_shexp` / `ffn_down_shexp` + a scalar sigmoid gate `ffn_gate_inp_shexp [n_embd]`. `shexp_out = silu(gate·x) * (up·x) → down`, then scaled by `sigmoid(gate_inp_shexp · x)`.
4. **Combine**: `out = moe_out + shexp_out`. Residual add is the caller's responsibility.
(`qwen35moe_internal.h:278–308`)
### GQA attention block (`qwen35moe_internal.h:160–196`)
Op order for attention layers:
1. `attn_q [2*head_dim*n_head, n_embd]` → split: first `head_dim` per head is Q, second is gate z.
2. QK-RMSNorm on Q (per head via `attn_q_norm [head_dim]`) and on K (via `attn_k_norm [head_dim]`), pre-RoPE.
3. `attn_k [head_dim*n_head_kv, n_embd]` and `attn_v [head_dim*n_head_kv, n_embd]`.
4. Partial NEOX RoPE on the first `rope_dim=64` dims of every Q and K head. Multi-section RoPE is parsed (`rope_sections[4]`, `qwen35moe.c:111–130`) but collapses to standard NEOX for text.
5. GQA: 16 Q heads broadcast across 2 KV heads (each KV head serves 8 Q heads), scale `1/sqrt(256)`. K/V appended to per-layer ring before attention.
6. Output gate: `attn_out *= sigmoid(z)`.
7. `attn_output [n_embd, head_dim*n_head]`.
### SSM (Gated-Delta-Net) block (`qwen35moe_ssm.c:229–351`)
Op order for SSM layers:
1. Projections from `attn_norm`-ed input: `attn_qkv → qkv [conv_dim=8192]`; `attn_gate → z [ssm_d_inner=4096]`; `ssm_beta → beta [32]` → sigmoid; `ssm_alpha → alpha [32]`.
2. Decay gate per V-head (`qwen35moe_ssm.c:86–93`): `g[h] = ssm_a[h] * softplus(alpha[h] + ssm_dt[h])`. `ssm_a` is stored as `−exp(A_log)` (negative), so `exp(g) ∈ (0,1)`.
3. Causal depthwise conv1d (kernel size 4) over `conv_dim` channels + SiLU (`qwen35moe_ssm.c:110–135`). Rolls `conv_state` in place (oldest-first, channel-major layout).
4. Split convolved output into q `[128*16]`, k `[128*16]`, v `[128*32]`; L2-normalize q/k per K-head (`qwen35moe_ssm.c:70–76`); GQA-tile from 16 K-heads to 32 V-heads using interleaved tiling (`vh % n_k_heads == kh`, `qwen35moe_ssm.c:311`).
5. Delta-rule recurrence per V-head (`qwen35moe_ssm.c:163–195`): for each of 32 `[128×128]` state matrices (stored transposed, row j = column j of S): decay → pred → delta → rank-1 write → read-after-write y. Scale `1/sqrt(128)` applied once at read.
6. Gated RMSNorm: `RMSNorm(y, ssm_norm) * silu(z)` per V-head (`qwen35moe_ssm.c:206–223`), then `ssm_out [n_embd, ssm_d_inner]`.
The recurrence is parallelized across V-heads via `fw_for` when `USE_THREADS` is set and `n_v_heads >= 4` (`qwen35moe_ssm.c:332–336`).
---
## 2-Die Split and Placement
### Constants (`qwen35moe_forward_cuda.c:89–95`)
```c
#define GPU_DIE_SPLIT_LAYER 40 /* default = disabled (all on one die) */
#define GPU_DIE_A_DEFAULT 0
#define GPU_DIE_B_DEFAULT 1
```
`GPU_DIE_SPLIT_LAYER=40` means no split by default; the single-die path is the validated baseline. A split is engaged only when `QWEN35_GPU_SPLIT_LAYER < n_layer` or explicit placement provides two dies.
### Boundary resolution (`gpu_split_layer`, `qwen35moe_forward_cuda.c:696–711`)
```
pn_load_on(2 dies) → split = n_layer/2 = 20
pn_load_on(1 die) → split = n_layer (single-die forced)
no placement → GPU_DIE_SPLIT_LAYER (40 = disabled)
QWEN35_GPU_SPLIT_LAYER env → overrides boundary (unless 1-die placement)
```
Clamped to `(0, n_layer]`.
### Die id resolution (`gpu_die_ids`, `qwen35moe_forward_cuda.c:717–736`)
Priority: `pn_load_on` explicit placement > `QWEN35_GPU_DIE_A` / `QWEN35_GPU_DIE_B` env vars > `GPU_DIE_A_DEFAULT=0` / `GPU_DIE_B_DEFAULT=1`.
Die ids are ordinals relative to `CUDA_VISIBLE_DEVICES`. On a K80 host, one K80 card exposes two GK110 dies; `CUDA_VISIBLE_DEVICES` maps them to ordinals 0 and 1.
### Weight placement
- Die A (`ctx_a`): layers `[0, split)` + token embedding (read host-side, row uploaded per token).
- Die B (`ctx_b`): layers `[split, n_layer)` + `output_norm` + LM head.
Each `gpu_ssm_layer.ctx` / `gpu_attn_layer.ctx` stores a back-pointer to its owning die context (`qwen35moe_forward_cuda.c:205`, `285`).
### 2-die multiplexing (`gpu_select_die`, `qwen35moe_forward_cuda.c:751–764`)
On legacy large-BAR K80 rigs (single MMIO slot), `barswap` time-multiplexes the two dies. The forward:
1. Acquires die A via `bs_acquire`.
2. Runs front layers on `ctx_a`.
3. Syncs, D2H copies the boundary `hidden [n_embd float32]`.
4. Acquires die B via `bs_acquire`.
5. H2D uploads `hidden` to `ctx_b`.
6. Runs back layers + LM head on `ctx_b`.
In small-BAR mode (modern driver, real dual-device), both contexts are live simultaneously and `g->bs == NULL`; no barswap or per-layer `gpu_select_die` is needed. The decode-slot bind path and the async path only run in small-BAR mode (`qwen35moe_forward_cuda.c:4655–4658`).
---
## KV Cache
### Allocation (`qwen35moe_kv_init`, `qwen35moe_forward.c:129–176`)
One `pn_qwen35moe_kv` per attention layer, 10 total. Layout `[n_head_kv][max_seq][head_dim]` (head-major).
**Q4 mode** (`QWEN35_KV_Q4`, default ON, `qwen35moe_forward.c:47–56`):
- Cache stored as ggml-Q4_0 rows: `kq [n_head_kv * max_seq * pn_kvq4_row_bytes(head_dim)]`, same for `vq`.
- fp16 sink shadow `k_sink/v_sink [n_head_kv * PN_KVQ4_SINKMAX * head_dim]` — attention sink tokens kept at fp16 to prevent argmax flips on near-ties.
- fp16 window shadow `k_win/v_win [n_head_kv * PN_KVQ4_WMAX * head_dim]` — the most recent W tokens similarly kept at fp16.
- PPL delta vs fp32 KV: 0.024% (validated, `qwen35moe_forward.c:45`).
- Disable with `QWEN35_KV_Q4=0`.
**fp32 mode**: `k/v [n_head_kv * max_seq * head_dim float32]`.
The reactive memory ladder rung 4a (`qwen35moe_forward.c:142–152`) auto-forces Q4 if the box is under memory pressure at allocation time even without the env var.
### KV window size
`PN_QWEN35MOE_MAX_SEQ` = 4096 (compile-time default, `qwen35moe_forward.c:70`).
`QWEN35_MAX_SEQ` env var overrides, clamped to:
- minimum: 16
- maximum: `PN_QWEN35MOE_MAX_SEQ_CAP` = 262 144 (`qwen35moe.h:29`)
The cap is the single source of truth shared between the CPU KV ring allocator (`qwen35moe_max_seq`, `qwen35moe_forward.c:76–91`) and the GPU admission gate (`gpu_admit_max_ctx`, `qwen35moe_forward_cuda.c:619–629`). Memory scales linearly: `n_head_kv * max_seq * head_dim * 8` bytes per attention layer on the host (same again on device in GPU mode).
### Ring behavior
KV is a fixed-size ring; `pos % max_seq` gives the slot. There is no dynamic resizing. If `pos >= max_seq` the oldest entry is overwritten. The ring is never grown after allocation.
### Deferred host mirror (B2)
After a GPU decode token, the GPU KV ring is the live source of truth. Writing back to the host mirror (`kv->k/v` or `kq/vq`) per token was too expensive. Instead, `kv->kv_mirror_stale = 1` is set (`qwen35moe_forward_cuda.c:3547`) and the D2H copy is deferred to `pn_qwen35moe_gpu_state_to_host()` (`qwen35moe_forward_cuda.c:6779`), which is called by the prefix cache on session eviction. After `pn_qwen35moe_state_reset()` the flag is cleared to 0 (`qwen35moe_forward.c:308`).
---
## Decode Slots / Multi-User Batching
### Slot capacity
`PN_QWEN35MOE_GPU_MAX_SLOTS` = 16 (`qwen35moe_forward_cuda.c:115`).
Memory cost: ~2.1 MB per SSM layer per slot for the per-slot `conv_slots / ssm_slots` images. At 16 slots across 30 SSM layers ≈ 1 GB on the die pair.
The reactive ladder rung 2 (`pn_qwen35moe_gpu_ladder_max_slots`, `qwen35moe_forward_cuda.c:4605–4613`) caps live slots: OK → 16, TIGHT → 8, CRITICAL → 1.
### Slot protocol
```
1. prefill each user's pn_state (single-token or batched)
2. immediately after each user's prefill:
pn_qwen35moe_decode_slot_bind(model, state_u, u, K)
— D2D snapshots the live GPU SSM state into slot u;
host mirror is stale after GPU prefill so bind must precede
any other user's forward
3. step all K users: pn_qwen35moe_decode_slots(model, tokens, states, K, logits)
— ONE forward pass, weights stream once for all K users
```
(`qwen35moe.h:126–138`)
Slot-major per-user SSM images live in `gpu_ssm_layer.conv_slots / ssm_slots` (`qwen35moe_forward_cuda.c:247–250`). Bind snapshots via D2D from the shared `conv_state_g / ssm_state_g` right after prefill (`qwen35moe_forward_cuda.c:4616–4638`). A slot-count change invalidates all existing bindings.
`pn_qwen35moe_decode_slot_move(src, dst, n_slots)` (`qwen35moe.h:154`) compacts the active set via D2D copy without a host round-trip.
### Async step machine (C2, `QWEN35_ASYNC_SCHED`, default ON)
Three-state enum (`qwen35moe_forward_cuda.c:7062–7065`):
- `QW_ASYNC_IDLE` — no step in flight
- `QW_ASYNC_WAIT_A` — 2-die only: front segment (die A) in flight, CUDA event pending
- `QW_ASYNC_WAIT_FINAL` — back segment + LM head in flight (or single-die full step)
`pn_qwen35moe_forward_step_begin()` launches the step and returns without blocking (returns 1 = not async-eligible and runs synchronously in that case). `pn_qwen35moe_forward_step_poll()` reaps via `fc_event_query`, advances to the next phase, and on completion sets `*logits_out` and increments `state->pos`. This lets a 2-core host host keep up to 8 cards' generations pipelined.
Requires: small-BAR mode (real dual-device, `g->bs == NULL`); LM head must have an on-device kernel (Q3_K/Q4_K/Q5_K/Q6_K/Q8_0/F32, `qwen35moe_forward_cuda.c:7083–7098`). Disable with `QWEN35_ASYNC_SCHED=0`.
---
## The MTP / nextn Trap
The GGUF metadata for Qwen3.5-MoE may advertise MTP (multi-token prediction) draft-head weights under `blk.N.nextn.*` keys. The weights **do not ship** with public GGUF conversions; `nextn_predict_layers` must always be 0.
The loader explicitly documents this and never touches any `blk.N.nextn.*` tensor:
```c
/* Honor the MTP trap: nextn_predict_layers=0 — never touch blk.N.nextn.* . */
```
(`qwen35moe.c:475`)
If a future GGUF includes these weights, loading them would require a separate code path not yet implemented. Attempting to load them without that path would silently skip them (no tensor lookup is performed) — so correctness is maintained but draft-head acceleration would simply not engage.
---
## Gotchas and Fragile Spots
### KV ring is fixed at allocation — no resize path
`qwen35moe_kv_init` (`qwen35moe_forward.c:129`) allocates the ring once at `max_seq`. There is no resize-on-demand path. If the ring reallocates (e.g., `cache_cap` changes on the GPU side under a geometry change), the segment graph must be dropped and recaptured (`qwen35moe_forward_cuda.c:4941`). Under memory pressure the reactive ladder forces Q4 at init time but cannot shrink the ring after the fact.
**Segfault risk**: if `pos >= max_seq` is not checked before the GPU KV-append kernel writes `pos % max_seq`, and the GPU ring's `cache_cap` was allocated for a different `max_seq` than the host ring, the write goes out of bounds. The GPU ring capacity is tracked separately as `d->cache_cap` (`qwen35moe_forward_cuda.c:306`); a mismatch after a partial realloc failure would be silent until the OOB write.
### kv_mirror_stale + session switch
After a GPU decode step, `kv->kv_mirror_stale = 1`. If a session is evicted without calling `pn_qwen35moe_gpu_state_to_host()` first (e.g., the prefix cache path is bypassed or fails partway), the host mirror remains stale. A subsequent resume re-syncs from the stale host mirror instead of the GPU ring, silently corrupting the KV cache with zeros or old content.
The `pn_qwen35moe_gpu_forget_state()` call (`qwen35moe.h:212`) must be paired with `pn_state_reset()` on session reuse; omitting it leaves the GPU SSM buffers binding to the old session and the next prefill reads stale GPU residue.
### GQA repeat tiling — interleaved, not blocked
The SSM's GQA repeat tiles K-heads to V-heads using `vh % n_k_heads == kh` (interleaved), **not** the blocked `vh / kv_groups` mapping (`qwen35moe_ssm.c:311–312`). This matches `ggml_repeat`'s tiling in llama.cpp. Getting this wrong produces plausible but numerically wrong y vectors — the error is hard to detect without the oracle diff.
### conv_state layout: channel-major, oldest-first
`conv_state[c * (ssm_conv-1) + t]`, t=0 oldest. The roll-forward in `qwen35moe_conv1d_silu` (`qwen35moe_ssm.c:131–133`) shifts t=0..km1-2 forward and appends the new raw input at t=km1-1. The GPU path must replicate this exactly or the conv state diverges after the first token.
### ssm_state is stored transposed
`ssm_state[h * S * S + j * S + i] == S_h[i][j]` — row j (contiguous over i) is column j of S (`qwen35moe_internal.h:214–215`). This matches llama.cpp's `gated_delta_net` layout. Any GPU kernel that reinterprets the layout as standard row-major will compute wrong dot products.
### Batched CUDA prefill and CUDA-graph capture ordering
The per-layer segment graph is only captured after the layer has run DIRECTLY (uncaptured) at least once — `body_ran` gates the attempt (`qwen35moe_forward_cuda.c:229–232`). A batched prefill that precedes the first single-token forward never sets `body_ran` for those layers' fused descriptors, so the first single-token decode after a batch-only session must run uncaptured to build the fused descs. Capturing mid-allocation (before the fused descs exist) would call `cuMemAlloc` inside the capture, invalidating the graph. The `body_ran` gate prevents this.
### Embedding: lazy vs materialized
For untied models (a separate `output.weight` exists), `token_embd` is left `NULL` and each row is dequantized on demand from the mmap (`pn_qwen35moe_model_embed_row`, `qwen35moe.c:620–629`). For tied models, the full ~2 GB F32 embedding is materialized and also serves as the LM head (`qwen35moe.c:437–453`). The fallback for untied models where `n_embd % block_size != 0` also materializes the full heap.
### Numeric status: A8 not done
`qwen35moe_forward.c` and `qwen35moe_forward_cuda.c` both carry an open `@todo A8: validate logits vs the llama.cpp oracle`. End-to-end numeric correctness has not been verified. Several operations in the CUDA path have no dedicated GPU kernel and fall back to host-side computation with a D2H/H2D round trip (marked `TODO kernel:` inline in the CUDA file): joint Q+gate split, GQA head broadcast + L2-norm, decay gate / softplus, sigmoid gates, SwiGLU for non-Q3_K experts, attention softmax.

193
docs/architecture/qwen35moe.md Normal file
View File

@ -0,0 +1,193 @@
# qwen35moe — Forward-Pass Design (Pantheon port)
Implementation blueprint for `src/models/qwen35moe.{h,c}`. Hybrid **Gated-Delta-Net SSM + sparse full-attention + shared-expert MoE** (Qwen3-Next lineage). Synthesized from llama.cpp `qwen35moe.cpp` + `delta-net-base.cpp` (math reference) and Pantheon `rwkvx.c` (structural template).
**Structure clones `rwkvx.c`; loading follows the GGUF/llama path (`pn_qweight`, `gguf.c`), NOT rwkvx's safetensors path.** Hybrid of both.
## Config (mini; Pro is the same arch, bigger)
| dim | value | source key |
|---|---|---|
| n_layer | 40 | `block_count` |
| full_attention_interval | 4 → **attn when `(i+1)%4==0`** (layers 3,7,…,39 = 10 attn; rest = 30 SSM) | `full_attention_interval` |
| n_embd | 2048 | `embedding_length` |
| n_head / n_head_kv | 16 / 2 (GQA) | `attention.head_count[_kv]` |
| head_dim | 256 | `attention.key_length`/`value_length` |
| rope_dim | **64** (partial — only first 64 of 256 rotated) | `rope.dimension_count` |
| n_expert / used | 256 / 8 (+1 shared) | `expert_count`/`expert_used_count` |
| expert FF / shared FF | 512 / 512 | `expert_feed_forward_length`/`…shared…` |
| **SSM:** state (S) | 128 | `ssm.state_size` |
| SSM n_group (K-heads) | 16 | `ssm.group_count` |
| SSM dt_rank (V-heads) | 32 | `ssm.time_step_rank` |
| SSM d_inner | 4096 | `ssm.inner_size` |
| SSM conv kernel | 4 | `ssm.conv_kernel` |
| conv_dim | key·2+value = 2048·2+4096 = **8192** | derived |
## Block scaffold — pre-norm sandwich (every layer, both types)
```
x → attn_norm → [SSM | full-attn block] → +x → post_attention_norm → shared-MoE FFN → +(post-attn x)
```
Final `output_norm` once after the stack → LM head. All norms RMS. Residuals add the *un-normalized* input. (Identical to llama.cpp's loop.)
## Full-attention block (layers `(i+1)%4==0`)
Input is `attn_norm`'d. Op order:
1. **Joint Q+gate** proj `attn_q` → `[2·head_dim·n_head]`; split: Q = first `head_dim` per head, gate `z` = second half.
2. **QK-RMSNorm on Q** (`attn_q_norm`, per-head, **before RoPE**).
3. `attn_k`,`attn_v` proj; reshape K to heads; **K-RMSNorm** (`attn_k_norm`, pre-RoPE).
4. **Partial RoPE** on Q,K — rotate only first `rope_dim=64` of each 256-dim head, pass the rest through (multi-section / IMRoPE; sections from `rope.dimension_sections`).
5. **GQA attention** (16 Q / 2 KV heads), scale `1/√head_dim`.
6. **Output gate:** `cur = attn_out * sigmoid(z)` ← note: **sigmoid** gate here.
7. Output proj `attn_output`. Residual added in the main loop.
## SSM block — Gated-Delta-Net (layers `(i+1)%4!=0`)
Input is `attn_norm`'d. **This is the hard new kernel.**
**Projections** (4 separate, from normed `x`):
- `attn_qkv` → fused **q|k|v** stream `[8192]` (the thing that gets convolved).
- `attn_gate` → **z** `[4096]` (output gate, used at the end).
- `ssm_beta` → `beta [32]`, then **`β = sigmoid(beta)`**.
- `ssm_alpha` → `alpha [32]`.
**Decay gate (scalar per V-head, "GDA"):**
```
g = (-exp(A_log)) · softplus(alpha + dt_bias) # ssm_a stored as -exp(A_log) (negative); ssm_dt = dt_bias
```
(`ssm_a` is consumed via plain multiply — pre-baked at load.)
**Causal conv1d** (`ssm_conv`, k=4): conv over the full 8192-channel q|k|v stream prepended with `conv_state` (last 3 timesteps) → **SiLU**. Then split into q`[128×16]`, k`[128×16]`, v`[128×32]`; **L2-normalize q and k** over head dim (not RMS); GQA-repeat q,k 16→32 heads.
**Per-token recurrence** (per V-head `h∈32`, state `S` is `[128,128]`, `i`=key dim, `j`=value dim):
```
S' = exp(g_t) · S_{t-1} # (a) scalar decay, whole state
pred_j = Σ_i S'[i][j] · k_t[i] # (b) read decayed state under key
delta_j = (v_t[j] − pred_j) · β_t
S_t[i][j] = S'[i][j] + k_t[i] · delta_j # (c) rank-1 delta-rule write
y_t[j] = (1/√128) · Σ_i S_t[i][j] · q_t[i] # (d) read-AFTER-write under query
```
**Scaling caution:** apply the q/output scale **once** — the autoregressive (single-token) path is what we want for decode; use the form above and do *not* also pre-scale q.
**Output path:**
```
gated = RMSNorm(y, ssm_norm) · silu(z) # gated RMSNorm — silu(z), NOT sigmoid
cur = ssm_out · gated # proj [4096→n_embd]
```
Residual added in the main loop.
## Shared-expert MoE FFN (every layer)
On the `post_attention_norm`'d tensor:
```
# routed:
moe_out = MoE(x, ffn_gate_inp, {ffn_gate/up/down_exps}, n_expert=256, used=8,
gate=SOFTMAX, norm_topk_prob=TRUE, scale=expert_weights_scale) # SwiGLU experts
# shared (always on):
shexp = SwiGLU(x, ffn_gate_shexp, ffn_up_shexp, ffn_down_shexp)
sgate = sigmoid(ffn_gate_inp_shexp · x) # 1 scalar/token
out = moe_out + sgate · shexp
```
Extends Pantheon's existing Mixtral MoE (routed top-k SwiGLU) with: **softmax+norm_topk_prob gating** and the **sigmoid-gated shared-expert** add.
## Dual state (per sequence, per layer)
- **SSM layers (×30):** conv_state `(4-1)·8192 = 24,576` f32 + ssm_state `128·4096 = 524,288` f32 = **548,864 f32/layer**.
- **Attention layers (×10):** KV cache, sized by `n_head_kv=2 × head_dim=256` per token (the *only* growing state).
- Mirror `rwkvx.c`'s dual top-level struct: parallel per-type arrays indexed by per-type index, + `block_order[]`/`block_index[]`.
## Pantheon mapping (from `rwkvx.c`)
**Clone wholesale:** dual-state top struct; `block_order`/`block_index` interleave + forward dispatch; model = per-type block-struct arrays + emb/output_norm/head; forward skeleton (embed → ping-pong per-layer loop → final norm → head); engine hooks (add `PN_ARCH_QWEN35MOE` enum + branches in the ~6 `engine.c` sites: `pn_load_ex`, `pn_get_forward_fn`/direct-set, `pn_state_new/free/reset`, `pn_model_free`, generate/stream tokenizer routing); adapter shim mirroring `rwkvx.h`; threadpool TLS; own preallocated workspace (not the arena); q8 matvec infra (reuse for expert matrices).
**Add (new):** SSM state struct (conv + ssm); the Gated-Delta kernel (recurrence above); shared-MoE FFN; **RoPE + GQA** on the attention block (rwkvx's attn block has neither — borrow RoPE from the llama path).
**Replace:** RWKV time-mix → SSM kernel; attn internals (add RoPE/GQA, sigmoid output gate); FFN → shared-MoE; RWKV trie tokenizer → Qwen **BPE** (`ft_tokenizer` via the GGUF path); `head_size=64` hardcode → config-driven.
**Loader:** GGUF (not safetensors). Parse all `qwen35moe.*` keys; honor `nextn_predict_layers=0` and never load `blk.N.nextn.*`. Layer type from `(i+1)%4==0`. Per-layer tensors via the gguf/`pn_qweight` conventions.
## Implementation order (= Bucket A / Phase 1 A2–A8)
1. **A2** arch+config+loader (detect, parse, GGUF, layer-type, nextn).
2. **A3** full-attention block (closest to known territory — GQA+QK-norm+partial RoPE+sigmoid gate).
3. **A4** SSM kernel (conv → recurrence → gated-norm). Validate this op in isolation first (hardest).
4. **A5** shared-MoE FFN (extend Mixtral path).
5. **A6** assemble hybrid + dual state + engine hooks.
6. **A8** validate vs the llama.cpp oracle (`temp 0`, `oracle_reference.md`).
## Correctness watch-list
- Decay is **scalar per V-head** (GDA), not per-dim.
- conv runs over the **full 8192** stream, *then* split; q,k get **L2-norm** post-conv.
- Partial RoPE: only first **64** dims.
- Attention layer gate = **sigmoid**; SSM layer gate = **silu** (gated RMSNorm). Don't mix them up.
- Recurrence is **read-after-write** (output reads post-update state).
- Apply the q/√S scale **once**.
- `norm_topk_prob=true` (renormalize the top-8 weights).
- **GQA head broadcast in the SSM (THE A8 gibberish bug — RESOLVED):** the 16 K/Q heads repeat over 32 V-heads via `ggml_repeat`, which **interleaves**: V-head `vh` pairs with K-head `vh % n_k_heads`, i.e. `vh = r*n_k_heads + kh` — **NOT** the blocked `vh = kh*kv_groups + r`. Same trap will appear in the GPU MoE/SSM port. Verified by per-layer activation bisection vs `llama-eval-callback`.
- **ChatML reasoning prefill:** assistant turn opens with `<think>` (`add_generation_prompt`); without it the first generated token diverges from the oracle. Pantheon: `PN_CHAT_CHATML_THINK`.
- ~~QK-norm / ssm_norm `(1+w)`~~ **RESOLVED (not the bug):** GGUF norm weights are ~1-centered (means 0.88–2.59, not ~0), so the converter baked `+1` into them → plain RMS is correct, matches llama.cpp. A8 should look at the **assembly wiring** (residual placement, norm points, kernel-handoff tensor layouts, state init), not the norm convention.
## A2 findings — apply in A3+ (verified against the real GGUF)
- **Exact tensor names** (the §blocks above used shorthand): `ssm_conv1d.weight`, `ssm_a` (no `.weight`), `ssm_dt.bias`, `ssm_alpha.weight`, `ssm_beta.weight`, `ssm_norm.weight`, `ssm_out.weight`; attention uses `attn_q/k/v.weight`, `attn_q_norm.weight`, `attn_k_norm.weight`, `attn_output.weight`, `attn_gate.weight`, `attn_qkv.weight`; MoE `ffn_gate_inp`, `ffn_{gate,up,down}_exps`, `ffn_{gate,up,down}_shexp`, `ffn_gate_inp_shexp`. **A2 loads by these real names.**
- **Q3_K added to the CPU path** — the engine lacked Q3_K dequant/`pn_qmv` (only Q4_K/Q5_K/Q6_K). A2 added it (validated: sane stats, no NaN). *Still needs oracle-level verification at A8 — a subtle bit error would only show as output mismatch.*
- **`rope.dimension_sections`** (the multi-section/IMRoPE array) is **not parsed in A2** — only `rope.dimension_count=64`. **A3 must parse the sections array** for partial multi-section RoPE.
- **`token_embd`** kept as dequantized F32 (owned, phi3 convention); `output.weight` (Q6_K) zero-copy `pn_qweight` with tied-embedding fallback.
- **File-split discipline (whale prevention):** A2 is one 607-LOC file. **A3 → `qwen35moe_attn.c`, A4 → `qwen35moe_ssm.c`, A5 → `qwen35moe_moe.c`** — do NOT grow `qwen35moe.c` into a 4th >1000-LOC whale. Keep `qwen35moe.c` for the model struct + load + forward orchestration.
## K80 GPU decode — status & performance
**N2-mini (Q3_K_M) runs on a K80 host** via `pn_qwen35moe_forward_cuda`, split across the
two dies of one card (small-BAR — both dies mapped at once, no barswap): front layers
`[0,split)` + token_embd on die A, back layers + output_norm + LM head on die B. Select with
`QWEN35_GPU_SPLIT_LAYER=20 QWEN35_GPU_DIE_A=0 QWEN35_GPU_DIE_B=1` (no barswap acquire —
`QWEN35_GPU_BARSWAP=1` opts back into the legacy single-slot path).
**Verified single-stream decode: ~13.7 tok/s** (~0.073 s/token, 2 dies, Q3_K_M), and the GPU output
**bit-matches the CPU forward** — argmax `11,1496,271,142,79,88,30,81,30,271,220,381`, token-0
logit 13.7252, every per-token logit ulp-0 identical. Up from a 0.2 tok/s first-light baseline (~68×).
K80 autoboost swings this ±~30% run-to-run. Actively being optimized (see headroom).
**What got it there** (all committed):
- GPU matvec kernels for **every quant type** (Q3_K/Q4_K/Q5_K/Q6_K/Q8_0/F32) — the mixed-quant
model was host-falling-back to `pn_qmv` (3.7 s of 5.2 s/token, led by the 248K-vocab Q6_K LM head).
- **Warp-per-row** matvec (one warp/row, 32 lanes coalesce each 256-elem superblock; was
thread-strided → scattered 32-byte segments at 2–8 GB/s → 14–41 GB/s).
- **Grouped MoE** — all 8 top-k experts' gate+up+SwiGLU in one launch + fused down/accumulate (~52→10 ops/layer).
- **Parallel router top-k** (the serial thread-0 selection sort was ~134 ms/token over 40 layers).
- **GPU-resident SSM state** (kills the per-token state round-trip) + per-die scratch pool (kills ~800 alloc/free per token).
- **Quant weights repacked at upload** (`fc_repack_weight`) into an aligned, scale-pre-unpacked,
transposed device layout → coalesced `u32` loads + shared-memory activation staging. Matvec
device-bandwidth ~2× (Q8_0 21→65, Q5_K 16→35, Q3_K 15→30 GB/s), +2 GB/die. Bit-exact (CPU-replica
verifier, ulp 0).
**Ceiling & headroom:** post-repack the matvecs are **instruction-issue-bound on Kepler** — ~65–70 GB/s
is near the sm_37 limit for *bit-exact* K-quant decode (the earlier ~170 GB/s figure was optimistic).
The remaining single-stream lever is **per-layer kernel fusion** to cut the ~18–20 ms/token of
tiny-kernel launch overhead (~100 F32 router/α/β + 94 Q8_0 launches, ~80 µs each vs ~35 µs of work)
→ ~15–20 tok/s. Beyond that, **"hundreds" is not a single-stream number on Kepler** — it comes from
**speculative block-verify** (DeepSeek DSpark: amortize the bandwidth-bound batch-1 decode across a
verified block) and **continuous batching** across concurrent users (serving-aggregate throughput).
## Batched prefill (the serving foundation)
`pn_qwen35moe_forward_batch_cuda` runs **n tokens of one sequence in a single pass**: token-major
`[n][dim]` activations, every per-token matvec batched into one `fc_matvec_*_nx` launch (each weight
block decoded once per 4-column tile — the dequant instruction stream amortizes over the batch), SSM
conv1d + delta-rule recurrence as **in-kernel scans** over the n positions, causal host GQA attention
per position, per-token MoE routing in batched grouped launches. Single-token decode path untouched.
**Bit-exact check** (2-die split @20, measured with an out-of-tree harness): per-position logits of a batched
12-token prefill are **memcmp-identical (248320 floats × 12)** to the sequential single-token forward
(compare with `QWEN35_FUSE=0x37` — the fused attn qknorm+rope kernel is ~1 ulp off the standalone
pair, a pre-existing fusion contraction property). Prefill → decode continuity verified: one decode
step on top of the batched state matches the fully sequential run exactly.
**Measured prefill throughput** (2-die split @20, boost clocks): n=32 — sequential 15.8 tok/s vs
batched **32.2 tok/s (2.04×)**; n=128 — sequential 17.8 tok/s vs batched **36.5 tok/s (2.05×)**.
Kernel-level nx speedups are 1.7–3.1× per pass (`foundry/cuda/tests/test_matvec_nx`). Remaining
prefill levers: wider column tiles, batching the host attention/Q-split glue onto the die, and
expert-grouped MoE batching (tokens sharing an expert stream its weights once).
**Host bring-up prereqs** (else CUDA falls back to CPU or the box hangs): quirk kernel (16-die
small-BAR map), `nvidiafb` blacklisted, `NVreg_EnableMSI=0`, `NVreg_DynamicPowerManagement=0`
(all on the GRUB cmdline), persistence mode — see `foundry/barswap/k80_quirk/README.md`. Profiler: `QWEN35_PROF=1`.
**Operational:** under sustained high-util load a marginal card can Xid-79 "fall off the bus"
(power/thermal — visible only now that the dies actually work); mitigate with `nvidia-smi -pl 110`
and/or forced-air cooling (K80s are passively cooled).

141
docs/architecture/rwkvx.md Normal file
View File

@ -0,0 +1,141 @@
# RWKV-X Architecture
RWKV-X is a hybrid model architecture that interleaves RWKV-7 recurrent blocks (~75% of layers) with sparse attention blocks (~25%). It is fundamentally different from transformer models and uses safetensors format instead of GGUF.
## Source
The RWKV-X engine in Pantheon (`src/models/rwkvx.c`, ~2,500 LOC) is copied from the battle-tested Foundry implementation (`foundry/rwkvx/rwkvx.c`), with Metal, CUDA, HybridMem, training, batch, instrumentation, and OpenMP support stripped. A thin adapter layer wraps it into Pantheon's `pn_model`/`pn_state`/`pn_forward_fn` interface.
## Model Architecture
For the 0.2B model (12 RWKV + 4 attention layers), block execution order is:
```
R0, R1, R2, A0, R3, R4, R5, A1, R6, R7, R8, A2, R9, R10, R11, A3
```
Where `R` = RWKV-7 recurrent block, `A` = sparse attention block.
### Forward Pass
```
embed(token)
→ for each block in interleaved order:
if RWKV-7 block:
LayerNorm → TimeMix (WKV-7 kernel) → residual
→ LayerNorm → ChannelMix FFN → residual
if attention block:
LayerNorm → Multi-Head Attention (with KV cache) → residual
→ LayerNorm → ChannelMix FFN → residual
→ LayerNorm → LM head → logits
```
### WKV-7 Kernel (Generalized Delta Rule)
The core recurrent kernel, computed per-head per-timestep:
```
for each timestep t, for each head h:
state *= diag(w[t]) // decay
state += (a[t] @ state) * k^T // delta rule bonus
state += v[t] @ k[t]^T // key-value update
output[t] = r[t] @ state // readout
```
State is a `[head_size, head_size]` matrix per head per RWKV-7 layer — fixed size regardless of sequence length.
### Attention Blocks
Standard multi-head attention with KV cache. Uses cached keys and values up to a configurable max length (default 2048). No RoPE — position information comes from the surrounding RWKV-7 blocks.
## Weight Format
RWKV-X uses safetensors (not GGUF). Weights are loaded via Foundry's `fs_open()` (zero-copy mmap) and routed by prefix:
| Prefix | Block Type |
|--------|-----------|
| `rwkv.blocks.N.*` | RWKV-7 recurrent block N |
| `moba.blocks.N.*` | Sparse attention block N |
| `emb.weight` | Token embeddings |
| `head.weight` | LM head |
| `ln_out.*` | Final layer norm |
Weight transformations at load time:
- Linear weights are transposed (safetensors stores `[out, in]`, inference needs `[in, out]`)
- Singleton dimensions are squeezed
- `att.r_k` is flattened from `[n_head, head_size]` to `[n_embd]`
- Embeddings are pre-normalized (L2 norm per row)
## INT8 Quantization
When `--quantize` is passed, all linear projection weights are quantized to INT8 at load time using per-row symmetric quantization:
```
scale[row] = max(abs(row)) / 127
quantized[row][col] = round(float[row][col] / scale[row])
```
Inference uses a NEON-accelerated INT8 dot product kernel (`fast_matvec_q8`) that pre-quantizes the input vector once, then computes INT8 dot products for all output rows. This gives ~3x speedup on ARM.
## Tokenizer
RWKV-X uses a **trie-based World tokenizer** with 65,536 tokens. The tokenizer operates on raw UTF-8 bytes with greedy longest-match — fundamentally different from BPE or SentencePiece.
The vocab is loaded from a text file (`rwkv_vocab_v20230424.txt`) via the `--vocab` CLI flag. Each line contains a token ID, token length, and the token's byte representation as a Python byte literal.
The tokenizer is stored inside the RWKV-X model struct (not in `pn_model->tokenizer`). The engine routes to it via `pn_rwkvx_encode()`/`pn_rwkvx_decode()` when `config.arch == PN_ARCH_RWKVX`.
## Dual State
Each inference session maintains two state types in `pn_state->kv_cache`:
- **Recurrent state**: Fixed-size per-layer matrix `[n_heads, head_size, head_size]` for RWKV-7 blocks
- **KV cache**: Key-value store for sparse attention blocks (capped at `max_cache_len`)
- **Workspace**: Pre-allocated scratch buffers (16 buffers + ping-pong pair + INT8 buffer)
All state is encapsulated in a single opaque struct. `pn_state_reset()` zeros both the recurrent state and KV cache.
## Adapter API
The adapter layer in `rwkvx.c` maps Pantheon's interface to internal RWKV-X functions:
| Pantheon API | Internal Function |
|-------------|-------------------|
| `pn_rwkvx_load()` | `rx_model_load_internal()` |
| `pn_rwkvx_free()` | `rx_model_free_internal()` + tokenizer free |
| `pn_rwkvx_forward()` | `rx_forward_internal()` |
| `pn_rwkvx_state_new()` | `rx_state_new_internal()` |
| `pn_rwkvx_state_free()` | `rx_state_free_internal()` |
| `pn_rwkvx_state_reset()` | `rx_state_reset_internal()` |
| `pn_rwkvx_encode()` | `rx_tokenizer_encode()` |
| `pn_rwkvx_decode()` | `rx_tokenizer_decode()` |
## Usage
```bash
# FP32 inference
./pantheon --model ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
--vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
--prompt "Hello world" --max-tokens 32
# INT8 quantized inference (~3x faster decode on ARM)
./pantheon --model ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
--vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
--prompt "Hello world" --max-tokens 32 --quantize
```
## What Was Stripped
The Foundry RWKV-X engine has additional features not yet in Pantheon:
| Feature | Why Stripped |
|---------|-------------|
| Metal GPU (`fm_*`) | Pantheon doesn't have Metal model integration yet |
| CUDA GPU (`fc_*`) | Pantheon doesn't have CUDA model integration yet |
| HybridMem (`hm_*`) | Separate feature, not part of base inference |
| Training (`rx_forward_ex`, callbacks) | Training not in Pantheon scope yet |
| Batch forward (`rx_forward_batch`) | Prefill optimization, add later |
| Instrumentation (attn_log, embed_capture) | Debug tooling, add later |
| OpenMP (`#pragma omp`) | Pantheon uses threadpool, not OpenMP |
These can be added incrementally in future phases.

553
docs/architecture/serve.md Normal file
View File

@ -0,0 +1,553 @@
# Serve Layer Architecture
Pantheon's serve layer implements an OpenAI-compatible HTTP inference server.
Entry point: `src/serve/serve_command.c`. The serve layer sits above the engine
(`src/core/engine.c`) and is reached via `pantheon serve`.
---
## Overview
At startup `pn_serve_main` (`serve_command.c:206`) parses one or more `--model`
flags, loads each model into an `mr_registry`, binds a TCP socket, and enters an
infinite `accept()` loop. Every accepted connection is handed to a detached
`pthread` (`spawn_connection`, `serve_command.c:185`); if `pthread_create` fails
it falls back to handling the request inline so no connection is ever silently
dropped.
The connection thread calls `handle_connection` → `dispatch`. Dispatch is a
straight O(route-count) `strcmp` chain (`serve_command.c:124`):
| Method | Path | Handler |
|--------|------|---------|
| `OPTIONS` | any | CORS preflight: `hs_send_json 200 {}` |
| `GET` | `/v1/models` | `oai_handle_models` |
| `POST` | `/v1/chat/completions` | `oai_handle_chat` |
| `POST` | `/admin/models/load` | `ma_handle_load` |
| `POST` | `/admin/models/unload` | `ma_handle_unload` |
| `GET` | `/admin/dies` | `ma_handle_dies` |
| `GET` | `/` | health: `{"service":"pantheon","status":"ok"}` |
`GET /v1/models` and admin routes never touch any generation lock; they can
respond concurrently with an in-flight generation on any model.
### Request lifecycle (socket → SSE)
```
accept()
└─ spawn detached pthread
└─ hs_read_request() (http_server.c — reads headers + body)
└─ dispatch()
└─ oai_handle_chat()
├─ jr_parse(body)
├─ parse_chat_request() (extracts model, params, session_id)
├─ route_model() (mr_acquire → entry + refcount++)
│
├─ [batched path: QWEN35_BATCH_SLOTS=1]
│ ├─ bs_slot_acquire_or_wait() (blocks up to 30 s)
│ └─ sg_run_completion_batched() → SSE
│
└─ [single-user path: sched == NULL]
├─ pthread_mutex_trylock(&entry->gen_lock) → 429 if busy
└─ sg_run_completion() → SSE
└─ mr_release(reg, entry) (refcount--)
```
---
## Model Registry and Routing
**Files:** `src/serve/model_registry.h`, `src/serve/model_registry.c`
### Data structures
```c
/* model_registry.h:54 */
typedef struct {
char name[MR_MAX_NAME]; /* served model id */
pn_model *model;
pn_state *state; /* single-user state (sched==NULL) */
pthread_mutex_t gen_lock; /* single-user generation serializer */
pn_batch_sched *sched; /* C3: non-NULL when batching is on */
pn_prefix_cache *prefix_cache; /* PC1: session prefix cache */
pthread_mutex_t pc_lock; /* serializes prefix-cache ops */
int dies[PN_MODEL_MAX_DIES];
int n_dies; /* 0 = CPU-resident */
uint64_t vram_bytes;
int refs; /* in-flight acquires */
bool unloading;
bool used;
} mr_entry;
/* model_registry.h:86 */
typedef struct {
mr_entry entries[MR_MAX_MODELS]; /* MR_MAX_MODELS = 16 */
int n;
pthread_mutex_t lock;
pthread_cond_t drained; /* signaled on every mr_release */
pthread_mutex_t upload_lock; /* A3: one GPU weight upload at a time */
} mr_registry;
```
### Loading (`mr_load`, `model_registry.c:90`)
Load is a four-phase operation designed to keep the registry serviceable
throughout:
1. **Slot reservation** (under `reg->lock`): a slot is marked `used=true` with
no model pointer yet — the name is taken but the entry is not yet acquirable
(`require_ready` skips it in `find_locked`).
2. **Placement + VRAM admission** (lock-free): auto-placement via the die manager
(`pn_dm_place`), or explicit dies from `--dies`. `pn_admit_model_load` gates
against the per-die budget using a conservative 8 k context guess
(`MR_ADMIT_CTX_GUESS = 8192`, `model_registry.c:36`). `pn_dm_bind` commits the
reservation.
3. **Weight upload** (under `reg->upload_lock`): `pn_load_on` runs while the
registry stays fully serviceable. `upload_lock` ensures at most one GPU upload
is in flight at a time, preventing two concurrent 2.4 GB transients stacking
on an 8 GB host (`model_registry.c:213`).
4. **Publish** (under `reg->lock`): model, state, locks, and die info are written
atomically and `reg->n` is incremented. After publish, the batch scheduler
(`bs_create`) and prefix cache (`pn_prefix_cache_new`) are created and
published under the lock. Both are fail-soft: failure leaves the model in
single-user mode or full-prefill mode respectively.
### Acquire / release refcounting
```c
mr_entry *mr_acquire(mr_registry *reg, const char *name); /* model_registry.c:322 */
void mr_release(mr_registry *reg, mr_entry *entry); /* model_registry.c:358 */
```
`mr_acquire` locks `reg->lock`, scans entries, increments `refs`, and returns
the pointer. Every user of an entry MUST pair `mr_acquire` with `mr_release`.
`mr_release` decrements `refs` and broadcasts `reg->drained`; `mr_unload` waits
on that condvar until `refs == 0` before freeing.
### Multi-card load-balancing (`model_registry.c:329`)
An operator may place one model id on several cards (`--dies 4,5` and `--dies
6,7` both under `--name my-model`). `mr_acquire` finds all ready entries that
share the requested name and picks the one with the fewest in-flight `refs` —
round-robin load balance at the reference granularity:
```c
for (int i = 0; i < MR_MAX_MODELS; i++) {
mr_entry *c = &reg->entries[i];
...
if (!e || c->refs < e->refs)
e = c;
}
```
Empty/null name resolves to the first (lowest slot index) resident entry —
the back-compat single-model shortcut (`model_registry.c:344`).
### Route model (`openai_chat.c:499`)
```c
static mr_entry *route_model(int client_fd, mr_registry *reg, const char *name)
{
mr_entry *entry = (mr_count(reg) <= 1) ? mr_acquire(reg, NULL)
: mr_acquire(reg, name);
...
}
```
With exactly one resident model, every request routes to it regardless of the
`model` field — preserving single-model server behavior. With more than one,
an exact name match is required; an unknown name returns 404, no loaded model
returns 503.
### `/v1/models` dedupe (`openai_chat.c:266`)
`oai_handle_models` calls `mr_snapshot` to get all resident entries, then
dedupes by name before serializing JSON — a model placed on two cards appears
as one logical id:
```c
bool seen = false;
for (int j = 0; j < i; j++)
if (strcmp(info[j].name, info[i].name) == 0) { seen = true; break; }
if (seen) continue;
```
### Unload (`mr_unload`, `model_registry.c:369`)
Marks `unloading=true` (blocking new acquires), decrements `reg->n`, then
blocks on `reg->drained` until `refs == 0`. Detaches all pointers under the
lock and frees them outside: prefix cache → batch scheduler → die binding →
state → model.
---
## Batched Decode (`batch_sched.c`)
**Feature gate:** `QWEN35_BATCH_SLOTS=1` (env var, CUDA build only).
`bs_enabled()` (`batch_sched.c:44`) caches the result.
When enabled, `mr_load` calls `bs_create(model, BS_MAX_USERS)` where
`BS_MAX_USERS = 8` (`batch_sched.h:173`). The scheduler owns 8 `pn_state*`
instances (one per slot) and a background driver thread.
### Slot lifecycle
```
IDLE -(bs_slot_acquire/acquire_or_wait)-> PREFILL
-(bs_slot_bind / bs_slot_bind_ex)-> BOUND
-(driver first step)--------------> DECODE
-(bs_slot_release)----------------> IDLE
```
Only the owner thread moves a slot back to IDLE. The driver marks `finished=true`
on EOS/budget but never calls `bs_slot_release` — it always signals the owner
to drain and release.
### GPU-ownership turnstile (`batch_sched.c:72`)
The model's single-user forward machinery (C2 async phase machine, shared live
SSM buffers, shared GPU KV ring) supports exactly one in-flight GPU forward at
a time. Pre-C3 the per-model `gen_lock` enforced this. The batch scheduler uses
a FIFO ticket turnstile instead — `gpu_ticket_next` / `gpu_ticket_serving` /
`cv_gpu` — which prevents the free-running L3 driver from barging ahead of a
waiting prefiller:
```c
static void bs_gpu_acquire_locked(pn_batch_sched *sched) { /* batch_sched.c:72 */
uint64_t my = sched->gpu_ticket_next++;
while (sched->gpu_ticket_serving != my)
pthread_cond_wait(&sched->cv_gpu, &sched->mu);
}
```
A turn owner never `cond_wait` (chunks and steps are bounded GPU work), so
the turnstile is deadlock-free by construction.
### D1 overflow queue + D3 admission backpressure
`bs_slot_acquire_or_wait` (`batch_sched.c:1475`, called from `openai_chat.c:577`
with a 30 s timeout) replaces the immediate 429:
- **Fast path:** slot free AND `pressure_level < BS_PRESSURE_HOLD_LEVEL` (3 =
CRITICAL) → acquire immediately.
- **Slow path:** enqueue (`n_queued++`) and block on `cv_idle`.
- Queue full + pressure < CRITICAL → 429 + `Retry-After: 5`.
- Queue full + CRITICAL → 503.
- Timeout after `wait_ms` → 429 + `Retry-After: 2`.
- Shutdown → 503.
`BS_QUEUE_DEPTH_DEFAULT = 32` (`batch_sched.h:274`). Override with `BS_QUEUE_DEPTH`.
`bs_notify_pressure` (`batch_sched.c:1589`) is called by the serve path before
every `bs_slot_acquire_or_wait` (`openai_chat.c:569`) to push the live
`pn_mem_state` into the scheduler, so the gate sees the current host state at
decision time.
### D2 — continuous batching / dynamic-width step
The driver compacts the active (BOUND/DECODE) slots into GPU indices
`[0, n_active)` each step. Idle slots pay zero GPU compute. When `active_set_changed`
is set (join, leave, or finish), `rebuild_gpu_map` (`batch_sched.c:237`) runs
a greedy permutation algorithm that handles arbitrary reorderings — including
cycles that require a scratch slot (`cap` index, `batch_sched.c:279`). This
fixed the K≥5-concurrent-joiners bug where the old downward-only move refusal
caused all active users to emit empty content.
### L3 — in-driver sampling (default on)
`bs_slot_bind_ex` (`batch_sched.c:1294`) passes the request's sampling params,
repeat-penalty window, and token budget to the driver. After each `decode_slots`
step the driver samples every slot itself — sequentially, deterministically —
and pushes the result to a per-slot output ring sized to the full budget:
```c
uint32_t next = pn_sample_next_token(lo[g], sched->n_vocab,
&sl->sp, sl->recent, sl->n_recent);
sl->ring[sl->ring_w++] = next; /* batch_sched.c:545 */
```
Owners drain the ring with `bs_slot_pop_token` and do detokenize/emitter/SSE
entirely off the step's critical path. A slow or stalled client cannot stall
the rest of the batch.
The ring is never resized after bind (`ring_cap == budget`), so it can never
overflow and the driver never blocks on a slow consumer.
Stop-string detection stays owner-side (text level); a slot may over-generate
past a stop string until its owner drains and releases — surplus ring tokens
are discarded.
Disable with `QWEN35_BS_DRIVER_SAMPLE=0` to restore legacy lockstep
logits/token exchange.
### L2 — chunked prefill with yield windows
Prefill runs in chunks of `QWEN35_BS_PREFILL_CHUNK` tokens (default 128).
After each chunk (except the last) `bs_prefill_yield` (`batch_sched.c:1040`)
makes the slot's host mirror authoritative via `gpu_state_to_host`, releases
the GPU turn, and re-acquires it FIFO — giving the driver one decode step
between chunks. This bounds a joiner's stall on concurrent decoders to one
chunk instead of one full prompt.
### Single-user fallback
When `QWEN35_BATCH_SLOTS` is not set (or the build has no CUDA), `entry->sched`
is `NULL`. The serve path (`openai_chat.c:658`) falls back to:
```c
if (pthread_mutex_trylock(&entry->gen_lock) != 0) {
hs_send_error(client_fd, 429, "server_busy",
"a generation is already in progress on this model; "
"retry shortly");
...
}
```
A second concurrent request to the same model gets an immediate 429. Requests
to different models run concurrently (each holds only its own entry's lock).
---
## Generation and Streaming (`serve_generation.c`)
### Prompt building
`build_prompt` (`serve_generation.c:120`) tokenizes the full `messages[]` array
into ChatML format using the model's tokenizer:
`<|im_start|>role\ncontent<|im_end|>\n` for each turn, with `tool` messages
wrapped in `<tool_response>`. Opens the assistant turn and prefills `<think>`
for reasoning models (`PN_CHAT_CHATML_THINK`).
### Admission
Before any prefill, `pn_admit` checks live host + die memory and either passes,
caps the context (sliding-window oldest-token drop), or denies. A denied
admission returns `SG_ADMIT_DENIED`, which the serve path converts to a 503
(`openai_chat.c:683`). A capped admission sets `result->context_capped = 1` and
disables prefix caching for that turn; the client sees `"context_capped": true`
on the response.
### Single-user completion (`sg_run_completion`, `serve_generation.c:525`)
```
build_prompt → pn_admit → [prefix cache acquire] → prefill chunks
→ sample first token → decode loop (sg_forward_step per token)
→ [prefix cache commit] → return sg_result
```
`sg_forward_step` (`serve_generation.c:411`) uses an async non-blocking launch +
event-poll when the model supports it (`pn_forward_step_begin/poll`), sleeping
50 µs → 500 µs between polls. This allows up to 8 per-card connection threads
to run their cards simultaneously from a 2-core host without blocking each other.
### Batched completion (`sg_run_completion_batched`, `serve_generation.c:744`)
Same flow as single-user but:
- Prefill takes the GPU turn via `bs_prefill_begin` or `bs_prefill_resume`.
- Chunked prefill with `bs_prefill_yield` windows (L2).
- After the last chunk, `bs_slot_bind_ex` (L3) or `bs_slot_bind` (legacy) hands
the slot to the driver.
- L3 path: drain the ring with `bs_slot_pop_token` → detokenize → emitter → SSE,
all off the critical path.
- `bs_slot_finalize` (if caching) then `bs_slot_release`.
### Streaming (SSE)
`run_streaming` / `run_streaming_batched` (`openai_chat.c:434,464`) call
`hs_begin_sse` and register `stream_delta_cb` as the `delta_fn`. The callback is
invoked by the emitter for each text fragment:
```c
static void stream_delta_cb(const char *text, int is_reasoning, void *ud) {
/* openai_chat.c:397 */
jw_raw(&jw, is_reasoning ? "\"reasoning_content\":" : "\"content\":");
jw_string(&jw, text);
send_chunk(sc, jw.data, NULL); /* hs_send_sse */
}
```
A final chunk carries `finish_reason`; then `hs_end_sse` writes `data: [DONE]`.
### Reasoning / stop-sequence emitter
The `emitter` struct (`serve_generation.c:179`) handles three channels:
- **in_think**: text before `</think>` → `reasoning_content`; switches to content
channel on `</think>`.
- **content**: after `</think>`, before a stop string or `<tool_call>`.
- **in_tool**: raw text after `<tool_call>` → captured in `tool` channel for
structured parsing by `tool_calling.c`.
A `carry` buffer holds the longest suffix that could start any marker or stop
string across a token boundary.
### Prefix cache (`src/core/prefix_cache.c`, used via `serve_generation.c:574`)
Keyed by `session_id` (from body field `session_id` or header `X-Session-Id`).
Single-user path: `pn_prefix_cache_acquire` returns the session's `pn_state` at
the delta boundary; on commit, `pn_prefix_cache_commit` records the full token
history. The `pc_lock` is not needed on the single-user path (`gen_lock` already
serializes), so `pcx.pc_lock = NULL` (`openai_chat.c:674`).
Batched path (PC-batched): session affinity is slot-resident. `bs_slot_acquire_biased`
prefers the slot that already holds this session's warm state. `bs_prefill_resume`
checks if the slot's committed history (`hist`) is an exact prefix of the new
prompt and skips the already-committed tokens. `bs_slot_finalize` snapshots the
GPU state to host and publishes the driver-maintained `ct` (committed-token log)
as `hist` for the next turn.
---
## Concurrency Model
| Lock | Scope | What it protects |
|------|-------|-----------------|
| `reg->lock` | `mr_registry` | slot table (used/refs/unloading/n); brief critical sections only |
| `reg->drained` | condvar on `reg->lock` | unload drain wait |
| `reg->upload_lock` | `mr_registry` | one GPU weight upload in flight at a time |
| `entry->gen_lock` | `mr_entry` | single-user generation serializer (trylock → 429) |
| `entry->pc_lock` | `mr_entry` | prefix cache ops on the single-user path (unused — gen_lock covers it; kept for the batched concurrent-slot case) |
| `sched->mu` | `pn_batch_sched` | ALL slot state: status words, token/logits flags, rings, turnstile counters, D1 queue depth, D3 pressure level |
| `sched->cv_gpu` (turnstile) | under `sched->mu` | one GPU forward in flight at a time (prefill chunk, bind, or decode step) |
| `sched->cv_work` | under `sched->mu` | driver wake (gate may have opened) |
| `sched->cv_idle` | under `sched->mu` | D1 waiter wake (slot freed or pressure dropped) |
| `sl->cv` | per-slot, under `sched->mu` | driver → owner wakeup (logits/ring/finished) |
Connection threads are otherwise independent: each owns its own
`hs_request`, `chat_request`, `sg_result`, and `emitter`. The sampler scratch
(`pn_sample_next_token`) is `_Thread_local`.
The `accept()` loop is single-threaded but non-blocking per iteration. Heavy
work (parsing, prefill, decode, SSE write) runs entirely in per-connection
threads, so cheap routes (`/v1/models`, `/`) are never queued behind an
in-flight generation.
---
## Admin API (`model_admin.c`)
| Route | Body | Action |
|-------|------|--------|
| `POST /admin/models/load` | `{"path":"..","name":"..","dies":[..],"quantize":false}` | `mr_load` — may block ~55 s for a GPU upload |
| `POST /admin/models/unload` | `{"name":".."}` | `mr_unload` — drains in-flight refs then frees |
| `GET /admin/dies` | — | `pn_dm_get` snapshot of all dies (budget/used/model) |
Error codes from `status_for_reason` (`model_admin.c:28`): admission/table-full
→ 507, duplicate/unloading → 409, not-found → 404, other → 500.
A runtime `/admin/models/load` arriving while a startup upload is in flight
blocks on `reg->upload_lock` and is refused promptly by the slot scan once the
prior load finishes — never stacks two 2.4 GB transients.
---
## Router Registration (`serve_registry.c`)
**Gating:** fully off unless `PANTHEON_REGISTRY_URL` is set. `rc_enabled()` is checked at
`serve_registry_start` entry; with it unset the function returns immediately and the
serve process is byte-for-byte unchanged (`serve_command.c:350`, `serve_registry.c:78`).
When enabled, at startup `serve_registry_start` (`serve_registry.c:76`) iterates every
loaded model (via `mr_snapshot`) and calls `rc_register` for each with:
```
model_id = info[i].name /* the shared served identity */
instance_id = host:port:model /* serve_registry.c:98 — unique per process */
endpoint = http://host:port
capacity = batch-slot count (1 if single-user)
```
A background heartbeat thread then re-acquires each entry every
`RW_HEARTBEAT_SECONDS` to read live load (`bs_get_load`) and POST it to the router.
### What this means for in-process replicas
Two replicas of `ornith` on dies 2 and 3 in **the same process** share the same
`host:port`, so they produce the same `instance_id = host:port:ornith` and **collapse
to one instance** at the router. The router sees one `ornith` endpoint, not two.
This is correct and intentional: **in-instance replica load-balancing is handled
entirely by `mr_acquire`** (least-busy across same-name entries — see above). The router
is the **multi-instance dispatch layer** for tying separate `pantheon serve` processes
together. A single self-routing instance does not need it.
> Cross-link: docs/README.md "Multi-Model Smart Routing" — "one instance is fully
> self-routing and needs nothing in front of it."
---
## Environment Variables
| Variable | Default | Effect |
|----------|---------|--------|
| `QWEN35_BATCH_SLOTS` | off | `1` enables multi-user batched decode (C3) |
| `QWEN35_BS_DRIVER_SAMPLE` | `1` | `0` restores legacy owner-side lockstep sampling |
| `QWEN35_BS_PREFILL_BATCH` | `1` | `0` disables batched chunk prefill (token-by-token) |
| `QWEN35_BS_PREFILL_CHUNK` | `128` | chunk size in tokens for L2 yield windows |
| `BS_QUEUE_DEPTH` | `32` | max D1 queue depth before 429 |
| `QWEN35_BS_STATS` | off | `1` logs per-step timing to stderr |
| `QWEN35_KV_Q4` | `1` | `0` disables Q4 KV quantization in admission math |
| `QWEN35_GPU_FULL_LOGITS` | off | Refused at serve startup — forces 248k-vocab D2H copy |
| `PN_ASYNC_POLL_US` | `500` | Max sleep between async forward-step event polls (µs) |
| `PANTHEON_REGISTRY_URL` | unset | Set to enable router registration (`serve_registry.c:78`); unset = no-op |
| `PANTHEON_NODE_ID` | `node0` | Node identifier sent in router registration (`serve_registry.c:83`) |
| `PANTHEON_REGISTRY_TRANSPORT` | `local` | Transport hint sent to router on registration (`serve_registry.c:84`) |
---
## Known Sharp Edges
- **`mr_unload` blocks its caller** until all in-flight `refs` drain
(`model_registry.c:389`). A long generation on the model being unloaded will
hold the admin response for its full duration. There is no timeout.
- **Duplicate names are explicitly allowed** (`model_registry.c:119`). An
operator placing the same model id on cards 4 and 5, then unloading by name,
will unload whichever slot `find_locked` finds first — the other remains. There
is no unload-by-slot-index API; the operator must call unload twice.
- **`make_completion_id` uses `rand()`** without seeding (`openai_chat.c:87`).
`rand()` is not seeded with `srand()`, so id generation is not
cryptographically random and starts from the same sequence on every process
restart. Collision risk is low (hex8 = 4B permutations) but completions ids
are not guaranteed unique across restarts.
- **`pc_lock` is allocated but never taken on the single-user path**
(`openai_chat.c:674`, `pcx.pc_lock = NULL`). On the batched path each slot
owns its own `hist`/`ct` structures guarded by `sched->mu`, so `pc_lock` is
also unused there. The lock exists as a slot for a future concurrent-slot
single-cache design.
- **Stop-string detection fires owner-side** after the driver has already
stepped tokens into the ring (L3 path). The driver steps until EOS/budget;
the owner drains until a stop string is detected in the decoded text.
Over-generation is bounded by `max_tokens` but the surplus ring tokens are
silently discarded (`serve_generation.c:930-942`). This means
`completion_tokens` in the usage field may be slightly lower than the number
of steps the driver ran.
- **GPU lazy init hangs if called too early** (`batch_sched.c:714`). GPU lane
registration (`bs_lazy_init_gpu`) is deferred to the first `bs_slot_bind` to
avoid a CUDA context timing issue right after a long weight upload. If the
context is still unsettled when the first request arrives, the bind will hang
or fail. The scheduler is fail-soft here (bind failure → -1 → request gets a
500), but a burst of concurrent first requests could all fail before the
context settles.
- **The `reg->lock` is brief but coarse** during `mr_load`'s slot reservation
phase (`model_registry.c:123`). A concurrent load + multiple request threads
all briefly contend on the same mutex. In practice this is negligible (the
critical sections are O(MR_MAX_MODELS) array scans at most), but worth noting
for any future lock-free refactor.
- **Cross-replica session affinity is not implemented.** `mr_acquire` selects the
least-busy replica by `refs` with no stickiness (`model_registry.c:329-341`). A
multi-turn session with `session_id` can hop replicas between turns, missing its
prefix cache on the new replica and paying a full reprefill. The single-die batched
path has slot-resident affinity (`bs_slot_acquire_biased`), but that only applies
within one replica. Cross-replica affinity is designed but not yet shipped.
- **`QWEN35_GPU_FULL_LOGITS` is a hard startup guard** (`serve_command.c:279`).
Setting it causes the server to refuse to start with an explicit error. It is
a debug knob that forces the full 248k-vocab logit tensor D2H per step and
caps throughput on the K80 host link (~0.4 GB/s). The guard is intentional and
correct; do not remove it.

204
docs/coding-best-practices.md Normal file
View File

@ -0,0 +1,204 @@
# Coding Best Practices — C
The standard code in this repo is held to. PRs are reviewed against it, and review
comments cite it by rule number (`blocker: rule 3, this file is 1,400 lines`).
Rule numbers are frozen. New rules append; they never renumber.
The PR process itself — size, evidence, comment conventions — is in
[`../CONTRIBUTING.md`](../CONTRIBUTING.md).
---
## Rules
**1. Descriptive names.** Every name says what it is at a glance — files, functions,
types, variables. The test: would you know what it is from the tree alone, without
opening it? `pn_prefix_cache_lookup()`, not `lookup()`. Abbreviate only what is
universal (`id`, `url`, `http`, `kv`, `ctx`).
**2. Small functions — ~100 lines of body.** Doc comments don't count. Past ~100, look
for the extraction point. One blank line between logical chunks, never two.
**3. Small files — under 1,000 lines of code.** Comments and headers don't count. Prefer
more files with fewer functions. Every function in a file must belong there.
**4. Library-first.** Write every function as if it will be extracted into a standalone
library — self-contained, minimal dependencies, no hard-coded project values in reusable
logic. Headers define the contract, sources hold the implementation. Extraction should
mean moving files, not rewriting them.
**5. Folder nesting is navigation.** Group by domain, not by file type. Each directory
has one clear purpose. A directory holding one file is usually wrong.
**6. Error transparency.** Never swallow an error. Return a code or log it — never both
nothing. No empty error paths. Callers get a typed error, not a guess.
**7. Delete dead code.** Don't comment it out, don't leave unused functions or includes.
Git has it if you need it back.
**8. Prove a test can fail.** A test you have never seen fail is not evidence. Break the
code it covers, watch it go red, report the failure count.
> **Apple's `make` is 3.81 (2006)** and compares mtimes at one-second granularity. A
> source edited in the same second as its object is reported `up to date` and is **not
> rebuilt** — so the obvious mutation loop can be running the previous binary and
> reporting a vacuous gate that isn't. When proving a test red, `rm` the specific `.o`
> **and** the linked binary first. Switching build configs needs `rm -rf build` entirely.
---
## File headers
Every source file opens with the SPDX pair, then the header block:
```c
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file prefix_cache.c
* @brief Longest-common-prefix KV reuse across turns.
*
* @status None
* @issues None
* @todo None
*/
```
`@status` / `@issues` / `@todo` are always present, `None` when empty — the empty slot is
the reminder that the slot exists. When you work in a file, clear resolved entries and
reset to `None`.
**Required on every new file.** Existing coverage is partial and is being backfilled;
until that lands, CI enforces the rule on added files only.
**Exempt:** test files, generated sources, and third-party-derived files (which keep their
upstream attribution — a Pantheon block may go below it, never in place of it).
---
## Documentation
**Public functions are documented in the header**, which is the API contract:
```c
/**
* Look up the longest cached prefix for a token sequence.
*
* @param cache Prefix cache handle
* @param tokens Token sequence to match
* @param n Length of tokens
* @return Matched prefix length, or 0 if nothing matched
*
* Thread-safe: yes (internal locking)
* Reentrant: no
* Signal-safe: no
*/
```
The three safety annotations are always present. `TBD` is an acceptable answer; omitting
them is not. Internal helpers need docs only when the name and signature don't carry it.
**Inline comments explain why, not what.** The code already says what it does.
```c
/* K80 has no __ldg for 8-byte types — split the load or it serialises */
```
---
## Layout
`include/` and `src/` mirror each other. `snake_case` filenames that name their contents
(`die_manager.c`, not `dm.c`). One header + one source per logical unit. Tests mirror the
source tree.
Every public symbol carries its module prefix — `pn_` for Pantheon, `fc_`/`tm_`/`ft_` for
Foundry. Anything not declared in a header is `static`. No exceptions.
---
## C conventions
**Formatting.** 4 spaces, no tabs. 100 columns. `char *ptr` — the star binds to the
variable. Opening brace on the **same line**, for functions and control flow alike.
```c
int pn_cache_lookup(pn_cache *cache, const uint32_t *tokens, size_t n) {
if (!cache || !tokens)
return -PN_EINVAL;
for (size_t i = 0; i < cache->count; i++) {
if (entry_matches(&cache->entries[i], tokens, n))
return (int)cache->entries[i].len;
}
return 0;
}
```
Omit braces on single-statement bodies — unless any branch of an `if`/`else` chain needs
them, in which case every branch gets them.
**Includes**, four groups separated by blank lines: this module's own header first (it
catches missing dependencies in that header at compile time), then standard library, then
system/platform, then project headers.
**Errors.** Functions return `int` (0 success, negative error) or a pointer (`NULL` on
failure). Use goto-cleanup for anything acquiring more than one resource — one exit path,
deterministic teardown. Error codes are a prefixed enum with a string conversion function.
**Memory.** `sizeof(*ptr)`, never `sizeof(Type)`. `calloc` or explicit zeroing. Document
ownership in the function doc — who allocates, who frees. Every `_open`/`_create`/`_new`
has a matching `_close`/`_destroy`/`_free`. `free(p); p = NULL;`. Arena-allocate groups
sharing a lifetime. Never return a stack address.
**const.** Every pointer parameter is `const` unless the function must modify what it
points at.
**Strings and buffers.** `sprintf`, `strcpy`, `strcat`, `gets`, and unbounded `scanf("%s")`
are banned — use `snprintf`, `strlcpy`, `strlcat`, `fgets`, width-limited `scanf`. Always
pass buffer sizes explicitly; check the `snprintf` return for truncation.
**Initialize everything.** `int rc = 0;`, `char *buf = NULL;`, `struct foo f = {0};`.
Declare close to first use, except where goto-cleanup requires it earlier.
**Opaque types** at public API boundaries — the struct definition stays in the `.c`.
Callers go through the API, never into fields.
**Header guards** use the project prefix (`PANTHEON_PREFIX_CACHE_H`), with the guard name
repeated in the `#endif` comment, and `extern "C"` in every public header.
**Asserts** document invariants that must never be false if the code is correct. Don't
assert things that legitimately fail — I/O, allocation, user input — those get error codes.
**Compiler.** The standard, modelled on SQLite and comparable C projects:
```
-std=c11 -Wall -Wextra -Werror -pedantic
```
plus `-Wshadow`, `-Wstrict-prototypes`, `-Wformat=2`, `-Wnull-dereference`,
`-Wconversion`, `-fstack-protector-strong`. Zero warnings, always.
The build does not yet meet this — it is `-Wall -Wextra -std=c11` today. Closing that gap
is tracked in the conformance spec in `plans/`; the standard above is the target the code
is held to, not a description of the current Makefile.
---
## Logging
| Level | Use |
|---|---|
| `ERROR` | operation failed, cannot continue |
| `WARN` | unexpected but recoverable |
| `INFO` | major state transitions, operation boundaries |
| `DEBUG` | decision points — which branch, and why |
| `TRACE` | entry/exit, raw values, wire data |
`DEBUG` and `TRACE` compile out in release builds. Log at boundaries and decisions, never
inside hot loops at `INFO` or above. Never log secrets, raw binary, or the success of
trivial operations. One line per message, greppable, with timestamp, level, and source
location.
The test: if you would add a `printf` to debug it, a `log_debug` should already be there.

162
docs/guides/chat.md Normal file
View File

@ -0,0 +1,162 @@
# Chat Templates & Interactive Chat
Pantheon v0.3.0 adds multi-turn interactive chat with automatic template detection. Each model family uses a different chat template — Pantheon detects the right one from GGUF metadata and formats messages with correct special tokens.
## Quick Start
```bash
./pantheon --model ../models/base/SmolLM2-135M-Instruct-Q8_0.gguf \
--chat --system "You are a helpful assistant."
```
```
Chat mode (ChatML). Type /quit to exit, /clear to reset.
> What is the capital of France?
The capital of France is Paris.
> And Germany?
The capital of Germany is Berlin.
> /quit
```
## CLI Flags
| Flag | Description |
|------|-------------|
| `--chat` | Enter interactive chat mode |
| `--system <text>` | System prompt (optional, chat mode only) |
| `--max-tokens <n>` | Max tokens per response (default: 128) |
Chat mode commands:
| Command | Action |
|---------|--------|
| `/quit` or `/exit` | Exit chat |
| `/clear` | Reset conversation (clears KV cache, starts fresh) |
## Supported Templates
| Style | Models | System Prompt |
|-------|--------|---------------|
| ChatML | Qwen2.5, SmolLM2 | Yes |
| Mistral | Mistral v0.x, Mixtral | No |
| Phi-3 | Phi-3-mini | Yes |
Detection is automatic — Pantheon reads the `tokenizer.chat_template` Jinja2 string from GGUF metadata and matches on key markers. Falls back to architecture-based detection if the template is missing.
## How It Works
### The Special Token Problem
BPE models (Qwen, SmolLM2) have "added tokens" like `<|im_start|>` that exist in the vocabulary but are **not produced by BPE merge rules**. Passing `"<|im_start|>user\nHello"` through the tokenizer splits `<|im_start|>` into individual characters — wrong.
Pantheon solves this by splitting formatted messages into **special** and **text** segments internally. Special tokens are looked up by exact string via `ft_token_to_id()`, text segments are encoded normally. The result is a correct `uint32_t[]` token array.
SentencePiece models (Mistral, Phi-3) don't have this problem — their Viterbi algorithm matches special tokens from the vocab naturally.
### Multi-Turn Flow
1. **Turn 0:** Format `system + user + assistant_prefix` → prefill → generate until EOS
2. **EOS sampled** but not fed through forward pass (existing behavior)
3. **Turn 1+:** Format `EOT + user + assistant_prefix` → prefill from current position → generate until EOS
4. **KV cache persists** across turns — no recomputation of earlier turns
### Template Formats
**ChatML** (Qwen2.5, SmolLM2):
```
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>
<|im_start|>user
{next_user}<|im_end|>
<|im_start|>assistant
```
**Mistral** (Mistral, Mixtral):
```
[BOS][INST] {user} [/INST]{response}</s> [INST] {next_user} [/INST]
```
**Phi-3**:
```
<|system|>
{system}<|end|>
<|user|>
{user}<|end|>
<|assistant|>
{response}<|end|>
<|user|>
{next_user}<|end|>
<|assistant|>
```
## C API
### `pn_stream_tokens()` — Pre-tokenized Streaming
```c
int pn_stream_tokens(pn_model *model, pn_state *state,
const uint32_t *tokens, int n_tokens,
int max_tokens,
pn_sampling_params *params,
pn_token_callback callback, void *user_data);
```
Like `pn_stream()` but takes pre-tokenized input. No BOS added — caller controls tokenization entirely. Returns number of tokens generated, or -1 on error.
### Chat Template Functions (`src/core/chat.h`)
```c
/* Detect style from Jinja2 template string */
pn_chat_style pn_chat_detect_style(const char *jinja_template);
/* Fallback: detect from architecture enum */
pn_chat_style pn_chat_style_from_arch(int arch);
/* Format first turn (system + user + assistant prefix) */
pn_token_seq pn_chat_format_first(pn_chat_style style, ft_tokenizer *tok,
const char *system, const char *user);
/* Format continuation turn (EOT + user + assistant prefix) */
pn_token_seq pn_chat_format_turn(pn_chat_style style, ft_tokenizer *tok,
const char *user);
```
### Embedding Chat in Your App
```c
#include "pantheon.h"
#include "src/core/engine.h"
#include "src/core/chat.h"
pn_model *model = pn_load("model.gguf");
pn_state *state = pn_state_new(model);
pn_sampling_params params = pn_sampling_defaults();
pn_chat_style style = (pn_chat_style)model->chat_style;
ft_tokenizer *tok = model->tokenizer;
/* First turn */
pn_token_seq seq = pn_chat_format_first(style, tok, "You are helpful.", "Hello");
pn_stream_tokens(model, state, seq.tokens, seq.n_tokens, 128, &params, on_token, NULL);
free(seq.tokens);
/* Second turn — KV cache preserved automatically */
seq = pn_chat_format_turn(style, tok, "Tell me more.");
pn_stream_tokens(model, state, seq.tokens, seq.n_tokens, 128, &params, on_token, NULL);
free(seq.tokens);
/* Reset for new conversation */
pn_state_reset(state);
```
## Tests
20 tests in `tests/test_chat.c` covering style detection (5), architecture mapping (6), style names (1), ChatML formatting (3), Phi-3 formatting (2), Mistral formatting (2), and passthrough (1).
```bash
make USE_BLAS=1 test # runs all 50 tests including chat
```

128
docs/guides/install.md Normal file
View File

@ -0,0 +1,128 @@
# Install
Build and run Pantheon on each supported platform. For a first run, see
[Quickstart](quickstart.md).
## Requirements
- A C11 compiler and `make`.
- Foundry, a git submodule at `./foundry`. Clone with
`git clone --recursive https://git.teamide.dev/David/Pantheon.git`; in an existing
clone run `git submodule update --init`. The Makefile looks for Foundry in this order:
`FOUNDRY=/path/to/foundry`, then `./foundry` (the submodule, or a symlink), then
`../foundry` (a sibling checkout). If none exists the build stops with
`Foundry not found at ...`.
- Nothing else for a CPU build. BLAS, threads and accelerators are opt-in flags.
## Linux (CPU)
```bash
sudo apt install libopenblas-dev # only for USE_BLAS=1
make USE_BLAS=1 USE_THREADS=1 pantheon
```
- `USE_BLAS=1` links OpenBLAS (`-lopenblas`).
- `USE_THREADS=1` adds the compute thread pool; size it with `PN_THREADS` (`0` = auto).
## macOS
```bash
make USE_BLAS=1 USE_THREADS=1 pantheon # Accelerate
make USE_BLAS=1 USE_THREADS=1 USE_METAL=1 pantheon # + Metal GPU backend
```
`USE_BLAS=1` uses Apple's Accelerate framework, so no extra install is needed.
`USE_METAL=1` (macOS only) compiles Foundry's Objective-C Metal backend and links the
Metal, MetalPerformanceShaders and Foundation frameworks.
## NVIDIA CUDA
```bash
make USE_CUDA=1 USE_BLAS=1 pantheon # Linux only
```
- **No CUDA toolkit or `nvcc` needed to build.** The kernels are embedded as PTX
(ISA 7.4, target `sm_37`) and the driver is loaded with `dlopen` at run time.
- **At run time** the process needs `libcuda.so.1` (the NVIDIA driver). The PTX needs a
driver that supports CUDA 11.4 or newer; Kepler cards such as the Tesla K80 top out at the
470 driver branch. cuBLAS (`libcublas.so.11`) is optional — without it a warning prints and
the custom kernels are used.
- With no driver or no card, the same binary runs on the CPU. `GET /` then reports
`"dies_visible": 0`.
- Pick devices with `--dies 0,1` or `CUDA_VISIBLE_DEVICES`; die numbers are relative to the
visible devices.
## Tenstorrent
**Ornith 1.5 9B is the only model supported on the card for now**; other models may load but
are not supported. The quick way is the two scripts at the repo root:
```bash
./install.sh # build pantheon-tt, set up card access (udev rule; asks before sudo)
./run_ornith.sh # download + check the model, serve on 127.0.0.1:8080
```
By hand:
```bash
make USE_TT=1 pantheon # Linux only, Blackhole p150a
```
- **No TT software needed to build.** Tensix programs are embedded in the binary
(`foundry/tt/ft_tt_programs.h`), so neither SFPI nor tt-metal is required.
- **At run time** the card must be reachable: by default through tt-kmd
(`/dev/tenstorrent/<n>`), or with `FT_TT_DRIVER=udrv` through Foundry's userspace driver,
which needs no kernel module but needs root or the udev rule
`foundry/tt/tools/60-foundry-blackhole.rules`. An explicit `FT_TT_DRIVER=udrv` that cannot
reach the card stops the process rather than falling back to the CPU.
- Read [accel-foundry.md §7](../architecture/accel-foundry.md) before running on the card:
what it can serve today, known limits, and hazards that cost a power cycle.
## Which binary you built
Each backend writes its own file and `pantheon` is a symlink to the last one built:
| Build | Binary |
|---|---|
| `make` | `pantheon-cpu` |
| `make USE_CUDA=1` | `pantheon-cuda` |
| `make USE_TT=1` | `pantheon-tt` |
| `make USE_METAL=1` | `pantheon-metal` |
Combined flags get combined names (e.g. `pantheon-cuda-tt`). Check what a binary is with
`./pantheon --backend` (prints `cuda`, `tt`, `metal` or `cpu`). Restore the link without
rebuilding: `ln -sfn pantheon-cuda pantheon`.
## Hardware notes
- **Build on the machine you run on.** The default flags include `-march=native`, so a binary
built on an AVX2 machine dies with `Illegal instruction` (SIGILL) on an older CPU such as a
no-AVX Pentium. `PORTABLE=1` builds a copyable binary instead (`-march=x86-64-v2` on x86-64,
the compiler default elsewhere), at some cost in speed.
- **Small hosts.** The `pantheon` binary is a single compiler invocation; `-j` only helps the
test build. On a 2-core host use `make -j2 ... test`.
- **Host RAM.** 8 GB is the design point: weights are mmap'd, and after a GPU upload the CUDA
paths drop the weight page cache (`QWEN35_DROP_WEIGHTS`, on by default).
## Running the tests
```bash
make USE_BLAS=1 test
```
Tests that need a model file print `[SKIP]` with the env var to set (e.g. `PN_QWEN35_MODEL`).
## Troubleshooting
| Symptom | Meaning / fix |
|---|---|
| `Illegal instruction` at start | Built on a newer CPU. Rebuild on the target machine. |
| `serve: REFUSED to start — ... has NO accelerator backend compiled in` | `--dies` was given to a binary without that backend. Check `./pantheon --backend`; build with `USE_CUDA=1` / `USE_TT=1`, or drop `--dies` to serve on the CPU. |
| `serve: REFUSED ... the die manager probed 0 dies` | Backend present but no device visible: driver not loaded, card missing, or `CUDA_VISIBLE_DEVICES` empty. |
| HTTP `429 server_busy` with `Retry-After` | Every decode slot is busy and the queue is full (or a 30 s queue wait timed out). Retry after the given seconds. |
| HTTP `429 model_at_session_capacity` | A new `session_id` while all `max_sessions` slots hold live sessions. Release idle sessions (`POST /session/release`) or raise `--max-sessions`. |
| HTTP `503 server_memory_constrained` | Host memory pressure: admission refused the prompt, or the queue is full under critical pressure. Transient; retry later. |
| HTTP `422 context_over_budget` | The prompt exceeds the per-session window (`kv_cap`). Not transient: shorten the conversation, then retry. Nothing is dropped silently. |
| `bpe: tokenizer.ggml.pre is unsupported; falling back to the qwen35/GPT-2 pre-tokenizer` | The model's pre-tokenizer is not implemented (e.g. SmolLM2). It still runs, but prompt token IDs may differ from the reference. See [supported-models.md](../reference/supported-models.md#known-issues). |
Full error list: [HTTP API](../reference/http-api.md#error-responses).

133
docs/guides/pth-convert.md Normal file
View File

@ -0,0 +1,133 @@
# PyTorch .pth Converter
Convert PyTorch checkpoint files (`.pth`) to safetensors format in pure C. No Python, no pip, no dependencies.
## Why?
PyTorch's `.pth` format uses Python's `pickle` module, which is fundamentally insecure — `pickle.load()` can execute arbitrary code. Every existing conversion tool requires Python + PyTorch + transformers (2-10 GB of dependencies).
Pantheon reads `.pth` files via a sandboxed C pickle interpreter that extracts tensor data without executing any embedded Python. It then writes safetensors, a secure mmap-friendly format.
## Build
```bash
cd Pantheon
make USE_BLAS=1 pantheon-convert
```
Or from Foundry (standalone, no Pantheon needed):
```bash
cd foundry
make pth-convert
```
## Usage
### Convert
```bash
# Convert .pth to safetensors
./pantheon-convert model.pth -o model.safetensors
```
### Inspect
```bash
# Print tensor table (names, shapes, dtypes, sizes)
./pantheon-convert model.pth --inspect
```
Output:
```
=== PyTorch Tensor Inspection ===
File: RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth
Tensors: 399
# Name DType Shape Size
--- -------------------------------- ------ ------------------------- --------
0 emb.weight BF16 [65536, 768] 96.0 MB
1 blocks.0.att.x_r BF16 [768] 1.5 KB
2 blocks.0.att.x_w BF16 [768] 1.5 KB
...
```
## Example: RWKV-7 0.1B
```bash
# 1. Download the .pth model from HuggingFace
huggingface-cli download BlinkDL/rwkv-7-world \
RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth \
--local-dir models/base/
# 2. Convert to safetensors
./pantheon-convert \
models/base/RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth \
-o models/base/RWKV-7-0.1B.safetensors
# 3. Run inference
./pantheon --model models/base/RWKV-7-0.1B.safetensors \
--vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
--prompt "Hello" --max-tokens 32 --temperature 0
```
## How It Works
A `.pth` file is a ZIP archive containing:
```
archive/
data.pkl # Pickle: OrderedDict mapping tensor names -> storage refs
data/0 # Raw bytes for storage #0
data/1 # Raw bytes for storage #1
...
```
The converter:
1. **ZIP reader** — mmaps the file, finds End-of-Central-Directory, parses entries
2. **Pickle interpreter** — stack-based opcode VM that extracts tensor metadata (names, shapes, dtypes, storage references). Security-whitelisted: only allows `torch._utils._rebuild_tensor_v2`, `collections.OrderedDict`, and torch storage classes. Unknown `GLOBAL`/`REDUCE` targets abort immediately.
3. **Storage resolver** — maps storage IDs to ZIP data entries, computes data pointers (zero-copy into mmap)
4. **Safetensors writer** — writes header JSON + aligned tensor data via Foundry's `fs_writer_*` API
## Supported Dtypes
| PyTorch Storage Class | Safetensors Dtype | Bits |
|----------------------|-------------------|------|
| `BFloat16Storage` | BF16 | 16 |
| `HalfStorage` | F16 | 16 |
| `FloatStorage` | F32 | 32 |
## Security
The pickle interpreter is deliberately minimal:
- **No code execution** — only extracts tensor metadata and raw data
- **Whitelisted classes** — `GLOBAL` only allows known torch/collections classes
- **Whitelisted callables** — `REDUCE` only allows `_rebuild_tensor_v2` and `OrderedDict`
- **Immediate abort** — any unrecognized opcode or unauthorized class triggers abort with warning
This is **not** a general-purpose pickle reader. It handles the specific pickle structure that `torch.save()` produces for state dicts. Adversarial pickles with embedded code will be rejected.
## C API
The pth reader is a reusable Foundry module (`pt_` prefix):
```c
#include "pth/pth.h"
pt_file *f = pt_open("model.pth");
int n = pt_tensor_count(f);
for (int i = 0; i < n; i++) {
const pt_tensor *t = pt_tensor_at(f, i);
printf("%s: %s [", t->name, fs_dtype_name(t->dtype));
for (int d = 0; d < t->ndim; d++)
printf("%s%lld", d ? ", " : "", (long long)t->shape[d]);
printf("] %zu bytes\n", t->data_size);
// t->data points directly into mmap'd storage (zero-copy)
}
pt_close(f);
```

160
docs/guides/quickstart.md Normal file
View File

@ -0,0 +1,160 @@
# Quickstart
Build Pantheon and generate text in under a minute.
## Build
```bash
git clone --recursive https://git.teamide.dev/David/Pantheon.git
cd Pantheon
make USE_BLAS=1 pantheon
```
Foundry comes in as a git submodule at `foundry/`; in a clone made without
`--recursive`, run `git submodule update --init` first.
Build flags:
| Flag | What | When |
|------|------|------|
| `USE_BLAS=1` | Accelerate (macOS) / OpenBLAS (Linux) | Always |
| `USE_THREADS=1` | Thread pool for parallel attention | 1B+ models |
| `DEBUG=1` | `-O0 -g` debug build | Debugging |
## Download a Model
Any GGUF model from HuggingFace works. RWKV-X safetensors models are also supported. Start small:
```bash
# SmolLM2 135M (fast, good for testing)
huggingface-cli download bartowski/SmolLM2-135M-Instruct-GGUF \
SmolLM2-135M-Instruct-Q8_0.gguf --local-dir ../models/base/
# RWKV-X 0.2B (hybrid recurrent model, safetensors format)
huggingface-cli download howard-hou/RWKV-X \
RWKV-X-0.2B-64k-Base.safetensors --local-dir ../models/base/
# Phi-3 mini (3.8B, good quality for size)
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-gguf \
Phi-3-mini-4k-instruct-Q4_K_M.gguf --local-dir ../models/base/
# Mistral 7B v0.3 (larger, needs USE_THREADS=1)
huggingface-cli download bartowski/Mistral-7B-Instruct-v0.3-GGUF \
Mistral-7B-Instruct-v0.3-Q4_K_M.gguf --local-dir ../models/base/
```
See [reference/supported-models.md](../reference/supported-models.md) for the full list of verified models.
## Generate Text
```bash
# GGUF model
./pantheon --model ../models/base/SmolLM2-135M-Instruct-Q8_0.gguf \
--prompt "The meaning of life is" --max-tokens 64
# RWKV-X safetensors model (requires --vocab)
./pantheon --model ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
--vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
--prompt "Hello world" --max-tokens 32
# RWKV-X with INT8 quantization (~3x faster on ARM)
./pantheon --model ../models/base/RWKV-X-0.2B-64k-Base.safetensors \
--vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
--prompt "Hello world" --max-tokens 32 --quantize
```
## CLI Flags
| Flag | Default | Description |
|------|---------|-------------|
| `--model <path>` | (required) | Model file (GGUF or safetensors) |
| `--vocab <path>` | none | RWKV World tokenizer vocab file (required for safetensors) |
| `--prompt <text>` | `"Hello"` | Input prompt |
| `--max-tokens <n>` | 128 | Maximum tokens to generate |
| `--temperature <f>` | 0.8 | Sampling temperature (0 = greedy) |
| `--top-p <f>` | 0.9 | Nucleus sampling threshold |
| `--top-k <n>` | 40 | Top-k sampling |
| `--quantize` | off | INT8 quantize weights at load time (RWKV-X) |
| `--chat` | off | Interactive multi-turn chat mode |
| `--system <text>` | none | System prompt (chat mode only) |
| `--bench` | off | Report tok/s, load time, memory |
| `--help` | | Show usage |
| `--version` | | Print version |
## Benchmark
```bash
./pantheon --model ../models/base/SmolLM2-135M-Instruct-Q8_0.gguf \
--prompt "The meaning of life is" --max-tokens 64 --bench --temperature 0
```
```
=== Benchmark ===
load: 41.6 ms
decode: 64 tokens in 787.3 ms (81.3 tok/s)
memory: 552.7 MB RSS
=================
```
## C API
Pantheon exposes a minimal C API for embedding into applications:
```c
#include "pantheon.h"
int main(void) {
/* GGUF model */
pn_model *model = pn_load("model.gguf");
/* Or: RWKV-X safetensors model */
pn_load_opts opts = { .vocab_path = "rwkv_vocab.txt", .quantize = true };
pn_model *rwkvx = pn_load_ex("model.safetensors", &opts);
pn_state *state = pn_state_new(model);
pn_sampling_params params = pn_sampling_defaults();
params.temperature = 0.0f; /* greedy */
char *output = pn_generate(model, state, "Hello world", 64, &params);
printf("%s\n", output);
free(output);
pn_state_free(state);
pn_model_free(model);
return 0;
}
```
### Streaming
```c
void on_token(const char *token, void *user_data) {
printf("%s", token);
fflush(stdout);
}
pn_stream(model, state, "Hello", 128, &params, on_token, NULL);
```
### API Functions
| Function | Description |
|----------|-------------|
| `pn_load(path)` | Load GGUF model, returns `pn_model*` |
| `pn_load_ex(path, opts)` | Load any model (GGUF or safetensors) with options |
| `pn_model_free(model)` | Free model and weights |
| `pn_state_new(model)` | Create inference state (KV cache + arena, or RWKV-X state) |
| `pn_state_free(state)` | Free state |
| `pn_state_reset(state)` | Clear state and position (reuse state) |
| `pn_generate(model, state, prompt, max_tokens, params)` | Generate text, returns `char*` (caller frees) |
| `pn_stream(model, state, prompt, max_tokens, params, cb, ctx)` | Streaming generation via callback |
| `pn_stream_tokens(model, state, tokens, n, max, params, cb, ctx)` | Streaming from pre-tokenized input (for chat) |
| `pn_sampling_defaults()` | Default sampling params (temp=0.8, top_p=0.9, top_k=40) |
## Run Tests
```bash
make USE_BLAS=1 test
```
62 tests across 7 suites: build sanity (5), GGUF dequant (11), arena (4), quantized matvec (4), SIMD kernels (6), chat templates (20), RWKV-X (12).

193
docs/reference/cli.md Normal file
View File

@ -0,0 +1,193 @@
# Pantheon — CLI Reference
Entry point: `cli/main.c`. `pantheon serve` runs the OpenAI-compatible server
(`src/serve/serve_command.c`), `pantheon router` the multi-instance dispatch layer
(`src/serve/router_command.c`); anything else is the one-shot / interactive CLI.
---
## `pantheon` (one-shot and chat)
```
pantheon --model <path> [--prompt <text> [--raw] | --chat [--system <text>]]
[--max-tokens <n>] [--temperature <f>] [--top-p <f>] [--top-k <n>]
[--config <path>] [--vocab <path>] [--quantize] [--bench]
pantheon --version | --backend | --help
```
| Flag | Argument | Default | Effect |
|------|----------|---------|--------|
| `--model` | `<path>` | — | GGUF or safetensors model. Without it, prints the version and checks Foundry links. |
| `--prompt` | `<text>` | `"Hello"` | One-shot generation. The model's chat template is applied unless `--raw`. |
| `--raw` | *(flag)* | off | One-shot `--prompt` without the chat template. |
| `--chat` | *(flag)* | off | Interactive multi-turn chat on stdin; template detected from the GGUF. |
| `--system` | `<text>` | unset | System prompt (chat mode only). |
| `--max-tokens` | `<n>` | `128` | Tokens to generate. |
| `--temperature` | `<f>` | `0.8` | Sampling temperature. |
| `--top-p` | `<f>` | `0.9` | Nucleus sampling threshold. |
| `--top-k` | `<n>` | `40` | Top-k sampling. |
| `--config` | `<path>` | unset | HuggingFace `config.json` (safetensors models). |
| `--vocab` | `<path>` | unset | RWKV World tokenizer vocab file. |
| `--quantize` | *(flag)* | off | INT8-quantize weights at load time. |
| `--bench` | *(flag)* | off | Report tok/s and memory. |
| `--version` | *(flag)* | — | `Pantheon v<ver>` then `backend=<b> build=<commit>`. |
| `--backend` | *(flag)* | — | Prints only the compiled backend token (`cuda`, `tt`, `metal`, `cpu`). |
| `-h` / `--help` | *(flag)* | — | Usage. |
---
## `pantheon serve`
```
pantheon serve --model <path> [--name <id>] [--dies <a[,b,...]>]
[--ctx-cap <n>] [--reasoning-budget <n>]
[--max-sessions <n>] [--max-concurrent <n>]
[--model <path> ...]
[--host <addr>] [--port <n>]
[--config <path>] [--vocab <path>]
[--quantize]
[--served-name <id>]
```
`--model` is repeatable. The per-model flags `--name` and `--dies` bind to
the **most recently seen** `--model`. There is no positional association;
flags are matched by order.
### Global flags
| Flag | Argument | Default | Effect |
|------|----------|---------|--------|
| `--model` | `<path>` (GGUF) | — *required, at least one* | Load and serve the model at this path. Repeatable; each occurrence starts a new model slot. Max `MR_MAX_MODELS=16`. |
| `--host` | `<addr>` | `127.0.0.1` | TCP bind address. Must be a dotted-IPv4 literal (`inet_pton`). |
| `--port` | `<n>` | `8080` | TCP port. Parsed with `atoi`. |
| `--config` | `<path>` | unset | HuggingFace `config.json` for safetensors models. |
| `--vocab` | `<path>` | unset | RWKV World tokenizer vocab file. |
| `--quantize` | *(flag)* | off | INT8-quantize weights at load time. |
| `--served-name` | `<id>` | unset | Back-compat alias: sets the **first** model's served name when it has no `--name`. Silently ignored if `--name` already set for model 0. |
| `-h` / `--help` | *(flag)* | — | Print usage and exit 0. |
### Per-model flags
These flags bind to the model specified by the **preceding** `--model`. If
used before any `--model` the server exits with an error.
| Flag | Argument | Default | Effect |
|------|----------|---------|--------|
| `--name` | `<id>` | basename of path | Served model id (the `model` field clients send in requests). Used for routing in multi-model mode. |
| `--dies` | `<a[,b,...]>` | none (auto-place) | Pin to accelerator dies (CUDA or Tenstorrent builds). CUDA ordinals are relative to `CUDA_VISIBLE_DEVICES`, not physical device numbers. Max `PN_MODEL_MAX_DIES=8`. N dies split the layers N ways (VRAM headroom, not speed). A `--dies` naming dies this binary has no backend for refuses to start; omit it to serve on the CPU. |
| `--ctx-cap` | `<n>` | `0` = engine default | Per-session context cap in tokens (the KV window). A longer prompt is refused with `context_over_budget`; nothing is silently dropped. Max 2^20. |
| `--reasoning-budget` | `<n>` | `0` = unbounded | After `<n>` tokens inside `<think>` without closing it, force `</think>` so the model answers. |
| `--max-sessions` | `<n>` | `0` = default 8 (`BS_MAX_USERS`) | Cap on live prefix-cache sessions; a new session over the cap is refused 429/503. |
| `--max-concurrent` | `<n>` | `0` = the resolved `--max-sessions` | Cap on simultaneous decode slots; surplus requests queue, then get 429/503. |
### Multi-model semantics
```
pantheon serve \
--model /models/big.gguf --name big --dies 0,1 \
--model /models/small.gguf --name small --dies 2
```
All models share the **single `--port`** (`serve_command.c:333`, one `hs_listen` call).
Each `--model` occupies an independent registry slot (`model_registry.c:117–122`). Per
slot, on load:
- Always: `pn_model` (weights) + `gen_lock` (non-batched serializer).
- Non-batched path (`QWEN35_BATCH_SLOTS` unset): also gets a `pn_state`; `gen_lock`
serializes single-user access — second concurrent request to the same model → 429.
- Batched path (`QWEN35_BATCH_SLOTS=1`): gets a `pn_batch_sched` instead
(`model_registry.c:272–282`, `bs_create`); the `gen_lock`+single-`pn_state`
serializer is bypassed. Up to `BS_MAX_USERS=8` concurrent users share one model.
Requests route by the `model` field in the JSON body (`openai_chat.c:616–629`,
`route_model` → `mr_acquire`). With exactly one loaded model, the field is ignored
(back-compat).
GPU uploads are serialized (one at a time) via `reg->upload_lock` to prevent stacking
the ~2.4 GB host transient.
`--served-name` affects only model 0 when `--name` was not also given for it.
#### Replicas (same name, different dies)
Repeating the **same `--name`** on different `--dies` creates a load-balanced replica
pool (`model_registry.c:117–119`):
```
pantheon serve \
--model /models/worker.gguf --name worker --dies 2 \
--model /models/worker.gguf --name worker --dies 3
```
`mr_acquire` (`model_registry.c:322–355`) picks the **least-busy** same-name entry
(fewest in-flight `refs`); each replica has its own die and `gen_lock`, so concurrent
requests run truly in parallel across those dies.
Session affinity across replicas within a single serve process is **not yet
implemented** — requests always go to the least-busy replica. See the
"Multi-Model Smart Routing" section in `docs/README.md`.
### Die index semantics
`--dies 0,1` means CUDA ordinals 0 and 1 as seen by the process — after
`CUDA_VISIBLE_DEVICES` filtering. If `CUDA_VISIBLE_DEVICES=4,5,6,7` is set,
`--dies 0,1` maps to physical cards 4 and 5. **The indices are NOT physical
device numbers.**
### Startup sequence
1. `pn_memwatch_start()` — host RAM poll thread starts.
2. Signal handlers installed (SIGINT/SIGTERM → clean CUDA teardown; crash
signals → `_exit()` so the kernel closes the nvidia fd and reclaims VRAM).
3. CUDA init (`fc_init()`, `pn_dm_init_cuda()`).
4. Model registry init.
5. For each `--model`: `mr_load()` (blocks; uploads weights to GPU).
6. TCP listen socket bound.
7. Accept loop: each connection spawns a detached thread.
### Runtime behavior
- Cheap routes (GET `/v1/models`, OPTIONS) are never queued behind a
generation.
- Per-model generation is serialized by `gen_lock`. A second request to the
same model while one is generating → immediate 429.
- With `QWEN35_BATCH_SLOTS=1`, the scheduler (`batch_sched`) replaces the
gen\_lock; up to `BS_MAX_USERS=8` concurrent users share one model.
- `QWEN35_GPU_FULL_LOGITS` being set causes immediate exit before any model
loads.
---
## `pantheon router`
```
pantheon router [--host <addr>] [--port <n>]
```
| Flag | Default | Effect |
|------|---------|--------|
| `--host` | `127.0.0.1` | Bind address. |
| `--port` | `8100` | Port. |
A host-side dispatch layer over several `pantheon serve` processes: no model loads, no
GPU. Serve instances register themselves when `PANTHEON_REGISTRY_URL` points at the router,
then heartbeat; instances whose heartbeat lapses are evicted. Requests go to the
least-busy live instance of the requested model, and a session (`X-Session-Id`) sticks to
the instance that served it so its prefix cache is reused. No authentication: bind it to
a trusted interface.
| Route | Purpose |
|-------|---------|
| `POST /registry/register` | A serve announces a model instance |
| `POST /registry/heartbeat` | Refresh liveness and load |
| `DELETE /registry/deregister` | Graceful removal |
| `GET /registry/fleet` | Model-level fleet view |
| `GET /v1/models` | Live models, deduplicated by model id |
| `POST /v1/chat/completions`, `POST /v1/completions` | Least-busy dispatch, SSE pass-through |
---
## `pantheon-convert` (cli/convert.c)
A separate binary (`make pantheon-convert`), not a subcommand: converts PyTorch `.pth`
checkpoints to safetensors. See [`../guides/pth-convert.md`](../guides/pth-convert.md).

158
docs/reference/env-flags.md Normal file
View File

@ -0,0 +1,158 @@
# Pantheon — Environment Flag Reference
Every `getenv()` call in `src/` and `foundry/` is listed here. Flags are
grouped by prefix. All flags are **process-constant** unless noted: they are
read once into a static cache on first call and never re-read.
---
## QWEN35\_\* — Qwen3.5-MoE model flags
### Core KV / context window
| Flag | Values | Default | Parse | Effect | Source |
|------|--------|---------|-------|--------|--------|
| `QWEN35_KV_Q4` | `0` or anything | ON | `env[0]=='0'` disables | KV-cache format: Q4\_0 (~4–7× smaller) vs fp32. Default changed to ON (B1). **Only `=0` disables.** | `qwen35moe_forward.c:51`, `engine.c:430` |
| `QWEN35_MAX_SEQ` | integer | `4096` | `atoi`, clamped `[16, 262144]` | KV-window size per attention layer. Memory scales linearly. Ceiling `PN_QWEN35MOE_MAX_SEQ_CAP=262144`. | `qwen35moe_forward.c:81`, `qwen35moe_forward_cuda.c:621`, `engine.c:454` |
| `QWEN35_KV_Q4_WIN` | integer 0–256 | `128` | `atoi`, clamped `[0, 256]` | fp16 recent-window exemption size inside the Q4 KV cache. `0` disables the window (pure Q4). | `qwen35moe_kv_q4.h:54` |
| `QWEN35_KV_Q4_SINK` | integer 0–8 | `4` | `atoi`, clamped `[0, 8]` | fp16 attention-sink prefix kept exempt from Q4 quantization. | `qwen35moe_kv_q4.h:67` |
### GPU placement / split
| Flag | Values | Default | Parse | Effect | Source |
|------|--------|---------|-------|--------|--------|
| `QWEN35_GPU_SPLIT_LAYER` | integer | `40` (`GPU_DIE_SPLIT_LAYER`) | `atoi`, must be `>0`; ignored when explicit 1-die placement | Layer boundary between die A and die B. Layers `[0, split)` → die A; `[split, n_layer)` → die B. Ignored when explicit single-die placement is authoritative. | `qwen35moe_forward_cuda.c:702` |
| `QWEN35_GPU_DIE_A` | integer (CUDA ordinal) | `0` | `atoi` | Die A CUDA device ordinal. Superseded by explicit `--dies` placement (warning printed). | `qwen35moe_forward_cuda.c:732` |
| `QWEN35_GPU_DIE_B` | integer (CUDA ordinal) | `1` | `atoi` | Die B CUDA device ordinal. Same supersession rule as `QWEN35_GPU_DIE_A`. | `qwen35moe_forward_cuda.c:733` |
| `QWEN35_GPU_MAX_LAYERS` | integer | unset (= all layers) | `atoi`, must be `>0` | Single-die only: upload/validate only the first N global layers. Ignored in 2-die mode. Also disables async scheduling (`QWEN35_ASYNC_SCHED`). | `qwen35moe_forward_cuda.c:1407`, `7164` |
| `QWEN35_GPU_BARSWAP` | any | unset | presence check (`== NULL` → small-BAR) | Legacy barswap (single MMIO slot) vs small-BAR mode (all dies live). Default is small-BAR (no barswap). Set to any non-empty value to restore the old barswap 2-die path. | `qwen35moe_forward_cuda.c:1416` |
| `QWEN35_DROP_WEIGHTS` | `0` or unset | unset (= drop) | `env[0]=='0'` opts out | After GPU upload, call `madvise(MADV_DONTNEED)` on the host weight mmap to free its page cache (~15 GB for a 35B MoE). **On by default** (qwen35 and llama CUDA paths); `=0` keeps the pages for host-fallback debugging. Host-fallback matvec refaults clean pages from disk. | `qwen35moe_forward_cuda.c:1935`, `llama_forward_cuda.c:211` |
### Batched decode (C3 scheduler)
| Flag | Values | Default | Parse | Effect | Source |
|------|--------|---------|-------|--------|--------|
| `QWEN35_BATCH_SLOTS` | `1` or anything | OFF | **`env[0]=='1'`** — only the character `'1'` enables | **Enable** multi-user batched decode. Any other value (including `=8`, `=4`, `=on`) silently keeps batching OFF. | `batch_sched.c:51` |
| `QWEN35_BS_STATS` | `1` or anything | OFF | `env[0]=='1'` | Print per-step driver timing to stderr (M6 instrumentation). | `batch_sched.c:110` |
| `QWEN35_BS_DRIVER_SAMPLE` | `0` or anything | ON | `env[0]=='0'` disables | In-driver sampling (default ON). Set `=0` to restore legacy owner-side lockstep (each request thread samples its own token). | `batch_sched.c:651` |
| `BS_QUEUE_DEPTH` | integer `>0` | `32` | `atoi` | Per-model overflow queue depth (D1). Requests queue here when all slots are busy before getting 429. | `batch_sched.c:725` |
| `QWEN35_BS_PREFILL_BATCH` | `0` or anything | ON | `env[0]=='0'` disables | Enable batched prefill (one GPU pass per chunk, ~2× vs token-by-token). `=0` restores the sequential token loop. | `serve_generation.c:471` |
| `QWEN35_BS_PREFILL_CHUNK` | integer `>=0` | `128` | `atoi`; `<0` → 0 | Prefill chunk size in tokens. `0` = entire prompt in one pass (unbounded scratch). | `serve_generation.c:482` |
### Logits / sampling
| Flag | Values | Default | Parse | Effect | Source |
|------|--------|---------|-------|--------|--------|
| `QWEN35_GPU_FULL_LOGITS` | any | unset | presence check | Force full n\_vocab (248K) D2H logits copy per token. **Serve refuses to start if this is set** — it cripples batched decode throughput (K×992 KB/step over host link). Debug/eval only. | `serve_command.c:279`, `qwen35moe_forward_cuda.c:2747` |
| `QWEN35_GPU_TOPK` | integer 0–256 | `40` | `atoi`, clamped `[0, FC_TOPK_MAX=256]` | On-die top-K reduction before D2H. `0` or forced by `QWEN35_GPU_FULL_LOGITS` → full copy. | `qwen35moe_forward_cuda.c:2751` |
### CUDA-graph / fusion
| Flag | Values | Default | Parse | Effect | Source |
|------|--------|---------|-------|--------|--------|
| `QWEN35_GRAPH` | `0` or anything | ON | `env[0]=='0'` disables | Master gate for CUDA-graph replay of SSM layers. Disabling also disables `QWEN35_GRAPH_FULL` and `QWEN35_GRAPH_SLOTS`. | `qwen35moe_forward_cuda.c:1899` |
| `QWEN35_GRAPH_FULL` | `0` or anything | ON | `env[0]=='0'` disables | Full-step segment graphs (C1): capture every layer of a die segment (SSM+attention interleaved) into one graph, replay with one `cuGraphLaunch`. Requires `QWEN35_GRAPH`. | `qwen35moe_forward_cuda.c:1916` |
| `QWEN35_GRAPH_SLOTS` | `0` or anything | ON | `env[0]=='0'` disables | Per-SSM-layer graphs on the batched-decode (slots) path (L9). Sub-gate of `QWEN35_GRAPH`. Attention layers stay direct. | `qwen35moe_forward_cuda.c:1931` |
| `QWEN35_GRAPH_CAPTURE_FAIL_INJECT` | int N | `0` (off) | `atoi(env)` | **Test only.** Forces the first N capture attempts of EVERY graph slot to be treated as failures, so the bounded-retry path can be exercised on real hardware. `N < 5` exercises recovery, `N >= 5` exercises the permanent give-up. Never set on a serving process. | `gpu_graph_capture_tracker.c:pn_graph_capture_inject_failures` |
| `QWEN35_FUSE` | hex bitmask | `0x3F` (all on) | `strtol(env, NULL, 0)` (hex/decimal) | Fusion-site bitmask: bit0=ssm proj4, bit1=ssm out+res, bit2=attn qkv, bit3=attn norm+rope, bit4=moe pre, bit5=moe post. Set specific bits `=0` to bisect regressions. | `qwen35moe_forward_cuda.c:1887` |
### Flash attention
| Flag | Values | Default | Parse | Effect | Source |
|------|--------|---------|-------|--------|--------|
| `QWEN35_FLASH` | `0` or anything | ON | `env[0]=='0'` disables | Flash attention for batched prefill. NOT bit-exact vs host softmax (~1e-6 relative drift). `=0` restores host-softmax path. | `qwen35moe_forward_cuda.c:3257` |
| `QWEN35_FLASH_DECODE` | `0` or anything | ON | `env[0]=='0'` disables | Flash attention for single-token decode. `=0` restores host-softmax decode path. | `qwen35moe_forward_cuda.c:3406` |
| `QWEN35_FLASH_DECODE_KPS` | integer `>=16` | `PN_QWEN35MOE_DEC_KEYS_PER_SPLIT` | `atoi`, floor 16 | Keys per decode split (tune for multi-split combine path in tests). | `qwen35moe_forward_cuda.c:3419` |
| `QWEN35_FLASH_SLOTS` | `0` or anything | ON | `env[0]=='0'` disables | Flash attention on the batched-decode (slots) path (L7). `=0` → host-softmax slots path. | `qwen35moe_forward_cuda.c:4917` |
| `QWEN35_ATTN_WINDOW` | integer `>0` | `0` (full) | `strtol` | StreamingLLM sliding-window attention recent-window size. `0`/unset = full attention. Applies to all 10 global attention layers. | `qwen35moe_forward_cuda.c:3272` |
| `QWEN35_ATTN_SINK` | integer `>=0` | `4` | `strtol` | StreamingLLM attention-sink prefix size (used with `QWEN35_ATTN_WINDOW`). | `qwen35moe_forward_cuda.c:3283` |
### 2-die pipeline / threading
| Flag | Values | Default | Parse | Effect | Source |
|------|--------|---------|-------|--------|--------|
| `QWEN35_ASYNC_SCHED` | `0` or anything | ON | `env[0]=='0'` disables | Non-blocking async dispatch for single-token decode (C2). `=0` restores fully synchronous path. Automatically disabled when `QWEN35_GPU_MAX_LAYERS` is set or LM head needs host fallback. | `qwen35moe_forward_cuda.c:7074` |
| `QWEN35_ASYNC_KVMIRROR` | any non-`0` | OFF | `env[0] && env[0]!='0'` | Defer host-KV-mirror stores in batched decode (L8). Experimental. | `qwen35moe_forward_cuda.c:5017` |
| `QWEN35_TWO_THREAD` | any non-`0` | OFF | `env[0] && env[0]!='0'` | Two-die two-thread decode (small-BAR, 2 dies, nslots≥1). | `qwen35moe_forward_cuda.c:5965` |
| `QWEN35_DIE_PIPELINE` | any non-`0` | OFF | `env[0] && env[0]!='0'` | Software-pipeline two batch groups across the die pair so both dies compute simultaneously (L5). Requires 2 dies + grouped MoE + nslots≥2. | `qwen35moe_forward_cuda.c:5979` |
| `QWEN35_VOCAB_SPLIT` | any non-`0` | OFF | `env[0] && env[0]!='0'` | Split LM head across 2 dies (T3). Only useful in 2-die non-barswap mode with peer access. | `qwen35moe_forward_cuda.c:5991` |
### SSM
| Flag | Values | Default | Parse | Effect | Source |
|------|--------|---------|-------|--------|--------|
| `QWEN35_SSM_2PASS` | any non-`0` | OFF | `env[0] && env[0]!='0'` | Route SSM recurrence to fused-reduction (2-pass) kernel: one LDS sweep, 3×S shared, slightly lossy by reassociation. Default OFF (bit-exact baseline). | `foundry/cuda/fc_cuda.c:3375` |
### Debug / profiling
| Flag | Values | Default | Parse | Effect | Source |
|------|--------|---------|-------|--------|--------|
| `QWEN35_DEBUG` | any | unset | presence check | Enable per-layer activation dumps to stderr. Also forces full-logits D2H (`gpu_topk_k()` returns 0). | `qwen35moe_forward.c:106`, `qwen35moe_forward_cuda.c:991` |
| `QWEN35_PROF` | any | unset | presence check | Per-token wall-clock profiler: accumulates matvec time by quant type and forward phase, printed per token. Adds `fc_sync` around timed ops. | `qwen35moe_forward_cuda.c:794` |
| `QWEN35_SPROF` | any | unset | presence check | Per-step batch-decode profiler: phase breakdown printed per step, cumulative summary every 10 steps. | `qwen35moe_forward_cuda.c:930` |
---
## PN\_\* — Pantheon server / memory flags
| Flag | Values | Default | Parse | Effect | Source |
|------|--------|---------|-------|--------|--------|
| `PN_THREADS` | integer | `0` (auto) | `atoi` | CPU thread-pool size for `USE_THREADS` builds. `0` = auto-detect. | `engine.c:772` |
| `PN_ASYNC_POLL_US` | integer 50–20000 | `500` | `atol`, clamped `[50, 20000]` | Async step poll interval in microseconds (exponential backoff up to this cap). | `serve_generation.c:422` |
| `PN_HOST_RESERVE_BYTES` | integer (bytes) | `max(1 GiB, 15% × MemTotal)` | `strtoull` | Override host RAM reserve held off-limits for admission. Values ≤0 or unparseable fall back to the formula. | `pn_memwatch.c:62` |
| `PN_DIE_RESERVE_BYTES` | integer (bytes) | `1073741824` (1 GiB) | `strtoull` | Override per-die VRAM reserve subtracted from each die's reported total before scheduling. | `pn_memwatch.c:72` |
| `PN_PSI_FULL_REJECT` | float (percent) | `15.0` | `strtod`; values ≤0 ignored | PSI-full avg10 threshold (% wall time) at/above which admission rejects new requests as CRITICAL. | `pn_memwatch.c:78` |
| `PN_LADDER_DISABLE` | any non-`0` | unset (enabled) | `env[0] && env[0]!='0'` | Disable the reactive degradation ladder entirely. The plan mirrors the config at all pressure levels. | `pn_ladder.c:21` |
| `PN_LOAD_SPIKE_BYTES` | integer (bytes) | `2516582400` (~2.4 GB) | `strtoull`; ≤0 falls back to default | Host anon RSS budget for GPU-upload transient (measured on a Tesla K80 host: ~2.4 GB for ~55 s). Admission checks this before starting a load. | `pn_admission.c:280` |
| `PN_VLM_DEBUG` | any | unset | presence check | Print top-3 logits + sampled token for the first 6 steps. Diagnoses empty VLM output. | `engine.c:1686` |
| `PN_GPU_PREFILL` | any | unset | presence check | (llama/Metal) Upload Q4\_K decoder weights + run prefill GEMMs on the GPU. Off by default (measured net loss for bulk OCR vs CPU+GPU overlap). | `llama.c:2068` |
| `PN_BATCH_GPU` | any | unset | presence check | (llama/Metal, batch≤32) Upload Q4\_K decoder weights for GPU-based batched GEMM decode. Experimental. | `llama.c:2398` |
---
## FC\_\* — foundry/cuda MoE kernel tuning
| Flag | Values | Default | Parse | Effect | Source |
|------|--------|---------|-------|--------|--------|
| `FC_MOE_DOWN_WPP` | `1`, `2`, `4`, or `8` | `2` (`FC_MOE_DOWN_WPP_DEFAULT`) | `atoi`; only exact power-of-2 values accepted | Warps-per-pair for the cooperative MoE down-projection CUDA kernel. Values outside {1,2,4,8} are silently ignored. | `foundry/cuda/fc_cuda.c:2645` |
| `FC_MOE_DOWN_THREADS` | `64`, `128`, `256`, or `512` | `128` (`FC_MOE_DOWN_THREADS_DEFAULT`) | `atoi`; must be multiple of 32×wpp | Thread count for the MoE down kernel. Only exact values {64,128,256,512} that are a multiple of `32×wpp` accepted; others silently ignored. | `foundry/cuda/fc_cuda.c:2652` |
---
## Distributed training (foundry/distributed)
| Flag | Values | Default | Parse | Effect | Source |
|------|--------|---------|-------|--------|--------|
| `WORLD_SIZE` | integer | `1` | `atoi` | Distributed training world size. Used by `fd_init_from_env()` in the foundry trainer — not the inference server. | `foundry/distributed/distributed.c:31` |
| `RANK` | integer | `0` | `atoi` | This process's rank. Same scope as `WORLD_SIZE`. | `foundry/distributed/distributed.c:32` |
---
## Key parsing gotchas
1. **`QWEN35_BATCH_SLOTS` is an enable flag, not a slot count.** Only `env[0]=='1'`
enables batching. `QWEN35_BATCH_SLOTS=8` leaves batching OFF.
(`batch_sched.c:51`)
2. **`QWEN35_KV_Q4` default is now ON.** Only `=0` disables. The old opt-in
value `=1` still works but is redundant. Any other value (e.g. `=true`)
also keeps it ON. (`qwen35moe_forward.c:51`)
3. **`QWEN35_GPU_FULL_LOGITS` is blocked at serve startup.** The server refuses
to start with this set — it is not silently ignored. (`serve_command.c:279`)
4. **`QWEN35_GPU_SPLIT_LAYER` is ignored when 1-die placement is active.**
`gpu_split_layer()` returns `n_layer` (single-die path) when `n_placed==1`
regardless of the env. (`qwen35moe_forward_cuda.c:703`)
5. **`QWEN35_GPU_DIE_A/B` are superseded by `--dies` / `pn_load_on` placement.**
A warning is printed but they are not honoured. (`qwen35moe_forward_cuda.c:724`)
6. **`FC_MOE_DOWN_WPP` and `FC_MOE_DOWN_THREADS` silently reject invalid values**
and keep the compiled default. No warning is emitted. (`foundry/cuda/fc_cuda.c:2645–2659`)
7. **`QWEN35_FUSE` parses via `strtol(env, NULL, 0)`** — it accepts hex (`0x3f`)
and decimal. (`qwen35moe_forward_cuda.c:1888`)
8. **`PN_LOAD_SPIKE_BYTES=0` falls back to the 2.4 GB default**, not zero — a
zero budget would silently disarm the host-RAM gate. (`pn_admission.c:284`)

821
docs/reference/http-api.md Normal file
View File

@ -0,0 +1,821 @@
# Pantheon — HTTP API Reference
The server speaks HTTP/1.1 over TCP. Every response carries
`Access-Control-Allow-Origin: *` and `Connection: close`. No authentication.
All bodies are JSON unless noted (SSE for streaming).
---
## Routes
| Method | Path | Handler | Audience |
|--------|------|---------|----------|
| `GET` | `/` | health | client / operator |
| `OPTIONS` | `*` | CORS preflight | client |
| `GET` | `/v1/models` | list models | client |
| `POST` | `/v1/chat/completions` | chat completion | client |
| `GET` | `/admin/models` | list live model instances | operator |
| `POST` | `/admin/models/load` | load a model | operator |
| `POST` | `/admin/models/unload` | unload a model | operator |
| `GET` | `/admin/dies` | die status | operator |
| `POST` | `/admin/session/export` | export a session's state as a blob | operator |
| `POST` | `/session/restore` | restore a session from a blob | client |
| `POST` | `/session/release` | free a session's slot | client |
| `GET` | `/session/estimate` | warm-restore vs re-prefill time estimate | client |
| `GET` | `/session/watch` | hold a session alive while connected | client |
Any other path → `404 {"error":{"message":"unknown route","type":"not_found","code":null}}`.
---
## GET /
Health check. Always 200.
**Response:**
```json
{"service":"pantheon","status":"ok","backend":"cuda",
"build":"5d818bfd1c9a","accelerator":true,"dies_visible":8}
```
| Field | Type | Meaning |
|-------|------|---------|
| `backend` | string | The accelerator backend this binary was COMPILED with: `cuda`, `tt`, `metal`, `cpu`. Compile-time constant, not a probe. |
| `build` | string | Commit this binary was built from (`-dirty` when the tree was not clean), or `unknown`. |
| `accelerator` | bool | False means every forward pass in this process runs on the host CPU, whatever the flags say. |
| `dies_visible` | int | Dies the die manager holds (`pn_dm_count()`). `0` on a CUDA build means no driver / no card / empty `CUDA_VISIBLE_DEVICES`. |
The cheapest route on the server is also the one that answers "what am I
running?" — see [`GET /v1/models`](#backend--build--what-this-binary-actually-is)
for why that question needed an endpoint at all.
---
## OPTIONS \*
CORS preflight. Always 200 with `{}` body.
---
## GET /v1/models
**Serve's own endpoint** (`openai_chat.c:oai_handle_models`, `serve_command.c:dispatch`):
same-name replica slots (multi-die LB copies) are deduplicated — clients see one
logical id per served name. The response does **not** include `live` or `free_slots`.
**Response — 200:**
```json
{
"object": "list",
"backend": "cuda",
"build": "5d818bfd1c9a",
"data": [
{
"id": "my-model",
"object": "model",
"created": 0,
"owned_by": "pantheon",
"max_sessions": 4,
"max_concurrent": 4,
"context_window": 32000,
"tool_choice_required_with_thinking": true,
"graphs": {"live": 30, "total": 40, "failed": 0,
"ineligible": 10, "pending": 0},
"attention": {"ondie": 10, "host": 0, "total": 10}
}
]
}
```
| Field | Type | Meaning |
|-------|------|---------|
| `backend` | string | **Top-level.** The accelerator backend this binary was COMPILED with: `cuda`, `tt`, `metal` or `cpu`. Compile-time constant, not a probe. |
| `build` | string | **Top-level.** The commit this binary was built from (`-dirty` suffixed when the tree was not clean), or `unknown`. |
| `id` | string | The routing key for `model` on a completion |
| `max_sessions` | int | RESIDENCY: how many conversations may hold KV at once |
| `max_concurrent` | int | CONCURRENCY: how many requests this model ACTUALLY decodes at once — the **served** value (`bs_get_load` capacity), not the configured cap. See below. |
| `context_window` | int | **Per-session context window in tokens** — see below. Omitted when unknown. |
| `tool_choice_required_with_thinking` | bool | `true` when `tool_choice: "required"` may be sent with thinking on: the think block is unconstrained and the turn ends in a parsed tool call (the think-gated grammar models ORNITH_JSON and QWEN_XML; `false` for MiniCPM's attribute dialect, which no grammar models, and for every model when `PANTHEON_TOOL_GRAMMAR=0`). |
| `graphs` | object | **CUDA-graph residency** — see below. Omitted when the model has no graph slots. |
| `attention` | object | **Attention-path residency** — see below. Omitted until the forward has reached an attention layer. |
### `backend` / `build` — what this binary actually is
Top-level on the response, not per-model, because it is a property of the
process: the accelerator backend it was **compiled** with (`cuda`, `tt`, `metal`,
`cpu`) and the commit it came from.
Compile-time constant, deliberately — not a runtime probe. A serve whose binary
has no CUDA compiled in cannot use a GPU whatever the driver reports, and the
completion text is identical either way, so nothing downstream can detect it
from the output.
The same values are on `GET /` (plus `dies_visible`), and out of band from
`pantheon --backend`, which prints one lowercase token for a deploy script to
assert against.
### `max_concurrent` — the SERVED number, not the configured cap
`max_concurrent` on `/v1/models` is what the model actually decodes at once
(`bs_get_load`'s capacity), which is **not** always what `--max-concurrent` asked
for. It reads `1` whenever there is no batch scheduler — no CUDA compiled in,
`QWEN35_BATCH_SLOTS` off, `bs_create` failed — and it drops below the cap when
the memory-pressure ladder clamps the slot count.
Same argument as `context_window` below: a client sizes its in-flight requests
from this number, so advertising a cap the serve cannot honour (e.g. `2` with no
scheduler, which turns every second concurrent chat into a `429`) is worse than
advertising nothing.
Operators still get the configured value: `/admin/models` carries **both**, as
`max_concurrent` (served) and `max_concurrent_configured` (asked for). Seeing
them differ is the signal.
### `context_window` — the advertised window
`context_window` is the window a client may actually plan a prompt against, in
tokens. It is the **effective** window (`pn_effective_ctx_window`): the operator's
`--ctx-cap` **after** the reactive memory ladder's runtime clamp — *not* the
configured cap, and not the model's theoretical maximum.
That distinction is the whole point of the field. Publishing the configured cap
while admission enforces a smaller one hands the client a prefill the serve then
rejects — on a K80 that rejection lands after minutes of cold-prefill TTFT, and it
lands mid-task.
Two properties clients depend on:
- **It is the same number a completion reports as `usage.kv_cap`.** The model list
is the client's INITIAL plan; `kv_cap` is the LIVE truth on every turn. A client
that reads both needs no polling and no extra endpoint to notice a mid-session
clamp.
- **It is additive and omitted when unknown.** A plain OpenAI-compatible client
ignores the extra key; a client that reads it never sees a `0` it has to
special-case. An absent field means *undetected*; a client should fall back to
a conservative window rather than guess.
Field name: Pantheon emits `context_window`. Clients that detect the window from
other servers' spellings (`context_length`, `max_model_len`, `n_ctx`, ...) need
to accept this one.
### `graphs` — CUDA-graph residency
Each layer of the hybrid stack can hold a captured CUDA graph that collapses its
~13-30 kernel launches into one replay. Across 40 layers those graphs are worth
about 1.8 ms/step. A capture that **fails** falls back to the direct kernel body
and produces **correct output** — so losing them costs only speed, silently.
This object is the only surface that reports it.
| Key | Meaning |
|-----|---------|
| `live` | Slots holding a captured graph right now |
| `total` | Slots the forward has actually reached (see below) |
| `failed` | Attempted and did not stick — **the only value that is a fault** |
| `ineligible` | Gated out; this model never captures. **Not a fault.** |
| `pending` | Eligible, warming up, no attempt yet |
| `gave_up` | Subset of `failed` whose bounded retry budget is spent (omitted when 0) |
**`ineligible` is normal.** The captured path requires the grouped MoE kernel,
which requires Q3_K expert tensors. A Q5_K_M model reports `live 0 / total 40 /
ineligible 40 / failed 0` — correct behaviour, zero exposure, and also zero of
the +4%. Read `failed`, not `live/total`, to decide whether something is wrong.
**`total` counts only exercised slots.** A serve decoding batched never touches
its single-token graph slots, so those are not reported as missing graphs.
A failed capture is retried on a bounded backoff and only stops after the budget
is spent, so a transient `failed` on one read that is `live` on the next is the
system self-healing, not a flapping metric.
### `attention` — on-die vs host-softmax residency
The same argument as `graphs`, only much sharper. Each full-attention layer
normally runs the on-die flash kernels. When it cannot it falls back to host-side
causal GQA softmax on the serve host's CPU — the exact wall this engine was built
to escape (it was 87.5% of an 8k prefill; moving attention on-die took that
prefill from 26 minutes to 3.82). The fallback returns the **same tokens**, logs
nothing per step, and on the depth bench costs **13.02 -> 1.22 tok/s** (K=2,
depth 4096) — and it worsens with context, because host softmax is O(context)
per layer per token. Without
this object a user simply waits, and nothing anywhere says why.
| Key | Meaning |
|-----|---------|
| `ondie` | Attention layers running the on-die flash kernels right now. |
| `host` | Layers on the host-softmax fallback — **any non-zero value is a fault** |
| `total` | Layers the forward has actually reached (same rule as `graphs.total`) |
| `gave_up` | Layers whose KV-ring allocation spent its retry budget (omitted when 0) |
| `impossible` | Subset of `gave_up`: the rings cannot fit this die at all (omitted when 0) |
`ondie + host == total`. Unlike `graphs`, there is no benign state here: `host >
0` always means this model is serving degraded, and the `reasons` histogram on
`/admin/models` says why. The two common causes are a per-slot KV-ring allocation
that did not fit (lower `--ctx-cap` or `--max-sessions`) and a session that has
run past the KV window, which wraps the ring.
A ring allocation that fails is retried on a bounded backoff, so a `host` reading
that clears on the next poll is the system self-healing. `gave_up` is the number
that should page someone.
No error cases (returns an empty `data` array when nothing is loaded).
**Router's `/v1/models`** (`registry.c:reg_models_json`) is a different shape: it
aggregates across live serve instances and adds `live` (count of live instances serving
that id) and `free_slots` (summed across live replicas). Do not conflate the two.
The router also emits `context_window`, aggregated the only honest way: the
**minimum** across that model_id's live replicas (a dispatch may land on any of
them), and **omitted entirely** if any live replica has not reported one. Serve
instances report their window on both `/registry/register` and every
`/registry/heartbeat`, because the ladder can move it at runtime.
---
## POST /v1/chat/completions
OpenAI-compatible chat completion. Supports streaming (SSE) and non-streaming.
Includes reasoning (`<think>…</think>` split) and tool-call support.
### Request headers
| Header | Required | Effect |
|--------|----------|--------|
| `Content-Type: application/json` | no | Not checked; the body is parsed as JSON regardless. |
| `X-Session-Id: <sid>` | no | Prefix-cache session key. A body key (`session_id`, `prompt_cache_key`, `promptCacheKey`) takes precedence. opencode sends this header on every request. |
### Request body
```json
{
"model": "my-model",
"messages": [
{"role": "system", "content": "..."},
{"role": "user", "content": "hello"}
],
"temperature": 0.7,
"top_p": 0.95,
"top_k": 40,
"max_tokens": 1024,
"stop": ["<|im_end|>"],
"stream": false,
"tools": [...],
"tool_choice": "auto",
"session_id": "abc123"
}
```
| Field | Type | Default | Notes |
|-------|------|---------|-------|
| `model` | string | — | Routing key. With one loaded model, any value (or absent) routes to it. With multiple models, must match a served name exactly; unknown → 404. |
| `messages` | array | *required* | Non-empty array of `{role, content}`. Roles: `system`, `user`, `assistant`, `tool`; a missing role is treated as `user`. `content` must be a string — any other type (e.g. an array of content parts) is read as empty. |
| `messages[].reasoning_content` | string | unset | Assistant turns only. The reasoning this serve returned for that turn; replayed into the prompt so a session can resume from its prefix cache. |
| `messages[].tool_calls` | array | unset | Assistant turns only. Rendered back into the prompt in the model's tool dialect. |
| `temperature` | float | `0.7` | `<= 0` is greedy argmax. |
| `top_p` | float | `0.95` | Nucleus sampling. |
| `top_k` | int | `40` | Top-K sampling. |
| `max_tokens` | int | `1024` | Max completion tokens. `0` or negative → `512`. Clamped to 1048576. |
| `max_completion_tokens` | int | `1024` | Alias; `max_tokens` wins if both are present. |
| `stop` | string or array | unset | Stop strings. |
| `stream` | bool | `false` | `true` → SSE (`text/event-stream`). |
| `stream_options.include_usage` | bool | `false` | Streaming only: send a terminal `usage` chunk before `[DONE]`. |
| `seed` | number | unset | Makes sampling reproducible. With `n` > 1, choice *i* uses `seed + i`. |
| `presence_penalty` | float | `0` | Clamped to [-2, 2]. |
| `frequency_penalty` | float | `0` | Clamped to [-2, 2]. |
| `logit_bias` | object | unset | `{"<token id>": bias}`; bias clamped to [-100, 100]. Non-numeric keys or values are skipped. |
| `logprobs` | bool | `false` | Return per-token log-probabilities (`choices[].logprobs.content[]`, with `token`, `logprob`, `bytes`). |
| `top_logprobs` | int | `0` | Alternatives per token; clamped to [0, 20]. |
| `n` | int | `1` | Number of completions. Each is a full separate decode (only the first uses the session's prefix cache). Capped at 8; a larger `n` returns 8 and sets `n_status: "capped_to_8"`. |
| `response_format` | object | `{"type":"text"}` | `json_object` constrains the answer to valid JSON. `json_schema` gets the same JSON constraint but the schema's shape is **not** enforced. See `response_format_status` below. |
| `enable_thinking` | bool | unset | Think-style templates: `true` opens a think block, `false` prefills a closed one. Absent → the `PANTHEON_NO_THINK` env decides. |
| `reasoning_budget` | int | per-model | Force `</think>` after N reasoning tokens. Overrides the model's `--reasoning-budget`; `0` or negative disables it. `max_reasoning_tokens` is an alias (`reasoning_budget` wins). |
| `tools` | array | unset | OpenAI-format tool definitions. Rendered into a system turn in the model's tool dialect (below). Ignored when `tool_choice` is `"none"`. |
| `tool_choice` | string or object | `"auto"` | `"none"` disables tools. `"required"` forces a call: the grammar is armed by default (`PANTHEON_TOOL_GRAMMAR=0` opts out). With thinking on the think block is unconstrained and the call is compelled from `</think>` (models advertising `tool_choice_required_with_thinking`). `{"type":"function","function":{"name":"X"}}` is enforced only when `PANTHEON_TOOL_GRAMMAR=1`; otherwise it behaves like `"auto"`. |
| `session_id` | string | unset | Prefix-cache session key (overrides `X-Session-Id`). Enables delta prefill on later turns of the same session. |
| `prompt_cache_key` / `promptCacheKey` | string | unset | Aliases for `session_id` (OpenAI's name; opencode's camelCase body field). Order: `session_id`, `prompt_cache_key`, `promptCacheKey`, then the `X-Session-Id` header. |
### Tool calling
Tools are rendered into a system turn (merged with a leading `system` message)
in the dialect the routed model emits. The dialect is chosen from the model name
and architecture (`tool_calling.c:tc_dialect_for_model`), not from the GGUF chat
template:
| Dialect | Used for | Call syntax |
|---------|----------|-------------|
| Qwen3-Coder XML | default | `<tool_call><function=NAME><parameter=KEY>VALUE</parameter></function></tool_call>` |
| MiniCPM XML | Llama-arch models whose template adds BOS (MiniCPM5) | `<function name="NAME"><param name="KEY">VALUE</param>…` |
| JSON | served names starting `ornith` | fenced or bare `{"function"\|"name": NAME, "arguments": {…}}` |
The model's output is parsed into OpenAI `tool_calls`, with arguments typed from
each tool's JSON schema. `PANTHEON_TOOL_GRAMMAR` decides which requests are
grammar-constrained:
| Value | Constrained requests |
|-------|----------------------|
| unset (default) | `tool_choice: "required"` only — after reasoning, the output is one well-formed call to an offered tool, arguments typed and `required` keys present. Everything else decodes **unconstrained**. |
| `1` | every request with tools: `"auto"` masks once a call opener appears; `"required"` and a named function compel a call. |
| `0` | none — `"required"` is unconstrained too. |
On the Tenstorrent serve a `"required"` turn keeps speculative decoding: drafts
are checked against the grammar and the output equals non-speculative
constrained decoding. On a `"required"` turn (every dialect, single-user and
batched) an end token is refused before the call opens and, inside the call,
allowed only where the call can still be closed; the serve then appends the
closing text. Any other call the model started but did not finish is left
out of `tool_calls` and reported in `tool_call_error` (`code`, optional
`function`, `parameter`, `message`) on the choice.
`tools` and `response_format` cannot both constrain one decode: when the request
carries tools, tools win and `response_format_status` is `"ignored_tools_present"`.
### Non-streaming response — 200
```json
{
"id": "chatcmpl-a1b2c3d4",
"object": "chat.completion",
"created": 1720000000,
"model": "my-model",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "Hello!",
"reasoning_content": "...",
"tool_calls": [...]
},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 12,
"completion_tokens": 7,
"total_tokens": 19,
"prompt_tokens_details": {"cached_tokens": 0},
"kv_used": 12,
"kv_cap": 32000
}
}
```
- `reasoning_content` — the text inside a `<think>…</think>` block; present only
when non-empty. Only think-style chat templates with thinking on produce it.
Non-think templates (plain ChatML such as SmolLM2, Mistral, Phi-3) return the
whole answer as `content`.
- `tool_calls` — present when a call was parsed; `finish_reason` is then
`"tool_calls"`. `content` is `null` for a pure tool call.
- `finish_reason` — `stop`, `length`, `tool_calls`, or `repetition_loop` (the
decode was stopped because the model was looping; outranks `tool_calls`).
- `logprobs` — on the choice, when requested.
- `prompt_tokens_details.cached_tokens` — prompt tokens reused from the prefix
cache rather than computed. Always present (`0` on a cold turn).
- `kv_used` — this session's KV occupancy in tokens after this turn.
- `kv_cap` — the **effective** per-session context window, the same number
`GET /v1/models` advertises as `context_window`, re-read every turn. If the
memory ladder moves the window mid-session, the next completion carries the
new value. Treat the model list as the initial plan and this as authoritative.
- `session_conflict: true` (inside `usage`) — two conversations are sharing one
session id, so neither can resume from cache.
- `response_format_status` (top-level, when the request sent `response_format`)
— `applied`, `schema_downgraded`, `ignored_tools_present`,
`ignored_unknown_type`, or `not_enforced` (a constraint was armed but the
output did not close a complete JSON object).
- `n_status` (top-level) — why fewer choices came back than `n` asked for:
`capped_to_8`, `partial_slot_unavailable`, `partial_generation_failed`.
- `usage` covers the whole request: the prompt is counted once, completion
tokens are summed across choices.
A prompt longer than the effective window is **refused** with
`context_over_budget` (below); nothing is silently dropped. The
`context_capped` field the writer supports is never set by the current
generation path.
### Streaming response — 200 (SSE)
`Content-Type: text/event-stream`. Each chunk is:
```
data: {"id":"chatcmpl-...", "object":"chat.completion.chunk", "created":...,
"model":"...", "choices":[{"index":0,"delta":{...},"finish_reason":null}]}
```
Delta fields emitted in order:
1. `{"role":"assistant"}` — first chunk of each choice.
2. `{"reasoning_content":"<fragment>"}` — inside a think block (think-style templates only).
3. `{"content":"<fragment>"}` — answer text. `logprobs` rides the chunk carrying those tokens.
4. Tool-call chunks (when a call was parsed):
- Opening: `{"tool_calls":[{"index":0,"id":"...","type":"function","function":{"name":"...","arguments":""}}]}`
- Arguments: `{"tool_calls":[{"index":0,"function":{"arguments":"..."}}]}`
5. Closing chunk: `{"delta":{},"finish_reason":"..."}`. Carries `tool_call_error`
on the choice and `response_format_status` top-level when applicable.
6. With `stream_options.include_usage`: a chunk with `"choices":[]` and `usage`.
7. Terminal sentinel: `data: [DONE]`
With `n` > 1 the choices stream one after another, told apart by `choices[].index`.
A client that disconnects aborts the turn and frees its slot.
Errors detected after the stream opened cannot change the status code:
`context_over_budget` is sent as a terminal event
`data: {"error":{"message":"...","type":"context_over_budget","code":null,"kv_used":N,"kv_cap":M}}`
and the stream ends; other generation failures close the choice with
`finish_reason: "stop"`.
### Error responses
| Status | `type` | Condition |
|--------|--------|-----------|
| 400 | `invalid_request_error` | Empty body, malformed JSON, body not an object, `messages` missing or empty |
| 404 | `model_not_found` | Several models loaded and `model` matches none |
| 422 | `context_over_budget` | Prompt exceeds the effective per-session window. Not transient: shorten the conversation and retry. Non-streaming: `kv_used`/`kv_cap` are in the message text; streaming: they are fields of the error object. |
| 429 | `server_busy` | Single-user path: a generation is already running on this model. Batched path: all slots busy and the queue full, or 30 s queue wait timed out. `Retry-After` when the scheduler supplies one. |
| 429 | `model_at_session_capacity` | A new `session_id` while every session slot (`max_sessions`) holds a live session. `Retry-After: 30`; the message estimates the re-prefill time. |
| 503 | `no_models_loaded` | No model loaded |
| 503 | `server_memory_constrained` | Host memory pressure: admission refused the prompt, or the queue is full under critical pressure |
| 503 | `server_shutting_down` | Server is draining for shutdown. `Retry-After: 1` when refused at the route. |
| 500 | `server_error` | Generation failed / out of memory |
Source: `openai_chat.c:oai_handle_chat`, `serve_generation.c:sg_run_completion`,
`serve_command.c:dispatch`.
---
## POST /admin/models/load
Operator-only. Load a model at runtime (Phase 7). Serialized against other
uploads via `reg->upload_lock` (one GPU upload in flight at a time).
### Request body
```json
{
"path": "/models/my-model.gguf",
"name": "my-model",
"dies": [0, 1],
"quantize": false,
"ctx_cap": 32768,
"max_sessions": 4,
"max_concurrent": 2,
"reasoning_budget": 1024
}
```
| Field | Type | Required | Notes |
|-------|------|----------|-------|
| `path` | string | yes | GGUF file path |
| `name` | string | no | Served id (defaults to file basename) |
| `dies` | int array | no | Device ordinals (same `CUDA_VISIBLE_DEVICES`-relative semantics as `--dies`). Max `PN_MODEL_MAX_DIES=8`. |
| `quantize` | bool | no | INT8-quantize at load time |
| `ctx_cap` | int | no | Per-session context cap in tokens; a prompt past it is refused with `context_over_budget`. `0`/absent = engine default. Mirrors `--ctx-cap`. |
| `max_sessions` | int | no | **Residency** — how many named sessions may hold KV at once. Over the cap a new session id gets `429 model_at_session_capacity`. `0`/absent = engine default. Mirrors `--max-sessions`. |
| `max_concurrent` | int | no | **Concurrency** — how many requests may decode simultaneously (batch-scheduler slots). Surplus queues, then `429`/`503`. Distinct from `max_sessions`. `0`/absent = the resolved `max_sessions`. Mirrors `--max-concurrent`. |
| `reasoning_budget` | int | no | Force `</think>` after N tokens of unclosed reasoning. `0`/absent = unbounded. Mirrors `--reasoning-budget`. |
Every per-model option available on the `serve` command line is settable here, so a model
added at runtime ends up configured identically to one launched with the process.
### Response — 200
```json
{
"id": "my-model",
"object": "model",
"path": "/models/my-model.gguf",
"dies": [0, 1],
"vram_bytes": 12345678,
"max_sessions": 4,
"max_concurrent": 4,
"context_window": 32000,
"context_window_configured": 32000,
"graphs": {
"live": 30, "total": 40, "failed": 0, "gave_up": 0,
"ineligible": 10, "pending": 0,
"captures": 30, "capture_failures": 0,
"batched_attn": {"live": 10, "total": 10, "failed": 0, "ineligible": 0, "pending": 0},
"batched_ssm": {"live": 20, "total": 30, "failed": 0, "ineligible": 10, "pending": 0}
},
"attention": {
"ondie": 10, "host": 0, "total": 10,
"gave_up": 0, "impossible": 0,
"host_engagements": 0, "ring_alloc_failures": 0
}
}
```
The operator surface carries **both** context numbers, because an operator's
question is not the client's: `context_window` is what sessions actually get right
now, `context_window_configured` is what `--ctx-cap` asked for. They are equal
unless the memory ladder is clamping this instant — and seeing them differ is
exactly the signal that memory pressure is costing context. (`/v1/models`
advertises only the effective one; a client has no use for a window it cannot
have.) Both are omitted when unknown.
Falls back to `{"status":"loaded"}` if the row cannot be found in the snapshot (race).
### Error responses
| Status | `type` | Condition |
|--------|--------|-----------|
| 400 | `invalid_request_error` | Missing `path`, malformed JSON, too many dies |
| 400 | `load_failed` | `dies` named on a serve with no accelerator backend compiled in, or with no dies visible. Not retryable — the fix is a different binary (`pantheon --backend`) or an omitted `dies`. |
| 409 | `load_failed` | Model already loaded / already unloading |
| 507 | `load_failed` | Admission refused (VRAM or host), die binding conflict, registry full |
| 404 | `load_failed` | File not found |
| 500 | `load_failed` | Other load failure |
Source: `model_admin.c:ma_handle_load`.
---
## POST /admin/models/unload
Operator-only. Unload a model by served name. Drains any in-flight generation
before freeing.
### Request body
```json
{"name": "my-model"}
```
| Field | Type | Required | Notes |
|-------|------|----------|-------|
| `name` | string | yes | Served model id to unload |
### Response — 200
```json
{"unloaded": "my-model"}
```
### Error responses
| Status | `type` | Condition |
|--------|--------|-----------|
| 400 | `invalid_request_error` | Missing `name`, malformed JSON |
| 404 | `unload_failed` | Name not loaded |
| 409 | `unload_failed` | Already unloading |
| 500 | `unload_failed` | Other failure |
Source: `model_admin.c:ma_handle_unload`.
---
## GET /admin/models
Operator-only. Every **live instance**, one row each. Unlike `/v1/models` these
are **not** deduplicated by name: two instances of one model on different dies
are two rows, because to an operator they are two different things.
The row is the same one `POST /admin/models/load` returns — this route exists so
its operator-only fields can be read at any time rather than only as a side
effect of loading a model.
### Response — 200
```json
{
"object": "list",
"data": [
{
"id": "my-model",
"object": "model",
"path": "/models/my-model.gguf",
"dies": [2, 3],
"vram_bytes": 12345678,
"max_sessions": 4,
"max_concurrent": 4,
"max_concurrent_configured": 4,
"context_window": 32000,
"context_window_configured": 32000,
"graphs": { "live": 30, "total": 40, "failed": 0, "gave_up": 0,
"ineligible": 10, "pending": 0,
"captures": 30, "capture_failures": 0,
"batched_attn": {"live": 10, "total": 10, "failed": 0,
"ineligible": 0, "pending": 0},
"batched_ssm": {"live": 20, "total": 30, "failed": 0,
"ineligible": 10, "pending": 0} },
"attention": { "ondie": 8, "host": 2, "total": 10,
"gave_up": 2, "impossible": 0,
"host_engagements": 2, "ring_alloc_failures": 10,
"reasons": {"ring_alloc_failed": 2} }
}
]
}
```
The operator row carries **both** concurrency numbers: `max_concurrent` is what
the model serves right now (`bs_get_load` capacity, the same value `/v1/models`
advertises) and `max_concurrent_configured` is what `--max-concurrent` asked for.
They differ when there is no batch scheduler at all (no CUDA compiled in,
`QWEN35_BATCH_SLOTS` off, `bs_create` failed) or when the memory-pressure ladder
clamped the slot count — the same split, for the same reason, as
`context_window` vs `context_window_configured`.
The operator `graphs` object extends the `/v1/models` one with `gave_up`, the
lifetime `captures` / `capture_failures` counters, and a per-class split:
`batched_attn` and `batched_ssm` (the multi-user decode path), `single_ssm` (the
one-user per-layer path) and `segment` (the per-die full-step graph). A class
the model has never exercised is omitted rather than reported as an empty
deficit. Field meanings: see `graphs` under `GET /v1/models`.
The operator `attention` object adds `gave_up` / `impossible`, the lifetime
`host_engagements` / `ring_alloc_failures` counters, and `reasons` — a
histogram, present only when `host > 0`, naming why each layer is off-die
(`ring_alloc_failed`, `dec_scratch_alloc_failed`, `kv_geometry_or_ring_wrap`,
`no_slot_decode_kernel`, `no_flash_kernels`, `no_batch_scratch`,
`flash_slots_off`). Field meanings: see `attention` under `GET /v1/models`.
No error cases (returns an empty `data` array when nothing is loaded).
Source: `model_admin.c:ma_handle_models`.
---
## GET /admin/dies
Operator-only. Snapshot of all probed accelerator dies, their VRAM budget/used,
and which model occupies them.
### Response — 200
```json
{
"object": "list",
"n_dies": 8,
"data": [
{
"die": 0,
"card": 0,
"model": "my-model",
"n_models": 1,
"vram_budget": 12884901888,
"vram_used": 11674869760
}
]
}
```
| Field | Type | Notes |
|-------|------|-------|
| `die` | int | Die manager ordinal (0-indexed across all probed dies) |
| `card` | int | Physical card index (`pn_dm_card_of(die)`) |
| `model` | string | Name of the model occupying this die, or empty string |
| `n_models` | int | Number of models on this die (typically 0 or 1) |
| `vram_budget` | int (bytes) | Die's total VRAM minus `PN_DIE_RESERVE_BYTES` |
| `vram_used` | int (bytes) | Bytes allocated by the model manager |
Source: `model_admin.c:ma_handle_dies`.
---
## Session routes
A session is a live prefix-cache / decode slot keyed by `session_id`. In each
route below the session id comes from the body's `session_id`, else the
`X-Session-Id` header (export reads the body only). `name` selects the model;
absent → the default model. Unknown model → `404 model_not_found`.
One session id may carry several conversations (opencode sends its title
request and its real turns under one id). Each keeps its own cache: a turn
resumes from the conversation its prompt continues. `max_sessions` counts ids,
not conversations. On the single-user path (no batch scheduler) export takes the
id's most recently used conversation, and restore replaces the conversation the
restored `tokens` continue (else adds one); release frees all of them.
### POST /admin/session/export
Operator. Serialize a live session's state so it can be restored later.
Request: `{"name": "my-model", "session_id": "abc123"}` (`session_id` required).
Response — 200:
```json
{"session_id":"abc123","weights_version":123,"n_tok":4096,
"blob_bytes":31457280,"blob_b64":"<base64>","tokens":[...]}
```
`tokens` is the committed history the blob covers; it must be sent back on
restore.
| Status | `type` | Condition |
|--------|--------|-----------|
| 400 | `invalid_request_error` | Empty/malformed body, missing `session_id` |
| 400 | `session_export_unsupported` | The model's architecture has no session snapshot format |
| 400 | `session_caching_disabled` | The model has no session cache |
| 404 | `session_not_found` | No live session with that id |
| 429 | `session_busy` | A turn is in flight on that session; retry after it completes |
| 500 | `server_error` | Serialize / out-of-memory failure |
Source: `model_admin.c:ma_handle_session_export`.
### POST /session/restore
Install an exported session into a live slot.
Request:
```json
{"name":"my-model","session_id":"abc123","blob":"<blob_b64 from export>","tokens":[...]}
```
`session_id`, `blob` (base64) and `tokens` are required. The blob must come from
the same model architecture and the same weights.
Response — 200: `{"restored":true,"session_id":"abc123","n_tok":4096}`
| Status | `type` | Condition |
|--------|--------|-----------|
| 400 | `invalid_request_error` | Empty/malformed body, missing `session_id`/`blob`/`tokens` |
| 400 | `session_caching_disabled` | The model has no session cache |
| 400 | `bad_session_blob` | Not base64, corrupt, wrong architecture, or does not fit this model |
| 409 | `weights_version_mismatch` | Blob captured under different weights — replay the conversation instead |
| 429 | `model_at_session_capacity` | No free session slot |
| 500 | `server_error` | Restore failed |
Source: `openai_chat.c:oai_handle_session_restore`.
### POST /session/release
Free a session's slot now instead of waiting for the idle timeout (12 h default,
`PN_SESSION_IDLE_TTL_MS`).
Request: `{"name": "my-model", "session_id": "abc123"}`
Response — 200: `{"released":true,"session_id":"abc123"}`. Idempotent: an unknown
or already-freed session also returns 200.
Errors: `400 invalid_request_error` (empty/malformed body, no session id),
`404 model_not_found`.
Source: `openai_chat.c:oai_handle_session_release`.
### GET /session/estimate
`GET /session/estimate?tokens=N[&name=M][&blob_bytes=B][&upload_bps=R]`
Arithmetic estimate of restoring a session from a blob vs re-prefilling it.
`tokens` (> 0) is required. Without `blob_bytes`, the blob is estimated at
16 KiB per token; without `upload_bps` (bytes/s), 12.5 MB/s is assumed.
Response — 200:
```json
{"model":"my-model","tokens":4096,"prefill_tok_per_s":17.00,
"cold_reprefill_s":240.94,"warm_restore_s":5.391,"blob_bytes_est":67108864}
```
`prefill_tok_per_s` is the serve's measured prefill rate (a fixed fallback
before one is measured). An unknown `name` is not an error; `model` is then `""`.
Errors: `400 invalid_request_error` when `tokens` is missing or not positive.
Source: `openai_chat.c:oai_handle_session_estimate`.
### GET /session/watch
`GET /session/watch?session_id=S[&name=M]` (or `X-Session-Id`).
Liveness hold. On success the server answers `200` with
`Content-Type: text/event-stream`, a `: watch established` comment, and keeps the
connection open. While it is held the session is never reaped for idleness;
when it closes, the session is released after a grace period (45 s default,
`PN_SESSION_WATCH_GRACE_MS`) unless a new watch for it arrives first.
| Status | `type` | Condition |
|--------|--------|-----------|
| 400 | `invalid_request_error` | No session id |
| 400 | `session_caching_disabled` | The model has no session cache |
| 404 | `model_not_found` | Unknown model |
| 503 | `watch_unavailable` | Watch manager not running, or its table is full |
Source: `serve_command.c:handle_session_watch`, `session_watch.c`.
---
## Error body format
All error responses use:
```json
{"error":{"message":"<text>","type":"<code>","code":null}}
```
The `code` field is always `null`.
---
## Concurrency and connection model
- One detached pthread per accepted TCP connection. `accept()` never blocks
behind a generation.
- On `pthread_create` failure the connection is handled inline (no request
is dropped).
- SIGPIPE is ignored so a client disconnect mid-stream cannot kill the server.
- Generation serialization:
- **Legacy path** (`sched == NULL`): `gen_lock` trylock per model. Second
concurrent request to the same model → immediate 429.
- **Batched path** (`QWEN35_BATCH_SLOTS=1`): the C3 scheduler owns
concurrency. Up to 8 concurrent users; overflow waits in a per-model FIFO
queue (default depth 32, `BS_QUEUE_DEPTH`). 30-second queue wait timeout.

370
docs/reference/supported-models.md Normal file
View File

@ -0,0 +1,370 @@
# Supported Models
Models verified to load and generate correctly with Pantheon, with the config each one
needs and a runnable command. This list grows as architecture support lands and new
GGUFs are tested.
Hardware in the **Tested** column is what the row was actually measured on — `M4 CPU` is
Apple Silicon, `K80 GPU` is a Tesla K80 (GK210, `sm_37`) via the CUDA backend, `TT p150a` is a
Tenstorrent Blackhole p150a via the `USE_TT=1` backend.
**Tenstorrent: Ornith-1.5-9B is the only supported model.** Other rows marked `TT p150a` ran on
the card during development; they are not supported there. To run Ornith 1.5 on a card, see the
README's "Ornith 1.5 on a Tenstorrent card".
**Tesla K80: Qwen3.6-35B-A3B and Nex-N2-mini (Q3_K_M, one model per card) are the supported
models**, run with `./k80_qwen36.sh` / `./k80_nexn2.sh` (README "Qwen3.6 and Nex-N2-mini on a
Tesla K80"). Other `K80 GPU` rows ran on the card; they are not supported there.
---
## Llama Architecture
| Model | Quant | Params | Layers | Heads (Q/KV) | Vocab | Context | Tested | Notes |
|-------|-------|--------|--------|--------------|-------|---------|--------|-------|
| SmolLM2-135M-Instruct | Q8_0 | 135M | 30 | 9/3 | 49,152 | 8,192 | M4 CPU | 81.3 tok/s decode, 42ms load. Pre-tokenizer falls back with a warning — see [Known Issues](#known-issues). |
| MiniCPM5-1B-SFT | Q8_0 | 1B | 24 | 16/2 | 130,560 | 131,072 | TT p150a | Ran on the card (not supported there) at 4.37 tok/s (see [`../architecture/accel-foundry.md`](../architecture/accel-foundry.md) §7 for its open correctness defect). Orchestrator. arch="llama" GGUF, head_dim=128 (decoupled: n_embd/n_head=96), rope_theta=5e6, rms_eps=1e-6. No new forward — config-only. ctx capped to 4096 for KV. |
**Source (SmolLM2):** `bartowski/SmolLM2-135M-Instruct-GGUF` on HuggingFace
**Source (MiniCPM5-1B):** `openbmb/MiniCPM5-1B-GGUF` (`MiniCPM5-1B-Q8_0.gguf`, ~1.15 GB, Apache-2.0). Loads into `PN_ARCH_LLAMA` — config-only, no new forward pass.
**Best config:** `make USE_BLAS=1` (no threads needed, model is small)
```bash
huggingface-cli download bartowski/SmolLM2-135M-Instruct-GGUF SmolLM2-135M-Instruct-Q8_0.gguf --local-dir models/base/
./pantheon --model models/base/SmolLM2-135M-Instruct-Q8_0.gguf \
--prompt "Hello, my name is" --max-tokens 32
```
---
## Qwen2 Architecture
| Model | Quant | Params | Layers | Heads (Q/KV) | Vocab | Context | Tested | Notes |
|-------|-------|--------|--------|--------------|-------|---------|--------|-------|
| Qwen2.5-0.5B-Instruct | Q8_0 | 494M | 24 | 14/2 | 151,936 | 32,768 | M4 CPU | Has attention biases, tied embeddings |
**Source:** `Qwen/Qwen2.5-0.5B-Instruct-GGUF` on HuggingFace
**Best config:** `make USE_BLAS=1` (threads may help at this size)
```bash
huggingface-cli download Qwen/Qwen2.5-0.5B-Instruct-GGUF qwen2.5-0.5b-instruct-q8_0.gguf --local-dir models/base/
./pantheon --model models/base/qwen2.5-0.5b-instruct-q8_0.gguf \
--prompt "Hello, how are you?" --max-tokens 32
```
---
## Mistral Architecture
| Model | Quant | Params | Layers | Heads (Q/KV) | Vocab | Context | Sliding Window | GGUF Ver | Tested | Notes |
|-------|-------|--------|--------|--------------|-------|---------|----------------|----------|--------|-------|
| Mistral-7B-Instruct-v0.3 | Q4_K_M | 7.2B | 32 | 32/8 | 32,768 | 32,768 | 4,096 | v3 | M4 CPU | BPE tokenizer, coherent output |
| Mistral-7B-Instruct-v0.1 | Q4_K_M | 7.2B | 32 | 32/8 | 32,000 | 32,768 | — | v2 | M4 CPU | SentencePiece tokenizer, GGUF v2 |
**Source:** `bartowski/Mistral-7B-Instruct-v0.3-GGUF`, `TheBloke/Mistral-7B-Instruct-v0.1-GGUF` on HuggingFace
**Best config:** `make USE_BLAS=1 USE_THREADS=1` (7B model needs threads)
```bash
# v0.3 (BPE, GGUF v3)
huggingface-cli download bartowski/Mistral-7B-Instruct-v0.3-GGUF \
Mistral-7B-Instruct-v0.3-Q4_K_M.gguf --local-dir models/base/
./pantheon --model models/base/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf \
--prompt "Hello, how are you?" --max-tokens 32
# v0.1 (SentencePiece, GGUF v2)
huggingface-cli download TheBloke/Mistral-7B-Instruct-v0.1-GGUF \
mistral-7b-instruct-v0.1.Q4_K_M.gguf --local-dir models/base/
./pantheon --model models/base/mistral-7b-instruct-v0.1.Q4_K_M.gguf \
--prompt "Hello" --max-tokens 32
```
---
## Phi-3 Architecture
| Model | Quant | Params | Layers | Heads (Q/KV) | Vocab | Context | Tested | Notes |
|-------|-------|--------|--------|--------------|-------|---------|--------|-------|
| Phi-3-mini-4k-instruct | Q4_K_M | 3.8B | 32 | 32/32 | 32,064 | 4,096 | M4 CPU | MHA, NeoX RoPE, fused QKV + gate/up |
**Source:** `microsoft/Phi-3-mini-4k-instruct-gguf` on HuggingFace
**Best config:** `make USE_BLAS=1` (threads help at this size)
```bash
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-gguf \
Phi-3-mini-4k-instruct-Q4_K_M.gguf --local-dir models/base/
./pantheon --model models/base/Phi-3-mini-4k-instruct-Q4_K_M.gguf \
--prompt "The capital of France is" --max-tokens 32
```
---
## Mixtral Architecture (MoE)
| Model | Quant | Params | Layers | Heads (Q/KV) | Vocab | Experts (total/used) | Tested | Notes |
|-------|-------|--------|--------|--------------|-------|---------------------|--------|-------|
| Mixtral_7Bx2_MoE | Q4_K_M | ~13B | 32 | 32/8 | 32,000 | 2/2 | M4 CPU | Merged expert tensors, SentencePiece tokenizer |
**Source:** `TheBloke/Mixtral_7Bx2_MoE-GGUF` on HuggingFace
**Best config:** `make USE_BLAS=1 USE_THREADS=1` (large model needs threads)
**Architecture notes:**
- MoE extends Llama: attention path is identical, only FFN differs
- Mixtral GGUFs use `general.architecture = "llama"`, detected via `llama.expert_count` metadata
- Per-layer FFN: router → softmax → top-k expert selection → SwiGLU per expert → weighted sum
- Supports merged (`ffn_gate_exps.weight` 3D) and split (`ffn_gate.E.weight` per expert) tensor formats
```bash
huggingface-cli download TheBloke/Mixtral_7Bx2_MoE-GGUF \
mixtral_7bx2_moe.Q4_K_M.gguf --local-dir models/base/
./pantheon --model models/base/mixtral_7bx2_moe.Q4_K_M.gguf \
--prompt "Hello" --max-tokens 32
```
---
## RWKV-X Architecture (Safetensors)
| Model | Format | Params | Layers (RWKV/Attn) | Heads | Embd | Vocab | Tested | Notes |
|-------|--------|--------|-------------------|-------|------|-------|--------|-------|
| RWKV-X-0.2B-64k-Base | safetensors | 0.2B | 12/4 | 12 | 768 | 65,536 | M4 CPU | FP32 and INT8 verified |
| RWKV-X-3.6B-64k-Base | safetensors | 3.6B | 32/8 | 40 | 2,560 | 65,536 | M4 CPU | 17.2 tok/s INT8, 2.6 tok/s FP32, 8.9 GB RSS |
**Source:** `howard-hou/RWKV-X` on HuggingFace
**Best config:**
- 0.2B: `make USE_BLAS=1` (no threads needed). Add `--quantize` for INT8 (~3x faster decode).
- 3.6B: `make USE_BLAS=1 USE_THREADS=1`. Add `--quantize` for INT8 (~6.4x faster decode).
**Requires:** RWKV World tokenizer vocab file (`--vocab` flag)
```bash
# Download models
huggingface-cli download howard-hou/RWKV-X \
RWKV-X-0.2B-64k-Base.safetensors --local-dir models/base/
huggingface-cli download howard-hou/RWKV-X \
RWKV-X-3.6B-64k-Base.safetensors --local-dir models/base/
# 0.2B FP32 inference
./pantheon --model models/base/RWKV-X-0.2B-64k-Base.safetensors \
--vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
--prompt "Hello world" --max-tokens 32
# 3.6B INT8 inference (recommended)
./pantheon --model models/base/RWKV-X-3.6B-64k-Base.safetensors \
--vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
--prompt "The capital of France is" --max-tokens 64 --quantize
```
**Architecture notes:**
- Hybrid model: RWKV-7 recurrent blocks (WKV-7 kernel) + sparse attention blocks
- 0.2B: 12 RWKV + 4 attention blocks, interleaved [R,R,R,A,R,R,R,A,...]
- 3.6B: 32 RWKV + 8 attention blocks, same interleaving pattern
- Dual state: fixed-size recurrent state per RWKV block + KV cache per attention block
- Trie-based World tokenizer (65K vocab, UTF-8 byte-level)
- Uses safetensors format (not GGUF)
---
## RWKV-7 Architecture (Converted from .pth)
| Model | Source Format | Params | Layers | Heads | Embd | Vocab | Tested | Notes |
|-------|-------------|--------|--------|-------|------|-------|--------|-------|
| RWKV-x070-World-0.1B-v2.8 | .pth → safetensors | 0.1B | 12 | 12 | 768 | 65,536 | M4 CPU | BF16, 399 tensors, converted via `pantheon-convert` |
**Source:** `BlinkDL/rwkv-7-world` on HuggingFace (original .pth format)
**Conversion:** Requires `pantheon-convert` to convert from PyTorch .pth to safetensors first. The .pth reader is a sandboxed pickle interpreter in pure C — no Python needed.
**Best config:** `make USE_BLAS=1` (small model, no threads needed)
**Requires:** RWKV World tokenizer vocab file (`--vocab` flag)
```bash
# Download .pth model
huggingface-cli download BlinkDL/rwkv-7-world \
RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth \
--local-dir models/base/
# Convert to safetensors (no Python needed)
./pantheon-convert models/base/RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth \
-o models/base/RWKV-7-0.1B.safetensors
# Run inference
./pantheon --model models/base/RWKV-7-0.1B.safetensors \
--vocab foundry/rwkvx/data/rwkv_vocab_v20230424.txt \
--prompt "Hello" --max-tokens 32
```
**Architecture notes:**
- Pure RWKV-7 (no sparse attention blocks, unlike RWKV-X hybrid)
- WKV-7 kernel with Generalized Delta Rule
- ChannelMix FFN with squared ReLU
- BFloat16 weights preserved through conversion (no precision loss)
- Uses same inference engine as RWKV-X (shared `pn_rwkvx_*` code path)
---
## qwen35moe Architecture (Hybrid Gated-Delta SSM + Shared-Expert MoE)
| Model | Quant | Params | Layers (SSM/Attn) | Heads (Q/KV) | Head dim | Embd | Experts (total/used+shared) | Vocab | Tested | Notes |
|-------|-------|--------|-------------------|--------------|----------|------|-----------------------------|-------|--------|-------|
| Nex-N2-mini | Q3_K_M | 35B-A3B | 30/10 (interval 4) | 16/2 | 256 | 2,048 | 256/8 + 1 | 248,320 | M4 CPU | **Oracle-validated** vs llama.cpp (layer-by-layer, `temp 0`). Reasoning model (`<think>`). load 376 ms, ~1.0 tok/s decode, 9.5 GB RSS — CPU **reference path**. |
| Nex-N2-mini | Q3_K_M | 35B-A3B | 30/10 (interval 4) | 16/2 | 256 | 2,048 | 256/8 + 1 | 248,320 | **K80 GPU (2-die)** | **Supported on K80** (`./k80_nexn2.sh`). Serve, 2026-10-20: ~21 tok/s decode at 1k context, prefill 107-119 tok/s up to 8k, 74 at 14k, 53 at 30k. Earlier single-stream CLI: ~13.7 tok/s decode — Q3_K_M split 20/20 across the two dies of one K80 (small-BAR, both dies mapped at once). GPU output **bit-matches the CPU forward** (argmax + every per-token logit, ulp 0) with `QWEN35_FUSE=0x37 QWEN35_FLASH=0`, measured by the author; default flash prefill is argmax-exact only. ~68× the CPU path; ~15–20 with kernel fusion (Kepler bit-exact ceiling). See [`../architecture/qwen35moe.md`](../architecture/qwen35moe.md). |
| Qwen3.6-35B-A3B | Q3_K_M | 35B-A3B | 30/10 (interval 4) | 16/2 | 256 | 2,048 | 256/8 + 1 | 248,320 | **K80 GPU (2-die)** | **Supported on K80** (`./k80_qwen36.sh`). Serve, 2026-10-20: ~21 tok/s decode at 1k context, prefill 116-121 tok/s up to 8k, 79 at 14k, 60 at 30k. |
**Source:** `nex-agi/Nex-N2-mini` (GGUF: `bartowski/nex-agi_Nex-N2-mini-GGUF`), `Qwen/Qwen3.6-35B-A3B`
(GGUF: `bartowski/Qwen_Qwen3.6-35B-A3B-GGUF`)
**Best config (M4, oracle):** `make USE_BLAS=1` (Accelerate) — the correctness reference, not a speed target.
**Best config (Tesla K80, the speed path):** `make USE_CUDA=1`; run the 2-die GPU decode with
`QWEN35_GPU_SPLIT_LAYER=20 QWEN35_GPU_DIE_A=0 QWEN35_GPU_DIE_B=1 ./pantheon --model <gguf> --prompt <text>`.
~13.7 tok/s single-stream; bit-matching the oracle under `QWEN35_FUSE=0x37 QWEN35_FLASH=0`. Multi-card K80 hosts may need platform setup
(PCI BAR sizing, NVIDIA driver options); Foundry's `barswap/k80_quirk/` documents it.
**Architecture notes:**
- Hybrid (Qwen3-Next lineage): **30 Gated-Delta-Net SSM** layers + **10 sparse full-attention** layers, attention when `(i+1) % 4 == 0`
- **Shared-expert MoE every layer:** softmax + top-8/256 routed experts (SwiGLU, `norm_topk_prob`) **plus** an always-on shared expert with a sigmoid gate
- SSM block: causal conv1d (k=4) → SiLU → L2-norm q/k → GQA head-broadcast → read-after-write delta-rule recurrence (scalar GDA decay) → gated RMSNorm
- Attention block: GQA 16/2, head_dim 256, QK-RMSNorm, **partial NEOX RoPE** (first 64 of 256 dims), sigmoid output gate
- Dual state per sequence: fixed recurrent SSM state (30 layers) + KV cache (10 layers), à la RWKV-X
- Tokenizer: Qwen BPE, `tokenizer.ggml.pre = "qwen35"` pre-tokenizer; ChatML with `<think>` reasoning prefill
- **MTP/nextn trap:** GGUF must carry `nextn_predict_layers=0`; the loader never reads `blk.N.nextn.*`
- The A8 gibberish bug was the SSM GQA head-broadcast tiling (`vh = r*n_k_heads + kh`, interleaved like `ggml_repeat`); see [`../architecture/qwen35moe.md`](../architecture/qwen35moe.md)
```bash
# Interactive (applies ChatML + <think>); --temperature 0 for oracle-style greedy
./pantheon --model models/nex-agi_Nex-N2-mini-Q3_K_M.gguf --chat --temperature 0
# One-shot (chat template applied automatically; --raw to send raw text)
./pantheon --model models/nex-agi_Nex-N2-mini-Q3_K_M.gguf \
--temperature 0 --max-tokens 90 --prompt "How much does the ball cost?"
```
---
## qwen35 Dense Architecture (Bonsai / Ornith)
A GGUF with `general.architecture = "qwen35"` and a dense FFN (`blk.0.ffn_gate.weight`, no
merged expert tensors) loads as `PN_ARCH_BONSAI` (`src/models/bonsai_forward.c`): the same
Gated-DeltaNet + sparse full-attention hybrid as qwen35moe, with a dense SwiGLU FFN in place of
the MoE. A `qwen35` GGUF with routed experts loads as qwen35moe instead.
| Model | Quant | Params | Layers (SSM/Attn) | Tested | Notes |
|-------|-------|--------|-------------------|--------|-------|
| Ternary-Bonsai-27B | Q2_0 (g128 ternary, GGUF type 142) | 27B | 48/16 | CPU | Oracle-equivalent vs PrismML's llama.cpp fork. |
| Ornith-1.5-9B | Q8_0 | 9B | 24/8 | TT p150a | **The supported Tenstorrent model.** `./install.sh && ./run_ornith.sh`. Serve: 42.9 tok/s decode, 1,630-token prompt in 2.43 s (670 tok/s), MTP speculative decoding on. Tool calls use the JSON dialect ([HTTP API](http-api.md#tool-calling)). |
| Ornith-1.0-9B | Q8_0 | 9B | — | K80 GPU, TT p150a | K80: 9.0 tok/s decode. TT (ran, not supported): output byte-identical to the CPU forward, 1.57 tok/s. Tool calls use the JSON dialect ([HTTP API](http-api.md#tool-calling)). |
**Backends:** CPU; CUDA (single die, falls back to CPU if the upload fails); Tenstorrent
(used only when no other accelerator took the model).
---
## Pixtral Vision Encoder (library only)
`src/models/pixtral_vision.c` runs a Pixtral / Mistral-Small-3.1 vision tower with the
LightOnOCR projector from an llama.cpp-style `clip` mmproj GGUF, producing image embeddings
for a ChatML decoder (`<|vision_start|>`, `<|image_pad|>`, `<|vision_end|>` tokens).
- **C API only:** `pn_attach_vision`, `pn_describe` / `pn_describe_stream` / `pn_describe_embeds`,
`pn_vision_encode_rgb`, and a continuous-batch decoder (`pn_batch_decoder_*`) for bulk OCR
(`include/pantheon.h`). The CLI and `pantheon serve` do not accept images; HTTP message
`content` must be a string.
- Decoding goes through the Llama-family path (`pn_llama_prefill` / `pn_llama_forward_batch`).
- Exercised by `tests/bench_vision.c`.
---
## Supported Quant Types
| Type | Bits/Weight | Block Size | Status |
|------|-------------|------------|--------|
| F32 | 32 | — | Supported |
| F16 | 16 | — | Supported |
| Q8_0 | 8 | 32 | Supported |
| Q4_0 | 4 | 32 | Supported |
| Q3_K | 3.4 | 256 | Supported |
| Q4_K | 4.5 | 256 | Supported |
| Q5_K | 5.5 | 256 | Supported |
| Q6_K | 6.5 | 256 | Supported |
| BF16 | 16 | — | Supported |
| Q2_0 (g128 ternary, type 142) | 2.1 | 128 | Supported (Bonsai) |
Q3_K_M, Q4_K_M and Q5_K_M GGUFs mix the K-quant tiers (e.g. Q3_K for most layers with Q4_K/Q6_K for attention) — all supported. Q3_K was added for the Nex-N2-mini bring-up.
---
## Hardware Tested
| Hardware | OS | Build Command | Notes |
|----------|----|---------------|-------|
| Apple M4 | macOS 15 | `make USE_BLAS=1` | Accelerate BLAS, NEON SIMD |
| Apple M4 | macOS 15 | `make USE_BLAS=1 USE_THREADS=1` | Add threads for 1B+ models |
| Tesla K80 ×8 (16 GK210 dies, sm_37) | Ubuntu 24.04 | `make USE_CUDA=1` | CUDA 11.4 driver. N2-mini 2-die GPU decode ~13.7 tok/s. |
| Tenstorrent Blackhole p150a | Linux | `make USE_TT=1` | See [`../architecture/accel-foundry.md`](../architecture/accel-foundry.md) §7. |
---
## Not Yet Tested (Should Work)
These models use supported architectures and quant types but haven't been verified yet.
**Llama family:**
- Llama-3.2-1B-Instruct (Q4_K_M, Q8_0)
- Llama-3.2-3B-Instruct (Q4_K_M)
- SmolLM2-360M-Instruct (Q8_0)
- TinyLlama-1.1B (Q4_0, Q8_0)
**Qwen2 family:**
- Qwen2.5-1.5B-Instruct (Q4_K_M, Q8_0)
- Qwen2.5-3B-Instruct (Q4_K_M)
- Qwen2.5-7B-Instruct (Q4_K_M)
- Qwen2.5-Coder-1.5B-Instruct (Q8_0)
**Mistral family:**
- Mistral-7B-Instruct-v0.2 (Q4_K_M, Q8_0)
**Phi-3 family:**
- Phi-3-mini-128k-instruct (Q4_K_M, Q8_0) — same arch, longer context
- Phi-3-small-8k-instruct (Q4_K_M) — may have different head config
- Phi-3-medium-4k-instruct (Q4_K_M) — larger model
**Mixtral family:**
- Mixtral-8x7B-Instruct-v0.1 (Q4_K_M) — full 8 experts, top-2 routing, ~28 GB
---
## Known Issues
- **Unsupported BPE pre-tokenizers fall back with a one-time warning.** Recognised
`tokenizer.ggml.pre` values are `qwen35`, `qwen2`, `gpt-2`, `default`, `llama-bpe`, `llama3`,
`llama-v3` (or unset). Anything else (e.g. SmolLM2's) uses the qwen35/GPT-2 pre-tokenizer and
prints once: `bpe: tokenizer.ggml.pre is unsupported; falling back to the qwen35/GPT-2
pre-tokenizer (token IDs may not match the reference for this model)`. Prompt token IDs can then
differ from the reference tokenizer.
---
## Not Yet Tested (RWKV-X)
No untested RWKV-X models remaining. Both 0.2B and 3.6B verified.
## Not Yet Supported
| Architecture | Status | Blocker |
|-------------|--------|---------|
| Phi-3/4 (other variants) | Phase 3.3+ | Tested mini-4k only |

19
evals/corpus/held_out.txt Normal file
View File

@ -0,0 +1,19 @@
The history of computing is a history of layered abstractions. At the bottom sit transistors, billions of them, each one a switch that knows nothing except on and off. Above them, logic gates combine switches into decisions; above the gates, arithmetic units and registers give those decisions memory and consequence. By the time a programmer types a line of Python, she is standing on perhaps a dozen layers of accumulated invention, each one built by people who mostly never met, each one designed so that the layer above could forget the details of the layer below. This forgetting is not a weakness. It is the entire point. A civilization advances, Alfred North Whitehead observed, by extending the number of important operations it can perform without thinking about them.
Consider what happens when a modern processor executes a single instruction. The instruction is fetched from a cache that predicts what will be needed before it is asked for, decoded into micro-operations that bear little resemblance to the published instruction set, scheduled out of order across execution ports, and retired in a way that preserves the illusion that everything happened one step at a time. The processor is, in effect, lying to the programmer constantly, and the lie is benevolent: it exists so that software written in 1985 still runs correctly on silicon designed forty years later. Backward compatibility is one of the great unsung engineering achievements of the modern era, and also one of its heaviest anchors.
Memory hierarchies tell the same story in a different register. A load from the L1 cache costs about four cycles. A load from main memory costs hundreds. A read from a spinning disk, when such things were common, cost the equivalent of millions of cycles: a delay so vast that, scaled to human time, it is the difference between grabbing a pencil from your desk and sailing to another continent to fetch one. Every performance-minded programmer eventually internalizes this pyramid, and much of the craft of optimization is simply arranging for the data you need next to be the data you touched recently.
The neural networks that now dominate machine learning inherit all of these constraints and add their own. A large language model is, mechanically, an enormous pile of matrix multiplications. Each layer transforms a vector of activations by a matrix of learned weights, applies a nonlinearity, and passes the result upward. The transformer architecture arranges these multiplications around an attention mechanism, which lets every position in a sequence gather information from every other position, weighted by learned relevance. Attention is quadratic in sequence length, which is why context windows were short for years and why so much recent ingenuity has gone into making them longer without paying the full quadratic price.
Quantization is one of the field's more elegant compromises. The weights of a trained network are stored as floating-point numbers, but most of that precision is wasted: the network's behavior is remarkably tolerant of coarse rounding, provided the rounding is done carefully. Group-wise quantization schemes store a small scale factor for each block of weights, spending a few high-precision bits so that many low-precision ones can follow. A model that needs seventy gigabytes in full precision can often run in under twenty with almost no measurable loss in quality, which is frequently the difference between a model that fits on your hardware and one that does not.
There is a useful analogy between quantization and the history of audio. Early digital audio engineers feared that sixteen bits per sample would be audibly crude, yet the compact disc proved transparent for practically all listeners because the quantization noise fell below the threshold of hearing. Neural networks exhibit a similar mercy. The signal that matters, the broad statistical structure learned during training, lives in the large-scale geometry of the weight matrices, not in their sixth decimal place. Perturb every weight slightly and the model's outputs barely move; the loss surface around a trained optimum is, in most directions, astonishingly flat.
Mixture-of-experts architectures push the economics further. Instead of one monolithic feed-forward network in each layer, the model keeps a population of smaller expert networks and a lightweight router that chooses a handful of them for every token. The result is a model whose parameter count is enormous but whose computational cost per token is modest, because only the chosen experts fire. The router's decisions are learned, not programmed, and inspecting them reveals a loose, emergent division of labor: some experts lean toward code, others toward punctuation, others toward particular languages, though the boundaries are blurry and the specialization is statistical rather than clean.
None of this machinery would matter without software to drive it, and the software has its own quiet elegance. An inference engine must stream tens of gigabytes of weights through a processor's arithmetic units, keeping every lane busy while never stalling on memory. It must maintain a key-value cache so that generating token one thousand does not require reprocessing the previous nine hundred ninety-nine. It must sample from the model's output distribution with controlled randomness: a temperature to sharpen or soften the probabilities, a nucleus cutoff to discard the improbable tail, penalties to discourage the model from repeating itself like a skipping record.
The economics of all this are shifting fast. A decade ago, running a capable language model required a data center; today, a quantized model of respectable ability runs on a laptop, and hobbyists coax them onto single-board computers and salvaged server hardware for the joy of it. The pattern is familiar to anyone who watched mainframes give way to minicomputers, minicomputers to desktops, desktops to phones. Computing power that begins in institutions ends in pockets. Whatever else the current wave of machine intelligence brings, it is already following the oldest trajectory in the industry: downward in cost, outward in reach, until the extraordinary becomes ambient and nobody thinks about it at all.
Language itself is the strangest layer in the stack. A model trained only to predict the next token, given mountains of text, ends up encoding grammar, facts, idioms, arithmetic of a halting sort, and something that behaves uncannily like reasoning. Whether that constitutes understanding is a question that philosophy has not settled and engineering has not needed to. The machines work. They translate contracts, draft code, summarize meetings, and tutor students, all by doing nothing more mysterious than continuing a sequence plausibly. The gap between the humility of the mechanism and the breadth of the behavior is the central surprise of the era, and we are still learning what to make of it.

241
evals/eval_quality.sh Executable file
View File

@ -0,0 +1,241 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
# eval_quality.sh — A/B quality gate for two engine builds whose logits are not
# bit-identical (relaxed float-reduction order, Q8_0 hot tensors, ...).
#
# Runs the fixed corpus perplexity and the greedy prompt suite through build A
# (reference) and build B (candidate) via tests/eval_ppl, then prints metrics
# and a single PASS/FAIL verdict.
#
# Usage:
# evals/eval_quality.sh <buildA_bin> <buildB_bin> [model.gguf]
#
# Env knobs:
# EVAL_MODEL model path (overrides arg 3)
# EVAL_MODEL_A/B per-side model path (overrides EVAL_MODEL) — for a
# model-file A/B: same binary twice, two GGUFs
# EVAL_PROMPTS prompt suite (default: evals/prompts/quality_suite.txt)
# EVAL_CORPUS ppl corpus (default: evals/corpus/held_out.txt)
# EVAL_MAX_NEW greedy tokens per prompt (default: 48)
# EVAL_PPL_TOKENS cap on corpus tokens, 0 = all (default: 0)
# EVAL_ENV_A/B extra env for each build, e.g. EVAL_ENV_A="QWEN35_GRAPH=0"
# (lets one binary A/B its own runtime variants)
# EVAL_DIE_ARG optional leading barswap die arg (legacy rigs)
# EVAL_OUT artifact dir (default: evals/out/<timestamp>)
# EVAL_SKIP_PPL=1 / EVAL_SKIP_GEN=1 skip a stage (for quick iteration)
#
# GPU runs: export the usual QWEN35_GPU_SPLIT_LAYER / QWEN35_GPU_DIE_A /
# QWEN35_GPU_DIE_B before invoking (shared by both builds), and use
# EVAL_ENV_A/EVAL_ENV_B for anything that must differ between them.
#
# ---------------------------------------------------------------------------
# Verdict thresholds (rationale)
#
# PPL_DELTA_MAX_PCT = 0.5 Perplexity is the core quality number. Reduction
# reordering / KV rounding moves ppl by ~0.01-0.1%;
# a real bug (wrong tensor, broken expert, bad scale)
# moves it percents to orders of magnitude. 0.5% is
# far above harness noise (0 for a fixed corpus —
# runs are deterministic) yet far below any real
# regression.
#
# EARLY_DIV_POS = 8 A benign near-tie flip almost never happens in the
# EARLY_DIV_MAX_FRAC = 0.10 first few forced tokens of a templated answer;
# systematic early divergence (>=10% of prompts
# diverging before token 8) means the candidate's
# logits are shifted, not just reordered.
#
# DIV_MAX_FRAC = 0.60 Once one near-tie token flips, all later tokens
# (warn at 0.30) differ textually, so "any divergence" over-counts;
# still, if >60% of a 60-prompt greedy suite diverges
# within 48 tokens the change is not a rounding-order
# effect. 30-60% prints a WARN: eyeball the TXT lines
# in the artifact dir for semantic equivalence.
#
# PASS = ppl delta < 0.5% AND early-divergence fraction < 10%
# AND diverged fraction <= 60%.
# A bit-exact pair (golden vs golden, GRAPH=0 vs GRAPH=1) must report
# 0 divergence and 0.000% ppl delta — that is the harness self-check.
# ---------------------------------------------------------------------------
set -u
PPL_DELTA_MAX_PCT=0.5
EARLY_DIV_POS=8
EARLY_DIV_MAX_FRAC=0.10
DIV_MAX_FRAC=0.60
DIV_WARN_FRAC=0.30
usage() { echo "usage: $0 <buildA_bin> <buildB_bin> [model.gguf]" >&2; exit 2; }
[ $# -ge 2 ] || usage
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
bin_a="$1"; bin_b="$2"
model="${EVAL_MODEL:-${3:-}}"
# A model-file A/B (one binary, two GGUFs — e.g. a retyped tensor) is the mirror
# image of the build A/B this script was written for: EVAL_MODEL_A/EVAL_MODEL_B
# override the shared model per side, and you pass the same binary twice.
model_a="${EVAL_MODEL_A:-$model}"
model_b="${EVAL_MODEL_B:-$model}"
prompts="${EVAL_PROMPTS:-$repo_root/evals/prompts/quality_suite.txt}"
corpus="${EVAL_CORPUS:-$repo_root/evals/corpus/held_out.txt}"
max_new="${EVAL_MAX_NEW:-48}"
ppl_tokens="${EVAL_PPL_TOKENS:-0}"
out_dir="${EVAL_OUT:-$repo_root/evals/out/$(date +%Y%m%d_%H%M%S)}"
[ -x "$bin_a" ] || { echo "FAIL: build A not executable: $bin_a" >&2; exit 2; }
[ -x "$bin_b" ] || { echo "FAIL: build B not executable: $bin_b" >&2; exit 2; }
[ -n "$model_a" ] && [ -f "$model_a" ] || { echo "FAIL: model A not found: '$model_a'" >&2; usage; }
[ -n "$model_b" ] && [ -f "$model_b" ] || { echo "FAIL: model B not found: '$model_b'" >&2; usage; }
[ -f "$prompts" ] || { echo "FAIL: prompts not found: $prompts" >&2; exit 2; }
[ -f "$corpus" ] || { echo "FAIL: corpus not found: $corpus" >&2; exit 2; }
mkdir -p "$out_dir"
echo "== eval_quality =="
echo "build A : $bin_a ${EVAL_ENV_A:+(env: $EVAL_ENV_A)}"
echo "build B : $bin_b ${EVAL_ENV_B:+(env: $EVAL_ENV_B)}"
echo "model A : $model_a"
echo "model B : $model_b"
echo "out : $out_dir"
# run_build <A|B> <mode> <input> <limit> <outfile>
run_build() {
side="$1"; mode="$2"; input="$3"; limit="$4"; outfile="$5"
bin="$bin_a"; extra="${EVAL_ENV_A:-}"; mdl="$model_a"
[ "$side" = B ] && { bin="$bin_b"; extra="${EVAL_ENV_B:-}"; mdl="$model_b"; }
# Perplexity needs the FULL logit vector: the GPU LM head defaults to a
# sparse top-k copy (-inf elsewhere), under which the true next token can be
# unrepresented -> -inf log-prob. Force full logits for ppl (the var is
# ignored on the CPU build). Greedy gen is unaffected — argmax survives top-k.
stage_env="$extra"
[ "$mode" = ppl ] && stage_env="$stage_env QWEN35_GPU_FULL_LOGITS=1"
echo "-- [$side] $mode ($(basename "$input"))"
# shellcheck disable=SC2086
if ! env $stage_env "$bin" ${EVAL_DIE_ARG:-} "$mdl" "$mode" "$input" "$limit" \
>"$outfile" 2>"$outfile.log"; then
echo "FAIL: build $side $mode run failed — see $outfile.log" >&2
tail -5 "$outfile.log" >&2
exit 1
fi
}
fail=0
# ---- perplexity ------------------------------------------------------------
ppl_delta_pct="skipped"
if [ "${EVAL_SKIP_PPL:-0}" != 1 ]; then
run_build A ppl "$corpus" "$ppl_tokens" "$out_dir/ppl_a.txt"
run_build B ppl "$corpus" "$ppl_tokens" "$out_dir/ppl_b.txt"
ppl_a=$(sed -n 's/.*ppl=\([0-9.eE+-]*\).*/\1/p' "$out_dir/ppl_a.txt")
ppl_b=$(sed -n 's/.*ppl=\([0-9.eE+-]*\).*/\1/p' "$out_dir/ppl_b.txt")
[ -n "$ppl_a" ] && [ -n "$ppl_b" ] || { echo "FAIL: missing PPL output" >&2; exit 1; }
ppl_delta_pct=$(awk -v a="$ppl_a" -v b="$ppl_b" \
'BEGIN { d = (b - a) / a * 100; printf "%.4f", (d < 0 ? -d : d) }')
echo "ppl A=$ppl_a B=$ppl_b |delta|=$ppl_delta_pct%"
awk -v d="$ppl_delta_pct" -v m="$PPL_DELTA_MAX_PCT" 'BEGIN { exit !(d < m) }' \
|| { echo "GATE: ppl delta $ppl_delta_pct% >= $PPL_DELTA_MAX_PCT%"; fail=1; }
# BATCHED perplexity — the path the serve actually runs.
#
# `ppl` above feeds one token at a time. Serve prefills a whole prompt in
# one batched call, and that is a DIFFERENT kernel path. Scoring only the
# sequential path is how the q4 window-ring hazard survived every gate:
# it cost +7.53% ppl batched (15x this threshold) while sequential ppl was
# bit-identical, so the gate that exists to catch exactly this saw nothing.
# A quality gate that measures a path the product does not use is not a
# quality gate.
run_build A pplb "$corpus" "$ppl_tokens" "$out_dir/pplb_a.txt"
run_build B pplb "$corpus" "$ppl_tokens" "$out_dir/pplb_b.txt"
pplb_a=$(sed -n 's/.*ppl=\([0-9.eE+-]*\).*/\1/p' "$out_dir/pplb_a.txt")
pplb_b=$(sed -n 's/.*ppl=\([0-9.eE+-]*\).*/\1/p' "$out_dir/pplb_b.txt")
[ -n "$pplb_a" ] && [ -n "$pplb_b" ] || { echo "FAIL: missing batched PPL output" >&2; exit 1; }
pplb_delta_pct=$(awk -v a="$pplb_a" -v b="$pplb_b" \
'BEGIN { d = (b - a) / a * 100; printf "%.4f", (d < 0 ? -d : d) }')
echo "ppl(batched) A=$pplb_a B=$pplb_b |delta|=$pplb_delta_pct%"
awk -v d="$pplb_delta_pct" -v m="$PPL_DELTA_MAX_PCT" 'BEGIN { exit !(d < m) }' \
|| { echo "GATE: batched ppl delta $pplb_delta_pct% >= $PPL_DELTA_MAX_PCT%"; fail=1; }
# Sequential-vs-batched spread WITHIN each build. A/B deltas cancel a defect
# present on both sides; this catches one that is present on both.
for side in A B; do
seq_v=$([ "$side" = A ] && echo "$ppl_a" || echo "$ppl_b")
bat_v=$([ "$side" = A ] && echo "$pplb_a" || echo "$pplb_b")
spread=$(awk -v a="$seq_v" -v b="$bat_v" \
'BEGIN { d = (b - a) / a * 100; printf "%.4f", (d < 0 ? -d : d) }')
echo "build $side: sequential=$seq_v batched=$bat_v spread=$spread%"
awk -v d="$spread" -v m="${PPL_BATCH_SPREAD_MAX_PCT:-1.0}" 'BEGIN { exit !(d < m) }' \
|| { echo "GATE: build $side batched-vs-sequential spread $spread% >= ${PPL_BATCH_SPREAD_MAX_PCT:-1.0}% — the batched path disagrees with itself"; fail=1; }
done
fi
# ---- A/B greedy divergence -------------------------------------------------
if [ "${EVAL_SKIP_GEN:-0}" != 1 ]; then
run_build A gen "$prompts" "$max_new" "$out_dir/gen_a.txt"
run_build B gen "$prompts" "$max_new" "$out_dir/gen_b.txt"
awk -v early_pos="$EARLY_DIV_POS" -v early_max="$EARLY_DIV_MAX_FRAC" \
-v div_max="$DIV_MAX_FRAC" -v div_warn="$DIV_WARN_FRAC" '
# Parse "GEN <i> prompt_tokens=<n>: id id ..." from both files, then
# report per-prompt first-diff position + flip counts and gate.
/^GEN / {
idx = $2
n = 0
for (f = 4; f <= NF; f++) tok[n++] = $f
if (FILENAME == ARGV[1]) {
a_len[idx] = n
for (i = 0; i < n; i++) a_tok[idx, i] = tok[i]
if (idx + 1 > n_prompts) n_prompts = idx + 1
} else {
b_len[idx] = n
for (i = 0; i < n; i++) b_tok[idx, i] = tok[i]
}
}
END {
if (n_prompts == 0) { print "GATE: no GEN output parsed"; exit 1 }
diverged = 0; early = 0; flips = 0; compared = 0
printf "%-8s %6s %6s %10s %6s\n", "prompt", "lenA", "lenB", "first_diff", "flips"
for (p = 0; p < n_prompts; p++) {
la = a_len[p]; lb = b_len[p]
ml = la < lb ? la : lb
fd = -1; fl = 0
for (i = 0; i < ml; i++)
if (a_tok[p, i] != b_tok[p, i]) { fl++; if (fd < 0) fd = i }
if (fd < 0 && la != lb) fd = ml # one build stopped early
flips += fl; compared += ml
if (fd >= 0) {
diverged++
if (fd < early_pos) early++
printf "%-8d %6d %6d %10d %6d\n", p, la, lb, fd, fl
}
}
div_frac = diverged / n_prompts
early_frac = early / n_prompts
flip_pct = compared ? flips * 100.0 / compared : 0
printf "prompts=%d diverged=%d (%.1f%%) early(<%d)=%d (%.1f%%) token_flips=%.2f%%\n",
n_prompts, diverged, div_frac * 100, early_pos, early,
early_frac * 100, flip_pct
rc = 0
if (early_frac >= early_max) {
printf "GATE: early divergence %.1f%% >= %.1f%%\n",
early_frac * 100, early_max * 100; rc = 1
}
if (div_frac > div_max) {
printf "GATE: diverged fraction %.1f%% > %.1f%%\n",
div_frac * 100, div_max * 100; rc = 1
} else if (div_frac >= div_warn)
printf "WARN: diverged fraction %.1f%% — review TXT lines for semantic equivalence\n",
div_frac * 100
exit rc
}' "$out_dir/gen_a.txt" "$out_dir/gen_b.txt" | tee "$out_dir/divergence.txt"
[ "${PIPESTATUS[0]}" -eq 0 ] || fail=1
# Side-by-side text for human review of diverged prompts.
grep '^TXT' "$out_dir/gen_a.txt" >"$out_dir/txt_a.txt" || true
grep '^TXT' "$out_dir/gen_b.txt" >"$out_dir/txt_b.txt" || true
fi
echo "=================================================="
if [ "$fail" -eq 0 ]; then
echo "VERDICT: PASS (ppl |delta|=$ppl_delta_pct% < $PPL_DELTA_MAX_PCT%, no systematic divergence)"
else
echo "VERDICT: FAIL — see gates above and artifacts in $out_dir"
fi
exit "$fail"

81
evals/flash_spec_gate.sh Executable file
View File

@ -0,0 +1,81 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
# flash_spec_gate.sh — regression gate for the on-die flash attention (prefill)
# path, covering the two cases the original 12-token argmax reference missed.
# Both broke output correctness silently while the 12-token gate stayed green.
#
# Pass 1 — MULTI-TILE PREFILL (no spec). The 12-token reference is smaller
# than one 16-key attention tile, so the cross-tile online-softmax combine
# was never exercised. The greedy suite's prompts are 27-35 tokens (2-3
# tiles), so QWEN35_FLASH=0 vs =1 on plain greedy decode gates the
# multi-tile prefill: argmax-exact => 0 divergence.
#
# Pass 2 — FLASH UNDER SPECULATIVE DECODE. spec-verify re-enters the BATCHED
# flash path mid-sequence (pos0 > 0), reusing the GPU KV cache. A partial
# accept leaves rejected draft rows in that GPU cache, and interleaved
# single-token host-path decodes update only the host KV mirror; the GPU
# mirror goes stale below the sync watermark. Flash+spec diverged 73% while
# flash-without-spec was clean. spec is designed lossless (GEN output is
# identical with spec on or off), so QWEN35_FLASH=0 vs =1 with
# QWEN35_SPEC=1 must also be 0 divergence.
#
# Pass 3 — BATCHED-PREFILL CAUSALITY. Reference-free: the first M per-position
# argmaxes of an M-token chunked prefill must equal the first M of an N-token
# one (N=384 > PN_KVQ4_WMAX). The q4 window ring once lapped inside a batched
# call, so rows read K/V from LATER rows; +7.53% batched perplexity while every
# A/B stayed green (plans/results/q4_win_ring_hazard.md). One build against
# itself, under the deployed cache config (QWEN35_KV_Q4=1).
# tests/test_qwen35_prefill_causality; red proof tests/causality_red/run_red.sh.
#
# NOTE: do NOT gate the flash path with a bit-exact logit memcmp
# of batched vs sequential prefill. The flash kernel reduces the softmax
# in tile order and drifts ~1e-6 vs the host's single-pass global max — it is
# argmax-exact by design, not bit-exact. The eval-harness divergence measure
# (argmax) is the correct criterion.
#
# Usage: evals/flash_spec_gate.sh [model.gguf]
# env: QWEN35_GPU_SPLIT_LAYER / QWEN35_GPU_DIE_A / QWEN35_GPU_DIE_B
# (defaults 20 / 0 / 1), EVAL_MODEL, EVAL_SKIP_PPL (default 0).
set -u
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
model="${EVAL_MODEL:-${1:-}}"
[ -n "$model" ] && [ -f "$model" ] || { echo "FAIL: model not found: '$model'" >&2; exit 2; }
eval_ppl="$repo_root/tests/eval_ppl"
[ -x "$eval_ppl" ] || { echo "FAIL: build tests/eval_ppl (make USE_CUDA=1 tests/eval_ppl)" >&2; exit 2; }
causality="$repo_root/tests/test_qwen35_prefill_causality"
[ -x "$causality" ] || { echo "FAIL: build tests/test_qwen35_prefill_causality (make USE_CUDA=1 tests/test_qwen35_prefill_causality)" >&2; exit 2; }
: "${QWEN35_GPU_SPLIT_LAYER:=20}"; export QWEN35_GPU_SPLIT_LAYER
: "${QWEN35_GPU_DIE_A:=0}"; export QWEN35_GPU_DIE_A
: "${QWEN35_GPU_DIE_B:=1}"; export QWEN35_GPU_DIE_B
skip_ppl="${EVAL_SKIP_PPL:-0}"
rc=0
echo "===== Pass 1: multi-tile prefill (FLASH=0 vs =1, plain greedy) ====="
EVAL_SKIP_PPL="$skip_ppl" EVAL_OUT="$repo_root/evals/out/flash_gate_prefill" \
EVAL_ENV_A="QWEN35_FLASH=0" EVAL_ENV_B="QWEN35_FLASH=1" EVAL_MODEL="$model" \
"$repo_root/evals/eval_quality.sh" "$eval_ppl" "$eval_ppl" || rc=1
echo "===== Pass 2: flash + speculative decode (FLASH=0 vs =1, QWEN35_SPEC=1) ====="
QWEN35_SPEC=1 EVAL_SKIP_PPL="$skip_ppl" EVAL_OUT="$repo_root/evals/out/flash_gate_spec" \
EVAL_ENV_A="QWEN35_FLASH=0" EVAL_ENV_B="QWEN35_FLASH=1" EVAL_MODEL="$model" \
"$repo_root/evals/eval_quality.sh" "$eval_ppl" "$eval_ppl" || rc=1
echo "===== Pass 3: batched-prefill causality (q4 window ring, N=384 > WMAX) ====="
caus_out="$repo_root/evals/out/causality_gate.log"
mkdir -p "$(dirname "$caus_out")"
# The test SKIPs (exit 0) without a CUDA forward, so require its PASS verdict line.
if QWEN35_KV_Q4="${QWEN35_KV_Q4:-1}" PN_QWEN35_MODEL="$model" \
"$causality" >"$caus_out" 2>&1 && grep -q "CAUSALITY path=cuda.*PASS" "$caus_out"; then
grep -E "CAUSALITY" "$caus_out"
else
grep -E "CAUSALITY|SKIP|pos=" "$caus_out"
echo "GATE: batched prefill is not causal (or did not run) -- see $caus_out"
rc=1
fi
[ "$rc" -eq 0 ] && echo "flash_spec_gate: PASS" || echo "flash_spec_gate: FAIL"
exit $rc

52
evals/prompts/coding_repetitive.txt Normal file
View File

@ -0,0 +1,52 @@
Add a one-line doc comment above each function and reproduce this C code otherwise unchanged:
static int clamp_int(int value, int low, int high) {
if (value < low) return low;
if (value > high) return high;
return value;
}
static float clamp_float(float value, float low, float high) {
if (value < low) return low;
if (value > high) return high;
return value;
}
static double clamp_double(double value, double low, double high) {
if (value < low) return low;
if (value > high) return high;
return value;
}
%%
Rename the variable `buf` to `buffer` everywhere and reproduce this C function otherwise unchanged:
int read_config_file(const char *path, char *buf, size_t buf_len) {
FILE *f = fopen(path, "rb");
if (!f) return -1;
size_t n = fread(buf, 1, buf_len - 1, f);
buf[n] = '\0';
fclose(f);
if (n == 0) return -1;
return (int)n;
}
%%
Write C getter functions int get_width(const shape_t *s), int get_height(const shape_t *s), int get_depth(const shape_t *s), int get_stride(const shape_t *s), int get_offset(const shape_t *s) for a struct shape_t with fields width, height, depth, stride, offset. Each getter returns the matching field.
%%
Continue this pattern up to case 9, keeping the exact same formatting:
switch (digit) {
case 0: return "zero";
case 1: return "one";
case 2: return "two";
%%
Translate this Python function to JavaScript, keeping the same names and structure:
def sum_even_numbers(numbers):
total = 0
for number in numbers:
if number % 2 == 0:
total = total + number
return total
%%
Fix the off-by-one bug (the loop should include the last element) and reproduce this C function otherwise unchanged:
float sum_array(const float *values, int count) {
float total = 0.0f;
for (int i = 0; i < count - 1; i++) {
total += values[i];
}
return total;
}

43
evals/prompts/graph_subset.txt Normal file
View File

@ -0,0 +1,43 @@
Write a C function that reverses a null-terminated string in place.
%%
This C code crashes. Find the bug:
char *dup = malloc(strlen(s));
strcpy(dup, s);
%%
Write a SQL query that returns the three highest-paid employees per department from a table employees(id, name, dept, salary).
%%
Complete this Python function:
def binary_search(arr, target):
lo, hi = 0, len(arr) - 1
%%
Write a C function that computes the dot product of two float arrays of length n.
%%
Explain the difference between a mutex and a semaphore in two sentences.
%%
Write a C function int popcount(uint32_t v) without using compiler builtins.
%%
If a train travels 60 km in 45 minutes, what is its average speed in km/h?
%%
Alice is taller than Bob. Bob is taller than Carol. Carol is taller than Dave. Who is the second tallest?
%%
If today is Wednesday, what day of the week will it be in 100 days?
%%
Sort these numbers in ascending order and explain your method: 34, 7, 23, 32, 5, 62.
%%
If 5 machines take 5 minutes to make 5 widgets, how long do 100 machines take to make 100 widgets?
%%
A rectangle has perimeter 20 and area 24. What are its side lengths?
%%
Which weighs more: a kilogram of feathers or a kilogram of steel? Explain in one sentence.
%%
What are three practical tips for improving sleep quality?
%%
Draft a short, polite email declining a meeting invitation because of a schedule conflict.
%%
Explain the rules of rock-paper-scissors to someone who has never played.
%%
Translate "Good morning, how are you today?" into French and Spanish.
%%
Write a two-sentence bedtime story about a robot who learns to paint.
%%
Explain inflation in economics using a simple everyday example.

133
evals/prompts/quality_suite.txt Normal file
View File

@ -0,0 +1,133 @@
Write a C function that reverses a null-terminated string in place.
%%
Write a Python function that returns the n-th Fibonacci number iteratively.
%%
Complete this C function:
int clamp(int v, int lo, int hi) {
%%
This C code crashes. Find the bug:
char *dup = malloc(strlen(s));
strcpy(dup, s);
%%
Write a bash one-liner that counts the number of lines containing "ERROR" in all .log files under /var/log.
%%
Explain what this code does:
x = x ^ y; y = x ^ y; x = x ^ y;
%%
Write a SQL query that returns the three highest-paid employees per department from a table employees(id, name, dept, salary).
%%
Debug this Python:
def add_item(item, items=[]):
items.append(item)
return items
Why does the list grow across calls?
%%
Write a JavaScript function debounce(fn, ms) from scratch.
%%
Complete this Python function:
def binary_search(arr, target):
lo, hi = 0, len(arr) - 1
%%
You have a tool named get_weather that takes {"city": string}. The user asks: "Do I need an umbrella in Paris today?" Respond with the tool call you would make, as JSON.
%%
You have tools read_file(path) and write_file(path, content). The user says: "Append a copyright line to LICENSE". List the tool calls you would make, in order, as JSON.
%%
Write a C function that computes the dot product of two float arrays of length n.
%%
What is wrong with this loop?
for (unsigned i = 10; i >= 0; i--) printf("%u\n", i);
%%
Write a Python generator that yields all primes below a given limit using a sieve.
%%
Explain the difference between a mutex and a semaphore in two sentences.
%%
Write a git command that shows the diff of the last commit only.
%%
Refactor this to remove the repeated condition:
if (a > 0) { x = 1; }
if (a > 0) { y = 2; }
if (a > 0) { z = 3; }
%%
Write a C function int popcount(uint32_t v) without using compiler builtins.
%%
What does the volatile keyword mean in C and when is it required?
%%
A farmer has 17 sheep. All but 9 run away. How many sheep are left?
%%
If a train travels 60 km in 45 minutes, what is its average speed in km/h?
%%
I have two ropes that each take exactly one hour to burn, but they burn unevenly. How can I measure 45 minutes?
%%
Which is larger: 2^10 or 10^3? Show your reasoning briefly.
%%
Alice is taller than Bob. Bob is taller than Carol. Carol is taller than Dave. Who is the second tallest?
%%
A bat and a ball cost $1.10 together. The bat costs $1.00 more than the ball. How much does the ball cost?
%%
What is 17 * 23? Compute it step by step.
%%
If today is Wednesday, what day of the week will it be in 100 days?
%%
Three friends split a restaurant bill of $75 equally, then add a 20% tip on the total. How much does each person pay?
%%
I am thinking of a number. Double it, add 10, and you get 36. What is my number?
%%
Sort these numbers in ascending order and explain your method: 34, 7, 23, 32, 5, 62.
%%
A snail climbs 3 meters up a wall each day but slides down 2 meters each night. The wall is 10 meters tall. On which day does it reach the top?
%%
How many times do the hour and minute hands of a clock overlap in 12 hours?
%%
If 5 machines take 5 minutes to make 5 widgets, how long do 100 machines take to make 100 widgets?
%%
What is the sum of the interior angles of a hexagon?
%%
Convert 0.625 to a fraction in lowest terms, showing the steps.
%%
A rectangle has perimeter 20 and area 24. What are its side lengths?
%%
Is 91 a prime number? Explain how you checked.
%%
You flip two fair coins. What is the probability of getting at least one head?
%%
Which weighs more: a kilogram of feathers or a kilogram of steel? Explain in one sentence.
%%
Explain photosynthesis to a 10-year-old in three sentences.
%%
Write a haiku about a mountain at dawn.
%%
What are three practical tips for improving sleep quality?
%%
Summarize the plot of Romeo and Juliet in two sentences.
%%
What is the capital of Australia, and why is it not Sydney?
%%
Draft a short, polite email declining a meeting invitation because of a schedule conflict.
%%
What is the difference between weather and climate?
%%
Recommend a simple dinner I can cook in 20 minutes with chicken, rice, and frozen vegetables.
%%
Explain the rules of rock-paper-scissors to someone who has never played.
%%
What causes seasons on Earth?
%%
Give me three creative names for a coffee shop next to a library.
%%
Translate "Good morning, how are you today?" into French and Spanish.
%%
What is a black hole, in plain language?
%%
List the planets of the solar system in order from the Sun.
%%
Write a two-sentence bedtime story about a robot who learns to paint.
%%
Why does bread rise when you bake it?
%%
What should I pack for a weekend hiking trip in cool weather?
%%
Explain inflation in economics using a simple everyday example.
%%
What is the difference between a virus and a bacterium?
%%
Describe the water cycle in four short steps.

212
examples/deploy.sh Executable file
View File

@ -0,0 +1,212 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
#
# deploy.sh — bring a Pantheon fleet up to a manifest, one model at a time.
#
# WHAT THIS IS
# A template. Copy it, point it at your own manifest, adapt the gates.
# It reconciles a running `pantheon serve` to models.conf by loading each
# missing model through POST /admin/models/load, one at a time, waiting for
# each to settle before starting the next.
#
# WHY ONE AT A TIME
# On a constrained host, loading everything at once is how you OOM. Each
# load is gated on free VRAM (via the engine's own /admin/dies) and host-RAM
# headroom, then settle-waited until the model actually appears in
# /v1/models. That makes bring-up idempotent and restartable: run it again
# after a failure and it picks up where it stopped.
#
# REQUIREMENTS
# bash + curl. Nothing else — no jq, no python, no nvidia-smi. GPU state
# comes from the engine, so this works the same on CPU, Metal and CUDA.
#
# USAGE
# ./deploy.sh reconcile to the manifest (idempotent)
# ./deploy.sh --plan dry run: print what would happen, change nothing
# ./deploy.sh --prune also unload anything running that isn't listed
# ./deploy.sh --file other.conf
# PANTHEON_URL=http://host:8080 ./deploy.sh
#
# EXIT CODES
# 0 all listed models are serving · 1 usage/config error · 2 a load failed
#
set -uo pipefail
URL="${PANTHEON_URL:-http://127.0.0.1:8080}"
MANIFEST="$(dirname "$0")/models.conf"
PLAN=0
PRUNE=0
# Free host RAM (MB) below which we refuse to start another load.
: "${MIN_HOST_FREE_MB:=1024}"
# Seconds to wait for a model to appear in /v1/models after a load returns.
: "${SETTLE_TIMEOUT:=300}"
while [ $# -gt 0 ]; do
case "$1" in
--plan) PLAN=1 ;;
--prune) PRUNE=1 ;;
--file) MANIFEST="${2:?--file needs a path}"; shift ;;
--url) URL="${2:?--url needs a URL}"; shift ;;
-h|--help) sed -n '2,32p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $1 (try --help)" >&2; exit 1 ;;
esac
shift
done
log() { printf '%s %s\n' "$(date +%H:%M:%S)" "$*"; }
die() { printf 'error: %s\n' "$*" >&2; exit 1; }
[ -r "$MANIFEST" ] || die "manifest not readable: $MANIFEST"
# --- engine state -----------------------------------------------------------
# Responses are small and flat, so grep/sed is enough and keeps the dependency
# list at zero. If your manifest grows past a few dozen models, switch to jq.
api() { curl -fsS --max-time 30 "$@" 2>/dev/null; }
serve_up() { api "$URL/v1/models" >/dev/null; }
loaded_models() {
api "$URL/v1/models" | tr ',' '\n' | sed -n 's/.*"id"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p'
}
model_loaded() { loaded_models | grep -qxF "$1"; }
# Free VRAM in bytes across all devices, from the engine itself. Prints
# nothing if the engine reports no devices (CPU/Metal builds) — callers treat
# an empty answer as "no GPU gate applies".
free_vram_bytes() {
local body budget used
body="$(api "$URL/admin/dies")" || return 0
printf '%s' "$body" | grep -q '"die"' || return 0
budget=$(printf '%s' "$body" | tr ',' '\n' | sed -n 's/.*"vram_budget"[[:space:]]*:[[:space:]]*\([0-9]*\).*/\1/p' | paste -sd+ - | bc 2>/dev/null)
used=$(printf '%s' "$body" | tr ',' '\n' | sed -n 's/.*"vram_used"[[:space:]]*:[[:space:]]*\([0-9]*\).*/\1/p' | paste -sd+ - | bc 2>/dev/null)
[ -n "${budget:-}" ] && [ -n "${used:-}" ] && echo $(( budget - used ))
}
# Free host RAM in MB. Linux via /proc, macOS via vm_stat. Prints nothing on
# platforms we cannot read, which disables the host gate rather than guessing.
host_free_mb() {
if [ -r /proc/meminfo ]; then
awk '/^MemAvailable:/ {print int($2/1024); exit}' /proc/meminfo
elif command -v vm_stat >/dev/null 2>&1; then
vm_stat | awk '
/page size of/ { for (i=1;i<=NF;i++) if ($i+0>1024) ps=$i+0 }
/Pages free/ { gsub(/\./,"",$3); f=$3 }
/Pages inactive/ { gsub(/\./,"",$3); f+=$3 }
END { if (ps && f) print int(f*ps/1048576) }'
fi
}
# --- manifest ---------------------------------------------------------------
# name | path | key=value ... -> JSON body for /admin/models/load
build_body() {
local name="$1" path="$2" opts="$3" json kv k v
json="{\"name\":\"$name\",\"path\":\"$path\""
for kv in $opts; do
k="${kv%%=*}"; v="${kv#*=}"
[ "$k" = "$kv" ] && { echo " ! ignoring malformed option '$kv'" >&2; continue; }
case "$k" in
dies) json="$json,\"dies\":[${v}]" ;;
quantize) json="$json,\"quantize\":${v}" ;;
ctx_cap|max_sessions|max_concurrent|reasoning_budget)
json="$json,\"$k\":${v}" ;;
*) echo " ! unknown option '$k' — skipped" >&2 ;;
esac
done
echo "$json}"
}
settle_wait() {
local name="$1" waited=0
while [ "$waited" -lt "$SETTLE_TIMEOUT" ]; do
model_loaded "$name" && return 0
sleep 2; waited=$(( waited + 2 ))
done
return 1
}
load_model() {
local name="$1" path="$2" opts="$3" body resp
body="$(build_body "$name" "$path" "$opts")"
local vram free_mb
vram="$(free_vram_bytes)"
[ -n "$vram" ] && log " gate: $(( vram / 1048576 )) MiB VRAM free"
free_mb="$(host_free_mb)"
if [ -n "$free_mb" ]; then
log " gate: ${free_mb} MiB host RAM free"
if [ "$free_mb" -lt "$MIN_HOST_FREE_MB" ]; then
log " REFUSED: host RAM below ${MIN_HOST_FREE_MB} MiB — stopping here"
return 1
fi
fi
resp="$(curl -fsS --max-time 900 -X POST "$URL/admin/models/load" \
-H 'Content-Type: application/json' -d "$body" 2>&1)" || {
log " LOAD FAILED: ${resp:-no response}"
return 1
}
settle_wait "$name" || { log " did not appear in /v1/models within ${SETTLE_TIMEOUT}s"; return 1; }
log " ok"
}
# --- main -------------------------------------------------------------------
serve_up || die "no serve responding at $URL — start one with: pantheon serve --model <path>"
log "manifest: $MANIFEST"
log "target: $URL"
[ "$PLAN" = 1 ] && log "PLAN MODE — nothing will be changed"
declare -a WANTED=()
rc=0
while IFS= read -r line || [ -n "$line" ]; do
line="${line%%#*}"
case "$line" in *[![:space:]]*) ;; *) continue ;; esac
IFS='|' read -r name path opts <<< "$line"
name="$(echo "${name:-}" | xargs)"
path="$(echo "${path:-}" | xargs)"
opts="$(echo "${opts:-}" | xargs)"
[ -n "$name" ] && [ -n "$path" ] || { echo "skipping malformed line: $line" >&2; continue; }
WANTED+=("$name")
if model_loaded "$name"; then
log "= $name already serving"
continue
fi
if [ "$PLAN" = 1 ]; then
log "+ would load $name from $path"
log " body: $(build_body "$name" "$path" "$opts")"
continue
fi
log "+ loading $name from $path"
load_model "$name" "$path" "$opts" || { rc=2; break; }
done < "$MANIFEST"
if [ "$PRUNE" = 1 ]; then
while IFS= read -r running; do
[ -n "$running" ] || continue
for w in ${WANTED+"${WANTED[@]}"}; do [ "$w" = "$running" ] && continue 2; done
if [ "$PLAN" = 1 ]; then
log "- would unload $running (not in manifest)"
else
log "- unloading $running (not in manifest)"
api -X POST "$URL/admin/models/unload" \
-H 'Content-Type: application/json' \
-d "{\"name\":\"$running\"}" >/dev/null \
|| log " unload failed"
fi
done < <(loaded_models)
fi
log "serving: $(loaded_models | paste -sd' ' -)"
[ "$rc" -ne 0 ] && log "stopped early — fix the failure above and re-run; loaded models are kept"
exit "$rc"

35
examples/models.conf Normal file
View File

@ -0,0 +1,35 @@
# models.conf — deploy manifest for examples/deploy.sh
#
# One model per line:
#
# name | /path/to/model.gguf | key=value key=value ...
#
# Lines are loaded top to bottom, one at a time. Put the model you least want
# to lose FIRST — on a constrained host, bring-up stops when memory runs out,
# and everything above the stopping point is already serving.
#
# Blank lines and #-comments are ignored. The options field may be empty.
#
# Recognised options (all optional; they mirror the serve command-line flags
# and POST /admin/models/load fields — see docs/reference/http-api.md):
#
# dies=0,1 pin to GPU device ordinals (CUDA only; relative to
# CUDA_VISIBLE_DEVICES). Omit to let the engine place it.
# ctx_cap=32768 per-session context cap in tokens
# max_sessions=4 how many named sessions may hold KV at once
# max_concurrent=2 how many requests may decode simultaneously
# reasoning_budget=1024 force </think> after N tokens of unclosed reasoning
# quantize=true INT8-quantize at load time
#
# ---------------------------------------------------------------------------
# A chat model with a bounded window and two decode slots.
chat | /models/chat.gguf | ctx_cap=32768 max_sessions=4 max_concurrent=2
# A small model, engine-placed, all defaults.
draft | /models/draft.gguf |
# Two replicas of the same worker on different GPUs. Same name = replicas,
# load-balanced least-busy. Requires CUDA.
# worker | /models/worker.gguf | dies=2 ctx_cap=8192
# worker | /models/worker.gguf | dies=3 ctx_cap=8192

1
foundry Submodule

@ -0,0 +1 @@
Subproject commit 775df8ab31a5bde31211090fba35b5ea8d0a4fce

580
include/pantheon.h Normal file
View File

@ -0,0 +1,580 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
#ifndef PANTHEON_H
#define PANTHEON_H
#include <stdint.h>
#include <stddef.h>
#ifdef __cplusplus
extern "C" {
#endif
/*
* Pantheon — Pure C Model Hosting Engine
*
* Public API. All types are opaque — implementation hidden.
* Supports GGUF and safetensors model formats.
* Supports Llama, Qwen, Phi, Mistral, RWKV-X architectures.
*
* Text generation: pn_generate(), pn_stream(), pn_stream_tokens()
* Multi-modal: pn_infer(), pn_transcribe(), pn_synthesize(),
* pn_describe(), pn_detect(), pn_embed()
*/
/* Opaque types */
typedef struct pn_model pn_model;
typedef struct pn_state pn_state;
/* Output types for multi-modal inference */
typedef enum {
PN_OUT_LOGITS, /* Text token logits: float[n_vocab] */
PN_OUT_EMBEDDING, /* Embedding vector: float[n_embd] */
PN_OUT_AUDIO, /* Audio waveform: float[n_samples] */
PN_OUT_BBOXES, /* Bounding boxes: float[n_boxes * 6] (x1,y1,x2,y2,conf,class) */
PN_OUT_MASK, /* Segmentation mask: uint8[height * width] */
PN_OUT_DEPTHMAP, /* Depth map: float[height * width] */
} pn_output_type;
/* Tagged output from pn_infer() */
typedef struct {
pn_output_type type;
float *data; /* Output buffer (caller frees via pn_output_free) */
int n; /* Number of primary elements (vocab size, n_samples, n_boxes, etc.) */
int width; /* Image width (vision outputs only, 0 otherwise) */
int height; /* Image height (vision outputs only, 0 otherwise) */
} pn_output;
/*
* Grammar-constrained decoding hook (n2 tool-calling Phase 5) — OPT-IN.
*
* When a sampling call carries a non-NULL `tool_grammar` (below), the sampler
* masks (logit = -INFINITY) every candidate token whose TEXT would take the
* grammar state off a valid tool-call prefix, then advances the grammar by the
* chosen token's text. This forces the model's trained XML tool-call format
* even under a heavy quant. It is entirely device-agnostic (a CPU-side logit
* mutation, like repeat_penalty) and depends only on this callback + an opaque
* grammar state — the engine does not link the grammar module or the tokenizer
* into the sampler.
*
* The hook is fully self-contained via callbacks, so the engine's sampler does
* NOT link the grammar module or the tokenizer:
* - `st` : opaque grammar state (a tg_state*, see tool_grammar.h).
* - `token_text` : id → UTF-8 text piece for a token (borrowed; NULL/empty →
* the token is treated as always-allowed, a no-op).
* - `allowed` : would appending this text keep a valid grammar prefix?
* - `advance` : walk the grammar state forward by the chosen token's text.
* - `user` : passed to token_text (e.g. the tokenizer/model handle).
* - `active` : gates the mask. While the model is still in the free
* reasoning/prose region (no tool-call opener yet) the caller
* leaves this 0 so NOTHING is masked (a legit prose answer is
* never forced into a call); it flips to 1 once a tool-call
* opener has appeared and the grammar has engaged.
* - `n_vocab` : vocab bound for the scan.
* When the whole hook pointer is NULL the sampler is byte-identical to a build
* without this feature.
*/
typedef struct {
void *st;
const char *(*token_text)(void *user, uint32_t id);
int (*allowed)(void *st, const char *text); /* 1 = allowed */
void (*advance)(void *st, const char *text);
void *user;
int active;
int n_vocab;
/* Optional self-refresh (may be NULL). When set, the sampler calls it at
* entry to recompute `active` from the current grammar state, so the hook
* gates itself WITHOUT the caller refreshing `active` before every step.
* This is what lets the BATCHED driver (which samples per-slot in its own
* thread from the slot's copied sampling params) run the grammar correctly:
* the driver never touches `active`, the hook refreshes it in-thread. The
* single-user path can also rely on it (it still refreshes externally too —
* harmless). Returns the new active value (1 = mask this step). */
int (*refresh)(void *st);
/* Optional (may be NULL): has the grammar reached a COMPLETE tool call?
* The BATCHED driver queries this AFTER advancing so it can finish the slot
* the instant "</function>"/"</tool_call>" closes — the completion-stop for
* the batched path (mirrors the single-user loop's break-on-complete), so a
* forced call terminates right after the call instead of running to the
* max_tokens budget. Unused (NULL) in the single-user path (its serve loop
* checks completion itself). */
int (*complete)(void *st);
/* Optional (may be NULL): is a tool-call skeleton currently being emitted
* under the grammar? The BATCHED driver's AUTO reasoning-budget force-close
* queries this so it never injects "</think>" mid-skeleton (which would
* desync the grammar walk) — the batched twin of the single-user
* sg_tg_in_call guard. NULL → treated as "not in a call" (the single-user
* path checks in_call itself and leaves this NULL). */
int (*in_call)(void *st);
/* Optional (may be NULL): tell the grammar its reasoning close was INJECTED
* by the serve's reasoning-budget force-close, not chosen by the model. The
* grammar cannot tell the two apart on its own — the injected "</think>" is
* fed through `advance` like any other text — so it would compel the tool
* opener from the very next char, while the model is still mid-sentence, and
* the rest of its thought (often a complete call) lands inside a parameter
* VALUE. Called by the BATCHED driver once the injected close has drained;
* the single-user loop calls the grammar directly. NULL → the pre-fix
* behavior (compel immediately). */
void (*reasoning_end_injected)(void *st);
/* Fail-CLOSED policy (forced-mode hardening). On a 0-survivor step (the mask
* would leave NO legal token) the sampler must NOT fall open to the model's
* own (garbage) top token — that is how empty/garbage calls escaped. When
* `fail_closed` is nonzero the sampler instead forces `forced_token` (an EOS /
* end id supplied by the caller), terminating the generation cleanly rather
* than emitting an out-of-grammar token. EOS is always a safe closed exit
* (the grammar guarantees a completed-or-abortable call, never a deadlock).
* When `fail_closed` is 0 the sampler keeps the legacy fail-open behavior
* (AUTO mode, or callers that did not set this) — so nothing regresses. */
int fail_closed; /* 1 = force forced_token on 0-survivor */
uint32_t forced_token; /* token forced on a 0-survivor step (EOS id) */
/* Optional (may be NULL): may a token with NO text (an end token: EOS,
* <|im_end|>) be drawn while the mask is active? NULL = always, the
* historical rule. A "required" walk answers no before its call opens (the
* turn must make a call) and yes inside it only when the call can still be
* closed — the serve then closes it (tgj_closing_text). */
int (*end_allowed)(void *st);
} pn_tool_grammar_hook;
/* One entry of an OpenAI `logit_bias` map: a token id and the bias (nominally
* -100..100) ADDED to that token's raw logit before any other sampling step. */
typedef struct {
uint32_t token_id;
float bias;
} pn_logit_bias_entry;
/*
* Per-token logprob sink (OpenAI `logprobs` / `top_logprobs`).
*
* OFF by default (NULL in pn_sampling_params). When set, the sampler computes a
* log-softmax over the RAW logits — before logit_bias, penalties, temperature,
* top-k/top-p and the grammar mask — so the reported numbers are the model's own
* distribution, which is what the OpenAI field means (they are NOT the
* post-filter probabilities the draw was actually taken from).
*
* - `n_top` : how many ALTERNATIVES to report per step (0..PN_LOGPROBS_MAX_TOP).
* - `emit` : called once per sampled token with the chosen token's logprob and
* the top-`n_top` alternatives (descending logprob). The arrays
* are borrowed and valid only for the duration of the call.
*
* INJECTED-TOKEN CONTRACT. A caller that force-decodes a token it did not
* sample (the reasoning-budget "</think>" close) must call `emit` for it with
* `chosen_logprob = NAN` and `n_top = 0`. The sink's record sequence is how a
* consumer pairs logprobs with the emitted token stream, so a token that
* silently produced no record would shift every later token's logprobs onto the
* wrong text — the placeholder keeps the two aligned.
*/
#define PN_LOGPROBS_MAX_TOP 20
typedef struct {
int n_top;
void (*emit)(void *user, uint32_t chosen, float chosen_logprob,
const uint32_t *top_ids, const float *top_logprobs, int n_top);
void *user;
} pn_logprob_sink;
/* Sampling parameters */
typedef struct {
float temperature;
float top_p;
int top_k;
float repeat_penalty;
int repeat_last_n;
/* OFF by default (NULL). Non-NULL → grammar-constrained tool decoding
* (Phase 5). With it NULL, sample_token is byte-identical to before. */
const pn_tool_grammar_hook *tool_grammar;
/* ── OpenAI sampling fields (all default to "off" = 0, so a caller that
* builds params with pn_sampling_defaults() is byte-identical to before) ── */
/* OpenAI `presence_penalty` / `frequency_penalty`, applied over the same
* recent-token window `repeat_last_n` feeds the repetition penalty:
* logit[t] -= frequency_penalty * count(t in window)
* logit[t] -= presence_penalty * (count(t in window) > 0)
* Unlike `repeat_penalty` (a SERVER default of 1.1) these are only ever
* non-zero because the CLIENT asked for them, so — also unlike repeat_penalty
* — they are applied on the greedy (temperature<=0) path too. See the
* ordering note in sample_token_core. */
float presence_penalty;
float frequency_penalty;
/* OpenAI `logit_bias`: borrowed array of id→bias entries (NULL / 0 = off).
* Added to the RAW logits first, before everything else. */
const pn_logit_bias_entry *logit_bias;
int n_logit_bias;
/* OpenAI `seed`: when `rng_seeded` is non-zero the random draw comes from
* `rng_state` (a per-request splitmix64/xorshift stream advanced in place)
* instead of the process-global rand(), so the same request + seed replays
* the same tokens. Zero → the global RNG, byte-identical to before. */
int rng_seeded;
uint64_t rng_state;
/* OpenAI `logprobs` / `top_logprobs` (NULL = off). */
const pn_logprob_sink *logprobs;
/* A backend that hands the sampler a TOP-N row (-INFINITY outside the N it named)
* gives a refill here: when the row cannot decide this draw exactly (the request's
* top-k could reach past the N after penalties), the sampler calls it for the dense
* row and samples that instead. NULL = rows are taken as given. Returns 0 on success. */
int (*dense_row)(void *ctx, float *logits, int n_vocab);
void *dense_ctx;
/* Optional, with dense_row: fetch only PART of the dense row — every 16-entry
* chunk i with need[i] != 0 (ids 16*i .. 16*i+15) is written into `logits` at
* its ids, the rest is left as it was. The grammar mask uses it to read just
* the chunks holding allowed tokens instead of the whole row. NULL = whole row
* only. Same ctx as dense_row. Returns 0 on success. */
int (*dense_chunks)(void *ctx, const unsigned char *need, float *logits, int n_vocab);
} pn_sampling_params;
/* Generation callback — called for each new token */
typedef void (*pn_token_callback)(const char *token, void *user_data);
/* Stop predicate — polled during generation; return non-zero to abort early. */
typedef int (*pn_stop_callback)(void *user_data);
/* Return default sampling parameters */
pn_sampling_params pn_sampling_defaults(void);
/* Load a model from GGUF or safetensors file */
pn_model* pn_load(const char *path);
/* ============================================================
* Pre-flight admission control (MemoryAware layer, §6)
* ============================================================ */
/* Verdict of pn_admit: whether an n_tokens prefill can run as-is, must be
* capped to a smaller window, or must be refused to protect the host. */
typedef enum {
PN_ADMIT_FIT = 0, /* run it as requested */
PN_ADMIT_CAPPED = 1, /* run only the first *fit_tokens_out tokens */
PN_ADMIT_DENIED = 2, /* refuse — host/die would be exhausted */
} pn_admit_status;
/*
* Decide whether to admit an n_tokens prefill for `model`, BEFORE any
* memory-growing work starts. Engine-level so both the server (serve_generation)
* and direct pn_model callers (evals / eval_ppl — the path that crash-looped)
* are covered. Computes the projected host + per-die cost, compares against the
* live memory state minus reserved headroom, and returns the verdict. On
* PN_ADMIT_CAPPED, *fit_tokens_out (if non-NULL) receives the largest prefill
* length that fits. Never allocates, never blocks. If the memory-watch layer is
* not running it returns PN_ADMIT_FIT (fail-open — no behavior change).
*
* Thread-safe: yes
* Reentrant: yes
*/
pn_admit_status pn_admit(pn_model *model, int n_tokens, int *fit_tokens_out);
/*
* The CONFIGURED per-session hard context cap for `model`, in tokens — the KV
* ring / max_seq ceiling clamped to the model's trained context, i.e. what the
* operator asked for (`--ctx-cap`). Returns 0 if unknown. Pure — never
* allocates.
*
* This is the OPERATOR's number. Never hand it to a client: under memory
* pressure admission enforces a smaller one — see pn_effective_ctx_window.
*
* Thread-safe: yes
* Reentrant: yes
*/
int pn_session_ctx_cap(const pn_model *model);
/*
* The EFFECTIVE per-session context window for `model`, in tokens — the
* configured cap above AFTER the reactive memory ladder's runtime clamp.
*
* The two differ exactly when the ladder has shrunk the admitted prefill window
* under memory pressure, and that difference is the whole reason this call
* exists: advertising the configured cap while admission enforces a smaller one
* gives a client a window the serve will then reject (and produced the
* self-contradictory "prompt 70000 > session cap 131072" message Phase 10
* fixed). Everything a CLIENT is told — `context_window` on GET /v1/models and
* GET /admin/models, `kv_cap` in a completion's usage object — comes from here.
*
* Returns 0 if unknown. Reads the live memwatch snapshot; never allocates.
*
* Thread-safe: yes
* Reentrant: yes
*/
int pn_effective_ctx_window(const pn_model *model);
/* Free a loaded model */
void pn_model_free(pn_model *model);
/* Keep a loaded model's accelerator resident while it is idle (the serve's idle tick, PM 5).
* Cheap and safe to call any time no generation is running on the model; a no-op for backends
* with nothing to keep alive. Returns >= 0, or -1 when a keepalive failed (the next use then
* re-arms as it always has). */
int pn_model_keepalive(pn_model *model);
/* Warm an accelerator-resident model once after load (re-arms idle families before the first
* request). Returns 0, or <0 on failure (the model still serves). No-op elsewhere. */
int pn_model_warm(pn_model *model);
/* Create inference state for a model */
pn_state* pn_state_new(pn_model *model);
/* Free inference state */
void pn_state_free(pn_state *state);
/* Reset inference state (clear KV cache, reset position) */
void pn_state_reset(pn_state *state);
/*
* Generate text from a prompt.
*
* Returns a newly allocated string (caller must free with free()).
* Returns NULL on error.
*/
char* pn_generate(pn_model *model, pn_state *state,
const char *prompt, int max_tokens,
pn_sampling_params *params);
/*
* Streaming generation — calls callback for each token.
*
* Returns number of tokens generated, or -1 on error.
*/
int pn_stream(pn_model *model, pn_state *state,
const char *prompt, int max_tokens,
pn_sampling_params *params,
pn_token_callback callback, void *user_data);
/*
* Streaming generation from pre-tokenized input.
* Prefills from token array, then generates until EOS or max_tokens.
* Does NOT add BOS — caller controls tokenization entirely.
* Returns number of tokens generated, or -1 on error.
*/
int pn_stream_tokens(pn_model *model, pn_state *state,
const uint32_t *tokens, int n_tokens,
int max_tokens,
pn_sampling_params *params,
pn_token_callback callback, void *user_data);
/* --- Multi-modal inference API --- */
/*
* Low-level forward pass returning typed output.
* For text models, returns PN_OUT_LOGITS with float[n_vocab].
* Caller must free result with pn_output_free().
* Returns NULL on error.
*/
pn_output* pn_infer(pn_model *model, pn_state *state, uint32_t token);
/* Free a pn_output returned by pn_infer() */
void pn_output_free(pn_output *out);
/*
* Speech-to-text: transcribe audio to text.
* Returns newly allocated string (caller frees with free()).
* Returns NULL if model does not support STT.
*/
char* pn_transcribe(pn_model *model, pn_state *state,
const float *audio_samples, int n_samples,
int sample_rate);
/*
* Text-to-speech: synthesize audio from text.
* Writes audio samples to *out_samples (caller frees with free()).
* Returns number of samples written, or -1 on error.
* Returns -1 if model does not support TTS.
*/
int pn_synthesize(pn_model *model, pn_state *state,
const char *text, float **out_samples,
int *out_sample_rate);
/*
* Attach a vision encoder (an llama.cpp-style "clip" mmproj GGUF) to a loaded
* text model so pn_describe() can run. Returns 0 on success, negative on error.
*/
int pn_attach_vision(pn_model *model, const char *mmproj_path);
/*
* Vision-language: describe an image given a text prompt.
* Returns newly allocated string (caller frees with free()).
* Returns NULL if model does not support VLM.
*/
char* pn_describe(pn_model *model, pn_state *state,
const uint8_t *image_rgb, int width, int height,
const char *prompt, int max_tokens,
pn_sampling_params *params);
/*
* Streaming variant of pn_describe: same result, but as generation proceeds it
* calls callback(full_text_so_far, user_data) after each token with the full
* decoded text up to that point (not just the new token). Pass NULL to behave
* exactly like pn_describe. If should_stop is non-NULL it is polled after each
* generated token; returning non-zero ends generation early and returns the text
* produced so far. Returns the final string (caller frees with free()).
*/
char* pn_describe_stream(pn_model *model, pn_state *state,
const uint8_t *image_rgb, int width, int height,
const char *prompt, int max_tokens,
pn_sampling_params *params,
pn_token_callback callback, void *user_data,
pn_stop_callback should_stop, void *stop_data);
/*
* Vision-language decode from PRE-SUPPLIED image embeddings, skipping the
* internal vision encode. Does exactly what pn_describe_stream does after the
* encode: builds the <|vision_start|> + <|image_pad|> * n_img + <|vision_end|> +
* prompt sequence, splices the supplied [n_img x embd_dim] decoder-space embeds
* at the image-pad positions, batched-prefill, and generates. embd_dim must
* equal the model's decoder embedding dimension. img_embds is NOT freed here
* (the caller owns it), so an external (e.g. Core ML GPU) vision encoder can
* supply them. callback / should_stop behave as in pn_describe_stream. Returns
* newly allocated text (caller frees with free()); NULL on error.
*/
char* pn_describe_embeds(pn_model *model, pn_state *state,
const float *img_embds, int n_img, int embd_dim,
const char *prompt, int max_tokens,
pn_sampling_params *params,
pn_token_callback callback, void *user_data,
pn_stop_callback should_stop, void *stop_data);
/*
* Run just the vision encoder for one RGB image, returning decoder-space
* embeddings [n_img x n_embd] (caller frees *embeds_out). The vision half of
* pn_describe_stream, split out so a bulk pipeline can stage embeds ahead of the
* batched decode. Returns 0 on success, -1 on error.
*/
int pn_vision_encode_rgb(pn_model *model, const uint8_t *image_rgb, int width, int height,
float **embeds_out, int *n_img_out, int *embd_dim_out);
/* --- Continuous-batch VLM decode (bulk OCR throughput) --- */
/*
* Prefill a state's KV cache from pre-supplied image embeddings (the ChatML
* vision turn) and sample the first generated token, WITHOUT running the decode
* loop. Leaves the state ready to be driven by the batch decoder below (or
* pn_llama_forward_batch). Writes *first_tok_out and returns 0 on success, -1 on
* error. img_embds is NOT freed (caller owns it).
*/
int pn_vlm_prefill_embeds(pn_model *model, pn_state *state,
const float *img_embds, int n_img, int embd_dim,
const char *prompt, pn_sampling_params *params,
uint32_t *first_tok_out);
/* Opaque continuous-batch decoder. */
typedef struct pn_batch_decoder pn_batch_decoder;
/* A finished page returned by the decoder: its id, its now-recyclable state, and
* the decoded UTF-8 transcription (caller frees text with free()). */
typedef struct {
int page_id;
pn_state *state;
char *text;
} pn_page_out;
/*
* Create a decoder that batches up to max_batch prefilled sequences through
* pn_llama_forward_batch, handling per-sequence sampling, EOS, and
* detokenization. max_tokens caps each page's output. Returns NULL on error.
*/
pn_batch_decoder* pn_batch_decoder_new(pn_model *model, int max_batch,
int max_tokens, const pn_sampling_params *params);
void pn_batch_decoder_free(pn_batch_decoder *d);
int pn_batch_decoder_capacity(const pn_batch_decoder *d); /* max_batch */
int pn_batch_decoder_active(const pn_batch_decoder *d); /* slots in use */
/*
* Admit a prefilled sequence into a free slot. state must already be prefilled
* (pn_vlm_prefill_embeds) and first_tok is that call's first_tok_out. Returns 0
* on success, -1 if the decoder is full. The decoder borrows state until it is
* handed back via a pn_page_out.
*/
int pn_batch_decoder_admit(pn_batch_decoder *d, pn_state *state, int page_id,
uint32_t first_tok);
/*
* Run ONE batched decode step over all active slots. Pages that finish (EOS or
* max_tokens) this step are written to out[] (up to max_out) with their text and
* their recyclable state; returns the number finished (>=0), or -1 if the model
* does not support batched decode (caller should fall back). Call repeatedly
* while pn_batch_decoder_active() > 0, admitting fresh pages into freed slots.
*/
int pn_batch_decoder_step(pn_batch_decoder *d, pn_page_out *out, int max_out);
/*
* Finalize ALL active slots with whatever they have decoded so far (early stop /
* drain), returning the number written to out[] (up to max_out).
*/
int pn_batch_decoder_flush(pn_batch_decoder *d, pn_page_out *out, int max_out);
/* ============================================================
* Batched lossless speculative decode
* ============================================================ */
/* Draft provider: given the accepted prefix generated so far for one sequence
* (prefix[0..n_prefix-1], the model's real tokens), fill draft_out with up to
* max_k proposed next tokens and return how many were produced (0..max_k).
* Returning 0 means "no proposal" (that slot takes a single plain greedy step
* this cycle). This is the seam where an external draft source plugs in. */
typedef int (*pn_draft_fn)(const uint32_t *prefix, int n_prefix,
uint32_t *draft_out, int max_k, void *user_data);
/*
* Configure the decoder for batched speculative decode. max_k is the per-slot
* draft width, min_accept_frac / probe_cycles drive the per-page fallback (after
* probe_cycles speculative cycles a slot whose running acceptance fraction is
* below min_accept_frac stops drafting and rides as a plain single-token slot).
* REQUIRES the decoder's sampling temperature <= 0 (greedy): speculation is only
* lossless for greedy decode, so this returns -1 for a sampling temperature.
* Allocates the ragged verify scratch. Returns 0 on success, -1 on error.
*/
int pn_batch_decoder_spec_config(pn_batch_decoder *d, int max_k,
double min_accept_frac, int probe_cycles);
/*
* Admit a prefilled sequence into a free slot with its own draft provider (as
* pn_batch_decoder_admit, plus draft_fn/draft_ud). Pass draft_fn = NULL to admit
* a plain (non-speculative) slot that still steps correctly under step_spec (it
* rides as a K=1 block). Requires pn_batch_decoder_spec_config first. Returns 0
* on success, -1 if full or not configured.
*/
int pn_batch_decoder_admit_spec(pn_batch_decoder *d, pn_state *state, int page_id,
uint32_t first_tok, pn_draft_fn draft_fn, void *draft_ud);
/*
* Run ONE batched speculative decode step. Each active slot proposes up to max_k
* drafts (via its draft_fn), all slots' [pending, drafts...] blocks are verified
* in one pn_llama_verify_batch pass, and each slot greedily accepts the drafts
* that already equal the model's own argmax, emitting the exact greedy
* continuation. The emitted token stream is therefore BIT-IDENTICAL to plain
* batched greedy decode (pn_batch_decoder_step) for ANY draft; drafts only change
* how many tokens each verify pass yields. Finished pages (EOS / max_tokens) are
* written to out[] (up to max_out); returns the number finished (>=0), or -1 if
* the model does not support batched verify (caller falls back to plain step).
*/
int pn_batch_decoder_step_spec(pn_batch_decoder *d, pn_page_out *out, int max_out);
/* Aggregate speculative-decode accounting since pn_batch_decoder_spec_config:
* verify passes run, draft tokens proposed, and draft tokens accepted (any of
* the out pointers may be NULL). */
void pn_batch_decoder_spec_stats(const pn_batch_decoder *d, long *passes,
long *drafted, long *accepted);
/*
* Object detection: find objects in an image.
* Returns bounding boxes as float[n_boxes * 6]: x1, y1, x2, y2, confidence, class_id.
* Caller frees returned array with free().
* Returns NULL if model does not support detection.
*/
float* pn_detect(pn_model *model, pn_state *state,
const uint8_t *image_rgb, int width, int height,
float confidence_threshold, int *n_boxes_out);
/*
* Embedding extraction: compute a vector representation.
* Returns float[n_embd] (caller frees with free()).
* Returns NULL if model does not support embeddings.
*/
float* pn_embed(pn_model *model, pn_state *state,
const char *text, int *n_dims_out);
#ifdef __cplusplus
}
#endif
#endif /* PANTHEON_H */

108
install.sh Executable file
View File

@ -0,0 +1,108 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
#
# Build Pantheon for a Tenstorrent Blackhole p150a and let this user drive the card,
# or (--k80) for a Tesla K80.
#
# ./install.sh TT: build, then set up card access (asks before using sudo)
# ./install.sh --yes same, without asking
# ./install.sh --k80 K80: build pantheon-cuda (no CUDA toolkit needed; the card needs
# the NVIDIA 470 driver). Then ./k80_qwen36.sh or ./k80_nexn2.sh.
#
# Card access needs no vendor kernel module: Foundry's userspace driver maps the card's
# PCI BARs. The udev rule (foundry/tt/tools/60-foundry-blackhole.rules) opens those to the
# `tenstorrent` group and sets two PCI config bits that otherwise need root on every run.
# Then run ./run_ornith.sh.
set -eu
cd "$(dirname "$0")"
RULES_SRC=foundry/tt/tools/60-foundry-blackhole.rules
RULES_DST=/etc/udev/rules.d/60-foundry-blackhole.rules
YES=0
K80=0
for arg in "$@"; do
case "$arg" in
--yes) YES=1 ;;
--k80) K80=1 ;;
*) echo "install: unknown option $arg (use --yes, --k80)" >&2; exit 2 ;;
esac
done
die() { echo "install: $*" >&2; exit 1; }
ask() {
[ "$YES" -eq 1 ] && return 0
printf '%s [y/N] ' "$1"
read -r reply
[ "$reply" = y ] || [ "$reply" = Y ]
}
check_tools() {
local missing=() tool
[ "$(uname -s)" = Linux ] || die "the Tenstorrent and K80 backends are Linux only"
for tool in cc make git curl sha256sum "$@"; do
command -v "$tool" > /dev/null || missing+=("$tool")
done
[ "${#missing[@]}" -eq 0 ] ||
die "missing: ${missing[*]} (Debian/Ubuntu: sudo apt install build-essential git curl pciutils)"
}
# build_pantheon <make flag> <backend>: e.g. USE_TT=1 tt -> ./pantheon-tt
build_pantheon() {
[ -f foundry/tensor/tensor.h ] || git submodule update --init foundry
echo "install: building pantheon-$2 (a few minutes on a small CPU)"
make "$1" -j"$(nproc)" "pantheon-$2"
[ "$(./pantheon-"$2" --backend)" = "$2" ] || die "built binary is not the $2 backend"
}
# Warn (don't fail) when the K80 can't run yet: the build itself needs no driver.
check_k80_driver() {
local driver
if ! command -v nvidia-smi > /dev/null || ! driver=$(nvidia-smi --query-gpu=driver_version \
--format=csv,noheader 2> /dev/null | head -1) || [ -z "$driver" ]; then
echo "install: WARNING: no NVIDIA driver found; a K80 needs the 470 driver to run"
return 0
fi
echo "install: NVIDIA driver $driver"
case "$driver" in 470.*) ;; *) echo "install: WARNING: Kepler (K80) is supported by the 470 driver branch" ;; esac
nvidia-smi -L | grep -q "K80" || echo "install: WARNING: no Tesla K80 listed by nvidia-smi -L"
}
# Compare rule lines only, so a reworded comment doesn't trigger sudo again.
card_access_ready() {
[ -f "$RULES_DST" ] &&
diff -q <(grep -v '^#' "$RULES_SRC") <(grep -v '^#' "$RULES_DST") > /dev/null &&
id -nG | tr ' ' '\n' | grep -qx tenstorrent
}
setup_card_access() {
if card_access_ready; then
echo "install: card access already set up"
return 0
fi
echo "install: card access needs sudo, once:"
echo " - create group 'tenstorrent' and add $USER to it"
echo " - install $RULES_DST, reload udev, re-trigger the card"
ask "Do it now?" || { echo "install: skipped; see the header of $RULES_SRC"; return 0; }
sudo groupadd -f tenstorrent
sudo usermod -aG tenstorrent "$USER"
sudo cp "$RULES_SRC" "$RULES_DST"
sudo udevadm control --reload
sudo udevadm trigger --subsystem-match=pci --attr-match=vendor=0x1e52
echo "install: done. Log out and back in so the 'tenstorrent' group applies."
}
if [ "$K80" -eq 1 ]; then
check_tools
build_pantheon USE_CUDA=1 cuda
check_k80_driver
echo "install: next, ./k80_qwen36.sh or ./k80_nexn2.sh"
else
check_tools lspci setpci
lspci -d 1e52:b140 | grep -q . ||
echo "install: WARNING: no Blackhole p150a (PCI 1e52:b140) found; building anyway"
build_pantheon USE_TT=1 tt
setup_card_access
echo "install: next, ./run_ornith.sh"
fi

38
k80_nexn2.sh Executable file
View File

@ -0,0 +1,38 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
#
# Run Nex-N2-mini (35B-A3B, Q3_K_M) on one Tesla K80 as an OpenAI-compatible server with
# prefix caching (model id "nex-n2-mini"). Run ./install.sh --k80 first.
#
# ./k80_nexn2.sh serve on 127.0.0.1:8082, dies 0,1
# HOST=0.0.0.0 PORT=8203 ./k80_nexn2.sh listen on the LAN
# DIES=2,3 ./k80_nexn2.sh use the second K80 card
# CTX_CAP=16384 MAX_SESSIONS=8 ./k80_nexn2.sh
# context window (default 32000) / cached sessions (5)
# NEXN2_DIR=/data/n2 ./k80_nexn2.sh where the model lives
# (default ~/.cache/pantheon/Nex-N2-mini)
# ./k80_nexn2.sh --download-only fetch and check the GGUF, don't serve
# extra args go to `pantheon serve`
#
# First run downloads 16.2 GB (resumable) and checks its sha256. Send `session_id` with each
# request, and send each earlier assistant turn back with its `reasoning_content`, so later
# turns reuse the cached prefix (a 4k-token turn 2: ~38 s cold, 2-5 s cached). Prefill is the
# slow part on a K80.
set -eu
cd "$(dirname "$0")"
NAME=nex-n2-mini
GGUF=nex-agi_Nex-N2-mini-Q3_K_M.gguf
GGUF_SHA=8b05a0a21746793c0be111b4dce0318a98f6468f315669fb5e13651245f7ee58
GGUF_GB=16
MIRROR="${NEXN2_MIRROR:-https://huggingface.co/dcherrera/Nex-N2-mini-Pantheon-K80/resolve/main}"
UPSTREAM="${NEXN2_UPSTREAM:-https://huggingface.co/bartowski/nex-agi_Nex-N2-mini-GGUF/resolve/main}"
DIR="${NEXN2_DIR:-$HOME/.cache/pantheon/Nex-N2-mini}"
CTX_CAP="${CTX_CAP:-32000}"
MAX_SESSIONS="${MAX_SESSIONS:-5}"
REASONING_BUDGET="${REASONING_BUDGET:-2000}"
PORT="${PORT:-8082}"
. scripts/k80_serve.sh
k80_main "$@"

38
k80_qwen36.sh Executable file
View File

@ -0,0 +1,38 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
#
# Run Qwen3.6-35B-A3B (Q3_K_M) on one Tesla K80 as an OpenAI-compatible server with prefix
# caching (model id "qwen36"). Run ./install.sh --k80 first.
#
# ./k80_qwen36.sh serve on 127.0.0.1:8081, dies 0,1
# HOST=0.0.0.0 PORT=8201 ./k80_qwen36.sh listen on the LAN
# DIES=2,3 ./k80_qwen36.sh use the second K80 card
# CTX_CAP=32768 MAX_SESSIONS=4 ./k80_qwen36.sh
# context window (default 131072) / cached sessions (2)
# QWEN36_DIR=/data/qwen36 ./k80_qwen36.sh where the model lives
# (default ~/.cache/pantheon/Qwen3.6-35B-A3B)
# ./k80_qwen36.sh --download-only fetch and check the GGUF, don't serve
# extra args go to `pantheon serve`
#
# First run downloads 17.1 GB (resumable) and checks its sha256. Send `session_id` with each
# request, and send each earlier assistant turn back with its `reasoning_content`, so later
# turns reuse the cached prefix (a 4k-token turn 2: ~38 s cold, 2-5 s cached). Prefill is the
# slow part on a K80.
set -eu
cd "$(dirname "$0")"
NAME=qwen36
GGUF=Qwen_Qwen3.6-35B-A3B-Q3_K_M.gguf
GGUF_SHA=17350b1355d32397cffe7d4b446fd9361dbb01aa26bbf41fc950aa595033794f
GGUF_GB=17
MIRROR="${QWEN36_MIRROR:-https://huggingface.co/dcherrera/Qwen3.6-35B-A3B-Pantheon-K80/resolve/main}"
UPSTREAM="${QWEN36_UPSTREAM:-https://huggingface.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF/resolve/main}"
DIR="${QWEN36_DIR:-$HOME/.cache/pantheon/Qwen3.6-35B-A3B}"
CTX_CAP="${CTX_CAP:-131072}"
MAX_SESSIONS="${MAX_SESSIONS:-2}"
REASONING_BUDGET="${REASONING_BUDGET:-8000}"
PORT="${PORT:-8081}"
. scripts/k80_serve.sh
k80_main "$@"

70
run_ornith.sh Executable file
View File

@ -0,0 +1,70 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
#
# Run Ornith-1.5-9B on a Tenstorrent Blackhole p150a as an OpenAI-compatible server
# (model id "ornith", /v1/chat/completions). Run ./install.sh first.
#
# ./run_ornith.sh serve on 127.0.0.1:8080
# HOST=0.0.0.0 PORT=8301 ./run_ornith.sh listen on the LAN
# CTX_CAP=32768 ./run_ornith.sh context window (default 131072)
# ./run_ornith.sh --reasoning-budget 2048 extra args go to `pantheon serve`
# ORNITH_DIR=/data/ornith ./run_ornith.sh where the model files live
# (default ~/.cache/pantheon/Ornith-1.5-9B)
# HF_TOKEN=... sent to Hugging Face if set
# ./run_ornith.sh --download-only fetch and check the files, don't serve
# ORNITH_MIRROR=<url> ORNITH_UPSTREAM=<url> other download locations
#
# First run: downloads ~10 GB from Hugging Face (resumable) and checks each file's sha256;
# if our mirror is unreachable, the GGUF comes from ornith-ai and the MTP draft head is built
# from it. Pantheon then converts the GGUF to the card format on the first load and caches
# it beside the GGUF (~10 GB more). Later runs start straight away.
set -eu
cd "$(dirname "$0")"
MIRROR="${ORNITH_MIRROR:-https://huggingface.co/dcherrera/Ornith-1.5-9B-Pantheon-TT/resolve/main}"
UPSTREAM="${ORNITH_UPSTREAM:-https://huggingface.co/ornith-ai/Ornith-1.5-9B-GGUF/resolve/main}"
DIR="${ORNITH_DIR:-$HOME/.cache/pantheon/Ornith-1.5-9B}"
GGUF=Ornith-1.5-9B-Q8_0.gguf
GGUF_SHA=22086870b009dbe9815ee752c48a82de930118a7c5ce5599590892ae03b8b010
MTP=Ornith-1.5-9B-Q8_0.bfp8.mtp.fttw
MTP_SHA=d9662d436973b0d28bd5ee09ab692a2050c12aaacdfa3aaf8027ec4173dc16fa
die() { echo "run_ornith: $*" >&2; exit 1; }
# The draft head is blk.32 of the GGUF; Pantheon does not build it on load, so build it here.
build_mtp() {
local venv="$DIR/.venv" conv=foundry/tt/convert/tools/build/ft_tt_convert_gguf
echo "run_ornith: building the MTP draft head from the GGUF"
command -v python3 > /dev/null || return 1
[ -x "$venv/bin/python" ] || python3 -m venv "$venv" || return 1
"$venv/bin/pip" -q install numpy gguf || return 1
make -C foundry/tt/convert/tools build/ft_tt_convert_gguf || return 1
"$venv/bin/python" tools/build_mtp_gguf.py "$DIR/$GGUF" "$DIR/mtp.gguf" || return 1
"$conv" "$DIR/mtp.gguf" "$DIR/$MTP" --ledger bfp8 || return 1
rm -f "$DIR/mtp.gguf"
touch "$DIR/$MTP.ok"
}
DOWNLOAD_ONLY=0
[ "${1:-}" = "--download-only" ] && { DOWNLOAD_ONLY=1; shift; }
[ "$DOWNLOAD_ONLY" -eq 1 ] || [ -x ./pantheon-tt ] || die "no ./pantheon-tt; run ./install.sh first"
. scripts/model_fetch.sh
mkdir -p "$DIR"
# Disk: ~10 GB for the GGUF if not here yet, ~10 GB for the card conversion unless
# --download-only or already converted, ~1 GB for the draft head and slack.
need_gb=1
[ -f "$DIR/$GGUF" ] || need_gb=$((need_gb + 10))
[ "$DOWNLOAD_ONLY" -eq 1 ] || [ -f "$DIR/Ornith-1.5-9B-Q8_0.bfp8.fttw" ] || need_gb=$((need_gb + 10))
[ "$need_gb" -gt 1 ] && mf_need_disk "$need_gb"
mf_fetch "$GGUF" "$GGUF_SHA" "$MIRROR" "$UPSTREAM" || die "could not download $GGUF"
mf_fetch "$MTP" "$MTP_SHA" "$MIRROR" || build_mtp ||
echo "run_ornith: WARNING: no MTP draft head; running without speculative decoding"
[ "$DOWNLOAD_ONLY" -eq 1 ] && { echo "run_ornith: files ready in $DIR"; exit 0; }
MTP_ENV=()
mf_verified "$MTP" && MTP_ENV=(PN_TT_MTP="$DIR/$MTP")
echo "run_ornith: starting on ${HOST:-127.0.0.1}:${PORT:-8080} (first start converts for the card)"
exec env FT_TT_DRIVER="${FT_TT_DRIVER:-udrv}" ${MTP_ENV[@]+"${MTP_ENV[@]}"} ./pantheon-tt serve \
--model "$DIR/$GGUF" --served-name ornith --ctx-cap "${CTX_CAP:-131072}" \
--host "${HOST:-127.0.0.1}" --port "${PORT:-8080}" "$@"

115
scripts/add-file-headers.sh Executable file
View File

@ -0,0 +1,115 @@
#!/usr/bin/env bash
# Insert the standard file-header block into sources that lack one.
#
# ./scripts/add-file-headers.sh # dry run — report only
# ./scripts/add-file-headers.sh --apply # write the blocks
# ./scripts/add-file-headers.sh --apply src/core # limit to a subtree
#
# Adds the SPDX pair when missing, then:
#
# /**
# * @file name.c
# * @brief TODO
# *
# * @status None
# * @issues None
# * @todo None
# */
#
# @brief is seeded from the paired header's @brief when there is one, otherwise
# left as TODO — a one-line summary is not mechanically derivable, and a wrong
# one is worse than an absent one. scripts/check-headers.sh rejects TODO briefs
# on files a PR adds.
#
# Exempt: tests, generated sources, third-party-derived files (those keep their
# upstream attribution).
set -euo pipefail
APPLY=0
[ "${1-}" = "--apply" ] && { APPLY=1; shift; }
SCOPE="${1-.}"
SPDX='/* SPDX-License-Identifier: AGPL-3.0-or-later */'
COPYRIGHT='/* Copyright (C) 2026 David (Corey) Herrera */'
is_exempt() {
case "$1" in
tests/*|*/tests/*) return 0 ;; # test files
*_ptx.h|*ptx_modules*) return 0 ;; # generated
*) ;;
esac
# third-party attribution present
head -5 "$1" | grep -qE 'SPDX-FileCopyrightText|Apache-2\.0' && return 0
return 1
}
added=0 todo=0 seeded=0 skipped=0
while IFS= read -r f; do
head -12 "$f" | grep -q '@file' && continue
if is_exempt "$f"; then skipped=$((skipped+1)); continue; fi
base=$(basename "$f")
brief="TODO"
pair="${f%.*}.h"
if [ "$pair" != "$f" ] && [ -f "$pair" ]; then
p=$(grep -m1 '@brief' "$pair" 2>/dev/null | sed 's/.*@brief[[:space:]]*//' || true)
[ -n "$p" ] && brief="Implementation of ${p%.}."
fi
[ "$brief" = "TODO" ] && todo=$((todo+1)) || seeded=$((seeded+1))
added=$((added+1))
[ "$APPLY" = "1" ] || continue
tmp=$(mktemp)
if head -1 "$f" | grep -q 'SPDX-License-Identifier'; then
# SPDX already present — insert the block after the copyright line
head -2 "$f" > "$tmp"
rest=3
else
printf '%s\n%s\n' "$SPDX" "$COPYRIGHT" > "$tmp"
rest=1
fi
{
printf '/**\n'
printf ' * @file %s\n' "$base"
printf ' * @brief %s\n' "$brief"
printf ' *\n'
printf ' * @status None\n'
printf ' * @issues None\n'
printf ' * @todo None\n'
printf ' */\n'
} >> "$tmp"
tail -n "+$rest" "$f" >> "$tmp"
mv "$tmp" "$f"
done < <(git ls-files "$SCOPE" | grep -E '\.(c|h|cu|cuh|m)$')
printf 'headers %s: %d file(s) — %d brief seeded from paired header, %d left TODO; %d exempt\n' \
"$([ "$APPLY" = 1 ] && echo written || echo needed)" "$added" "$seeded" "$todo" "$skipped"
# --- SPDX pass -------------------------------------------------------------
# Independent of the header block: a file can carry @file and still have no SPDX
# line. Only files that are unambiguously ours get stamped. Anything whose header
# cites upstream sources is listed for manual triage instead — a port of an
# Apache-2.0 source needs Apache-2.0 plus the upstream copyright, and that is a
# per-file judgement, not a sed.
spdx_added=0 triage=0
while IFS= read -r f; do
head -3 "$f" | grep -q 'SPDX-License-Identifier' && continue
case "$f" in tests/*|*/tests/*|*_ptx.h) continue ;; esac
if head -20 "$f" | grep -qiE 'tt-metal|tt-kmd|Tenstorrent|transliteration|rendering of|upstream'; then
triage=$((triage+1))
[ "$APPLY" = 1 ] || printf ' triage: %s\n' "$f"
continue
fi
spdx_added=$((spdx_added+1))
[ "$APPLY" = 1 ] || continue
tmp=$(mktemp)
printf '%s\n%s\n' "$SPDX" "$COPYRIGHT" > "$tmp"
cat "$f" >> "$tmp"
mv "$tmp" "$f"
done < <(git ls-files "$SCOPE" | grep -E '\.(c|h|cu|cuh|m)$')
printf 'SPDX %s: %d file(s); %d need manual triage (upstream-derived)\n' \
"$([ "$APPLY" = 1 ] && echo written || echo needed)" "$spdx_added" "$triage"
[ "$APPLY" = 1 ] || printf 'dry run — pass --apply to write\n'

67
scripts/assert_backend.sh Executable file
View File

@ -0,0 +1,67 @@
#!/bin/sh
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
#
# Refuse to deploy a Pantheon binary whose COMPILED backend is not the one the
# fleet expects. One line to call, exit 0 or exit 1.
#
# scripts/assert_backend.sh <binary> <expected-backend>
# scripts/assert_backend.sh /opt/pantheon/pantheon cuda
#
# Why this exists: `make`, `make USE_CUDA=1` and `make USE_TT=1` used to write
# the same file, so whichever ran last owned the deployed path. A CPU-only
# binary served a 35B-A3B MoE on a 2-core no-AVX Pentium with all 8 K80 dies
# idle at 0 MiB for an unknown period, and a deploy script launched it onto a GPU
# fleet manifest and reported success. The binary knew its own backend the whole
# time — nothing asked it.
#
# The binaries are now per-backend (pantheon-cuda / pantheon-tt / pantheon-cpu)
# and `pantheon` is a symlink, so this checks what the symlink RESOLVES TO by
# asking the binary itself rather than trusting the name. `--backend` prints one
# lowercase token (cuda|tt|metal|cpu) on stdout and exits 0; it loads no model
# and opens no device, so it is free to run on every deploy.
#
# A binary too old to know the flag fails CLOSED: it is by definition from
# before per-backend names existed, which is exactly the era this guards.
set -eu
BIN="${1:-}"
WANT="${2:-}"
if [ -z "$BIN" ] || [ -z "$WANT" ]; then
echo "usage: $0 <binary> <expected-backend> e.g. $0 ./pantheon cuda" >&2
exit 2
fi
if [ ! -x "$BIN" ]; then
echo "assert_backend: '$BIN' is not an executable" >&2
exit 1
fi
GOT=$("$BIN" --backend 2>/dev/null) || GOT=""
if [ -z "$GOT" ]; then
echo "assert_backend: '$BIN' does not support --backend — it predates" >&2
echo " per-backend build names and cannot be trusted to be a $WANT build." >&2
echo " Rebuild: make USE_CUDA=1 (produces ./pantheon-cuda)" >&2
exit 1
fi
if [ "$GOT" != "$WANT" ]; then
echo "assert_backend: REFUSING to deploy '$BIN' — it is a '$GOT' build, the" >&2
echo " fleet expects '$WANT'. A '$GOT' binary would load the models and" >&2
echo " answer every request on the host CPU with every die idle." >&2
echo " Fix: point the deploy path at the $WANT binary —" >&2
echo " ln -sfn pantheon-$WANT pantheon (no rebuild needed if it exists)" >&2
if [ "$WANT" = cpu ]; then
echo " or build one: make (a plain make IS the CPU build)" >&2
else
# The flag is the token uppercased: cuda -> USE_CUDA=1, tt -> USE_TT=1.
echo " or build one: make USE_$(printf '%s' "$WANT" | tr 'a-z' 'A-Z')=1" >&2
fi
exit 1
fi
echo "assert_backend: '$BIN' is a '$GOT' build ($("$BIN" --version | tail -1))"
exit 0

68
scripts/check-cla.sh Executable file
View File

@ -0,0 +1,68 @@
#!/bin/sh
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
#
# Verify every commit author in a range appears on the CLA signature roster.
# See CLA.md (individuals) and CLA-CORPORATE.md (organizations).
#
# scripts/check-cla.sh [range] # default: origin/main..HEAD
#
# Run it locally before opening a pull request; CI runs the same script.
set -eu
cd "$(git rev-parse --show-toplevel)"
RANGE="${1:-${BASE_REF:-origin/main}..HEAD}"
ROSTER=".cla/signatures.md"
if [ ! -f "$ROSTER" ]; then
echo "check-cla: $ROSTER is missing" >&2
exit 1
fi
# Signed addresses: every `git-email:` value that is not inside a comment.
signed=$(grep -v '^[[:space:]]*#' "$ROSTER" |
sed -n 's/^[[:space:]]*-\{0,1\}[[:space:]]*git-email:[[:space:]]*//p' |
tr -d '\r' | tr '[:upper:]' '[:lower:]' | sort -u)
authors=$(git log --no-merges --format='%ae' "$RANGE" |
tr '[:upper:]' '[:lower:]' | sort -u)
if [ -z "$authors" ]; then
echo "check-cla: no commits in $RANGE — nothing to check"
exit 0
fi
fail=0
for email in $authors; do
if printf '%s\n' "$signed" | grep -qxF "$email"; then
echo "ok $email"
else
echo "FAIL $email not on the CLA roster"
fail=1
fi
done
if [ "$fail" -ne 0 ]; then
cat <<'EOF'
Contributions are accepted under the Contributor License Agreement in CLA.md.
Sign it by adding your entry to .cla/signatures.md in this pull request:
- name: Your Full Name
git-email: you@example.com
forge-username: your-username
date: YYYY-MM-DD
agreement: ICLA v1.0
statement: I have read the Pantheon ICLA and I agree to it.
Use the same address you commit with. If you commit under more than one
address, list each on its own git-email: line. Organizations contributing on
behalf of employees sign CLA-CORPORATE.md by email instead.
EOF
exit 1
fi
echo "check-cla: all authors have signed"

66
scripts/check-dco.sh Executable file
View File

@ -0,0 +1,66 @@
#!/bin/sh
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
#
# Verify every commit in a range carries a Developer Certificate of Origin
# sign-off matching its author. See CONTRIBUTING.md.
#
# scripts/check-dco.sh [range] # default: origin/main..HEAD
#
# Run it locally before opening a pull request; CI runs the same script.
set -eu
cd "$(git rev-parse --show-toplevel)"
RANGE="${1:-${BASE_REF:-origin/main}..HEAD}"
commits=$(git rev-list --no-merges "$RANGE")
if [ -z "$commits" ]; then
echo "check-dco: no commits in $RANGE — nothing to check"
exit 0
fi
fail=0
for sha in $commits; do
author_email=$(git show -s --format='%ae' "$sha")
author_name=$(git show -s --format='%an' "$sha")
subject=$(git show -s --format='%s' "$sha")
signoffs=$(git show -s --format='%B' "$sha" |
sed -n 's/^[Ss]igned-off-by:[[:space:]]*//p' |
tr '[:upper:]' '[:lower:]')
if [ -z "$signoffs" ]; then
echo "FAIL $sha $subject"
echo " no Signed-off-by line"
fail=1
continue
fi
want=$(printf '%s' "$author_email" | tr '[:upper:]' '[:lower:]')
if printf '%s\n' "$signoffs" | grep -qF "<$want>"; then
echo "ok $sha $author_name <$author_email>"
else
echo "FAIL $sha $subject"
echo " sign-off does not match the author <$author_email>"
printf ' found: %s\n' "$signoffs"
fail=1
fi
done
if [ "$fail" -ne 0 ]; then
cat <<'EOF'
Every commit needs a Developer Certificate of Origin sign-off matching its
author -- see https://developercertificate.org/ and CONTRIBUTING.md.
git commit -s -m "..." new commits
git commit --amend -s --no-edit the most recent commit
git rebase --signoff <base> a whole branch
EOF
exit 1
fi
echo "check-dco: all commits signed off"

98
scripts/check-headers.sh Executable file
View File

@ -0,0 +1,98 @@
#!/bin/sh
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
#
# Verify every source file ADDED in a range carries the SPDX pair and the file
# header block. See docs/coding-best-practices.md.
#
# scripts/check-headers.sh [range] # default: origin/main..HEAD
#
# Only files the range ADDS are checked. Files that already exist are left
# alone deliberately: the rule is that everyone submitting code stays in
# compliance, not that the whole tree gets retrofitted.
#
# Exempt: tests, generated sources, and files carrying upstream third-party
# attribution (those keep their own header).
#
# Run it locally before opening a pull request; CI runs the same script.
set -eu
cd "$(git rev-parse --show-toplevel)"
RANGE="${1:-${BASE_REF:-origin/main}..HEAD}"
files=$(git diff --diff-filter=A --name-only "$RANGE" -- \
'*.c' '*.h' '*.cu' '*.cuh' '*.m')
if [ -z "$files" ]; then
echo "check-headers: ok (no source files added)"
exit 0
fi
fail=0
note() {
echo "check-headers: $1" >&2
echo " $2" >&2
fail=$((fail + 1))
}
OLDIFS=$IFS
IFS='
'
for f in $files; do
IFS=$OLDIFS
# Added and then removed again within the range.
[ -f "$f" ] || continue
case "$f" in
tests/*|*/tests/*) continue ;;
*_ptx.h|*_ptx.cuh) continue ;;
esac
# Third-party-derived files keep their upstream attribution.
if head -5 "$f" | grep -qE 'SPDX-FileCopyrightText|Apache-2\.0'; then
continue
fi
if ! head -3 "$f" | grep -q 'SPDX-License-Identifier'; then
note "$f: missing the SPDX identifier" \
"add: /* SPDX-License-Identifier: AGPL-3.0-or-later */"
continue
fi
if ! head -16 "$f" | grep -q '@file'; then
note "$f: missing the @file header block" \
"run: ./scripts/add-file-headers.sh --apply $f"
continue
fi
brief=$(head -16 "$f" | sed -n 's|.*@brief[[:space:]]*||p' | head -1)
case "$brief" in
'')
note "$f: @brief is empty" \
"one line saying what this file is for"
;;
TODO*|todo*)
note "$f: @brief is still TODO" \
"replace it with one line saying what this file is for"
;;
esac
for tag in @status @issues @todo; do
if ! head -16 "$f" | grep -q -- "$tag"; then
note "$f: header block is missing $tag" \
"all three of @status/@issues/@todo are present, 'None' when empty"
fi
done
done
IFS=$OLDIFS
if [ "$fail" -gt 0 ]; then
echo "check-headers: $fail problem(s) — see docs/coding-best-practices.md" >&2
exit 1
fi
echo "check-headers: ok"

43
scripts/k80_serve.sh Normal file
View File

@ -0,0 +1,43 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
#
# Shared body of the Tesla K80 run scripts (k80_qwen36.sh, k80_nexn2.sh). Sourced, not run.
# The caller sets NAME, GGUF, GGUF_SHA, GGUF_GB, MIRROR, UPSTREAM, DIR, CTX_CAP, MAX_SESSIONS,
# REASONING_BUDGET and PORT, then calls `k80_main "$@"`.
#
# One model takes both dies of one K80 (DIES, default 0,1). Prefix caching is on: the serve
# keeps MAX_SESSIONS session slots, and a client that sends `session_id` (or X-Session-Id) and
# replays each assistant turn's `reasoning_content` only prefills the new tokens on later turns.
die() { echo "$NAME: $*" >&2; exit 1; }
# The flags the K80 fleet runs with (measured settings, not defaults).
k80_env() {
K80_ENV=(PN_THREADS=2 PN_DIE_RESERVE_BYTES=512M PANTHEON_TOOL_GRAMMAR=1
QWEN35_KV_Q4=1 QWEN35_BATCH_SLOTS=1 QWEN35_VOCAB_SPLIT=1
QWEN35_VOCAB_SPLIT_FULL_LOGITS=1 QWEN35_PREFILL_PIPELINE=1 QWEN35_MOE_DEDUP=1
QWEN35_SSM_WARP=1 PN_SPARSE_SAMPLER=1 PN_GRAMMAR_ARM_WINDOW=1
FC_MOE_DEDUP_SHX=1 FC_MOE_DEDUP_COMPACT=1 FC_ATTN_KTILE_REG=1 FC_ATTN_VSTAGE=1)
}
k80_main() {
local download_only=0
[ "${1:-}" = "--download-only" ] && { download_only=1; shift; }
if [ "$download_only" -eq 0 ]; then
[ -x ./pantheon-cuda ] || die "no ./pantheon-cuda; run ./install.sh --k80 first"
command -v nvidia-smi > /dev/null && nvidia-smi -L > /dev/null 2>&1 ||
die "no NVIDIA driver/GPU visible (nvidia-smi -L); a K80 needs the 470 driver"
fi
. scripts/model_fetch.sh
mkdir -p "$DIR"
[ -f "$DIR/$GGUF" ] || mf_need_disk "$((GGUF_GB + 1))"
mf_fetch "$GGUF" "$GGUF_SHA" "$MIRROR" "$UPSTREAM" || die "could not download $GGUF"
[ "$download_only" -eq 1 ] && { echo "$NAME: files ready in $DIR"; exit 0; }
k80_env
echo "$NAME: starting on ${HOST:-127.0.0.1}:$PORT, dies ${DIES:-0,1} (loading takes a few minutes)"
exec env "${K80_ENV[@]}" ./pantheon-cuda serve \
--model "$DIR/$GGUF" --name "$NAME" --dies "${DIES:-0,1}" \
--ctx-cap "$CTX_CAP" --max-sessions "$MAX_SESSIONS" --reasoning-budget "$REASONING_BUDGET" \
--host "${HOST:-127.0.0.1}" --port "$PORT" "$@"
}

52
scripts/model_fetch.sh Normal file
View File

@ -0,0 +1,52 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 David (Corey) Herrera
#
# Download-and-check helpers for the model run scripts (run_ornith.sh, k80_*.sh). Sourced,
# not run. The caller sets DIR (where model files live) and defines die().
#
# A file is good once its sha256 matched (or it was built locally); a `<file>.ok` marker
# records that so later runs don't re-hash many GB. Downloads resume into `<file>.part` and
# move into place only after the hash matches. HF_TOKEN, if set, is sent to Hugging Face.
# mf_verified <file>: already checked.
mf_verified() {
[ -f "$DIR/$1" ] && [ -f "$DIR/$1.ok" ]
}
# mf_verify <file> <sha256>: hash it; keep and mark it only if it matches.
mf_verify() {
echo "fetch: checking $1"
[ "$(sha256sum "$DIR/$1" | cut -d' ' -f1)" = "$2" ] || { rm -f "$DIR/$1"; return 1; }
touch "$DIR/$1.ok"
}
# mf_download <base-url> <file> <sha256>
mf_download() {
local auth=()
echo "fetch: downloading $2"
[ -n "${HF_TOKEN:-}" ] && auth=(-H "Authorization: Bearer $HF_TOKEN")
curl -fL --retry 5 -C - ${auth[@]+"${auth[@]}"} -o "$DIR/$2.part" "$1/$2" || return 1
mv "$DIR/$2.part" "$DIR/$2"
mf_verify "$2" "$3"
}
# mf_fetch <file> <sha256> <mirror-url> [upstream-url]: make <file> present and checked.
# An upstream URL is tried when the mirror fails. Returns 1 if every source failed.
mf_fetch() {
mf_verified "$1" && return 0
[ -f "$DIR/$1" ] && mf_verify "$1" "$2" && return 0
mf_download "$3" "$1" "$2" && return 0
rm -f "$DIR/$1.part"
[ -n "${4:-}" ] || return 1
echo "fetch: mirror failed, trying $4"
mf_download "$4" "$1" "$2" && return 0
rm -f "$DIR/$1.part"
return 1
}
# mf_need_disk <gb>: die unless DIR has <gb> free.
mf_need_disk() {
local free_gb
free_gb=$(df -Pk "$DIR" | awk 'NR == 2 { print int($4 / 1048576) }')
[ "$free_gb" -ge "$1" ] || die "$DIR has ${free_gb} GB free; this needs about $1 GB (set the model dir)"
}

95
src/accel/backend_identity.c Normal file
View File

@ -0,0 +1,95 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file backend_identity.c
* @brief Implementation of the compile-time backend identity (see
* backend_identity.h).
*
* Everything here is decided by the preprocessor and returned as a string
* literal. That is deliberate: the whole value of this module is that it CANNOT
* disagree with the code that was compiled. A runtime probe ("is there a CUDA
* driver?") answers a different question and is exactly the one that was
* already misleading — the incident binary had no CUDA compiled in at all, so
* no probe would have run and no probe would have complained.
*
* The combination arms (CUDA+TT etc.) exist because the Makefile permits
* `make USE_CUDA=1 USE_TT=1`. They are enumerated rather than assembled at
* runtime so every value is a literal and the accessors stay signal-safe.
*
* @status Complete.
* @issues None
* @todo None
*/
#include "backend_identity.h"
/* The Makefile stamps the build commit in. A build with no git (a tarball, a
* copied tree) still compiles — it just reports "unknown" rather than lying. */
#ifndef PN_BUILD_COMMIT
#define PN_BUILD_COMMIT "unknown"
#endif
/* One enumeration, two spellings. Written out per combination instead of
* concatenated at runtime so both accessors return literals. */
#if defined(USE_CUDA)
# if defined(USE_TT)
# define PN_BACKEND_NAME_STR "CUDA+TT"
# define PN_BACKEND_TOKEN_STR "cuda+tt"
# elif defined(USE_METAL)
# define PN_BACKEND_NAME_STR "CUDA+METAL"
# define PN_BACKEND_TOKEN_STR "cuda+metal"
# else
# define PN_BACKEND_NAME_STR "CUDA"
# define PN_BACKEND_TOKEN_STR "cuda"
# endif
#elif defined(USE_TT)
# if defined(USE_METAL)
# define PN_BACKEND_NAME_STR "TT+METAL"
# define PN_BACKEND_TOKEN_STR "tt+metal"
# else
# define PN_BACKEND_NAME_STR "TT"
# define PN_BACKEND_TOKEN_STR "tt"
# endif
#elif defined(USE_METAL)
# define PN_BACKEND_NAME_STR "METAL"
# define PN_BACKEND_TOKEN_STR "metal"
#else
# define PN_BACKEND_NAME_STR "CPU"
# define PN_BACKEND_TOKEN_STR "cpu"
#endif
const char *pn_backend_name(void)
{
return PN_BACKEND_NAME_STR;
}
const char *pn_backend_token(void)
{
return PN_BACKEND_TOKEN_STR;
}
const char *pn_backend_build_commit(void)
{
return PN_BUILD_COMMIT;
}
bool pn_backend_has_accelerator(void)
{
#if defined(USE_CUDA) || defined(USE_TT) || defined(USE_METAL)
return true;
#else
return false;
#endif
}
bool pn_backend_supports_die_placement(void)
{
/* Metal is deliberately excluded: die_manager.h has pn_dm_init_cuda and
* pn_dm_init_tt and nothing else, so on a Metal build `--dies 0` names an
* index no init path can ever register. */
#if defined(USE_CUDA) || defined(USE_TT)
return true;
#else
return false;
#endif
}

102
src/accel/backend_identity.h Normal file
View File

@ -0,0 +1,102 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file backend_identity.h
* @brief What accelerator backend THIS binary was compiled with, as a value the
* process can print. Compile-time constants only — no probing, no
* driver, no allocation, safe to call from anywhere at any time.
*
* Why this exists: `make`, `make USE_CUDA=1` and `make USE_TT=1` are
* three different programs, and until the per-backend output names
* landed they all wrote the same file. A CPU-only binary sitting at the
* deployed path served a 35B MoE on a 2-core host with all 8 K80 dies
* idle for an unknown period, and answering "what am I running?" took
* `ldd` and `strings` on a remote box. The backend is a compile-time
* constant; the process should simply say so — in its startup log, on
* /v1/models, on the health route, and through `pantheon --backend` so a
* deploy script can assert on it before launching anything.
*
* pn_backend_name() is the human string ("CUDA"), pn_backend_token() the
* machine one ("cuda") — a deploy script compares the token, a log line
* prints the name. They are the same enumeration, never derived from
* each other at runtime, so neither can drift into a lie.
*
* @status Complete. CUDA/TT/METAL arms are compile-time only, so they are
* proved by the preprocessor, not by a hardware run.
* @issues None
* @todo None
*/
#ifndef PN_BACKEND_IDENTITY_H
#define PN_BACKEND_IDENTITY_H
#include <stdbool.h>
#ifdef __cplusplus
extern "C" {
#endif
/**
* Human-readable name of the compiled accelerator backend: "CUDA", "TT",
* "METAL", "CUDA+TT", ... or "CPU" when none is compiled in.
*
* @return A string literal — never NULL, never freed.
*
* Thread-safe: yes (returns a literal)
* Reentrant: yes
* Signal-safe: yes
*/
const char *pn_backend_name(void);
/**
* Machine-readable form of the same value, lowercase and shell-comparable:
* "cuda", "tt", "metal", "cuda+tt", "cpu". This is what `pantheon --backend`
* prints and what a deploy script asserts against.
*
* @return A string literal — never NULL, never freed.
*
* Thread-safe: yes (returns a literal)
* Reentrant: yes
* Signal-safe: yes
*/
const char *pn_backend_token(void);
/**
* The commit this binary was built from (short hash, with a "-dirty" suffix
* when the tree had uncommitted changes), or "unknown" when the build had no
* git available. Stamped in by the Makefile as -DPN_BUILD_COMMIT.
*
* @return A string literal — never NULL, never freed.
*
* Thread-safe: yes (returns a literal)
* Reentrant: yes
* Signal-safe: yes
*/
const char *pn_backend_build_commit(void);
/**
* Whether ANY GPU/accelerator backend is compiled in. False means every forward
* pass in this process runs on the host CPU, whatever the flags say.
*
* Thread-safe: yes Reentrant: yes Signal-safe: yes
*/
bool pn_backend_has_accelerator(void);
/**
* Whether this binary carries a backend the DIE MANAGER can probe — i.e. one
* for which `--dies` means anything. CUDA and Tenstorrent have pn_dm_init_*
* paths; Metal does not (it is one implicit device, never placed by index), so
* a Metal build answers false here and true for pn_backend_has_accelerator().
*
* This is the predicate the serve's refuse-to-start guard asks, because the
* question there is not "is there a GPU" but "can a die index be honoured".
*
* Thread-safe: yes Reentrant: yes Signal-safe: yes
*/
bool pn_backend_supports_die_placement(void);
#ifdef __cplusplus
}
#endif
#endif /* PN_BACKEND_IDENTITY_H */

69
src/accel/die_layer_split.c Normal file
View File

@ -0,0 +1,69 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file die_layer_split.c
* @brief Implementation of the shared N-die layer partitioner.
*
* @status Complete.
* @issues None
* @todo None
*/
#include "die_layer_split.h"
#include <stddef.h>
/* The boundary before die `idx` of an n_die-way split of n_layer layers.
* Computed in `long` because n_layer * idx overflows int for absurd inputs
* long before the caller's validation would notice, and a wrapped boundary is
* a silently-wrong split rather than a crash. */
static int split_boundary(int n_layer, int n_die, int idx)
{
return (int)((long)n_layer * (long)idx / (long)n_die);
}
/* Shared validation for both entry points. */
static int split_args_ok(int n_layer, int n_die)
{
return n_layer > 0 && n_die >= 1 &&
n_die <= PN_DIE_LAYER_SPLIT_MAX_DIES && n_die <= n_layer;
}
int pn_plan_die_layer_split(int n_layer, int n_die, int override_boundary,
pn_die_layer_split *out)
{
if (!out || !split_args_ok(n_layer, n_die))
return -1;
for (int i = 0; i < n_die; i++) {
out->layer_lo[i] = split_boundary(n_layer, n_die, i);
out->layer_hi[i] = (i == n_die - 1) ? n_layer
: split_boundary(n_layer, n_die, i + 1);
}
/* A single boundary describes a 2-die split completely and nothing wider,
* so the override applies at exactly that width. Clamped inside (0,n_layer)
* so an out-of-range knob narrows the split rather than emptying a die. */
if (n_die == 2 && override_boundary > 0) {
int b = override_boundary;
if (b >= n_layer)
b = n_layer - 1;
out->layer_hi[0] = b;
out->layer_lo[1] = b;
}
out->n_die = n_die;
return 0;
}
int pn_die_layer_range(int n_layer, int n_die, int idx, int *lo_out,
int *hi_out)
{
if (!split_args_ok(n_layer, n_die) || idx < 0 || idx >= n_die)
return -1;
if (lo_out)
*lo_out = split_boundary(n_layer, n_die, idx);
if (hi_out)
*hi_out = (idx == n_die - 1) ? n_layer
: split_boundary(n_layer, n_die, idx + 1);
return 0;
}

100
src/accel/die_layer_split.h Normal file
View File

@ -0,0 +1,100 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file die_layer_split.h
* @brief How a model's transformer layers divide across an N-die placement.
* ONE definition of the layer ranges, shared by the pre-load admission
* projection and the GPU forward that actually uploads them.
*
* The two used to derive the ranges separately — model_registry.c projected
* `n_layer*idx/n_placed` per die while qwen35moe_forward_cuda.c split at
* `n_layer/2` regardless of how many dies were placed. They agreed only at
* n_placed == 2, so a `--dies 5,6,7` placement bound three dies, uploaded a
* two-way split, under-projected each die's share by a third, and OOM'd. A
* projection that disagrees with placement is the bug class this module exists
* to make unrepresentable: admission and the forward call the SAME function.
*
* Quant-agnostic by construction: this module partitions LAYER INDICES. It
* never sees a tensor, a byte count or a quantization type, so the same split
* carries to Q3_K, Q4_K, Q6_K and F16 unchanged.
*
* @status Complete. Ranges, the N=2 compatibility guarantee and the refusal
* boundaries are covered by tests/test_die_layer_split.c.
* @issues None
* @todo None
*/
#ifndef PN_DIE_LAYER_SPLIT_H
#define PN_DIE_LAYER_SPLIT_H
/* Widest layer split this partitioner will plan. The die manager and the
* memwatch state carry 16 dies (PN_MEM_MAX_DIES), and PN_MODEL_MAX_DIES caps
* what one model may be PLACED on; this is the split planner's own ceiling and
* is deliberately the same number, so "the placement fit" and "the split fit"
* cannot disagree. Widths above it are refused, never silently narrowed. */
#define PN_DIE_LAYER_SPLIT_MAX_DIES 16
/* A planned N-way layer split. Die i owns global layers [layer_lo[i],
* layer_hi[i]); the ranges are contiguous, non-empty, cover [0, n_layer)
* exactly, and differ in length by at most one. The LM head and final norm
* ride on die n_die-1 (the last), which is a convention this header states
* rather than a field: nothing here can enforce what the uploader does with
* the head, so both sides name the same rule. */
typedef struct {
int n_die;
int layer_lo[PN_DIE_LAYER_SPLIT_MAX_DIES];
int layer_hi[PN_DIE_LAYER_SPLIT_MAX_DIES];
} pn_die_layer_split;
/**
* Plans how `n_layer` layers divide across `n_die` dies.
*
* The boundary for die i is `n_layer * i / n_die` in integer arithmetic, which
* is the flattest split available and — the property the 2-die path depends on
* — reduces to exactly `n_layer / 2` at n_die == 2, the boundary the CUDA
* forward has always used. So a 2-die placement plans byte-identically to the
* hardcoded pair it replaces, for even and odd layer counts alike. Any
* remainder lands on the LATER dies, which is what makes that identity hold
* (41 layers over 2 dies is 20 + 21, not 21 + 20).
*
* @param n_layer Total transformer layers (> 0)
* @param n_die Dies to split across (1 .. PN_DIE_LAYER_SPLIT_MAX_DIES)
* @param override_boundary Manual first boundary, or <= 0 for none. Honoured
* ONLY at n_die == 2 (where a single boundary is the
* whole plan); IGNORED for every other width, because
* one number cannot describe N-1 boundaries. Clamped
* to [1, n_layer-1] so it can never empty a die.
* @param out Receives the plan. Untouched on refusal.
* @return 0 on success, -1 if refused — n_layer <= 0, n_die < 1, n_die above
* PN_DIE_LAYER_SPLIT_MAX_DIES, or n_die > n_layer (a die with no layers
* is not a narrower split, it is a bug). Callers must REFUSE on -1
* rather than fall back to a narrower split: mis-splitting silently is
* the failure this replaces.
*
* Thread-safe: yes (pure)
* Reentrant: yes
*/
int pn_plan_die_layer_split(int n_layer, int n_die, int override_boundary,
pn_die_layer_split *out);
/**
* The layer range for ONE die of an N-way split, without materializing the
* whole plan — the shape admission wants when it is pricing die `idx` alone.
* Identical arithmetic to pn_plan_die_layer_split; it is the same function
* viewed through one index.
*
* @param n_layer Total transformer layers (> 0)
* @param n_die Dies to split across (1 .. PN_DIE_LAYER_SPLIT_MAX_DIES)
* @param idx Which die (0 .. n_die-1)
* @param lo_out Receives the first global layer on this die (may be NULL)
* @param hi_out Receives one past the last (may be NULL)
* @return 0 on success, -1 on the same refusals as pn_plan_die_layer_split
* (plus an out-of-range idx). Outputs untouched on refusal.
*
* Thread-safe: yes (pure)
* Reentrant: yes
*/
int pn_die_layer_range(int n_layer, int n_die, int idx, int *lo_out,
int *hi_out);
#endif /* PN_DIE_LAYER_SPLIT_H */

441
src/accel/die_manager.c Normal file
View File

@ -0,0 +1,441 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file die_manager.c
* @brief Implementation of the die manager (see die_manager.h). One
* process-global table (dies + bindings) behind a mutex, mirroring the
* singleton style of pn_memwatch. Placement policy detail lives in
* pn_dm_place below; everything else is straight accounting.
*
* @status Logical path tested on the Mac; CUDA and TT probes gated on GPU hardware.
* @issues None
* @todo None
*/
#include "die_manager.h"
#include <pthread.h>
#include <stdio.h>
#include <string.h>
#ifdef USE_CUDA
#include "cuda/fc_cuda.h"
#include "../core/mem/pn_memwatch.h"
#endif
#ifdef USE_TT
#include "tt/ft_tt_coord.h"
#include "tt/ft_tt_device.h"
#include "tt/ft_tt_runtime.h"
#include "../core/mem/pn_memwatch.h"
#endif
/* One resident model's reservation: which dies it holds and how many bytes it
* put on each (total split evenly across the set). */
typedef struct {
bool used;
char name[PN_DM_NAME_MAX];
int dies[PN_DM_MAX_DIES];
int n_dies;
uint64_t bytes_per_die;
} dm_binding;
typedef struct {
bool present;
uint64_t vram_budget;
uint64_t vram_used;
} dm_die;
static struct {
pthread_mutex_t lock;
bool running;
int n_dies;
dm_die die[PN_DM_MAX_DIES];
dm_binding binding[PN_DM_MAX_BINDINGS];
} g_dm = { .lock = PTHREAD_MUTEX_INITIALIZER };
/* --- init / shutdown --------------------------------------------------- */
int pn_dm_init_logical(int n_dies, uint64_t budget_bytes)
{
if (n_dies <= 0 || n_dies > PN_DM_MAX_DIES || budget_bytes == 0)
return -1;
pthread_mutex_lock(&g_dm.lock);
if (g_dm.running) { /* idempotent while running */
pthread_mutex_unlock(&g_dm.lock);
return 0;
}
memset(g_dm.die, 0, sizeof(g_dm.die));
memset(g_dm.binding, 0, sizeof(g_dm.binding));
g_dm.n_dies = n_dies;
for (int i = 0; i < n_dies; i++) {
g_dm.die[i].present = true;
g_dm.die[i].vram_budget = budget_bytes;
}
g_dm.running = true;
pthread_mutex_unlock(&g_dm.lock);
return 0;
}
#ifdef USE_CUDA
int pn_dm_init_cuda(int n_dies)
{
if (n_dies <= 0 || n_dies > PN_DM_MAX_DIES)
return -1;
/* Driver-only gate on purpose: this runs at serve startup, before any
* model has a placement, so it must not be the thing that forces a
* process-default context onto whatever die happens to be ordinal 0. */
if (!fc_driver_available())
return -1;
pthread_mutex_lock(&g_dm.lock);
if (g_dm.running) {
int n = g_dm.n_dies;
pthread_mutex_unlock(&g_dm.lock);
return n;
}
memset(g_dm.die, 0, sizeof(g_dm.die));
memset(g_dm.binding, 0, sizeof(g_dm.binding));
int found = 0;
uint64_t reserve = pn_die_reserve_bytes();
for (int i = 0; i < n_dies; i++) {
/* Transient bare context per die — created, read, destroyed. Probing
* with fc_cuda_init_device() stranded a full context (cuBLAS + 12 JIT'd
* PTX modules, ~124 MiB) on EVERY die the serve enumerated, including
* dies owned by another serve. A probe must leave no trace. */
size_t die_free = 0, die_total = 0;
if (fc_cuda_probe_device(i, &die_free, &die_total) != 0 || die_total == 0)
break; /* ordinals are contiguous — stop */
uint64_t budget = die_total > reserve ? die_total - reserve : 0;
g_dm.die[i].present = true;
g_dm.die[i].vram_budget = budget;
/* Anything already gone from `free` beyond the reserve is occupancy we
* did not create (another process / stranded VRAM) — count it so
* placement doesn't over-commit the die. */
uint64_t external = (die_total - die_free) > reserve
? (die_total - die_free) - reserve : 0;
g_dm.die[i].vram_used = external < budget ? external : budget;
/* 0 bytes ours: the probe context is already destroyed, and no model
* has been placed on this die yet. */
if (pn_memwatch_running())
pn_memwatch_set_die(i, die_free, budget, 0);
found++;
}
if (found == 0) {
pthread_mutex_unlock(&g_dm.lock);
return -1;
}
g_dm.n_dies = found;
g_dm.running = true;
pthread_mutex_unlock(&g_dm.lock);
fprintf(stderr, "die_manager: tracking %d CUDA die%s\n",
found, found == 1 ? "" : "s");
return found;
}
#endif /* USE_CUDA */
#ifdef USE_TT
bool pn_dm_tt_driver_explicit(void)
{
ft_tt_runtime_driver driver = FT_TT_RUNTIME_DRIVER_KMD;
bool explicit_choice = false;
/* A REFUSED value (neither kmd nor udrv) counts as explicit: somebody asked
* for a transport and did not get one, which must not read as "no card". */
if (ft_tt_runtime_driver_select(&driver, &explicit_choice) != FT_TT_OK)
return true;
return explicit_choice;
}
/* Capacity of one Blackhole device, in bytes, or 0 if it cannot be measured.
*
* The probe leaves no trace: it opens the device, reads the grid and closes
* again. That matters for the same reason it did on CUDA — this runs at serve
* startup, before any model has a placement, and a probe that held a context
* or a TLB window would be holding it against whichever job takes the card.
*
* It goes through ft_tt_runtime_open() rather than ft_tt_init_device() so it
* probes over WHICHEVER TRANSPORT the process is configured for. Opening the
* chardev here would have refused a machine with no vendor module before the
* serve ever reached the seam — which is exactly the defect this round exists
* to remove, one layer up from where it was first seen.
*
* The capacity is NOT a constant and is deliberately not defaulted. Bank count
* comes from ARC telemetry and bank size from the architecture table selected
* at open; ft_tt_runtime_dram_bytes() answers 0 rather than reporting the
* nominal eight banks as if it had measured them, so a telemetry failure ends
* the scan instead of registering a die with an invented budget. */
static uint64_t tt_probe_device_bytes(int device_index)
{
ft_tt_runtime runtime;
uint64_t total;
if (ft_tt_runtime_open(device_index, &runtime) != FT_TT_OK)
return 0;
/* ft_tt_runtime_open() already refuses a topology ft_tt_coord cannot map,
* on both transports, so a card we would not dare address never gets here
* (spec §7.3). */
total = ft_tt_runtime_dram_bytes(&runtime);
ft_tt_runtime_close(&runtime);
return total;
}
int pn_dm_init_tt(int n_devices)
{
if (n_devices <= 0 || n_devices > PN_DM_MAX_DIES)
return -1;
pthread_mutex_lock(&g_dm.lock);
if (g_dm.running) { /* idempotent while running */
int n = g_dm.n_dies;
pthread_mutex_unlock(&g_dm.lock);
return n;
}
memset(g_dm.die, 0, sizeof(g_dm.die));
memset(g_dm.binding, 0, sizeof(g_dm.binding));
int found = 0;
uint64_t reserve = pn_die_reserve_bytes();
for (int i = 0; i < n_devices; i++) {
uint64_t total = tt_probe_device_bytes(i);
if (total == 0)
break; /* ordinals are contiguous — stop */
g_dm.die[found].present = true;
g_dm.die[found].vram_budget = total > reserve ? total - reserve : 0;
/* No external-occupancy term, unlike the CUDA probe: the card exposes
* no cross-process allocation view, so `free` from a fresh context is
* always `total`. A second process holding DRAM is invisible here and
* is a reason to run one serve per card, not a number to fake. */
g_dm.die[found].vram_used = 0;
if (pn_memwatch_running())
pn_memwatch_set_die(found, total, g_dm.die[found].vram_budget, 0);
found++;
}
if (found == 0) {
pthread_mutex_unlock(&g_dm.lock);
return -1;
}
g_dm.n_dies = found;
g_dm.running = true;
pthread_mutex_unlock(&g_dm.lock);
fprintf(stderr, "die_manager: tracking %d Tenstorrent device%s (one die each)\n",
found, found == 1 ? "" : "s");
return found;
}
#endif /* USE_TT */
void pn_dm_shutdown(void)
{
pthread_mutex_lock(&g_dm.lock);
g_dm.running = false;
g_dm.n_dies = 0;
memset(g_dm.die, 0, sizeof(g_dm.die));
memset(g_dm.binding, 0, sizeof(g_dm.binding));
pthread_mutex_unlock(&g_dm.lock);
}
/* --- queries ------------------------------------------------------------ */
int pn_dm_count(void)
{
pthread_mutex_lock(&g_dm.lock);
int n = g_dm.running ? g_dm.n_dies : 0;
pthread_mutex_unlock(&g_dm.lock);
return n;
}
/* Locked helper: does `b` cover die `id`? */
static bool binding_has_die(const dm_binding *b, int id)
{
for (int i = 0; i < b->n_dies; i++)
if (b->dies[i] == id)
return true;
return false;
}
int pn_dm_get(int die_id, pn_dm_die *out)
{
if (!out)
return -1;
pthread_mutex_lock(&g_dm.lock);
if (!g_dm.running || die_id < 0 || die_id >= g_dm.n_dies ||
!g_dm.die[die_id].present) {
pthread_mutex_unlock(&g_dm.lock);
return -1;
}
memset(out, 0, sizeof(*out));
out->id = die_id;
out->vram_budget = g_dm.die[die_id].vram_budget;
out->vram_used = g_dm.die[die_id].vram_used;
for (int i = 0; i < PN_DM_MAX_BINDINGS; i++) {
if (!g_dm.binding[i].used || !binding_has_die(&g_dm.binding[i], die_id))
continue;
if (out->n_models == 0) {
strncpy(out->model_name, g_dm.binding[i].name,
PN_DM_NAME_MAX - 1);
out->model_name[PN_DM_NAME_MAX - 1] = '\0';
}
out->n_models++;
}
pthread_mutex_unlock(&g_dm.lock);
return 0;
}
/* --- placement ----------------------------------------------------------- */
/* Locked helpers. */
static uint64_t die_free_bytes(int id)
{
const dm_die *d = &g_dm.die[id];
return d->vram_budget > d->vram_used ? d->vram_budget - d->vram_used : 0;
}
static bool die_occupied(int id)
{
return g_dm.die[id].vram_used > 0;
}
/* The card mate of a die under the fixed K80 pairing (2i <-> 2i+1); -1 when
* the mate is outside the tracked range. */
static int die_mate(int id)
{
int mate = (id % 2 == 0) ? id + 1 : id - 1;
return (mate >= 0 && mate < g_dm.n_dies) ? mate : -1;
}
int pn_dm_place(uint64_t need_bytes, int n_dies_wanted, int *dies_out)
{
if (!dies_out || n_dies_wanted < 1 || n_dies_wanted > 2)
return -1; /* >2-die pipeline placement arrives with Phase 6.1 */
pthread_mutex_lock(&g_dm.lock);
if (!g_dm.running) {
pthread_mutex_unlock(&g_dm.lock);
return -1;
}
uint64_t per_die = need_bytes / (uint64_t)n_dies_wanted;
int rc = -1;
if (n_dies_wanted == 2) {
/* Same-card pair with room on both halves. Prefer a fully-free pair;
* tie-break on the most free space (least loaded). */
int best = -1;
bool best_free = false;
uint64_t best_room = 0;
for (int a = 0; a + 1 < g_dm.n_dies; a += 2) {
int b = a + 1;
if (!g_dm.die[a].present || !g_dm.die[b].present)
continue;
if (die_free_bytes(a) < per_die || die_free_bytes(b) < per_die)
continue;
bool pair_free = !die_occupied(a) && !die_occupied(b);
uint64_t room = die_free_bytes(a) + die_free_bytes(b);
if (best < 0 || (pair_free && !best_free) ||
(pair_free == best_free && room > best_room)) {
best = a;
best_free = pair_free;
best_room = room;
}
}
if (best >= 0) {
dies_out[0] = best;
dies_out[1] = best + 1;
rc = 0;
}
} else {
/* Single die: pack onto a card whose mate is already occupied so free
* same-card pairs stay available for 2-die models; among candidates
* take the least-loaded (most free). */
int best = -1;
bool best_packed = false;
uint64_t best_room = 0;
for (int i = 0; i < g_dm.n_dies; i++) {
if (!g_dm.die[i].present || die_free_bytes(i) < per_die)
continue;
int mate = die_mate(i);
bool packed = (mate >= 0 && die_occupied(mate)) || die_occupied(i);
uint64_t room = die_free_bytes(i);
if (best < 0 || (packed && !best_packed) ||
(packed == best_packed && room > best_room)) {
best = i;
best_packed = packed;
best_room = room;
}
}
if (best >= 0) {
dies_out[0] = best;
rc = 0;
}
}
pthread_mutex_unlock(&g_dm.lock);
return rc;
}
/* --- bind / release ------------------------------------------------------ */
int pn_dm_bind(const int *dies, int n, const char *model_name,
uint64_t total_bytes)
{
if (!dies || n < 1 || n > PN_DM_MAX_DIES || !model_name || !*model_name)
return -1;
pthread_mutex_lock(&g_dm.lock);
if (!g_dm.running)
goto refuse;
/* Same-name multi-card placement: one model id may be bound to several
* DISJOINT die-sets (one binding per card) for load-balanced serving.
* Overlapping dies are still rejected by the per-die room check below, so
* here we only need a free binding slot — no duplicate-name refusal. Each
* placement gets its own binding; the registry releases one per entry on
* unload (pn_dm_release is by-name, first-match — N entries → N releases). */
dm_binding *slot = NULL;
for (int i = 0; i < PN_DM_MAX_BINDINGS; i++) {
if (!g_dm.binding[i].used) { slot = &g_dm.binding[i]; break; }
}
if (!slot)
goto refuse; /* binding table full */
uint64_t per_die = total_bytes / (uint64_t)n;
for (int i = 0; i < n; i++) {
int id = dies[i];
if (id < 0 || id >= g_dm.n_dies || !g_dm.die[id].present ||
die_free_bytes(id) < per_die)
goto refuse; /* unknown die / no room */
}
slot->used = true;
strncpy(slot->name, model_name, PN_DM_NAME_MAX - 1);
slot->name[PN_DM_NAME_MAX - 1] = '\0';
slot->n_dies = n;
slot->bytes_per_die = per_die;
for (int i = 0; i < n; i++) {
slot->dies[i] = dies[i];
g_dm.die[dies[i]].vram_used += per_die;
}
pthread_mutex_unlock(&g_dm.lock);
return 0;
refuse:
pthread_mutex_unlock(&g_dm.lock);
return -1;
}
int pn_dm_release(const char *model_name)
{
if (!model_name)
return -1;
pthread_mutex_lock(&g_dm.lock);
for (int i = 0; i < PN_DM_MAX_BINDINGS; i++) {
dm_binding *b = &g_dm.binding[i];
if (!b->used || strncmp(b->name, model_name, PN_DM_NAME_MAX) != 0)
continue;
for (int j = 0; j < b->n_dies; j++) {
dm_die *d = &g_dm.die[b->dies[j]];
d->vram_used = d->vram_used > b->bytes_per_die
? d->vram_used - b->bytes_per_die : 0;
}
memset(b, 0, sizeof(*b));
pthread_mutex_unlock(&g_dm.lock);
return 0;
}
pthread_mutex_unlock(&g_dm.lock);
return -1;
}

173
src/accel/die_manager.h Normal file
View File

@ -0,0 +1,173 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file die_manager.h
* @brief Minimal die manager — the placement primitive under multi-model
* serving. Tracks per-die
* state (bound model names, VRAM budget/occupancy) and assigns models
* to die-sets, topology-aware: a K80 card is the die pair (2i, 2i+1)
* linked by a PLX switch, so 2-die models get a same-card pair and
* 1-die models pack onto cards whose mate is already occupied (keeping
* whole pairs free for the next 2-die model).
*
* Three init paths, one per accelerator family: pn_dm_init_logical()
* creates N logical dies with a given budget (the CPU/Mac/test path —
* placement bookkeeping needs no GPU), pn_dm_init_cuda() (hardware) probes
* real K80 dies via fc_cuda_probe_device, and pn_dm_init_tt() (hardware)
* probes Tenstorrent Blackhole cards via ft_tt_init_device + ARC
* telemetry. All three feed pn_memwatch_set_die so MemoryAware
* admission sees the same per-die truth. This module is bookkeeping
* only — it never uploads weights; pn_load_on() does the binding of a
* model to the die-set this module picks.
*
* @status Logical path exercised on the Mac (tests/test_die_manager.c). The
* CUDA probe compiles under USE_CUDA and is gated on GPU hardware (needs live
* dies to validate budgets + memwatch feed). The TT probe compiles
* under USE_TT and is gated on GPU hardware the same way.
* @issues Placement assumes the fixed K80 pairing (card = dies 2i/2i+1); a
* different topology needs a card-map input. On a TT card the pairing
* is inert (one die per device) rather than wrong.
* @todo
* - [ ] (hardware) Validate pn_dm_init_cuda budgets vs nvidia-smi on 16 dies.
*/
#ifndef PN_DIE_MANAGER_H
#define PN_DIE_MANAGER_H
#include <stdbool.h>
#include <stdint.h>
#ifdef __cplusplus
extern "C" {
#endif
/* Hard caps: the K80 test machine has 16 GK210 dies (mirrors PN_MEM_MAX_DIES); one
* binding per resident model. */
#define PN_DM_MAX_DIES 16
#define PN_DM_MAX_BINDINGS 16
#define PN_DM_NAME_MAX 64
#define PN_DM_DIES_PER_CARD 2
/* Copy-out view of one die (pn_dm_get / status surfaces). */
typedef struct {
int id;
char model_name[PN_DM_NAME_MAX]; /* first bound model, "" if free */
int n_models; /* bindings touching this die */
uint64_t vram_budget; /* usable bytes (total - reserve) */
uint64_t vram_used; /* bytes reserved by bindings */
} pn_dm_die;
/* Start the manager with n_dies LOGICAL dies of budget_bytes each — the
* CPU/test path (a model can be "placed" without CUDA present). Idempotent
* while running (returns 0 without re-init). Returns 0 on success, -1 on a
* bad argument.
*
* Thread-safe: yes */
int pn_dm_init_logical(int n_dies, uint64_t budget_bytes);
#ifdef USE_CUDA
/* (hardware) Start the manager from real dies: probe CUDA ordinals [0, n_dies)
* via fc_cuda_init_device + fc_mem_info_ctx, budget = total −
* pn_die_reserve_bytes(), and feed each die into pn_memwatch_set_die when the
* watcher is running. Requires fc_init() done and fc_available() true —
* returns -1 otherwise (caller falls back to pn_dm_init_logical or CPU-only).
* Idempotent while running. Returns the number of dies probed (>0) or -1. */
int pn_dm_init_cuda(int n_dies);
#endif
#ifdef USE_TT
/* (hardware) Start the manager from Tenstorrent card(s): probe device ordinals
* [0, n_devices) via ft_tt_init_device + ARC telemetry, register ONE die per
* DEVICE with budget = total − pn_die_reserve_bytes(), and feed each into
* pn_memwatch_set_die when the watcher is running. Ordinals are contiguous, so
* the first one that will not open ends the scan. Idempotent while running.
* Returns the number of devices probed (>0) or -1.
*
* ONE-DEVICE-ONE-DIE, deliberately, and NOT core-group-as-unit (Blackhole spec
* §9.3). The unit this module models is a MEMORY DOMAIN — every field it
* stores is a byte budget or a byte occupancy — and a p150a has exactly one:
* ~31.88 GiB of GDDR6 that all 140 Tensix cores address over the same NoC. A
* core group is a COMPUTE partition, not a memory domain, so registering N
* groups as N dies would divide one real budget into N fictional ones (and
* refuse a model that fits the card), and would assert an independence the
* hardware does not have — Phase 6.7 measured a per-NoC-row DRAM ceiling
* shared across groups, and readers on one side of the grid slowing readers on
* the other.
*
* The K80 topology is deliberately not ported forward: pn_dm_card_of()'s
* die-pair model describes two GK210s with private VRAM behind a PLX switch,
* and there is no analogue here. At one die per device it degenerates to the
* identity rather than to something wrong.
*
* Multi-model isolation is a real requirement and it already has a mechanism
* one layer down: ft_tt_group_partition() hands each group disjoint cores AND
* disjoint DRAM banks with its own allocator. When several models share a
* card, the field to add is a CORE COUNT on the binding beside the single
* memory die — memory and compute are separate resources on this part, which
* is the one structural difference from a K80 die.
*
* Note there is no cross-process occupancy signal: ft_tt_mem_info()'s `free`
* is total minus THIS context's own allocations, so unlike the CUDA probe
* there is no stranded-memory term and vram_used starts at 0.
*
* Thread-safe: yes */
int pn_dm_init_tt(int n_devices);
/**
* Whether this process was TOLD which Tenstorrent transport to use
* (`FT_TT_DRIVER=kmd|udrv`) — including having named one that does not exist.
*
* The caller needs this because "no Tenstorrent device" has two meanings and
* only one of them is benign. On a machine with no card a failed pn_dm_init_tt
* is expected and serving CPU-resident is correct. When somebody named a
* transport, the same failure means the card they asked for could not be
* reached — and answering on the CPU would be the silent fallback this project
* has shipped twice, returning a plausible answer from the wrong thing. It was
* measured on 2026-08-17 that the card and the CPU fallback return
* BYTE-IDENTICAL completion text, so nothing downstream can catch it.
*
* Thread-safe: yes */
bool pn_dm_tt_driver_explicit(void);
#endif
/* Stop tracking and forget all dies/bindings. Safe to call if never started. */
void pn_dm_shutdown(void);
/* Number of tracked dies (0 = manager not running → CPU-only placement). */
int pn_dm_count(void);
/* Copy die_id's state into *out. Returns 0, or -1 if unknown/not running. */
int pn_dm_get(int die_id, pn_dm_die *out);
/* The card index a die sits on (K80 pairing: die/2). Pure. */
static inline int pn_dm_card_of(int die_id) { return die_id / PN_DM_DIES_PER_CARD; }
/* Pick a die-set for a model needing need_bytes across n_dies_wanted dies
* (bytes split evenly). Topology-aware: n=2 → a same-card pair, preferring a
* fully-free pair, least-loaded tie-break; n=1 → pack onto a die whose card
* mate is already occupied, least-loaded among those, else the least-loaded
* free-fitting die. Writes the chosen ids to dies_out[0..n) and returns 0, or
* -1 when nothing fits (or the manager is not running). Does NOT reserve —
* call pn_dm_bind to commit.
*
* Thread-safe: yes */
int pn_dm_place(uint64_t need_bytes, int n_dies_wanted, int *dies_out);
/* Commit model_name to the die-set: re-checks capacity under the lock and
* adds total_bytes/n to each die's vram_used. Rejects a duplicate name, an
* unknown die, or a set that no longer fits. Returns 0 on success, -1 on
* refusal (nothing is partially bound).
*
* Thread-safe: yes */
int pn_dm_bind(const int *dies, int n, const char *model_name,
uint64_t total_bytes);
/* Release model_name's binding, returning its bytes to each die. Returns 0,
* or -1 if the name is unknown. */
int pn_dm_release(const char *model_name);
#ifdef __cplusplus
}
#endif
#endif /* PN_DIE_MANAGER_H */

259
src/accel/simd.c Normal file
View File

@ -0,0 +1,259 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file simd.c
* @brief NEON / AVX2+FMA / plain-C implementations of the CPU vector kernels.
*
* Written from a behaviour specification (plans/cleanroom_r2_simd_spec.md); no ggml or
* llama.cpp source was consulted.
*
* Release builds use -ffast-math, so output bits depend on the shape of this source (which
* loops are explicit SIMD, accumulator count, statement split), not only on the maths.
* Change the shape of any loop and the bits change.
*
* @status None
* @issues - -ffast-math re-associates silu*up into (up*x)/(1+e); release and debug differ.
* - x86 vectorized silu uses a reciprocal estimate; its scalar tail divides exactly.
* - NEON Q4_0 scales per element, scalar Q4_0 gets d hoisted by the compiler, Q8_0
* scales at the end: the paths do not agree bit-for-bit.
* - The first FMA into a zero accumulator folds to a multiply (keeps -0).
* - NaN payload choice follows compiled operand order.
* - rms computes NaN for n == 0 (nothing is written).
* - No NULL or range checks.
* - pn_vmul_f32, pn_vscale_f32, pn_vsum_f32 have no callers.
* - The shipping x86 target (the bank, no AVX/FMA) never compiles the AVX2 path.
* @todo None
*/
#include "simd.h"
#include <math.h>
#include "../formats/gguf_dequant.h"
#if defined(__ARM_NEON) || defined(__ARM_NEON__)
#define PN_SIMD_PATH_NEON 1
#include <arm_neon.h>
#elif defined(__AVX2__) && defined(__FMA__)
#define PN_SIMD_PATH_AVX2 1
#include <immintrin.h>
#endif
#ifdef PN_SIMD_PATH_AVX2
/* Low half + high half, then two horizontal pair-adds; lane 0 holds the total. */
static float avx2_horizontal_sum(__m256 v) {
__m128 low = _mm256_castps256_ps128(v);
__m128 high = _mm256_extractf128_ps(v, 1);
low = _mm_add_ps(low, high);
low = _mm_hadd_ps(low, low);
low = _mm_hadd_ps(low, low);
return _mm_cvtss_f32(low);
}
#endif
float pn_dot_f32(const float *a, const float *b, int n) {
int i = 0;
float sum;
#if defined(PN_SIMD_PATH_NEON)
float32x4_t acc0 = vdupq_n_f32(0.0f);
float32x4_t acc1 = vdupq_n_f32(0.0f);
for (; i + 7 < n; i += 8) {
float32x4_t a0 = vld1q_f32(a + i);
float32x4_t b0 = vld1q_f32(b + i);
float32x4_t a1 = vld1q_f32(a + i + 4);
float32x4_t b1 = vld1q_f32(b + i + 4);
acc0 = vfmaq_f32(acc0, a0, b0);
acc1 = vfmaq_f32(acc1, a1, b1);
}
acc0 = vaddq_f32(acc0, acc1);
sum = vaddvq_f32(acc0);
#elif defined(PN_SIMD_PATH_AVX2)
__m256 acc0 = _mm256_setzero_ps();
__m256 acc1 = _mm256_setzero_ps();
for (; i + 15 < n; i += 16) {
__m256 a_lo = _mm256_loadu_ps(a + i);
__m256 a_hi = _mm256_loadu_ps(a + i + 8);
__m256 b_lo = _mm256_loadu_ps(b + i);
__m256 b_hi = _mm256_loadu_ps(b + i + 8);
acc0 = _mm256_fmadd_ps(a_lo, b_lo, acc0);
acc1 = _mm256_fmadd_ps(a_hi, b_hi, acc1);
}
acc0 = _mm256_add_ps(acc0, acc1);
sum = avx2_horizontal_sum(acc0);
#else
sum = 0.0f;
#endif
for (; i < n; i++)
sum += a[i] * b[i];
return sum;
}
void pn_vadd_f32(const float *a, const float *b, float *out, int n) {
int i = 0;
#if defined(PN_SIMD_PATH_NEON)
for (; i + 3 < n; i += 4) {
float32x4_t va = vld1q_f32(a + i);
float32x4_t vb = vld1q_f32(b + i);
vst1q_f32(out + i, vaddq_f32(va, vb));
}
#elif defined(PN_SIMD_PATH_AVX2)
for (; i + 7 < n; i += 8) {
__m256 va = _mm256_loadu_ps(a + i);
__m256 vb = _mm256_loadu_ps(b + i);
_mm256_storeu_ps(out + i, _mm256_add_ps(va, vb));
}
#endif
for (; i < n; i++)
out[i] = a[i] + b[i];
}
void pn_vmul_f32(const float *a, const float *b, float *out, int n) {
int i = 0;
#if defined(PN_SIMD_PATH_NEON)
for (; i + 3 < n; i += 4) {
float32x4_t va = vld1q_f32(a + i);
float32x4_t vb = vld1q_f32(b + i);
vst1q_f32(out + i, vmulq_f32(va, vb));
}
#elif defined(PN_SIMD_PATH_AVX2)
for (; i + 7 < n; i += 8) {
__m256 va = _mm256_loadu_ps(a + i);
__m256 vb = _mm256_loadu_ps(b + i);
_mm256_storeu_ps(out + i, _mm256_mul_ps(va, vb));
}
#endif
for (; i < n; i++)
out[i] = a[i] * b[i];
}
void pn_vscale_f32(const float *a, float scale, float *out, int n) {
int i = 0;
#if defined(PN_SIMD_PATH_NEON)
float32x4_t vscale = vdupq_n_f32(scale);
for (; i + 3 < n; i += 4) {
float32x4_t va = vld1q_f32(a + i);
vst1q_f32(out + i, vmulq_f32(va, vscale));
}
#elif defined(PN_SIMD_PATH_AVX2)
__m256 vscale = _mm256_set1_ps(scale);
for (; i + 7 < n; i += 8) {
__m256 va = _mm256_loadu_ps(a + i);
_mm256_storeu_ps(out + i, _mm256_mul_ps(va, vscale));
}
#endif
for (; i < n; i++)
out[i] = a[i] * scale;
}
float pn_vsum_f32(const float *a, int n) {
int i = 0;
float sum;
#if defined(PN_SIMD_PATH_NEON)
float32x4_t acc = vdupq_n_f32(0.0f);
for (; i + 3 < n; i += 4)
acc = vaddq_f32(acc, vld1q_f32(a + i));
sum = vaddvq_f32(acc);
#elif defined(PN_SIMD_PATH_AVX2)
__m256 acc = _mm256_setzero_ps();
for (; i + 7 < n; i += 8)
acc = _mm256_add_ps(acc, _mm256_loadu_ps(a + i));
sum = avx2_horizontal_sum(acc);
#else
sum = 0.0f;
#endif
for (; i < n; i++)
sum += a[i];
return sum;
}
/* Plain loop on every path: the compiler vectorizes it (and, under -ffast-math, reorders
* the arithmetic); hand-written SIMD here would change the output bits. */
void pn_vsilu_mul_f32(const float *gate, const float *up, float *out, int n) {
for (int i = 0; i < n; i++) {
float x = gate[i];
float s = x / (1.0f + expf(-x));
out[i] = s * up[i];
}
}
void pn_vrms_norm_f32(const float *x, const float *w, float *out, int n, float eps) {
/* Double accumulation: callers rely on its precision over long rows. */
double ss = 0.0;
for (int k = 0; k < n; k++)
ss += (double)x[k] * (double)x[k];
float r = (float)(1.0 / sqrt(ss / (double)n + (double)eps));
int i = 0;
#if defined(PN_SIMD_PATH_NEON)
float32x4_t vr = vdupq_n_f32(r);
for (; i + 3 < n; i += 4) {
float32x4_t vx = vld1q_f32(x + i);
float32x4_t vw = vld1q_f32(w + i);
vst1q_f32(out + i, vmulq_f32(vmulq_f32(vx, vr), vw));
}
#elif defined(PN_SIMD_PATH_AVX2)
__m256 vr = _mm256_set1_ps(r);
for (; i + 7 < n; i += 8) {
__m256 vx = _mm256_loadu_ps(x + i);
__m256 vw = _mm256_loadu_ps(w + i);
_mm256_storeu_ps(out + i, _mm256_mul_ps(_mm256_mul_ps(vx, vr), vw));
}
#endif
for (; i < n; i++)
out[i] = x[i] * r * w[i];
}
float pn_dot_q4_0_f32(const void *block, const float *vec) {
const pn_block_q4_0 *blk = (const pn_block_q4_0 *)block;
float df = pn_half_to_float(blk->d);
#if defined(PN_SIMD_PATH_NEON)
float32x4_t acc0 = vdupq_n_f32(0.0f);
float32x4_t acc1 = vdupq_n_f32(0.0f);
float32x4_t vdf = vdupq_n_f32(df);
float32x4_t voffset = vdupq_n_f32(8.0f);
for (int j = 0; j < 16; j += 4) {
float lo[4];
float hi[4];
for (int k = 0; k < 4; k++) {
lo[k] = (float)(blk->qs[j + k] & 0x0F);
hi[k] = (float)(blk->qs[j + k] >> 4);
}
float32x4_t vlo = vmulq_f32(vsubq_f32(vld1q_f32(lo), voffset), vdf);
float32x4_t vhi = vmulq_f32(vsubq_f32(vld1q_f32(hi), voffset), vdf);
acc0 = vfmaq_f32(acc0, vlo, vld1q_f32(vec + j));
acc1 = vfmaq_f32(acc1, vhi, vld1q_f32(vec + j + 16));
}
return vaddvq_f32(vaddq_f32(acc0, acc1));
#else
float sum = 0.0f;
for (int j = 0; j < 16; j++) {
int lo = blk->qs[j] & 0x0F;
int hi = blk->qs[j] >> 4;
sum += df * (float)(lo - 8) * vec[j];
sum += df * (float)(hi - 8) * vec[j + 16];
}
return sum;
#endif
}
float pn_dot_q8_0_f32(const void *block, const float *vec) {
const pn_block_q8_0 *blk = (const pn_block_q8_0 *)block;
float df = pn_half_to_float(blk->d);
#if defined(PN_SIMD_PATH_NEON)
float32x4_t acc0 = vdupq_n_f32(0.0f);
float32x4_t acc1 = vdupq_n_f32(0.0f);
for (int j = 0; j < 32; j += 8) {
int16x8_t q16 = vmovl_s8(vld1_s8(blk->qs + j));
float32x4_t q_lo = vcvtq_f32_s32(vmovl_s16(vget_low_s16(q16)));
float32x4_t q_hi = vcvtq_f32_s32(vmovl_s16(vget_high_s16(q16)));
acc0 = vfmaq_f32(acc0, q_lo, vld1q_f32(vec + j));
acc1 = vfmaq_f32(acc1, q_hi, vld1q_f32(vec + j + 4));
}
return df * vaddvq_f32(vaddq_f32(acc0, acc1));
#else
float sum = 0.0f;
for (int j = 0; j < 32; j++)
sum += (float)blk->qs[j] * vec[j];
sum *= df;
return sum;
#endif
}

151
src/accel/simd.h Normal file
View File

@ -0,0 +1,151 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file simd.h
* @brief CPU vector kernels: dot products, element-wise ops, RMS norm, Q4_0/Q8_0 block dots.
*
* Written from a behaviour specification (plans/cleanroom_r2_simd_spec.md); no ggml or
* llama.cpp source was consulted.
*
* One code path is chosen at compile time: NEON (arm64), AVX2+FMA (x86 built with both),
* otherwise plain C that the compiler may auto-vectorize. There is no runtime dispatch.
*
* Common contract for every function taking `n`: for n <= 0 nothing is read or written and
* reductions return +0.0f. Pointers need only float (4-byte) alignment. In-place calls are
* supported: out == a, out == b, out == gate, out == up, out == x, w == x, a == b.
* No NULL checks: a NULL pointer with n > 0 crashes.
*
* @status None
* @issues None
* @todo None
*/
#ifndef PN_SIMD_H
#define PN_SIMD_H
#include <stddef.h>
/**
* Dot product of two float vectors.
*
* @param a First vector, n floats
* @param b Second vector, n floats
* @param n Element count
* @return Sum of a[i] * b[i]; +0.0f when n <= 0
*
* Thread-safe: yes (pure)
* Reentrant: yes
* Signal-safe: yes
*/
float pn_dot_f32(const float *a, const float *b, int n);
/**
* Element-wise add: out[i] = a[i] + b[i].
*
* @param a First input, n floats
* @param b Second input, n floats
* @param out Output, n floats (may equal a or b)
* @param n Element count
*
* Thread-safe: yes (pure apart from writing out)
* Reentrant: yes
* Signal-safe: yes
*/
void pn_vadd_f32(const float *a, const float *b, float *out, int n);
/**
* Element-wise multiply: out[i] = a[i] * b[i].
*
* @param a First input, n floats
* @param b Second input, n floats
* @param out Output, n floats (may equal a or b)
* @param n Element count
*
* Thread-safe: yes (pure apart from writing out)
* Reentrant: yes
* Signal-safe: yes
*/
void pn_vmul_f32(const float *a, const float *b, float *out, int n);
/**
* SwiGLU combine: out[i] = silu(gate[i]) * up[i], silu(x) = x / (1 + expf(-x)).
*
* @param gate Gate activations, n floats
* @param up Up projections, n floats
* @param out Output, n floats (may equal gate or up)
* @param n Element count
*
* Thread-safe: yes (pure apart from writing out)
* Reentrant: yes
* Signal-safe: yes (libm expf only)
*/
void pn_vsilu_mul_f32(const float *gate, const float *up, float *out, int n);
/**
* RMS norm: out[i] = x[i] * r * w[i], r = 1 / sqrt(mean(x^2) + eps), mean taken in double.
*
* @param x Input, n floats
* @param w Per-element weight, n floats (may equal x)
* @param out Output, n floats (may equal x)
* @param n Element count
* @param eps Added to the mean square before the square root; not clamped
*
* Thread-safe: yes (pure apart from writing out)
* Reentrant: yes
* Signal-safe: yes (libm sqrt only)
*/
void pn_vrms_norm_f32(const float *x, const float *w, float *out, int n, float eps);
/**
* Dot of one Q4_0 block (32 elements, 18 bytes) with 32 floats.
*
* @param block Q4_0 block (pn_block_q4_0); needs 2-byte alignment
* @param vec 32 floats
* @return Sum over j of (code_j - 8) * d * vec[j]
*
* Thread-safe: yes (pure)
* Reentrant: yes
* Signal-safe: yes
*/
float pn_dot_q4_0_f32(const void *block, const float *vec);
/**
* Dot of one Q8_0 block (32 elements, 34 bytes) with 32 floats.
*
* @param block Q8_0 block (pn_block_q8_0); needs 2-byte alignment
* @param vec 32 floats
* @return d * sum over j of qs[j] * vec[j]
*
* Thread-safe: yes (pure)
* Reentrant: yes
* Signal-safe: yes
*/
float pn_dot_q8_0_f32(const void *block, const float *vec);
/**
* Scale a vector: out[i] = a[i] * scale.
*
* @param a Input, n floats
* @param scale Multiplier
* @param out Output, n floats (may equal a)
* @param n Element count
*
* Thread-safe: yes (pure apart from writing out)
* Reentrant: yes
* Signal-safe: yes
*/
void pn_vscale_f32(const float *a, float scale, float *out, int n);
/**
* Sum of a float vector.
*
* @param a Input, n floats
* @param n Element count
* @return Sum of a[i]; +0.0f when n <= 0
*
* Thread-safe: yes (pure)
* Reentrant: yes
* Signal-safe: yes
*/
float pn_vsum_f32(const float *a, int n);
#endif /* PN_SIMD_H */

1659
src/accel/tt/pn_tt_attention.c Normal file
View File

File diff suppressed because it is too large Load Diff

103
src/accel/tt/pn_tt_column_runs.c Normal file
View File

@ -0,0 +1,103 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_column_runs.c
* @brief Implementation of the column-tile permutation arithmetic. See pn_tt_column_runs.h,
* which carries why a joint projection is de-interleaved in the WEIGHT.
*
* No device, no allocation, no I/O — deliberately, so it is provable with no card.
*
* @status None
* @issues None
* @todo None
*/
#include "pn_tt_column_runs.h"
uint32_t pn_tt_column_runs_total(const pn_tt_column_run *runs, uint32_t run_count)
{
uint32_t total = 0u;
uint32_t index;
if (!runs || run_count == 0u || run_count > PN_TT_COLUMN_RUNS_MAX)
return 0u;
for (index = 0; index < run_count; index++) {
if (runs[index].count == 0u)
return 0u;
/* A destination wider than any weight this backend admits is a caller bug that would
* otherwise wrap and produce a plausible small number. */
if (total > UINT32_MAX - runs[index].count)
return 0u;
total += runs[index].count;
}
return total;
}
bool pn_tt_column_runs_are_injective(const pn_tt_column_run *runs, uint32_t run_count,
uint32_t source_columns)
{
uint32_t index;
if (pn_tt_column_runs_total(runs, run_count) == 0u || source_columns == 0u)
return false;
for (index = 0; index < run_count; index++) {
uint32_t other;
if (runs[index].first >= source_columns ||
runs[index].count > source_columns - runs[index].first)
return false;
/* Pairwise rather than a bitmap: `run_count` is at most 128, so this is at worst
* 8,128 comparisons ONCE at model load, and a bitmap would need a bound on
* `source_columns` that this module has no business having. */
for (other = 0; other < index; other++) {
const uint32_t a_end = runs[index].first + runs[index].count;
const uint32_t b_end = runs[other].first + runs[other].count;
if (runs[index].first < b_end && runs[other].first < a_end)
return false;
}
}
return true;
}
bool pn_tt_column_runs_map(const pn_tt_column_run *runs, uint32_t run_count,
uint32_t destination_column, uint32_t *out_source_column,
uint32_t *out_contiguous)
{
uint32_t walked = 0u;
uint32_t index;
if (pn_tt_column_runs_total(runs, run_count) == 0u || !out_source_column ||
!out_contiguous)
return false;
for (index = 0; index < run_count; index++) {
if (destination_column < walked + runs[index].count) {
const uint32_t inside = destination_column - walked;
*out_source_column = runs[index].first + inside;
*out_contiguous = runs[index].count - inside;
return true;
}
walked += runs[index].count;
}
return false;
}
uint32_t pn_tt_column_runs_field(uint32_t groups, uint32_t stride, uint32_t offset,
uint32_t width, pn_tt_column_run *out_runs, uint32_t max_runs)
{
uint32_t index;
if (!out_runs || groups == 0u || width == 0u || groups > max_runs ||
groups > PN_TT_COLUMN_RUNS_MAX)
return 0u;
/* The field has to fit inside the record, or consecutive groups would overlap and the
* injectivity check downstream would report a defect this function created. */
if (stride < width || offset > stride - width)
return 0u;
for (index = 0; index < groups; index++) {
out_runs[index].first = index * stride + offset;
out_runs[index].count = width;
}
return groups;
}

124
src/accel/tt/pn_tt_column_runs.h Normal file
View File

@ -0,0 +1,124 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_column_runs.h
* @brief A PERMUTATION of a weight's output columns, expressed as runs of COLUMN TILES —
* the whole of the "de-interleave a joint projection" problem, with no card in it.
*
* ---------------------------------------------------------------------------
* WHY THIS EXISTS: A JOINT PROJECTION IS INTERLEAVED PER HEAD
* ---------------------------------------------------------------------------
* Ornith's `blk.N.attn_q.weight` emits Q **and** the attention output gate from one matmul,
* interleaved per head:
*
* [ head 0 Q (head_dim) | head 0 gate (head_dim) | head 1 Q | head 1 gate | ... ]
*
* Every op downstream wants them apart and compact. The QK-RMSNorm reads head `h` as row
* block `h` at tiles `h*tile_cols`, consecutive; flash decode reads a KV group as ONE
* contiguous run of `q_rows*dim_tiles` tiles. Over the joint strip, neither lands: group
* `g`'s run would be head `2g`'s Q followed by head `2g`'s GATE.
* `foundry/tt/tests/test_attention_ops.c` asserts exactly that, so this is a measured
* requirement rather than a tidiness one.
*
* ---------------------------------------------------------------------------
* AND WHY IT IS SOLVED IN THE WEIGHT RATHER THAN PER TOKEN
* ---------------------------------------------------------------------------
* The three places it could be fixed, and what each costs:
*
* 1. a per-token card-side gather — a new op, 512 tiles a layer, 8 layers a token;
* 2. a stride word in `resident_attention_reader.c` AND one in `resident_reduce_reader.c` —
* two kernel source changes, a program rebuild, and the reduce one is under the model's
* OWN RMS norms on the shipped default path;
* 3. a PERMUTED UPLOAD of the weight at load — no kernel change, no per-token cost, and
* every downstream op runs at its already-proven shape.
*
* (3), and this file is its arithmetic. An N-major artifact lays column tile `c` as a
* contiguous `tile_rows * tile_bytes` run, so a column permutation is a permutation of
* contiguous byte ranges and the upload is one `ft_tt_fttw_upload_range()` per run per
* shard. The cost is a second copy of the tensor in card DRAM — 71.3 MiB a layer for
* Ornith's `attn_q`, 570 MiB over its 8 attention layers, against 32 GB of board memory.
*
* Nothing here touches a device, so it is provable with no card:
* `tests/test_tt_column_runs.c`.
*
* @status None
* @issues None
* @todo None
*/
#ifndef PN_TT_COLUMN_RUNS_H
#define PN_TT_COLUMN_RUNS_H
#include <stdbool.h>
#include <stdint.h>
/** How many runs one destination weight may be assembled from. 32 heads x 2 halves is 64;
* this is the ceiling a caller's array has to satisfy and it is checked, not assumed. */
#define PN_TT_COLUMN_RUNS_MAX 128u
/** One contiguous run of SOURCE column tiles, in the order the DESTINATION wants them. */
typedef struct pn_tt_column_run {
uint32_t first; /**< first source column tile */
uint32_t count; /**< consecutive source column tiles, > 0 */
} pn_tt_column_run;
/**
* Destination column tiles this run list produces.
*
* @return the sum of the counts, or 0 for an empty or malformed list
*
* Thread-safe: yes Reentrant: yes Signal-safe: yes
*/
uint32_t pn_tt_column_runs_total(const pn_tt_column_run *runs, uint32_t run_count);
/**
* Do these runs stay inside a `source_columns`-wide weight, and does no source column appear
* twice?
*
* The second half is what makes a wrong run list a REFUSAL rather than a weight with one
* column duplicated and another missing — which would be a smoothly wrong projection with
* nothing on the card able to see it.
*
* @return true when every run is in range and the destination uses each source column at
* most once
*
* Thread-safe: yes Reentrant: yes Signal-safe: yes
*/
bool pn_tt_column_runs_are_injective(const pn_tt_column_run *runs, uint32_t run_count,
uint32_t source_columns);
/**
* Where destination column `destination_column` comes from, and how many destination columns
* after it are contiguous in the SOURCE too.
*
* The second output is what turns a permutation into a small number of large transfers: a
* caller uploads `*out_contiguous` columns in one range and asks again.
*
* @param out_source_column the source column tile it reads
* @param out_contiguous destination columns from here that share the run, >= 1
* @return false when `destination_column` is past the end
*
* Thread-safe: yes Reentrant: yes Signal-safe: yes
*/
bool pn_tt_column_runs_map(const pn_tt_column_run *runs, uint32_t run_count,
uint32_t destination_column, uint32_t *out_source_column,
uint32_t *out_contiguous);
/**
* Build the run list that lifts ONE interleaved field out of a joint projection.
*
* Field `offset..offset+width` of each of `groups` records of `stride` column tiles. For
* Ornith's `attn_q` at head_dim 256 (dim_tiles 8): `groups` 32, `stride` 16, `width` 8, and
* `offset` 0 for Q or 8 for the gate.
*
* @param out_runs receives `groups` runs
* @param max_runs capacity of `out_runs`
* @return runs written, or 0 on a bad shape or too small an array
*
* Thread-safe: yes Reentrant: yes Signal-safe: yes
*/
uint32_t pn_tt_column_runs_field(uint32_t groups, uint32_t stride, uint32_t offset,
uint32_t width, pn_tt_column_run *out_runs,
uint32_t max_runs);
#endif /* PN_TT_COLUMN_RUNS_H */

209
src/accel/tt/pn_tt_core_boot_guard.c Normal file
View File

@ -0,0 +1,209 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_core_boot_guard.c
* @brief Implementation of the pre-boot readiness gate (see pn_tt_core_boot_guard.h).
*
* @status New 2026-08-24.
* @issues None
* @todo None
*/
#ifdef USE_TT
#include "pn_tt_core_boot_guard.h"
#include <stdio.h>
#include <string.h>
#include <time.h>
#include "pn_tt_residency.h"
#include "tt/ft_tt_core_readiness.h"
#include "tt/ft_tt_launch.h"
#include "tt/ft_tt_memmap.h"
/**
* Milliseconds this guard will wait for a busy core to come free.
*
* IT IS LONGER THAN THE STOP'S GRACE ON PURPOSE, AND THE DIFFERENCE IS MEASURED.
* `pn_tt_residency_stop()` is issued straight after a refusal, so what it waits out is the
* CB-wait bound — 100,000,000 spins — and 20 s covers it. This guard waits out whatever an
* EARLIER PROCESS left, and a residency abandoned by a process that DIED sits in
* `ftq_wait_for_work()` until the IDLE bound instead: 400,000,000 spins, recorded in
* `pn_tt_device.c`'s own death-stop warning as **over 40 s**. A 20 s guard would refuse a
* core that was 30 s from clearing, and that refusal costs the whole model the
* ~26 s/position host forward — far more than the wait.
*
* The cost of the margin is zero on a healthy core: this wait does not run unless the card
* has already said a processor is executing.
*/
#define PN_TT_BOOT_GUARD_WAIT_MS 90000u
/** Milliseconds between readiness re-reads while waiting out a spin.
*
* The stop polls at one millisecond because it is waiting on a queue field it can read in
* 3 us. A readiness read is a mailbox snapshot plus fourteen more reads, ~400 us, and the
* thing it waits for takes seconds — so polling it at 1 ms would spend a third of the wait
* inside the instrument. 50 ms costs at most 50 ms of extra wait on a spin that is orders
* of magnitude longer. */
#define PN_TT_BOOT_GUARD_POLL_MS 50u
/** Names in FT_TT_PROC_* order, for the ticking mask. */
static const char *const PN_TT_BOOT_GUARD_PROCESSOR_NAME[FT_TT_PROC_COUNT] = {
"BRISC", "NCRISC", "TRISC0", "TRISC1", "TRISC2"
};
/** "TRISC0 TRISC2" into `out`, or "none". Never NULL, never unterminated. */
static void describe_ticking(uint32_t mask, char *out, size_t out_bytes)
{
unsigned processor;
size_t used = 0;
if (!out || out_bytes == 0u)
return;
out[0] = '\0';
for (processor = 0; processor < FT_TT_PROC_COUNT; processor++) {
int written;
if ((mask & (1u << processor)) == 0u)
continue;
written = snprintf(out + used, out_bytes - used, "%s%s", used ? " " : "",
PN_TT_BOOT_GUARD_PROCESSOR_NAME[processor]);
if (written < 0 || (size_t)written >= out_bytes - used)
return;
used += (size_t)written;
}
if (used == 0u)
snprintf(out, out_bytes, "none");
}
/**
* The whole refusal, in one block, on stderr.
*
* IT IS LONG ON PURPOSE. The failure mode this guard sits in front of is not "the arm did
* not happen" — it is "the arm did not happen, the model quietly ran on the CPU at
* ~26 s/position, and the numbers looked plausible for eleven minutes". A refusal that reads
* like one more warning would reproduce exactly that. Every line here answers a question the
* person reading it is about to ask: which core, what did the card say, can I wait, what do
* I run, and what happens to the model now.
*/
static void print_refusal(const ft_tt_core_readiness *readiness, uint32_t core_index,
const char *who, const char *core_env, unsigned waited_ms)
{
char ticking[64];
describe_ticking(readiness->ticking_mask, ticking, sizeof(ticking));
fprintf(stderr,
"pn_tt: ==================================================================\n"
"pn_tt: %s — Tensix %u is NOT READY, so the %s residency is not armed.\n"
"pn_tt: the card says: %s\n"
"pn_tt: verdict '%s' · reset 0x%08x · instrn_buf 0x%08x · go 0x%02x\n"
"pn_tt: resident queue %s · control heartbeat %s · executing: %s\n",
PN_TT_BOOT_GUARD_REFUSAL_TOKEN, core_index, who,
ft_tt_core_readiness_string(readiness->state),
ft_tt_core_verdict_string(readiness->status.verdict),
readiness->status.state.reset_register, readiness->status.state.instrn_buf_status,
readiness->status.state.go_messages[0].signal,
readiness->status.queue_present ? "present" : "absent",
readiness->status.queue_heartbeat_ticking ? "ticking" : "frozen", ticking);
if (waited_ms != 0u)
fprintf(stderr, "pn_tt: waited %u ms for it to clear and it did not.\n", waited_ms);
if (readiness->state == FT_TT_CORE_NOT_READY_STILL_EXECUTING) {
fprintf(stderr,
"pn_tt: THIS IS A LEFTOVER FROM AN EARLIER PROCESS. A refused descriptor\n"
"pn_tt: leaves a compute thread spinning out its 400,000,000-iteration\n"
"pn_tt: bound, and that thread outlives the process that started it. It\n"
"pn_tt: WILL finish — leave the core alone and try again.\n");
}
if (readiness->state == FT_TT_CORE_NOT_READY_INSTRUCTION_FIFO) {
fprintf(stderr,
"pn_tt: THIS CORE IS ALREADY WEDGED. Rung 1 does NOT clear it. Only a\n"
"pn_tt: whole-card reset does, and that resets the WHOLE CARD:\n"
"pn_tt: sudo -n env FT_TT_DRIVER=udrv ./tt/tools/build/ft_tt_core \\\n"
"pn_tt: recover %u --rung 3 --write --then-probe --driver udrv\n",
core_index);
}
/* NOTHING WAS WRITTEN, and saying so is load-bearing: the next question after a refusal
* is always "is the card in a worse state now than before I ran this". */
fprintf(stderr,
"pn_tt: NOTHING WAS WRITTEN TO THIS CORE. Booting firmware onto it is the\n"
"pn_tt: wedge (instrn_buf 0x66), which is what this refusal prevented.\n"
"pn_tt: CONSEQUENCE: with this family unarmed the model may fall back to the\n"
"pn_tt: HOST forward at ~26 s/position. Watch for 'using CPU forward' and\n"
"pn_tt: treat plausible-but-very-slow numbers after this line as INVALID.\n");
if (core_env)
fprintf(stderr, "pn_tt: To arm somewhere else instead, set %s to a free core.\n",
core_env);
fprintf(stderr,
"pn_tt: `ft_tt_core status %u --detail` reads this core without touching it.\n"
"pn_tt: ==================================================================\n",
core_index);
}
/** One millisecond of sleep, `count` times. */
static void nap_ms(unsigned count)
{
struct timespec nap = { 0, 1000000L };
unsigned index;
for (index = 0; index < count; index++)
(void)nanosleep(&nap, NULL);
}
bool pn_tt_core_boot_is_permitted(ft_tt_runtime *runtime, uint32_t core_index, const char *who,
const char *core_env)
{
ft_tt_core_readiness readiness;
unsigned waited_ms = 0u;
if (!runtime || !who)
return true;
/* A READ THAT FAILS IS NOT A REFUSAL.
*
* This guard catches one positively-identified condition: a core the card says is busy.
* Everything else must pass. If the coordinate does not resolve or the reading itself
* errors, refusing would let a diagnostic path's own failure take the whole model to the
* ~26 s/position host forward — the exact silent degradation this round is defending
* against, arriving through the defence. The boot below has its own error path for a
* core that genuinely cannot be reached. */
if (ft_tt_core_readiness_read(&runtime->noc, &runtime->grid, core_index, &readiness) !=
FT_TT_OK)
return true;
if (readiness.ready)
return true;
/* WAIT ONLY WHEN THE READING SAYS WAITING IS THE ANSWER. A spun-out bound finishes; a
* 0x66 core, a stalled kernel and a stuck boot do not, and PN_TT_BOOT_GUARD_WAIT_MS
* spent in front of an inevitable refusal is a minute and a half of a model load that
* nobody gets back. */
if (readiness.waiting_can_help) {
fprintf(stderr, "pn_tt: Tensix %u is still executing — %s waits up to %u ms. This is a "
"leftover spinning compute thread, and it is bounded.\n",
core_index, who, (unsigned)PN_TT_BOOT_GUARD_WAIT_MS);
while (waited_ms < (unsigned)PN_TT_BOOT_GUARD_WAIT_MS) {
nap_ms(PN_TT_BOOT_GUARD_POLL_MS);
waited_ms += PN_TT_BOOT_GUARD_POLL_MS;
if (ft_tt_core_readiness_read(&runtime->noc, &runtime->grid, core_index,
&readiness) != FT_TT_OK)
return true;
if (readiness.ready) {
fprintf(stderr, "pn_tt: Tensix %u came free after %u ms — %s arms normally\n",
core_index, waited_ms, who);
return true;
}
/* The reading may CHANGE under us: a spin that ends inside a wedged core stops
* ticking and starts reading 0x66. Re-asking every poll is what lets the refusal
* name the state the core is actually in rather than the one it started in. */
if (!readiness.waiting_can_help)
break;
}
}
print_refusal(&readiness, core_index, who, core_env, waited_ms);
return false;
}
#endif /* USE_TT */

109
src/accel/tt/pn_tt_core_boot_guard.h Normal file
View File

@ -0,0 +1,109 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_core_boot_guard.h
* @brief The one question asked before ANY of this backend boots firmware onto a Tensix:
* **is that core ready, or is something still running on it?**
*
* ---------------------------------------------------------------------------
* THE FAILURE THIS EXISTS FOR — IT CROSSES A PROCESS BOUNDARY
* ---------------------------------------------------------------------------
* A refused descriptor makes a resident reader abort and leaves the compute thread spinning
* out its 400,000,000-iteration bound. That is the family's design. `pn_tt_residency_stop()`
* waits it out (20 s of grace) and `pn_tt_residency_retired()` reports whether it did, so the
* process that CAUSED the spin will not re-arm onto it — those guards landed and they fire.
*
* **They protect one process. The spinning thread outlives it.** A `serve` restart or a model
* reload is a NEW process with an empty residency table, and its ordinary first arm boots
* firmware onto a core that is still running. MEASURED ON SILICON 2026-08-21/22: that is a
* WEDGE — core 90 came back `instrn_buf_status = 0x66`, twice, and rung 1 does NOT clear it;
* only a whole-card `ft_tt_core recover --rung 3` does. Nothing in a fresh process's own
* memory can see this, so the CARD is asked. `ft_tt_core_readiness.h` is that reading, and it
* is the same verdict `ft_tt_core status` has printed since 2026-08-12.
*
* ---------------------------------------------------------------------------
* IT SITS AT THE BOOT, NOT AT ONE FAMILY'S ARM
* ---------------------------------------------------------------------------
* `pn_tt_core_holder()` already refuses a core another residency of THIS process holds, by
* name. This is the other half — the core the CARD says is busy — and it has to cover every
* family, so it is called at all three places that boot firmware:
*
* - `pn_tt_prepare_family_on_core()`, which is reduce, eltwise, topk, rope, memory,
* attention, gdn, gdnc, the GDN head-split members, the render directions and both
* head-norm slots;
* - `pn_tt_matmul_group_boot()`, members 1..N;
* - `pn_tt_device_open()`'s own core, which is matmul member 0 and the FIRST boot a serve
* takes — the one a restart lands on.
*
* ---------------------------------------------------------------------------
* WAIT, THEN REFUSE — AND THE READING DECIDES WHICH
* ---------------------------------------------------------------------------
* "Not ready" is two different things. A core spinning out a bound WILL finish, so it is
* WAITED for, on the same grace `pn_tt_residency_stop()` uses because it is the same spin. A
* core at `instrn_buf_status = 0x66` never will, and waiting on it is dead time in front of
* an inevitable refusal. `ft_tt_core_readiness`'s `waiting_can_help` separates them; this
* file does not guess.
*
* Recovery is deliberately NOT attempted. Resetting a core out of a blocked wait is how cores
* get wedged in the first place (`pn_tt_residency.h` hazard 2), and rung 3 resets the WHOLE
* CARD — which is an operator's decision, not an arming path's.
*
* ---------------------------------------------------------------------------
* THE REFUSAL IS LOUD, AND THAT IS A REQUIREMENT RATHER THAN A STYLE
* ---------------------------------------------------------------------------
* A family that fails to arm can take the WHOLE MODEL to the ~26 s/position CPU forward, and
* it did so behind a single line — `qwen35 Tenstorrent upload skipped/failed; using CPU
* forward` — which burned 11 minutes of a run before anyone noticed, because the numbers that
* came out were plausible and merely very slow. A silent degradation caused BY a reliability
* fix would be worse than the bug it fixes. So the refusal prints a block, names the core,
* quotes what the card said, gives the recovery command verbatim, and states the consequence.
*
* **`PN_TT_BOOT_GUARD_REFUSAL_TOKEN` is the stable grep handle** for a timing script's abort
* guard, next to the "using CPU forward" one every script already carries.
*
* @status New 2026-08-24. The classifier under it is covered off-hardware by
* foundry/tt/tests/test_core_readiness.c.
* @issues None
* @todo None
*/
#ifndef PN_TT_CORE_BOOT_GUARD_H
#define PN_TT_CORE_BOOT_GUARD_H
#ifdef USE_TT
#include <stdbool.h>
#include <stdint.h>
#include "tt/ft_tt_runtime.h"
/** The one string a script needs to grep for. Do not reword it without saying so. */
#define PN_TT_BOOT_GUARD_REFUSAL_TOKEN "REFUSED TO BOOT"
/**
* May firmware be booted onto `core_index` right now?
*
* Reads the core. If it is busy in a way that clears itself, waits for it — up to
* `PN_TT_RESIDENCY_STOP_GRACE_MS`, the same bound the stop uses for the same spin — and
* re-reads. Returns false with a LOUD, multi-line explanation on stderr if it is still not
* ready, or if it never could be.
*
* Writes nothing to the card, resets nothing and parks nothing. Reads only.
*
* @param runtime the open card; borrowed
* @param core_index Tensix index, as `ft_tt_coord_tensix_by_index` numbers them
* @param who what wants the core, for the message, e.g. "gdn" or "matmul member 3"
* @param core_env the environment variable that would move this family to another core,
* or NULL when there is not one
* @return true when the core may be booted. **A read that itself fails returns
* true**: this guard exists to catch a specific, positively-identified
* busy core, and turning an unreadable status into a refusal would let a
* diagnostic path's failure take the whole model to the CPU forward
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
bool pn_tt_core_boot_is_permitted(ft_tt_runtime *runtime, uint32_t core_index, const char *who,
const char *core_env);
#endif /* USE_TT */
#endif /* PN_TT_CORE_BOOT_GUARD_H */

319
src/accel/tt/pn_tt_descriptor_census.c Normal file
View File

@ -0,0 +1,319 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_descriptor_census.c
* @brief Implementation of the (family, function, tile count) descriptor census. See
* pn_tt_descriptor_census.h, which carries why the per-residency report is not
* enough for the elementwise family and does not see the render pair at all.
*
* No device, no allocation, no I/O — deliberately, so it is provable with no card.
*
* @status None
* @issues None
* @todo None
*/
#include "pn_tt_descriptor_census.h"
#include <string.h>
/**
* Is `value` finite, asked of the EXPONENT BITS THROUGH A VOLATILE SLOT.
*
* `isfinite()` is foldable to a literal 1 under `-ffast-math` (which implies
* `-ffinite-math-only`), and this tree compiles with `-ffast-math`. Asking the bits is the
* recorded fix — AND THE RECORDED FIX IS NOT ENOUGH ON ITS OWN.
*
* Measured here, Apple clang, `-O3 -ffast-math`: a plain `memcpy`-then-shift written inline
* in the calling translation unit answers **1 for a positive infinity and 1 for a NaN**. The
* compiler recognises the idiom, applies `-ffinite-math-only` to it, and folds it — the exact
* fold the bit test was adopted to defeat. It is not constant propagation: it survives the
* value arriving through a `volatile uint64_t`. The same code compiled without `-ffast-math`
* answers 0 for both.
*
* The `volatile double` slot below fixes it (as does reading through a `const double *`, or
* `__attribute__((optnone))`): a volatile store forbids the compiler from reasoning about the
* value it just wrote. `tests/parity_score.c`'s `parity_double_is_finite()` has the unguarded
* shape and is safe only because it lives in its own translation unit with no LTO — turning
* LTO on would silently re-open the eleventh vacuous-gate class.
*/
static int census_double_is_finite(double value)
{
volatile double opaque = value;
double copy = opaque;
uint64_t bits;
memcpy(&bits, &copy, sizeof(bits));
return ((bits >> 52) & 0x7FFu) != 0x7FFu;
}
/** Copy `source` into a key field, always NUL-terminated, truncating rather than refusing. */
static void census_copy_key(char *destination, const char *source)
{
size_t index = 0;
if (!source) {
destination[0] = '\0';
return;
}
while (index + 1u < PN_TT_DESCRIPTOR_CENSUS_NAME_BYTES && source[index] != '\0') {
destination[index] = source[index];
index++;
}
destination[index] = '\0';
}
/** Does `row` already carry this key? The key is all three fields, never two. */
static bool census_row_matches(const pn_tt_descriptor_census_row *row, const char *family,
const char *function, uint32_t tile_count)
{
char wanted_family[PN_TT_DESCRIPTOR_CENSUS_NAME_BYTES];
char wanted_function[PN_TT_DESCRIPTOR_CENSUS_NAME_BYTES];
if (row->tile_count != tile_count)
return false;
census_copy_key(wanted_family, family);
census_copy_key(wanted_function, function);
return strcmp(row->family, wanted_family) == 0 &&
strcmp(row->function, wanted_function) == 0;
}
/**
* Find or create the row for a key, or say why not.
*
* @param out_row set on success only
* @return 0, or -1 with `refused` or `dropped` already accounted
*/
static int census_row_for_key(pn_tt_descriptor_census *census, const char *family,
const char *function, uint32_t tile_count, double seconds,
pn_tt_descriptor_census_row **out_row)
{
uint32_t index;
/* A zero width or a non-finite duration is a defect in the CALLER, and folding it into a
* row would make the fit answer with authority about a number nobody measured. */
if (tile_count == 0u || !census_double_is_finite(seconds) || seconds < 0.0) {
census->refused++;
return -1;
}
for (index = 0; index < census->row_count; index++) {
if (census_row_matches(&census->rows[index], family, function, tile_count)) {
*out_row = &census->rows[index];
return 0;
}
}
if (census->row_count >= PN_TT_DESCRIPTOR_CENSUS_ROWS) {
/* NOT an "other" bucket. A truncated census that still sums to the family total is
* indistinguishable from a complete one. */
census->dropped++;
census->dropped_seconds += seconds;
return -1;
}
index = census->row_count++;
memset(&census->rows[index], 0, sizeof(census->rows[index]));
census_copy_key(census->rows[index].family, family);
census_copy_key(census->rows[index].function, function);
census->rows[index].tile_count = tile_count;
*out_row = &census->rows[index];
return 0;
}
int pn_tt_descriptor_census_record(pn_tt_descriptor_census *census, const char *family,
const char *function, uint32_t tile_count, double seconds)
{
pn_tt_descriptor_census_row *row = NULL;
if (!census ||
census_row_for_key(census, family, function, tile_count, seconds, &row) != 0)
return -1;
row->ops++;
row->seconds += seconds;
return 0;
}
int pn_tt_descriptor_census_record_recovered(pn_tt_descriptor_census *census,
const char *family, const char *function,
uint32_t tile_count, double seconds,
uint64_t rearms)
{
pn_tt_descriptor_census_row *row = NULL;
if (!census)
return -1;
/* A "recovered" descriptor that re-armed nothing is a CLEAN one under the wrong name, and
* accepting it would let a caller move time out of the fit by mislabelling it. */
if (rearms == 0u) {
census->refused++;
return -1;
}
if (census_row_for_key(census, family, function, tile_count, seconds, &row) != 0)
return -1;
row->recovered_ops++;
row->recovered_seconds += seconds;
row->rearms += rearms;
return 0;
}
bool pn_tt_descriptor_census_row_at(const pn_tt_descriptor_census *census, uint32_t index,
const pn_tt_descriptor_census_row **out_row)
{
if (!census || index >= census->row_count)
return false;
if (out_row)
*out_row = &census->rows[index];
return true;
}
/** Does `row` belong to `family`? A NULL family means every row. */
static bool census_row_in_family(const pn_tt_descriptor_census_row *row, const char *family)
{
char wanted[PN_TT_DESCRIPTOR_CENSUS_NAME_BYTES];
if (!family)
return true;
census_copy_key(wanted, family);
return strcmp(row->family, wanted) == 0;
}
/**
* Distinct tile counts among a family's CLEAN rows, counted the O(n^2) way; n is at most 64.
*
* Clean because this is the fit's shape guard: a row that only ever refused contributes no
* equation, so counting it would let a one-point fit past the first of the two redundant
* guards and leave the determinant test alone with the job.
*/
static uint32_t census_distinct_shapes(const pn_tt_descriptor_census *census,
const char *family)
{
uint32_t distinct = 0;
uint32_t index;
for (index = 0; index < census->row_count; index++) {
uint32_t earlier;
bool seen = false;
if (!census_row_in_family(&census->rows[index], family) || census->rows[index].ops == 0u)
continue;
for (earlier = 0; earlier < index && !seen; earlier++)
seen = census_row_in_family(&census->rows[earlier], family) &&
census->rows[earlier].ops != 0u &&
census->rows[earlier].tile_count == census->rows[index].tile_count;
if (!seen)
distinct++;
}
return distinct;
}
int pn_tt_descriptor_census_family_totals(const pn_tt_descriptor_census *census,
const char *family, uint64_t *out_ops,
double *out_seconds, uint32_t *out_shapes)
{
uint64_t ops = 0;
double seconds = 0.0;
uint32_t rows = 0;
uint32_t index;
if (!census)
return -1;
for (index = 0; index < census->row_count; index++) {
if (!census_row_in_family(&census->rows[index], family))
continue;
ops += census->rows[index].ops;
seconds += census->rows[index].seconds;
rows++;
}
if (rows == 0u)
return -1;
if (out_ops)
*out_ops = ops;
if (out_seconds)
*out_seconds = seconds;
if (out_shapes)
*out_shapes = census_distinct_shapes(census, family);
return 0;
}
int pn_tt_descriptor_census_family_recovery(const pn_tt_descriptor_census *census,
const char *family, uint64_t *out_recovered_ops,
double *out_recovered_seconds,
uint64_t *out_rearms)
{
uint64_t recovered_ops = 0;
double recovered_seconds = 0.0;
uint64_t rearms = 0;
uint32_t rows = 0;
uint32_t index;
if (!census)
return -1;
for (index = 0; index < census->row_count; index++) {
if (!census_row_in_family(&census->rows[index], family))
continue;
recovered_ops += census->rows[index].recovered_ops;
recovered_seconds += census->rows[index].recovered_seconds;
rearms += census->rows[index].rearms;
rows++;
}
/* Only "the family never ran" refuses. A family that ran and never refused answers zeros,
* because a reader has to be able to print "0 re-arms" rather than print nothing. */
if (rows == 0u)
return -1;
if (out_recovered_ops)
*out_recovered_ops = recovered_ops;
if (out_recovered_seconds)
*out_recovered_seconds = recovered_seconds;
if (out_rearms)
*out_rearms = rearms;
return 0;
}
int pn_tt_descriptor_census_fit(const pn_tt_descriptor_census *census, const char *family,
double *out_fixed_seconds, double *out_seconds_per_tile,
uint32_t *out_shapes)
{
double weight_sum = 0.0, weighted_x = 0.0, weighted_y = 0.0;
double weighted_xx = 0.0, weighted_xy = 0.0;
uint32_t shapes;
uint32_t index;
double determinant;
if (!census)
return -1;
shapes = census_distinct_shapes(census, family);
/* TWO UNKNOWNS NEED TWO EQUATIONS. A family that ran at one width cannot say which part
* of its descriptor is fixed, and answering anyway is how "958 ns per single-tile op"
* was produced from a division. */
if (shapes < 2u)
return -1;
for (index = 0; index < census->row_count; index++) {
const pn_tt_descriptor_census_row *row = &census->rows[index];
double weight, x, y;
if (!census_row_in_family(row, family) || row->ops == 0u)
continue;
weight = (double)row->ops;
x = (double)row->tile_count;
y = row->seconds / weight;
weight_sum += weight;
weighted_x += weight * x;
weighted_y += weight * y;
weighted_xx += weight * x * x;
weighted_xy += weight * x * y;
}
determinant = weight_sum * weighted_xx - weighted_x * weighted_x;
/* Scale-free degeneracy test: the absolute determinant of a fit over 7,760-tile shapes is
* enormous while the same fit over 4-tile shapes is tiny, so a fixed epsilon would refuse
* one family and accept a collinear other. */
if (!census_double_is_finite(determinant) ||
determinant <= 1e-12 * weight_sum * weighted_xx)
return -1;
if (out_seconds_per_tile)
*out_seconds_per_tile = (weight_sum * weighted_xy - weighted_x * weighted_y) /
determinant;
if (out_fixed_seconds)
*out_fixed_seconds = (weighted_xx * weighted_y - weighted_x * weighted_xy) /
determinant;
if (out_shapes)
*out_shapes = shapes;
return 0;
}

234
src/accel/tt/pn_tt_descriptor_census.h Normal file
View File

@ -0,0 +1,234 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_descriptor_census.h
* @brief Attribute on-card descriptor time by (FAMILY, FUNCTION, TILE COUNT) — one level
* below the per-residency report, with no card and no device in it.
*
* ---------------------------------------------------------------------------
* WHY THIS EXISTS: "293.8 us EACH" IS THE SAME AVERAGE, ONE LEVEL DOWN
* ---------------------------------------------------------------------------
* `qwen35_tt_report_residencies()` was written because "162 descriptors a token, 247 us
* each" averaged over families whose work differs by three orders of magnitude. It replaced
* that with a line per residency — and the line it produced for the elementwise family reads
* **136 descriptors/token, 293.8 us each**, which is the identical mistake at the next level:
*
* - the eltwise residency runs SIX different functions (silu, sigmoid, add, mul, sub,
* silu_mul), unary and binary, over at least THREE different widths (`n_embd/32`, the
* FFN width, the attention gate's width);
* - a unary SIGMOID over the gate and a binary SILU_MUL over the FFN are both "one
* descriptor", and one reads half the bytes of the other;
* - the two RENDER directions run the same three kernel sources in two other programs at
* two other page sizes, and they are **not in the residency table at all** — the
* bridge's ~0.022 s/token has only ever been a SUBTRACTION between runs.
*
* So the on-card term's largest non-matmul family has never been decomposed, and the second
* largest is invisible to the instrument. This file is the missing key.
*
* ---------------------------------------------------------------------------
* AND WHAT THE KEY BUYS THAT A COUNT CANNOT
* ---------------------------------------------------------------------------
* A family that runs the SAME program over SEVERAL widths prices its own per-descriptor
* fixed cost, from one ordinary run, with no extra card arm:
*
* seconds_per_descriptor(width) = fixed + per_tile * tile_count
*
* `pn_tt_descriptor_census_fit()` solves that line by weighted least squares over a family's
* rows. **The intercept is what merging descriptors removes and the slope is what it does
* not** — which is the one question a "N descriptors x 293.8 us" figure cannot answer, and
* the one that three levers here have already been scoped wrong from (two of which measured
* exactly 0 when built).
*
* The intercept is NOT "the host round trip" and must not be reported as one. It is
* everything that does not scale with the strip: the parameter write, the ring write, the
* head bump, the first poll's PCIe latency, the kernel's own per-descriptor prologue, the CB
* rebase and the completion post. Separating THOSE needs
* `pn_tt_residency_submit_repeat()`, which is a different instrument and a card arm.
*
* ---------------------------------------------------------------------------
* WHAT IT REFUSES, AND WHY EACH REFUSAL IS LOUD
* ---------------------------------------------------------------------------
* A census that silently drops rows reads exactly like a census of a cheaper model. So:
*
* - a table overflow does NOT fold into an "other" bucket — it increments `dropped`, and
* any reader that prints a census must print `dropped` beside it;
* - a fit over ONE distinct width is refused, not answered. Two unknowns need two
* equations, and a line through one point is the arithmetic that produced the retired
* "958 ns per single-tile op". TWO guards enforce this — the named shape count and the
* normal equation's determinant — and they are REDUNDANT WITH EACH OTHER: mutating either
* one away leaves the suite green and only removing BOTH goes red (4 failures). They are
* both kept, and that redundancy is stated here rather than left as a check nobody can
* prove red on its own;
* - a non-finite duration is refused at record time, because `failed = value > limit` is
* false for a NaN and a NaN that reaches a fit poisons every figure downstream.
*
* ---------------------------------------------------------------------------
* A DESCRIPTOR THAT REFUSED AND WAS REPLAYED IS NOT A DESCRIPTOR'S COST
* ---------------------------------------------------------------------------
* Every resident family in this backend submits under a recovery: if the residency declines
* the descriptor the host RE-ARMS it — a full teardown and relaunch — and submits again, and
* the caller's stopwatch spans all three. Folding that bracket into `seconds / ops` prices a
* teardown as arithmetic, and it has already produced a wrong headline number: PHASE CONV read
* **4,502 us a descriptor against 738.2** for no reason but a circular-buffer write pointer
* that made every CONV after a DELTA refuse. The recorded "CONV is DATA-DEPENDENT, 4,571 us on
* a degenerate stream" has the same shape and no surviving harness.
*
* So `ops` and `seconds` are the CLEAN descriptors — first attempt, retired — and a replayed
* one goes to `recovered_ops` / `recovered_seconds` / `rearms` instead. Three consequences,
* and the middle one is the point:
*
* - `pn_tt_descriptor_census_fit()` sees clean rows only, so a run with refusals fits the
* op rather than the recovery;
* - a reader that prints `us each` MUST print `rearms` beside it, exactly as it must print
* `dropped` — a contaminated figure that looks like a clean one is the whole defect;
* - `pn_tt_descriptor_census_family_totals()` answers the CLEAN totals, so wall-time
* accounting that wants everything adds `pn_tt_descriptor_census_family_recovery()`.
*
* No device, no allocation, no I/O — deliberately, so it is provable with no card.
*
* @status None
* @issues None
* @todo None
*/
#ifndef PN_TT_DESCRIPTOR_CENSUS_H
#define PN_TT_DESCRIPTOR_CENSUS_H
#include <stdbool.h>
#include <stdint.h>
/** Distinct (family, function, tile count) keys the table holds. Ornith's decode path uses
* well under half of these; the margin is for a hybrid arm that runs more shapes. */
#define PN_TT_DESCRIPTOR_CENSUS_ROWS 64u
/** Key strings are COPIED, not borrowed, so a caller may pass a stack buffer. */
#define PN_TT_DESCRIPTOR_CENSUS_NAME_BYTES 24u
/** One key and everything measured against it. */
typedef struct pn_tt_descriptor_census_row {
char family[PN_TT_DESCRIPTOR_CENSUS_NAME_BYTES]; /**< "eltwise", "render-widen", ... */
char function[PN_TT_DESCRIPTOR_CENSUS_NAME_BYTES]; /**< "add", "silu_mul", "widen", ... */
uint32_t tile_count; /**< tiles the descriptor covers */
uint64_t ops; /**< CLEAN descriptors on this key — first attempt, retired */
double seconds; /**< summed submit wall time of those clean descriptors */
/** Descriptors the residency REFUSED and the host replayed after a re-arm, and the whole
* bracket they cost. Never folded into `seconds`: a teardown is not a descriptor's work,
* and `seconds / ops` is quoted as one. */
uint64_t recovered_ops;
double recovered_seconds;
/** Re-arms performed inside those brackets. `>= recovered_ops`; a caller that re-arms more
* than once for one descriptor says so. */
uint64_t rearms;
} pn_tt_descriptor_census_row;
/** The whole instrument. Zero-initialise it; there is no constructor. */
typedef struct pn_tt_descriptor_census {
pn_tt_descriptor_census_row rows[PN_TT_DESCRIPTOR_CENSUS_ROWS];
uint32_t row_count;
/** Descriptors that matched no row and found no free slot, and their time. NEVER folded
* into a row: an "other" bucket makes a truncated census look complete. */
uint64_t dropped;
double dropped_seconds;
/** Descriptors refused at record time — a bad width or a non-finite duration. Kept apart
* from `dropped` because they mean a defect upstream, not a table too small. */
uint64_t refused;
} pn_tt_descriptor_census;
/**
* Add one descriptor's time to its key, creating the row on first sight.
*
* @param census the table; NULL is refused
* @param family residency-level name, truncated to the key width
* @param function what the descriptor computed, truncated to the key width
* @param tile_count tiles the descriptor covers; 0 is refused
* @param seconds measured submit wall time; non-finite or negative is refused
* @return 0, or -1 with `refused` or `dropped` incremented
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
int pn_tt_descriptor_census_record(pn_tt_descriptor_census *census, const char *family,
const char *function, uint32_t tile_count, double seconds);
/**
* Add one REPLAYED descriptor's whole bracket — refusal, re-arm, resubmit — to its key.
*
* Identical keying and identical refusals to `pn_tt_descriptor_census_record()`; the only
* difference is which pair of counters moves, and that difference is the point. Nothing this
* function records ever reaches `seconds / ops` or the fit.
*
* @param census the table; NULL is refused
* @param family residency-level name, truncated to the key width
* @param function what the descriptor computed, truncated to the key width
* @param tile_count tiles the descriptor covers; 0 is refused
* @param seconds the WHOLE bracket, not the replay alone; non-finite or negative refused
* @param rearms re-arms performed inside the bracket; 0 is refused, because a recovered
* descriptor that re-armed nothing is a caller that mislabelled a clean one
* @return 0, or -1 with `refused` or `dropped` incremented
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
int pn_tt_descriptor_census_record_recovered(pn_tt_descriptor_census *census,
const char *family, const char *function,
uint32_t tile_count, double seconds,
uint64_t rearms);
/**
* Read row `index` back, in first-sight order.
*
* @return false once `index` reaches `row_count`, which is the loop's terminator
*/
bool pn_tt_descriptor_census_row_at(const pn_tt_descriptor_census *census, uint32_t index,
const pn_tt_descriptor_census_row **out_row);
/**
* Sum one family's rows.
*
* @param family matched exactly against the recorded key; NULL sums EVERY row
* @return 0, or -1 when the family has no rows (so a caller cannot print zeros
* for a family that never ran and read it as "free")
*/
int pn_tt_descriptor_census_family_totals(const pn_tt_descriptor_census *census,
const char *family, uint64_t *out_ops,
double *out_seconds, uint32_t *out_shapes);
/**
* The same family's REPLAYED descriptors — what `..._family_totals()` deliberately leaves out.
*
* A reader that prints a per-descriptor figure has to print `out_rearms` beside it. It is
* answered as 0/0/0 rather than refused when the family ran clean, because "the family has no
* recovery" and "the family never ran" are different facts and only the second is a refusal.
*
* @param family matched exactly against the recorded key; NULL sums EVERY row
* @param out_recovered_ops descriptors that refused and were replayed
* @param out_recovered_seconds their whole brackets, summed
* @param out_rearms re-arms performed inside them
* @return 0, or -1 when the family has no rows at all
*/
int pn_tt_descriptor_census_family_recovery(const pn_tt_descriptor_census *census,
const char *family, uint64_t *out_recovered_ops,
double *out_recovered_seconds,
uint64_t *out_rearms);
/**
* Solve `seconds_per_descriptor = fixed + per_tile * tile_count` for one family, by least
* squares over its rows weighted by descriptor count.
*
* REFUSED unless the family shows at least two DISTINCT tile counts. Read the header's
* "what the key buys" block before quoting the intercept as a host round trip — it is not
* one, and this function cannot tell you that it is.
*
* CLEAN ROWS ONLY. A row whose descriptors all refused and were replayed contributes no
* equation, so a family that never retired a first attempt is refused rather than fitted to
* its own recovery path.
*
* @param out_fixed_seconds intercept, seconds per descriptor at zero tiles
* @param out_seconds_per_tile slope, seconds per tile
* @param out_shapes distinct tile counts the fit used; NULL to ignore
* @return 0, or -1 (no such family, one shape only, or a degenerate
* normal equation)
*/
int pn_tt_descriptor_census_fit(const pn_tt_descriptor_census *census, const char *family,
double *out_fixed_seconds, double *out_seconds_per_tile,
uint32_t *out_shapes);
#endif /* PN_TT_DESCRIPTOR_CENSUS_H */

2112
src/accel/tt/pn_tt_device.c Normal file
View File

File diff suppressed because it is too large Load Diff

1340
src/accel/tt/pn_tt_device.h Normal file
View File

File diff suppressed because it is too large Load Diff

948
src/accel/tt/pn_tt_device_internal.h Normal file
View File

@ -0,0 +1,948 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_device_internal.h
* @brief The layout of `struct pn_tt_device`, shared by the two source files that own it.
*
* pn_tt_device.c owns the card, the artifact residency and the matmul; pn_tt_oncard.c
* owns the reduce/elementwise/sampling residencies and the DRAM activation buffers
* they pass to each other. They are one object and two files because they are two
* subjects, and the struct is here rather than in the public header so a caller
* still cannot reach into it.
*
* @status None
* @issues None
* @todo None
*/
#ifndef PN_TT_DEVICE_INTERNAL_H
#define PN_TT_DEVICE_INTERNAL_H
/** Mask tiles for verify attention in ROWS mode: up to 4 x (32 / q_rows) rows. */
#define PN_TT_ATTN_ROWS_CHUNKS 4u
#ifdef USE_TT
#include <stdio.h>
#include "pn_tt_descriptor_census.h"
#include "pn_tt_device.h"
#include "pn_tt_eltwise_split.h"
#include "pn_tt_fp32_strip.h"
#include "pn_tt_gdn.h"
#include "pn_tt_matmul_group.h"
#include "pn_tt_residency.h"
#include "tt/ft_tt_buffer.h"
#include "tt/ft_tt_fttw.h"
#include "tt/ft_tt_op_memory.h"
#include "tt/ft_tt_op_vector.h"
#include "tt/ft_tt_queue.h"
#include "tt/ft_tt_runtime.h"
#include "pn_tt_stage_pipeline.h"
/** Staging slots for PHASE DELTA — one per 16-row sub-chunk of a 32-row pass. */
#define PN_TT_GDNC_STAGE_SLOTS 2u
/** Most cores the prefill attention chunk mixer may split its Q heads across. */
#define PN_TT_ATTN_MAX_MEMBERS 8u
/** Most cores one per-head norm width may split its heads across. */
#define PN_TT_HEAD_NORM_MAX_MEMBERS 8u
/** Bounds every wait. There is no untimed path to the card (spec §7.3). */
#define PN_TT_OP_TIMEOUT_MS 60000u
#define PN_TT_TILE_DIM FT_TT_OP_TILE_DIM
#define PN_TT_TILE_BF16 FT_TT_OP_TILE_BYTES_BF16
#define PN_TT_TILE_FP32 FT_TT_OP_TILE_BYTES_FP32
/** THE CEILING IS THE DST SECTION, NOT L1. A band only pays if `MB` grows with it, and
* `MB * NB <= 8` (resident_matmul_block_protocol.h) makes MB=2/NB=4 the only point on the
* lattice that beats the shipped MB=1/NB=8 — so a third band buys nothing the second did not
* and costs another pass of both chunk mixers. 4 is slack for the arm that proves that. */
#define PN_TT_STRIP_MAX_TILE_ROWS 64u
/** How many distinct elementwise tile counts ONE arming may be asked to serve. The forward
* asks for two (n_embd and n_ff) and the slack is so a third width needs no header change.
* The shapes are kept rather than consumed because a residency is BOUNDED — an idle kernel
* retires itself — and re-arming one needs the shapes it was armed with. */
#define PN_TT_ONCARD_MAX_ELTWISE_SHAPES 8u
/** How many displaced KV sequence keys are remembered, i.e. how deep an interleave this
* detects. A key that comes back after being displaced is two sequences alternating on ONE
* region set; beyond this depth the detector goes blind, which is why it is a stated bound
* and not a guarantee. */
#define PN_TT_KV_DISPLACED_KEYS 8u
/** One resident weight plus the name it answers to. */
typedef struct {
char name[FT_TT_ARTIFACT_NAME_BYTES];
pn_tt_weight weight;
} pn_tt_resident;
/** One fp32 buffer of the GDN family in card DRAM. A bank and an offset, which is everything
* `ft_tt_op_operand` needs — the tile size is the family's, not the buffer's, because every
* circular buffer of the `gdn` program is Float32. */
/** PM 7 ITEM 5: most GDN GATE members one device arms. */
#define PN_TT_GDNGATE_MAX_MEMBERS 8u
typedef struct pn_tt_gdn_buffer {
uint32_t bank;
uint32_t address;
} pn_tt_gdn_buffer;
/** GDN residencies one head-split may hold, member 0 included.
*
* EIGHT because that is where the measurement stops, not where the card does: the core
* scaling probe reached 7.12x on 8 and the next doubling was never taken. Raising it is a
* measurement, not an edit — every extra member is a Tensix `pn_tt_prepare_family_on_core()`
* has to be able to refuse against the matmul group by name. */
/** L3: most top-k split members, member 0 included. */
#define PN_TT_TOPK_MAX_MEMBERS 8u
#define PN_TT_GDN_MAX_MEMBERS 16u
/** CHUNKWISE residencies one core split may hold, member 0 included.
*
* SIXTEEN rather than the decode family's eight because the two phases divide by different
* quantities and the smaller one still admits it: PHASE CONV has **256 tile columns** at
* Ornith's conv_dim of 8192 and PHASE DELTA has **32 v-heads**, so 16 members own 16 columns
* or 2 heads each and both divide exactly. It is a ceiling on the LADDER, not a claim about
* the knee — the decode family's knee was 4 with 8 reproducibly behind it. Every extra member
* is a Tensix `pn_tt_prepare_family_on_core()` has to be able to refuse by name. */
#define PN_TT_GDNC_MAX_MEMBERS 16u
/** One SSM layer's GDN buffers. `weight` is a constant staged once; the other two are the
* carried state, READ AND WRITTEN by their descriptors and never downloaded. */
typedef struct pn_tt_gdn_region {
pn_tt_gdn_buffer weight; /**< taps TAP-MAJOR planes, transposed at load */
pn_tt_gdn_buffer window; /**< taps-1 planes, the same layout, rolled on the card */
pn_tt_gdn_buffer state; /**< [v_heads, S, S] fp32, oracle-transposed */
} pn_tt_gdn_region;
/** One layer's K and V regions in card DRAM. Identical geometry and two separate bases —
* contract F1 and N4 (two regions rather than one with a fixed offset, so each is a plain
* ft_tt_op_operand). Internal because no public call takes or returns one. */
typedef struct pn_tt_kv_region {
uint32_t k_bank, k_address;
uint32_t v_bank, v_address;
} pn_tt_kv_region;
/** Distinct HEAD widths one model may need per-head reduces at.
*
* TWO, because Ornith needs exactly two and they are not the same number: its
* Gated-Delta-Net per-head norms are over S = 128 (Wt 4) and its attention QK-RMSNorms are
* over head_dim = 256 (Wt 8). A reduce residency is PLACED for one Wt and refuses any
* other, so these cannot share a core — see pn_tt_head_norm.h. Raising this is one more
* Tensix, which the card has; it is bounded so the collision refusal has a fixed set to
* check. */
#define PN_TT_HEAD_NORM_MAX_WIDTHS 2u
/** One per-head reduce residency, the ONE width it was placed for, and the ONE page size —
* which is also which reduce PROGRAM it runs, since the two are the same choice. */
typedef struct pn_tt_head_norm_residency {
pn_tt_residency residency;
uint32_t tile_cols; /**< width/32; 0 for a slot that never armed */
uint32_t tile_bytes; /**< PN_TT_TILE_FP32 (`reduce_fp32`) or PN_TT_TILE_BF16 (`reduce`) */
/** THE HEAD SPLIT (`PN_TT_HEAD_NORM_CORES`): members 1..N-1 run the same program and take
* disjoint HEAD ranges. Heads are independent row blocks and the wire carries no absolute
* block index, so a member is the same descriptor at an offset base — host-only. */
pn_tt_residency member[PN_TT_HEAD_NORM_MAX_MEMBERS - 1u];
uint32_t members; /**< 1 = no split */
} pn_tt_head_norm_residency;
/** The two directions of the activation strip bridge, in the order `render[]` holds them. */
enum { PN_TT_RENDER_WIDEN = 0, PN_TT_RENDER_NARROW = 1, PN_TT_RENDER_DIRECTIONS = 2 };
/** Residencies ONE DIRECTION of the bridge may hold, member 0 included.
*
* EIGHT and not sixteen, because the two directions are split independently and both sets
* have to fit above every core any other family can reach: at eight apiece they occupy
* 122..128 and 129..135 of the 122..139 the elementwise split's map leaves free. Eight also
* divides both of Ornith's render widths (256 widen, 128 narrow) and leaves the narrow slice
* at 16 tiles, still two blocks of depth 8. Raising it is more Tensix and a re-check of that
* map, not an edit here. */
#define PN_TT_RENDER_SPLIT_MAX_MEMBERS 8u
/** One direction of the bridge: a residency, the two page sizes it converts between, and the
* ZERO operand that turns `kVecFnAdd` into a copy. See pn_tt_render.h. */
typedef struct pn_tt_render_residency {
pn_tt_residency residency;
/** Members 1..N-1 of the split; member 0 is `residency` above. See pn_tt_render.h. */
pn_tt_residency member[PN_TT_RENDER_SPLIT_MAX_MEMBERS - 1u];
uint32_t members; /**< live residencies, member 0 included; 1 when unsplit */
/** Per-member AWAIT seconds. Kept apart from the census, which keys on the WHOLE
* descriptor's width so its `fixed + per_tile * tiles` fit stays comparable across a
* member ladder — what that cannot show is the SHAPE of the set, and "it stopped at N"
* is exactly the finding that needs it. */
double member_seconds[PN_TT_RENDER_SPLIT_MAX_MEMBERS];
uint32_t in_tile_bytes;
uint32_t out_tile_bytes;
uint32_t tiles; /**< the widest strip placed for; 0 when this direction never armed */
uint32_t depth; /**< the ONE block depth every descriptor of it carries */
uint32_t zero_bank;
uint32_t zero_address; /**< `tiles` input-format tiles, allocated zeroed, never written */
} pn_tt_render_residency;
/** THE MERGED GDN OUTPUT GATE — `silu(bf16 z) * fp32 y_norm -> bf16 gated` under a MIXED CB
* table, which is what lets `render_narrow` leave the decode token. No zero operand: this one
* is a real binary op, not a copy dressed as one. See pn_tt_gate.h. */
typedef struct pn_tt_gate_residency {
pn_tt_residency residency;
uint32_t tiles; /**< the widest gate placed for; 0 when it never armed */
uint32_t depth; /**< the ONE block depth every descriptor of it carries */
} pn_tt_gate_residency;
struct pn_tt_device {
ft_tt_runtime runtime;
ft_tt_runtime_images firmware;
ft_tt_runtime_images program_bfp8;
ft_tt_runtime_images program_bfp4;
bool have_bfp4;
/* The resident triad: the same three kernels, built to not return between ops. */
ft_tt_runtime_images program_resident;
bool have_resident; /**< the images loaded and the firmware matches */
bool resident_requested; /**< PN_TT_DISPATCH did not say `oneshot` */
bool resident_running; /**< a residency is live on the core RIGHT NOW */
/** Member 0's work queue. It lives here rather than in the group because the device
* still owns exactly one residency's lifetime; the group holds the rest. */
ft_tt_queue queue;
/** The cores one projection is split across. One member is the pre-group path. */
pn_tt_matmul_group group;
pn_tt_red_split red_split; /**< RED PROOF: break member 1 in one named way */
uint32_t poll_backoff_us; /**< sleep between completion polls; 0 = tight loop */
uint32_t resident_tile_bytes; /**< the ONE weight tile size the residency serves */
uint32_t resident_depth; /**< the ONE prefetch depth every descriptor carries */
uint32_t artifact_tile_bytes; /**< the artifact's single matmul tile size, 0 if mixed */
uint32_t artifact_kt_gcd; /**< every matmul Kt divides by this */
uint64_t red_bad_descriptor; /**< RED PROOF: malform this matvec's parameters, 0 = never */
/** RED PROOF `PN_TT_RED=fast_check`: ask the one-line completion check about one
* descriptor further on than the poll waited for. It must REFUSE; green would mean the
* fast path's check is inert and the whole member-count ladder measures nothing. */
bool red_fast_check;
/** RED PROOF `PN_TT_RED=param_cache_hash`: collapse the cache's hash to a constant. The
* memcmp still decides, so this arm must be `cmp` IDENTICAL — which is the claim that
* the hash is a bucket index and nothing more. */
bool red_param_cache_hash;
/** RED PROOF `PN_TT_RED=param_cache_blind`: take the first bucket occupant WITHOUT the
* memcmp. Paired with the constant hash every projection reuses block 0, so this arm
* must produce a `cmp` that DIFFERS. Green would mean the words never varied. */
bool red_param_cache_blind;
uint64_t resident_ops; /**< matvecs executed through the queue */
uint64_t resident_restarts; /**< times the residency had to be re-armed */
uint16_t core_x, core_y;
uint32_t core_index;
ft_tt_allocator allocator;
ft_tt_noc_coord dram[FT_TT_MAX_DRAM_BANKS];
uint32_t dram_bank_count;
ft_tt_fttw artifact;
bool artifact_open;
/** STEP 4 S2: a second artifact (the MTP draft head) whose weights are placed on demand. */
ft_tt_fttw extra_artifact;
bool extra_open;
pn_tt_resident *resident;
uint32_t resident_count;
uint64_t resident_bytes;
/* ------------------------------------------------------------
* The ON-CARD residencies — one core per op family
* ------------------------------------------------------------
* A resident binary is linked at a fixed slab, so one program per core. The forward
* needs three families in sequence per layer, so it gets three cores and they pass
* activations through DRAM instead of through the host.
*/
pn_tt_residency reduce; /**< RMS norm (the reduce family) */
/** L4: the second RMS-norm member — each owns half of every row's final chain and
* output (kRedFinal byte 3); both reduce the whole row. `PN_TT_REDUCE_CORES=2`. */
pn_tt_residency reduce_member;
/** MEASURE ONLY (2026-10-05, PHASE G pre-work): `PN_TT_IDLE_FILL=N` arms N extra reduce
* residencies on cores 0..N-1 that never receive work — the price of RESIDENT-BUT-IDLE
* cores, which a full-grid prefill would add. */
pn_tt_residency idle_fill[64];
uint32_t idle_fill_count;
uint32_t reduce_members;
pn_tt_residency eltwise; /**< SwiGLU gate and residual add */
pn_tt_residency sample; /**< exact top-N over the logit row */
pn_tt_residency rope; /**< NeoX rotary, on q and k */
pn_tt_residency memory; /**< the KV append (no compute thread: DM0|DM1 only) */
/** STEP 3 (A): the on-card puller (BRISC only, `PN_TT_PULLER_CORE`, default 89). */
pn_tt_residency puller;
pn_tt_residency attention; /**< flash decode, one descriptor per KV group */
/** THE CHUNK MIXER'S HEAD SPLIT (`PN_TT_ATTN_CORES`). Members 1..N-1 run the SAME program
* on their own cores and take disjoint Q-head ranges of a prefill chunk; decode never
* uses them. Every word of an attention descriptor is a DRAM address with the head folded
* in by the host, so the split needs no kernel change. */
pn_tt_residency attention_member[PN_TT_ATTN_MAX_MEMBERS - 1u];
uint32_t attention_members; /**< 1 = no split */
double attention_member_seconds[PN_TT_ATTN_MAX_MEMBERS];
pn_tt_residency gdn; /**< the Gated-Delta-Net block, two descriptors per SSM layer */
/** THE CHUNKWISE Gated-Delta-Net block — the same two phases over C <= 32 POSITIONS in one
* descriptor, for prefill. It carries NO STATE OF ITS OWN: the conv window and the
* recurrent state it advances are `gdn_region[]`'s, at their addresses, in their layout,
* which is what makes the prefill->decode seam free. See pn_tt_gdnc.h. */
pn_tt_residency gdnc;
/** EXTRA CHUNKWISE residencies, behind `PN_TT_GDNC_CORES`. The work divides the same two
* ways the decode family's does and the WIRE HAS CARRIED THE SLICE SINCE THIS FAMILY
* SHIPPED (`ft_tt_op_gdnc_request.first_col/.col_count`, `kGdncColSlice`): PHASE CONV
* splits TILE COLUMNS — 256 of them at Ornith's conv_dim of 8192, and no column reads
* another column's operands or writes another column's tile — and PHASE DELTA splits
* V-HEADS, exactly as `gdn_member[]` does. Every member carries the IDENTICAL
* circular-buffer placement, because `resolve_cb_pages()` is sized from `state_tiles` and
* `conv_taps` and from neither slice. `gdnc` itself is member 0, so this array holds one
* fewer. See pn_tt_gdnc.c's PN_TT_GDNC_CORES block. */
pn_tt_residency gdnc_member[PN_TT_GDNC_MAX_MEMBERS - 1u];
uint32_t gdnc_members; /**< 1 when unsplit — `gdnc` alone */
/** EXTRA GDN residencies for the PHASE DELTA head split, ON BY DEFAULT AT 4 behind
* `PN_TT_GDN_HEAD_CORES`. PHASE DELTA divides EXACTLY by v-head — measured
* `59.8 us + N x 103.4` — so N members each owning `v_heads/N` heads is the whole
* lever. **It parallelises 1.83 / 2.70 / 2.65x on 2 / 4 / 8, not the 1.96 / 3.79 /
* 7.12x the core-scaling probe reported**: that probe gave every core a DIFFERENT
* layer's state, so no two descriptors shared DRAM, where a real split shares the
* staged q|k|v and the gate block and scatters into the same four `y` tiles. Every
* member carries the IDENTICAL circular-buffer placement, because nothing this family
* places is sized from `v_heads`. `gdn` itself is member 0, so this array holds one
* fewer. */
pn_tt_residency gdn_member[PN_TT_GDN_MAX_MEMBERS - 1u];
uint32_t gdn_members; /**< 1 when unsplit — `gdn` alone */
/** EXTRA ELTWISE residencies, behind `PN_TT_ELTWISE_CORES`. Every descriptor of the
* family is split into `eltwise_members` contiguous, DISJOINT tile ranges — no shared
* buffer, no shared DRAM byte, no reduction — and the set is enqueued before any member
* is awaited. `eltwise` itself is member 0, so this array holds one fewer. See
* pn_tt_eltwise_group.h for the core map and why 106 upward cannot collide. */
pn_tt_residency eltwise_member[PN_TT_ELTWISE_SPLIT_MAX_MEMBERS - 1u];
uint32_t eltwise_members; /**< 1 when unsplit — `eltwise` alone; 0 before it arms */
/** PHASE W1a: how many of `eltwise_members` serve EVERY descriptor (the decode split).
* Members past it are WIDE members (`PN_TT_ELTWISE_WIDE_CORES`), which only descriptors
* of `PN_TT_ELTWISE_WIDE_MIN_TILES` or more use — the prefill SwiGLU. */
uint32_t eltwise_decode_members;
/** PM 7: the NEXT eltwise descriptor is enqueued and NOT awaited
* (`pn_tt_eltwise_group_defer_next`); it stays pending until
* `pn_tt_eltwise_group_settle()` (or the next eltwise descriptor) awaits it. Streamed,
* that moves its puller AWAIT from straight after it to where a consumer needs it. */
bool eltwise_defer_next;
uint32_t eltwise_pending_members; /**< members of the pending descriptor, 0 = none */
uint32_t eltwise_pending_words[PN_TT_ELTWISE_SPLIT_MAX_MEMBERS][FT_TT_OP_VECTOR_WORDS];
ft_tt_op_eltwise_request eltwise_pending_request;
double eltwise_pending_started;
uint64_t eltwise_deferred; /**< descriptors deferred over the process */
/** PM 7 ITEM 5: the GDN GATE family (pn_tt_gdngate.h) — members, the per-layer constants
* table, the per-layer gate-block arrays, and the last issue (for a refused member's
* replay). 0 members = not armed: the host gate path stays. */
pn_tt_residency gdngate[PN_TT_GDNGATE_MAX_MEMBERS];
uint32_t gdngate_members, gdngate_layers, gdngate_heads, gdngate_max_bands;
uint32_t gdngate_scale_bits, gdngate_block_stride;
pn_tt_gdn_buffer gdngate_constants, gdngate_blocks;
uint32_t gdngate_pending[PN_TT_GDNGATE_MAX_MEMBERS];
uint32_t gdngate_next_member;
uint32_t gdngate_cores[PN_TT_GDNGATE_MAX_MEMBERS];
bool gdngate_failed;
uint64_t gdngate_issues;
uint64_t gdngate_flats; /**< ITEM 6: decode/verify FLAT gate descriptors */
uint64_t gdngate_layer_issues, gdngate_settles;
double gdngate_settle_seconds;
struct {
uint32_t layer, rows, sub_rows;
pn_tt_activation alpha, beta;
} gdngate_last;
/** PHASE W: the reader thread (pn_tt_reader.h), started on first use; `reader_failed`
* once it could not be (no reader window bank), so nothing retries per call. */
struct pn_tt_reader *reader;
bool reader_failed;
/** Each member's own AWAIT time, for the report that diagnoses the knee. The awaits run
* in member order after every enqueue, so a balanced set puts nearly all of it on member
* 0 and a STRAGGLER is the one member with time of its own. */
double eltwise_member_seconds[PN_TT_ELTWISE_SPLIT_MAX_MEMBERS];
/** EXTRA reduce residencies, one per distinct HEAD width, placed for that width rather
* than the model's. The GDN's per-head norms are Wt = S/32 = 4 and attention's
* QK-RMSNorms are Wt = head_dim/32 = 8, against the model's 128; a reduce residency is
* placed for one Wt — see pn_tt_head_norm.h for why they must not share a core rather
* than merely need not. */
pn_tt_head_norm_residency head_norm[PN_TT_HEAD_NORM_MAX_WIDTHS];
uint32_t head_norm_count; /**< live slots, in the order they armed */
/** THE ACTIVATION STRIP BRIDGE — one residency per DIRECTION, `pn_tt_render.h`. They are
* two programs and not one because a writer is built for exactly one output page size,
* so a single triad cannot both widen and narrow. */
pn_tt_render_residency render[PN_TT_RENDER_DIRECTIONS];
/** THE MERGED GATE that deletes the narrow half of that bridge, `pn_tt_gate.h`. Default
* off; when it is up, `render[PN_TT_RENDER_NARROW]` issues no decode descriptors. */
pn_tt_gate_residency gate;
bool oncard[PN_TT_ONCARD_COUNT];
bool oncard_requested[PN_TT_ONCARD_COUNT];
/** What `PN_TT_ONCARD` unset — or `all` — selects, PER MODEL FAMILY. It is not a
* constant because on-card attention is a different forward on each family and the
* evidence for it is per family: it is measured good on the qwen35/bonsai hybrid
* (chat parity 0.986080, +0.4% perplexity, faster at every length) and measured to FAIL
* the same gate on MiniCPM5-1B through `llama_forward_tt.c`. One global default would
* have shipped the second as a side effect of shipping the first.
* pn_tt_device_oncard_default() sets it; pn_tt_device_open() seeds it. */
bool oncard_default[PN_TT_ONCARD_COUNT];
/** Requested because the OPERATOR NAMED IT, as opposed to because it is the default.
* The two want opposite failure behaviour and `oncard_requested` cannot tell them
* apart — see pn_tt_device_oncard_named(). */
bool oncard_named[PN_TT_ONCARD_COUNT];
uint32_t reduce_tile_cols; /**< the ONE Wt every reduce descriptor carries */
uint32_t eltwise_depth; /**< the ONE block depth every eltwise descriptor carries */
/** The caller's promise that every strip holds ONE live row (row 0) right now — a
* single-token forward. pn_tt_device_set_single_row(). */
bool single_row;
/** S2c: a verify's live rows are rows 0..verify_rows-1 (0 = no promise). <= 16: the
* half-tile levers (matmul M5 faces 0-1, reduce M3) hold; <= 4: eltwise E1 too (its SFPU
* row-0 body covers rows 0-3). */
uint32_t verify_rows;
uint32_t topk_row; /**< S2c: the logits row the next top-k selects over (0 = decode) */
/** The matmul K-block depth THIS MODEL wants when `PN_TT_PREFETCH_TILES` is unset
* (0 = the global default). pn_tt_device_prefer_prefetch_tiles(). */
uint32_t prefetch_tiles_preferred;
uint32_t sample_tile_count; /**< the vocabulary row's tile count */
uint32_t sample_depth;
uint32_t sample_result_address; /**< L1 on the sampling core */
uint32_t sample_state_address; /**< L1 on the sampling core */
/** L3 TOP-K SPLIT (`PN_TT_TOPK_CORES`): members 1..n-1 each select over a contiguous
* vocabulary slice; the host merges by (key desc, index asc), the kernel's own order. */
pn_tt_residency sample_member[PN_TT_TOPK_MAX_MEMBERS - 1u];
uint32_t sample_members;
uint32_t sample_member_result[PN_TT_TOPK_MAX_MEMBERS - 1u];
uint32_t sample_member_state[PN_TT_TOPK_MAX_MEMBERS - 1u];
/* The shapes each family was armed with, kept so a retired residency can be re-armed
* from them. reduce_tile_cols and sample_tile_count above serve the same purpose for
* their families; the elementwise counts have nowhere else to live. */
uint32_t oncard_eltwise_tiles[PN_TT_ONCARD_MAX_ELTWISE_SHAPES];
uint32_t oncard_eltwise_count;
uint64_t oncard_restarts; /**< times an on-card residency had to be re-armed */
uint32_t strip_bank_cursor; /**< round-robins DRAM activation strips across banks */
/** `PN_TT_STRIP_TILE_ROWS` — bands of 32 rows every activation strip is allocated with,
* resolved ONCE (0 means "not read yet") so a strip allocated late cannot disagree with
* one allocated early. 1 is the shipped height and the pre-Mt allocation byte for byte. */
uint32_t strip_tile_rows;
/** The height used when `PN_TT_STRIP_TILE_ROWS` is unset; 0 = 1. Set by a MODEL whose
* batched prefill can run more than one band (pn_tt_device_prefer_strip_tile_rows()). */
uint32_t strip_tile_rows_preferred;
/** PHASE G option 3: the height of the PREFILL-ONLY strip set (`PN_TT_PREFILL_TILE_ROWS`),
* 0 when prefill shares the decode strips. Decode strips stay `strip_tile_rows`. */
uint32_t prefill_tile_rows;
/** `PN_TT_ACTIVATION_REPLICAS` — how many DRAM banks the HOT activation strips are
* mirrored into, so N matmul members do not all read one ~59 GB/s bank. 1 is off and
* is the pre-replication forward byte for byte. See pn_tt_replica_map.h. */
uint32_t activation_replicas;
/** Round-robins WEIGHTS across banks, one step per tensor. It lives on the device rather
* than in `pn_tt_device_load_artifact()` because a weight re-staged after load — a
* de-interleaved half of a joint projection — must continue the same rotation rather
* than restart it and pile every restaged tensor onto one bank. Starts at 2. */
uint32_t weight_bank_cursor;
/* ------------------------------------------------------------
* The attention triple — RoPE, KV append, flash decode
* ------------------------------------------------------------
* The shapes are kept, not consumed: every one of the three residencies is BOUNDED and
* retires on idle, so re-arming one needs what it was armed with.
*/
ft_tt_kv_cache_layout kv_layout; /**< K and V share it (contract F1) */
pn_tt_kv_region *kv_region; /**< one per layer */
uint32_t kv_layer_count;
uint32_t attn_q_heads, attn_kv_heads, attn_head_dim, attn_rope_dim;
uint32_t attn_q_rows; /**< Q heads per KV group = q_heads / kv_heads */
uint32_t attn_dim_tiles; /**< head_dim / 32 */
uint32_t attn_block_tiles; /**< the ONE prefetch depth the residency was placed for */
uint32_t attn_embed_tiles; /**< the memory family's OTHER group size, required non-zero */
double attn_rope_theta;
/** The constants operand: the reduce scaler, 1/sqrt(head_dim) and the 32 tail masks.
* Built once per model. */
uint32_t attn_const_bank, attn_const_address;
/** ONE row block of the rotary table, rebuilt per token so its ROW 0 carries this
* token's position — which is what lets the whole forward stay at tile row 0 instead of
* carrying the position's row phase through every op (pn_tt_attention.c). */
uint32_t rope_table_bank, rope_table_address;
/** A SECOND chunk-only rotary table, so two bands of a pass (first positions p and p+32)
* each keep their block cached instead of evicting each other. Odd chunk indices use it;
* decode never does. */
uint32_t rope_chunk1_bank, rope_chunk1_address;
uint32_t rope_chunk1_first; /**< first position it holds; UINT32_MAX = none */
/** PHASE G: a whole-prompt pass rotates up to 64 chunks per attention layer, and two slots
* evicted each other 51 times a layer (+0.14 s). Chunks 2.. get a block each, built once
* per prefill and keyed by first position. Allocated lazily. */
uint32_t rope_many_bank[64], rope_many_address[64], rope_many_first[64];
/** S2c: the verify's rotary block (rows 0..k-1 = positions first..), decode-formula. */
uint32_t rope_rows_bank, rope_rows_address;
uint32_t rope_rows_key_first, rope_rows_key_rows; /**< the block it holds (rows 0 = none) */
/** S2f: the attention ROWS-mode staircase mask tile (on the constants' bank) and the
* host copy of the constants it is assembled from. */
uint32_t attn_rows_mask_address;
/** (first % 32) | rows << 8 | 1 << 16 when built, one a CHUNK: a verify wider than
* 32 / q_rows rows runs as several ROWS descriptors, each with its own mask tile. */
uint32_t attn_rows_mask_key[PN_TT_ATTN_ROWS_CHUNKS];
unsigned char *attn_constants_host;
uint32_t rope_table_position; /**< the position row 0 currently holds; UINT32_MAX = none */
/** True while the rotary block holds a CHUNK's 32 rows rather than a decode token's row
* 0 plus 31 zeros. Decode's next rebuild has to re-zero before it writes row 0, or the
* stated rows-1..31-are-zero invariant is only true by accident (of the ACTIVATION's
* padding, not the table's). */
bool rope_table_is_chunk;
uint32_t rope_table_chunk_first; /**< first position the chunk block holds, when is_chunk */
/* ------------------------------------------------------------
* The GATED-DELTA-NET family — one core, and the recurrence's state
* ------------------------------------------------------------
* The per-layer regions are the only DRAM in this backend that is READ AND WRITTEN by the
* card and never downloaded. The five staging buffers below them are SHARED across every
* layer, because a token runs one layer at a time and nothing in them carries.
*/
pn_tt_gdn_shape gdn_shape;
pn_tt_gdn_region *gdn_region; /**< one per SSM layer */
/** STEP 4 S2: snapshot slots of every layer's window + state for the speculative verify's
* rollback, `gdn_snapshot[slot * layers + layer]`, allocated on first use. */
pn_tt_gdn_region *gdn_snapshot;
uint32_t gdn_snapshot_slots;
/** Multi-turn: every layer's window + state copied at a prefill pass boundary
* (pn_tt_gdn_checkpoint.c), one region a layer, allocated on the first capture. */
pn_tt_gdn_region *gdn_checkpoint;
uint32_t gdn_state_out_plus1; /**< 0 = in place; else slot + 1 takes the state writes */
uint32_t gdn_row; /**< S2c: the CONV/DELTA row select, 0 = decode */
/** S2c: deferred awaits — split CONV/DELTA descriptors are enqueued and awaited together
* by pn_tt_gdn_await_deferred(); one gate block a row so queued DELTAs never share one. */
bool gdn_defer;
uint32_t gdn_pending_members;
uint64_t keepalive_nops; /**< PM 5: NOPs the keepalive has sent */
uint32_t gdn_pending_depth; /**< deferred descriptors a member holds since its last await */
uint64_t gdn_defer_drains; /**< full-ring drains the deferred rows took (flow control) */
pn_tt_gdn_buffer gdn_gate_row[16];
uint32_t gdn_gate_rows, gdn_gate_stride;
pn_tt_gdn_buffer gdn_stream_in; /**< CONV: the qkv projection, a row-0 strip */
/** CONV's OUTPUT **and** DELTA's INPUT — ONE buffer, because they are one geometry. Both
* are `conv_dim/32` row-0 tiles and `conv_out`'s q, k and v slices sit exactly where
* `ftg_delta_stream_base()` looks for them, so the delta rule reads the convolution's
* answer in place. All the per-head L2 norm has to do is write its answer back over the
* first `2*k_heads*state_tiles` tiles of it. */
pn_tt_gdn_buffer gdn_stream_out;
pn_tt_gdn_buffer gdn_gate; /**< DELTA: decay[H] then beta[H], flat fp32 */
pn_tt_gdn_buffer gdn_y; /**< DELTA: y, a row-0 strip, head-major */
unsigned char *gdn_stage; /**< host tile staging; grown on demand */
size_t gdn_stage_bytes;
/** `PN_TT_GDN_SELFCHECK`: read the staged qkv row and conv weight back out of DRAM once
* and compare, so a wrong GDN answer can be attributed to the host layout or to the
* card rather than to both. `gdn_weight_oracle` is layer 0's `ssm_conv1d`, borrowed. */
bool gdn_self_checked;
bool gdn_out_checked;
/** `PN_TT_GDN_SELFCHECK`: DELTA descriptors scored against the closed form so far. It is
* a COUNT and not a flag because position 0 is the one place the carried state is
* provably zero — so every SSM layer of the FIRST token has a closed form, and one layer
* cannot say whether an error is the op's or that layer's data. */
uint32_t gdn_delta_checks;
/** A HOST SHADOW of SSM layer 0's rolling conv window, plus the tokens scored against it.
* Position 0 is the ONE position where the window is trivially zero, so a check that
* stops there says nothing about the ROLL — and the roll is the only part of PHASE CONV
* that carries state across tokens. */
float *gdn_window_shadow;
uint32_t gdn_window_checks;
/** Has the injective-stream map already run? `PN_TT_GDN_RAMP` poisons the layer it probes,
* so it runs once. */
bool gdn_ramp_probed;
/** Has the CONV data arm echoed the stream it is about to convolve? Once a run — an arm
* that silently did not fire reads exactly like an arm that fired and cost nothing, and
* those two are the whole question the arm exists to separate. */
bool gdn_synthetic_reported;
/** Whole-tile strip checks taken on the SHIPPED on-card route, capped so three layers
* over four tokens is a probe and not a log. */
uint32_t gdn_strip_checks;
/** Has `PN_TT_GDN_CORES`' multi-core sweep already run? It borrows the carried state of
* the layers it uses, so it runs ONCE per device and at position 0. */
bool gdn_probe_done;
/** PER-PHASE on-card accounting, indexed by `FT_TT_OP_GDN_PHASE_*` (1 and 2; slot 0 is
* never written). `oncard_seconds` is a SUM over every family, so the plan's "+0.110 s
* over 48 GDN descriptors" was a SUBTRACTION between two runs — and a subtraction cannot
* tell CONV from DELTA, which do different work and are the two things a re-pricing has
* to separate. These are measured directly, in one run. */
double gdn_phase_seconds[3];
uint64_t gdn_phase_ops[3];
/** THE SAME PHASES, FOR DESCRIPTORS THAT REFUSED AND WERE REPLAYED. The submit stopwatch
* spans the refusal, the re-arm — a residency teardown and relaunch — and the resubmit,
* so a refusing run prices a teardown as the op's arithmetic. That is exactly how PHASE
* CONV once read **4,502 us a descriptor against 738.2**, and it is the shape the
* unsourced "CONV is DATA-DEPENDENT, 4,571 us on a degenerate stream" also has. Kept
* apart so `gdn_phase_seconds / gdn_phase_ops` is the CLEAN figure and a reader that
* quotes it prints `gdn_phase_rearms` beside it. */
double gdn_phase_recovered_seconds[3];
uint64_t gdn_phase_recovered_ops[3];
uint64_t gdn_phase_rearms[3];
/** Every SSM layer's qkv, as it was uploaded, so a wrong tensor coming back out of the
* card can be NAMED rather than described. `layer_count * conv_dim` floats. */
float *gdn_qkv_history;
const float *gdn_weight_oracle;
/* ------------------------------------------------------------
* The CHUNKWISE family's own operands — see pn_tt_gdnc.h
* ------------------------------------------------------------
* Three buffers, and every one of them is a CONSTANT or a per-chunk constant. The chunk's
* activations are the gdn family's staging strips and its carried state is the gdn
* family's regions, so nothing here is read back and nothing here carries.
*/
pn_tt_gdn_buffer *gdnc_constant; /**< one per SSM layer: ROW-REPLICATED tap planes,
* then the 2*taps band tiles at the end of them */
/** PM 8: copies of each layer's conv constant in OTHER DRAM banks, so the CONV members
* do not all read one bank (`PN_TT_GDNC_CONST_REPLICAS`, 1 = none). Member m reads copy
* `m % replicas` (copy 0 = `gdnc_constant`). `layer_count * (replicas - 1)` entries. */
pn_tt_gdn_buffer *gdnc_constant_replica;
uint32_t gdnc_constant_replicas;
uint32_t gdnc_conv_layer; /**< the layer of the CONV descriptor being submitted */
/** THE DELTA STAGING, IN TWO SLOTS — one per sub-chunk index (`first_row / 16`), so the
* host can stage sub-chunk 1 while the card is still reading sub-chunk 0's. Each slot
* is a whole constant region (I, 2I, the strict mask, scale, then one decay-ratio tile
* per v-head) plus its gate block PINNED to the same bank, which they share a
* coordinate word with. */
pn_tt_gdn_buffer gdnc_delta_constant[PN_TT_GDNC_STAGE_SLOTS];
pn_tt_gdn_buffer gdnc_gate[PN_TT_GDNC_STAGE_SLOTS];
/** What each slot holds ON THE CARD, byte for byte, so an upload can send only the units
* that changed and the card's bytes still equal the builders' by construction. */
unsigned char *gdnc_shadow_decay[PN_TT_GDNC_STAGE_SLOTS];
float *gdnc_shadow_gate[PN_TT_GDNC_STAGE_SLOTS];
bool gdnc_shadow_valid[PN_TT_GDNC_STAGE_SLOTS];
/** A slot staged ahead of its descriptor (`pn_tt_gdnc_delta_prestage_at()`), consumed by
* the descriptor whose (layer, first_row, chunk_rows) it matches. */
struct {
bool valid;
uint32_t layer, first_row, chunk_rows;
} gdnc_prestaged[PN_TT_GDNC_STAGE_SLOTS];
/** The worker that builds a staging's heads while this thread uploads them
* (`PN_TT_GDNC_STAGE_THREAD`). Stopped in pn_tt_gdnc_release(). */
pn_tt_stage_pipeline gdnc_stage_pipeline;
/** ONE-SHOT host work to run between the next split's enqueue and its await. */
int (*gdnc_overlap_fn)(void *context);
void *gdnc_overlap_context;
/** A SECOND one-shot slot, run after the first in the same window (the prefill's
* alpha/beta prefetch; the first belongs to the DELTA staging). It may re-arm itself. */
int (*gdnc_overlap_aux_fn)(void *context);
void *gdnc_overlap_aux_context;
uint64_t gdnc_stage_writes; /**< DRAM write calls the staging made */
uint64_t gdnc_stage_hidden_ops; /**< stagings that ran under a split's card time */
uint64_t gdnc_stage_verify_checks; /**< PN_TT_GDNC_STAGE_VERIFY read-backs */
double host_clock_seconds[PN_TT_HOST_CLOCK_COUNT]; /**< pn_tt_device_host_clock_add() */
uint64_t gdnc_stage_verify_mismatches; /**< units that differed from the builders */
unsigned char *gdnc_decay_stage; /**< v_heads fp32 tiles, contiguous, so the 128 KiB
* a layer a chunk is ONE write */
float *gdnc_gate_stage; /**< v_heads * FT_TT_OP_GDNC_GATE_STRIDE */
/** PER-PHASE accounting, indexed by `FT_TT_OP_GDNC_PHASE_*`. Separate from the decode
* family's because the two run different work on different cores and a prefill that
* mixes chunks with a `1..taps-2` tail runs BOTH. */
double gdnc_phase_seconds[3];
uint64_t gdnc_phase_ops[3];
/** PER-MEMBER AWAIT, so a core split is visible as `members` numbers and not only as one
* that moved. The awaits are issued in order after every enqueue, so a BALANCED set puts
* nearly all of the wall on member 0 and leaves the rest near zero — the same signature
* the matmul group's per-member wait line carries, and the discriminator between a set
* that ran concurrently and one this host serialised. */
double gdnc_member_seconds[PN_TT_GDNC_MAX_MEMBERS];
uint64_t gdnc_member_ops[PN_TT_GDNC_MAX_MEMBERS];
/** And the replayed ones, for the same reason the decode family keeps them apart — this
* is the family the 4,502 us reading was actually taken on. */
double gdnc_phase_recovered_seconds[3];
uint64_t gdnc_phase_recovered_ops[3];
uint64_t gdnc_phase_rearms[3];
/** THE DELTA STAGING, WHICH NO PHASE LINE COULD SEE. Every DELTA descriptor transposes
* `v_heads` decay runs and gate runs on the host and writes 151,552 B of them to DRAM
* BEFORE `gdnc_submit()` starts its stopwatch, so the cost is inside the caller's
* `sequential` bracket and attributed to nothing. Two clocks, not one: BUILD is host
* arithmetic and UPLOAD is the x1 Gen2 link, and only the second is what a smaller
* operand would move. `bytes` is carried so the rate is derived here rather than from an
* assumed operand size. */
double gdnc_stage_build_seconds;
double gdnc_stage_upload_seconds;
uint64_t gdnc_stage_ops;
uint64_t gdnc_stage_bytes;
/** THE SPLIT'S COMPLETION SHAPE, and the clock that says what it cost. `gdnc_group` is
* true only when every armed member carries the group binding AND the block read back
* with its magic — a partial arm is not a shape, it is a set of members some of which
* post and some of which do not, and it falls back rather than running. The sequence is
* the GROUP's notion of "item N" and is stamped identically on every member of one
* issue; it never resets, so a slot holding a previous issue's number can never satisfy
* this one. */
bool gdnc_group;
uint32_t gdnc_sequence;
double gdnc_group_seconds;
uint64_t gdnc_group_ops;
/** Which host sequence owns the ONE region set, and the keys it displaced. A displaced
* key coming back is two sequences alternating, and it is refused by name. */
const void *kv_key;
const void *kv_displaced[PN_TT_KV_DISPLACED_KEYS];
uint32_t kv_displaced_count;
uint64_t kv_rebinds;
/* Scratch, on the card: one activation strip in, one result strip out. Used only by
* the host-in/host-out pn_tt_matvec(); the on-card forward names its own strips. */
ft_tt_buffer activation;
ft_tt_buffer result;
/* Scratch, on the host. Only the two ROWS: a matvec moves logical row 0 of each tile
* and nothing else, so the full tile staging buffers this used to keep are gone. */
uint16_t *row_in; /**< in_features bf16 */
uint16_t *row_out; /**< out_features bf16 */
uint32_t max_in_features, max_out_features, max_tile_rows, max_tile_cols;
uint32_t max_row_features; /**< the widest of the two; both host rows are this */
/* Scratch for the MULTI-ROW strip transfers (pn_tt_activation_{upload,download}_rows).
* A batched prefill fills rows 0..M-1 of the same tiles a matvec leaves as zero padding,
* so those two move WHOLE tiles and need the tile-shaped staging buffer the row-0 path
* deliberately does not keep. Grown on demand and freed with the device; NULL until a
* batched call asks, so a decode-only run pays nothing for them. */
unsigned char *tile_stage; /**< tiles * 2048 B, face-ordered */
size_t tile_stage_bytes;
uint16_t *rows_stage; /**< rows * count bf16, row-major — the tilizer's input */
size_t rows_stage_bytes;
/* THE MULTI-CORE CORRECTNESS GATE. `PN_TT_MATVEC_DUMP=<file>` appends logical row 0 of
* EVERY projection's output, in dispatch order, as raw bf16 — so one run at N=1 and one
* at N=4 are two files that must be byte-identical. It is a byte comparison and not a
* score on purpose: two members overwriting each other completed every kernel, kept
* every status word clean and left the aggregate GB/s unchanged, and only bytes saw it
* (foundry/tt/DEV_NOTES.md §3). Off by default; it costs a readback per op. */
FILE *matvec_dump;
uint64_t matvec_dump_bytes;
uint64_t matvec_dump_rows;
/** WHAT HAS BEEN ISSUED AND NOT AWAITED, so a card refusal can be REPLAYED onto a
* re-armed residency rather than costing the token. Chained projections are
* independent by contract, so replaying them is the same computation in the same
* order — which is why the recovery can be the plain synchronous path. */
struct {
const pn_tt_weight *weight;
pn_tt_activation input;
pn_tt_activation output;
} pending[PN_TT_MATVEC_CHAIN_CEILING];
uint32_t pending_count;
uint64_t matvec_count;
double matvec_seconds;
double upload_seconds; /**< bf16 convert + the row-0 host->card write */
double dispatch_seconds; /**< ft_tt_op_matmul_run: image load, CB config, go, poll */
/** `dispatch` split at the seam that matters: ISSUE is the host writing parameter words
* and ring entries over MMIO, WAIT is the completion poll and everything the card does
* under it. Phase 3 removes issue outright and one poll per wait; whatever is left of
* wait is the card's own work and survives it. */
double dispatch_issue_seconds;
double dispatch_wait_seconds;
uint64_t dispatch_issues;
uint64_t dispatch_waits;
uint64_t dispatch_wait_descriptors;
double readback_seconds; /**< the row-0 card->host read + bf16 convert */
double oncard_seconds; /**< enqueue + wait for the reduce/eltwise/sample residencies */
uint64_t oncard_ops;
/** The SAME descriptors as `oncard_seconds`, keyed by (family, function, tile count).
* `oncard_seconds` is one number over every family and the per-residency report is one
* number over every SHAPE — the elementwise residency runs six functions over three
* widths and reports a single 293.8 us mean, and the two render residencies are not in
* that report at all. See pn_tt_descriptor_census.h. */
pn_tt_descriptor_census census;
uint64_t bytes_read, bytes_written;
};
/** Seconds since an arbitrary epoch. Shared by the two files that time this device. */
double pn_tt_now_seconds(void);
/**
* Charge ONE retired on-card descriptor to the aggregate counters AND to the census.
*
* Every family's submit path ended in the same three lines, and the census would have been
* the fourth — copied five times, which is five places for a family to be counted in one
* instrument and not the other. It is one call now, and the two can no longer disagree.
*
* @param family residency-level name, e.g. "eltwise", "render-widen"
* @param function what the descriptor computed, e.g. "silu_mul"
* @param tiles tiles the descriptor covers — the census fit's x axis
* @param seconds measured wall time inside submit
*/
void pn_tt_oncard_account(pn_tt_device *device, const char *family, const char *function,
uint32_t tiles, double seconds);
/** `name`'s value as an unsigned, or `fallback` when unset or unparsable. */
uint32_t pn_tt_env_u32(const char *name, uint32_t fallback);
/** E3 SPLIT PROBE: OR `PN_TT_VEC_PROBE` (kVecProbe* bits) into the top byte of word 3 of an
* ELTWISE or REDUCE parameter block. Timing only — an armed run's text is not evidence. */
void pn_tt_vec_probe_apply(uint32_t *words);
/**
* One matmul with BOTH operands already resident in card DRAM — no host transfer at all.
*
* This is the dispatch half of pn_tt_matvec(), split out so the on-card forward can chain
* projections through DRAM strips instead of through the host.
*
* @return 0, or -1 with the reason on stderr
*/
int pn_tt_matvec_dispatch(pn_tt_device *device, const pn_tt_weight *weight,
const pn_tt_activation *input, const pn_tt_activation *output);
/* ================================================================
* Shared by the two residency owners — pn_tt_oncard.c and pn_tt_attention.c
* ================================================================ */
/** Every processor of a triad. The memory family is the exception: it has no compute
* thread and enables DM0|DM1 only (foundry/tt/ft_tt_op_memory.h PART 2). */
#define PN_TT_ONCARD_TRIAD_MASK \
((1u << FT_TT_PROC_DM0) | (1u << FT_TT_PROC_DM1) | (1u << FT_TT_PROC_MATH0) | \
(1u << FT_TT_PROC_MATH1) | (1u << FT_TT_PROC_MATH2))
/** The GDN triad's program name. Here rather than in pn_tt_gdn.c because TWO files need it:
* that one arms the family and pn_tt_oncard.c owns the refusal when a family asked for by
* name has no program to arm. */
/** The RESIDENT matmul triad, built for a Bfp8_b weight. One program, not two: the
* production ledger puts every matmul weight of an artifact at Bfp8_b, so a run needs one
* resident binary and the Bfp4_b variant has no tensor to serve. That is a property of the
* ARTIFACT and is checked at load, never assumed — see pn_tt_matmul_group_arm().
*
* HERE RATHER THAN IN pn_tt_device.c since 2026-08-27: the matmul group's DST-section
* interlock reads the ARMED image's recorded `defines=` to decide whether eight DST tiles
* are legal, so the name of the program it will arm is no longer private to the file that
* loads it. Two files naming the same program in two string literals is the shape this
* backend has already been bitten by. */
#define PN_TT_RESIDENT_PROGRAM "rtriad_bfp8"
#define PN_TT_GDN_PROGRAM "gdn"
/** The CHUNKWISE triad's program name. Same reason it is here: pn_tt_gdnc.c arms the family
* and pn_tt_oncard.c owns the core-collision refusal that has to name it. */
#define PN_TT_GDNC_PROGRAM "gdnc"
/** Just the two data-movement processors. */
#define PN_TT_ONCARD_DM_MASK ((1u << FT_TT_PROC_DM0) | (1u << FT_TT_PROC_DM1))
/** Bytes written per host->card call while zeroing or filling a DRAM region. */
#define PN_TT_ZERO_CHUNK_BYTES 65536u
/** The reduce family's program name. Two files arm a residency of it now — pn_tt_oncard.c
* at the model's width and pn_tt_head_norm.c at the head width. */
#define PN_TT_REDUCE_PROGRAM_NAME "reduce"
/** THE SAME TRIAD PLACED AT Float32 — `kernels/cb_tables/reduce_float32.cbtable`, a
* `TILE_BYTES=4096` build of the same three sources, no kernel change.
*
* It is a SECOND program and not a replacement for two reasons. The bf16 one is 9x past the
* 1e-3 gate on a NARROW norm and inside it on the model's — the error is per reduce, so the
* members that matter are the ones with few elements to average it over. And doubling the
* page halves the widest residency the arena serves, 107 tile columns against 217, so the
* model's `n_embd/32` = 128 RMS norms DO NOT FIT at Float32 and stay on `reduce`. */
#define PN_TT_REDUCE_FP32_PROGRAM_NAME "reduce_fp32"
/**
* Build and upload ONE reduce member's constants buffer.
*
* The scaler tile, the eps tile and gamma all depend on WHICH MEMBER, so a buffer built for
* one is wrong for another — which is why the function is a parameter rather than baked in.
* `pn_tt_norm_constants_upload()` is this at FT_TT_OP_REDUCE_FN_RMS_NORM.
*
* THE PAGE SIZE IS THE RESIDENCY'S, NOT THIS FUNCTION'S. The model's reduce residency is
* placed by `reduce_float16b.cbtable` at 2048 bytes and the per-head one by
* `reduce_float32.cbtable` at 4096; the reader issues `TILE_BYTES` reads and nothing on the
* card can catch a constants buffer built at the other one.
*
* @param gamma `width` values, or NULL for a member whose chain has no gamma step
* @param eps what the member's reduce should see added to its accumulator
* @param tile_bytes PN_TT_TILE_BF16 or PN_TT_TILE_FP32 — the page the residency was placed at
* @return 0, or -1 with the reason on stderr
*/
int pn_tt_reduce_constants_upload(pn_tt_device *device, ft_tt_op_reduce_function function,
const float *gamma, uint32_t width, float eps,
uint32_t tile_bytes, pn_tt_norm_constants *out_constants);
/** Is `name` the PN_TT_RED mutation this process was asked to run? Shared so the attention
* family's red proofs read the same knob as the on-card ones. */
bool pn_tt_red(const char *name);
/** Allocate a region shaped like SSM layer `layer`'s live window + state, on the live
* region's banks (one coordinate word carries both). Left uninitialised. 0 or -1. */
int pn_tt_gdn_alloc_twin(pn_tt_device *device, uint32_t layer, pn_tt_gdn_region *out);
/** The bytes of one layer's carried conv window and recurrent state (unrounded). */
void pn_tt_gdn_carried_bytes(const pn_tt_device *device, size_t *window, size_t *state);
/** Write `bytes` of zeros at a DRAM address, in bounded chunks. */
int pn_tt_zero_dram(pn_tt_device *device, uint32_t bank, uint32_t address, size_t bytes);
/** `PN_TT_IDLE_RETIRE_ONLY` (diagnosis): may the residency `name` be retired for idleness? */
bool pn_tt_idle_retire_allowed(const char *name);
/** `pn_tt_zero_dram` with every byte `value` (a test poison; 0 is the zero). */
int pn_tt_fill_dram(pn_tt_device *device, uint32_t bank, uint32_t address, size_t bytes,
unsigned char value);
/** Write `bytes` from host memory to a DRAM address, in bounded chunks. */
int pn_tt_write_dram(pn_tt_device *device, uint32_t bank, uint32_t address, const void *data,
size_t bytes);
/** Read `bytes` from a DRAM address into host memory, in bounded chunks. */
int pn_tt_read_dram(pn_tt_device *device, uint32_t bank, uint32_t address, void *out,
size_t bytes);
/** An explicitly-named program directory to read INSTEAD of the embedded blob. */
#define PN_TT_PROGRAM_ROOT_ENV "PN_TT_PROGRAM_DIR"
/**
* The program root this process was told to use, or NULL for the embedded blob.
*
* NULL IS THE DEFAULT AND THE DEFAULT IS THE BINARY'S OWN PROGRAMS. There is no fallback
* path in either direction: an unset variable means embedded, and a set one means that
* directory or a refusal (`ft_tt_program_resolve.h`). The hardcoded
* absolute programs directory this replaced made Pantheon a one-machine program and
* turned a program built into the wrong root into a silent 1.085x — the `rtriad_bfp8`
* incident.
*
* One reader for the whole backend so the answer cannot differ between the matmul programs
* and the on-card families of the same run.
*
* @return the directory, or NULL when the variable is unset or empty
*
* Thread-safe: yes Reentrant: yes Signal-safe: no
*/
const char *pn_tt_program_override_root(void);
/** Name the place programs are coming from, for a message that has to say one or the other.
* Takes the root so a caller cannot describe one arm while resolving through another. */
const char *pn_tt_program_where(const char *root);
/**
* Resolve `program` by NAME and boot its firmware on this family's core.
*
* SEPARATE FROM THE START, and the order is load-bearing: pn_tt_residency_load() zeroes the
* whole handle, so a caller that placed its circular buffers first would have them silently
* erased. Prepare, THEN place, THEN start.
*/
bool pn_tt_prepare_family(pn_tt_device *device, pn_tt_residency *residency, const char *program,
const char *name, const char *core_env, uint32_t core_default,
uint32_t processor_mask);
/** The same, on a core the CALLER names — for arming several residencies of one family at
* once, where a single environment variable cannot say a different core for each. `core_env`
* is still taken, but only so the refusal message can name the knob that would move it. */
bool pn_tt_prepare_family_on_core(pn_tt_device *device, pn_tt_residency *residency,
const char *program, const char *name, const char *core_env,
uint32_t core, uint32_t processor_mask);
/** Kick a prepared residency whose circular buffers the caller has now placed. */
bool pn_tt_start_family(pn_tt_residency *residency);
/** The family last started on the core at NoC (x, y), or "?" (PM 8, AWAIT census). */
const char *pn_tt_core_label(uint16_t x, uint16_t y);
/** Does this binary (or the named override root) carry the program `program`? */
bool pn_tt_program_is_present(const char *program);
/**
* The ONE block depth a residency can run for every tile count it will see.
*
* Both conditions are the CB-cycle invariant and neither is optional: `depth` must divide each
* run so a block is contiguous, and `2 * depth` — the page count the buffers are placed at —
* must divide it too, so each descriptor returns the write pointer to the base. A descriptor
* that leaves it elsewhere walks the next one up L1 with no error.
*
* Shared because TWO files place elementwise-shaped buffers now: pn_tt_oncard.c for the
* model's eltwise and sampling residencies, pn_tt_render.c for the two strip-bridge ones.
*
* @return the depth, or 0 when no depth serves every count
*/
uint32_t pn_tt_common_depth(const uint32_t *tile_counts, uint32_t count, uint32_t cap);
/** Ceiling on a resident reader's block depth. Not a tuning knob: two blocks of `depth` pages
* on each of two input streams is 4 * depth * 2048 bytes of a 1.29 MiB arena, and a deeper
* block buys nothing once the reader is no longer transaction-bound. */
#define PN_TT_ONCARD_MAX_DEPTH 32u
#endif /* USE_TT */
#endif /* PN_TT_DEVICE_INTERNAL_H */

2558
src/accel/tt/pn_tt_dispatch.c Normal file
View File

File diff suppressed because it is too large Load Diff

75
src/accel/tt/pn_tt_dispatch.h Normal file
View File

@ -0,0 +1,75 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_dispatch.h
* @brief STEP 3 (A) — on-card dispatch: the RECORDER. Captures every host<->card access a
* decode token makes, at the transport, classified by address.
*
* @status A1: observe mode (`PN_TT_DISPATCH_RECORD=<token>`): wraps the runtime's three
* accessors for one decode token and prints what the token is made of — queue writes
* (parameter blocks, ring slots, head bumps), completion polls, and every OTHER access
* (the segment boundaries a puller has to hand back to the host).
* A1b (`PN_TT_DISPATCH_SHADOW=<forward>`): learn a model from two tokens and predict
* every later token's queue writes and awaited counts, compared byte for byte.
* @issues None
* @todo A2: the puller's dry run; A3: live replay.
*/
#ifndef PN_TT_DISPATCH_H
#define PN_TT_DISPATCH_H
#ifdef USE_TT
#include <stdbool.h>
#include <stdint.h>
#include "pn_tt_device.h"
/** Call at the start of a decode forward (before the embedding upload); `forward` counts
* forwards from 1. Arms the recorder when `forward` is the one asked for. */
void pn_tt_dispatch_token_begin(pn_tt_device *device, uint64_t forward);
/** Call after the head (top-k read back). Disarms and, after the recorded token, reports. */
void pn_tt_dispatch_token_end(pn_tt_device *device, uint64_t forward);
/** A new sequence (the carried state was zeroed): the model is re-learned on its forwards. */
void pn_tt_dispatch_sequence_reset(void);
/** No stream, no replay and no recording is in progress: the host may issue outside a token
* (the keepalive's NOPs). */
bool pn_tt_dispatch_quiet(void);
/** STEP 4 S2 (verify data movers on the puller, armed on demand):
* row `src_row` of every tile of `src` -> row `dst_row` of `dst` (all of dst's mirrors). */
int pn_tt_row_copy(pn_tt_device *device, const pn_tt_activation *src, uint32_t src_row,
const pn_tt_activation *dst, uint32_t dst_row);
/** FILL on the puller (`resident_puller.c` kOpFill): every 32-bit word of DRAM
* [address, address + bytes) of `bank` becomes `value`, card side. address and bytes 64 B
* multiples. -1 (nothing written) when the puller cannot arm, mid-stream, or misaligned. */
int pn_tt_fill_dram_oncard(pn_tt_device *device, uint32_t bank, uint32_t address,
size_t bytes, uint32_t value);
/** COPY on the puller (`resident_puller.c` kOpCopy): DRAM [src_address, + bytes) of
* `src_bank` -> [dst_address, + bytes) of `dst_bank`, card side (the multi-turn checkpoint of
* the carried GDN state). Addresses and bytes 64 B multiples. Flushes first. -1 when the
* puller cannot arm, mid-stream, misaligned, or the descriptor failed (a puller program
* without the opcode refuses it: the copy is then not done, and the caller must not trust
* the destination). */
int pn_tt_copy_dram_oncard(pn_tt_device *device, uint32_t src_bank, uint32_t src_address,
uint32_t dst_bank, uint32_t dst_address, size_t bytes);
/** Cumulative S2 row-copy count and host wall time. */
void pn_tt_move_stats(uint64_t *rows, double *row_ms);
/** S2d: STREAMED segments around a speculative verify / MTP drafting (pn_tt_dispatch.c).
* `PN_TT_SPEC_STREAM=0` off. begin returns false when not streaming; end returns false if a
* segment failed on the card (the caller must then treat its work as failed). */
bool pn_tt_dispatch_stream_begin(pn_tt_device *device);
bool pn_tt_dispatch_stream_end(void);
void pn_tt_dispatch_stream_report(double cycles);
/** PM 6: launch what the stream has captured so far WITHOUT waiting (the card starts on it
* while the host waits on something else). true when nothing was streaming or the launch
* went; false if the stream has failed. */
bool pn_tt_dispatch_stream_kick(void);
/** A1b: print the shadow's tallies (`PN_TT_DISPATCH_SHADOW=<first model forward>`). */
void pn_tt_dispatch_report(void);
#endif /* USE_TT */
#endif /* PN_TT_DISPATCH_H */

685
src/accel/tt/pn_tt_eltwise_group.c Normal file
View File

@ -0,0 +1,685 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_eltwise_group.c
* @brief Implementation of the elementwise core split. See pn_tt_eltwise_group.h — the core
* map, the knobs and the argument for why this family splits cleanly live there, and
* the tile arithmetic lives in pn_tt_eltwise_split.c where a test can reach it.
*
* @status Behind `PN_TT_ELTWISE_CORES`, default 1.
* @issues None
* @todo None
*/
#ifdef USE_TT
#include "pn_tt_eltwise_group.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "pn_tt_device_internal.h"
#include "pn_tt_name_list.h"
/** Members, and where they start. See the header for why 106 cannot collide.
*
* **DEFAULT 2, AND 2 IS THE KNEE — MEASURED 2026-08-24 ON THE SHIPPED BLOB.** The wall over
* the ladder is 5.01 / **5.17** / 4.20 / 3.18 tok/s at N = 1 / 2 / 8 / 16, against a control
* re-taken the same hour on the same binary. It is not a shape anyone predicted, and the
* census fit says exactly why:
*
* N 1 2 8 16
* us/tile 1.999 0.998 0.000 -0.001
* fixed -100.5 +42.2 +563.0 +1125.2 us/descriptor
*
* **THE PER-TILE TERM SCALES PERFECTLY** — 1.999 -> 0.998 is 2.00x at two members, and by
* eight there is no per-tile term left at all. The cores are not the bind. What stops it is
* the INTERCEPT, which grows ~+75-85 us for every member added, flat, whatever the width.
* So a member pays for itself only on a descriptor wide enough: saving is
* `1.999 * W * (1/N - 1/(N+1))` against ~80 us, i.e. **W >= ~143 tiles at the first
* doubling** — and Ornith's elementwise widths are 128 and 384. Measured per width at two
* members: SwiGLU Wt=384 **667.2 -> 425.6 us (1.57x)**, and the residual ADD at Wt=128
* **110.0 -> 147.0 us, WORSE**, exactly as that line predicts. At four members the
* break-even is 572 tiles and nothing in the family is that wide.
*
* **THE INTERCEPT IS THE HOST'S SERIAL ENQUEUE, AND THE PER-MEMBER AWAITS PROVE IT.** Every
* member above 0 reports **0.0079-0.0080 s/token** — identical to three digits at N=2 and at
* N=16, where its slice is eight times smaller. A cost that does not move with the work is
* not the card. It is 58 us a descriptor of parameter write, ring write and head bump that
* the host pays for member `m` before member `m+1` is told anything. Member 0's own await
* falls 0.0202 -> 0.0081 across that same ladder, which is its work halving away.
*
* So the lever above this one is NOT more cores: it is the ISSUE path — the parameter cache
* `pn_tt_matmul_group.c` already uses to make a repeated projection cost zero device
* accesses, applied to a family that re-writes 64 bytes for all 136 descriptors of a token.
*
* **THE BLOCK DEPTH IS NOT THE CONFOUND, AND THAT WAS MEASURED RATHER THAN ASSUMED.** The
* depth moves with the member count (32 unsplit, 8 at eight members), so the ladder walks
* two knobs. `PN_TT_ELTWISE_DEPTH=8` at ONE member is 5.02 tok/s and fits -103.5 + 2.003,
* against the control's 5.01 and -100.5 + 1.999. The depth is worth **0.0%**. */
#define PN_TT_ELTWISE_CORES_ENV "PN_TT_ELTWISE_CORES"
#define PN_TT_ELTWISE_CORES_DEFAULT 2u
#define PN_TT_ELTWISE_MEMBER_CORE_ENV "PN_TT_ELTWISE_MEMBER_CORE_FIRST"
/* 112 since the PX7 placement shipped (was 106, inside the 16-member matmul's 96..110). */
#define PN_TT_ELTWISE_MEMBER_CORE_DEFAULT 112u
/**
* PHASE W1a — WIDE MEMBERS for the prefill's whole-prompt descriptors.
*
* A 1,629-row prefill's SwiGLU is ONE descriptor of 51 x 384 = 19,584 tiles: 15.65 ms on the
* two decode members, 33 of them a prompt = 0.52 s of the 1.57 s `batched` line, while the
* free Tensix of the 0-70 region sit idle (2026-10-07). The fit above says the per-tile term
* scales perfectly and the cost of a member is its ~45 us serial enqueue, so a descriptor that
* long pays for many members where a decode one pays for two.
*
* `PN_TT_ELTWISE_WIDE_CORES=N` arms N MORE eltwise residencies from
* `PN_TT_ELTWISE_WIDE_CORE_FIRST` (58: 58..70 is clear of every family and of the 48-core
* prefill grid on 0..47), at member 0's placement. An SFPU descriptor of at least
* `PN_TT_ELTWISE_WIDE_MIN_TILES` (2048; 16x that for ADD/MUL) goes over EVERY member, dealt out in whole blocks of
* `2 * depth` (`pn_tt_eltwise_split_slice_blocks()`), so no width has to divide; every
* narrower descriptor takes the decode split exactly as before. Same per-element arithmetic
* on a different core: bit-exact by construction. The mirrors of each wide slice are keyed in
* that member's own replica table (`pn_tt_replica_map.c`).
*/
#define PN_TT_ELTWISE_WIDE_CORES_ENV "PN_TT_ELTWISE_WIDE_CORES"
#define PN_TT_ELTWISE_WIDE_CORE_ENV "PN_TT_ELTWISE_WIDE_CORE_FIRST"
#define PN_TT_ELTWISE_WIDE_CORE_DEFAULT 58u
#define PN_TT_ELTWISE_WIDE_MIN_TILES_ENV "PN_TT_ELTWISE_WIDE_MIN_TILES"
#define PN_TT_ELTWISE_WIDE_MIN_TILES_DEFAULT 2048u
/** CAP on the block depth, so a member-count ladder can be run at a FIXED depth. The header
* has the confound this exists to separate. */
#define PN_TT_ELTWISE_DEPTH_ENV "PN_TT_ELTWISE_DEPTH"
/**
* WHICH FUNCTIONS THE SPLIT IS APPLIED TO — because THE KNEE WAS NEVER A WIDTH, AND IT MOVES
* WITH THE COMPLETION SHAPE.
*
* RE-MEASURED ON SILICON 2026-08-26, per SHAPE, at one member and at two, on TWO completion
* shapes. `K` is the break-even recovered from each pair by `T(2) = T(1)/2 + K/2`:
*
* ---- window, 5 settle reads ---- ---- line, 0 ----
* shape N=1 N=2 K N=1 N=2 K
* SILU_MUL Wt=384 634.5 361.0 87.5 619.8 332.6 45.4
* SILU_MUL Wt=128 235.3 161.7 88.1 220.1 132.6 45.1
* SIGMOID Wt=128 203.6 144.6 85.6 189.0 115.7 42.4
* ADD Wt=128 77.5 82.5 87.5 (LOSES) 62.3 53.5 44.7 (WINS)
* render-widen Wt=256 125.4 105.1 84.6 110.3 77.1 43.9
* -------------------------------------------------------------------------------
* **87.2 +- 1.3** **44.3 +- 1.1**
*
* **TWO SHAPES AT Wt=128 WIN AND TWO AT Wt=128 LOSE on the window, so the discriminator is
* not the width** — the recorded "W >= 143 tiles" was this statement seen through the
* family's pooled line (which re-measures at W >= 110). A descriptor is worth splitting in
* two iff its UNSPLIT TIME is above K.
*
* **AND K IS A PROPERTY OF THE AWAIT, NOT OF THE FAMILY.** 87.2 -> 44.3 when the completion
* shape went `window` -> `line`: 42.9 us, i.e. 21.5 us off each of the two members, against
* the 5 x 3.31 = 16.6 us of settle reads the line removes. Five shapes across TWO families
* (`eltwise` and `render-widen`, different programs at different page sizes) recover the same
* constant to +-2.5%, twice.
*
* **SO ON THE SHIPPED `line` DEFAULT EVERY SHAPE IN THIS FAMILY IS ABOVE THE BAR** — the
* cheapest is the residual ADD at 62.3 us against K = 44.3 — and the filter DEFAULTS TO
* UNRESTRICTED. It is kept, with the table, because it is the only way to run the family
* under `window` or `control` without paying 8.8 us on each of 64 residual adds, and because
* a knee that moves with another knob is exactly the thing a later round will re-derive
* wrongly from one row of this table.
*
* A LIST AND NOT A THRESHOLD, deliberately. A threshold read out of the live census would
* make some descriptors of one key split and others not, and the key's mean is what the
* `fixed + per_tile * tiles` fit is solved from — the instrument this whole finding came out
* of would start reporting a blend.
*
* CONTAINS, NOT SELECTS: this is a FILTER, so an unset or empty value does not restrict.
* `PN_TT_ELTWISE_SPLIT_FUNCTIONS=NOTHING_MATCHES_THIS` is the arm that proves it is live —
* measured, it puts member 1 on **0.0 descriptors/token** and every census row back on the
* one-member line (per_tile 0.977 -> 2.000).
*/
#define PN_TT_ELTWISE_SPLIT_FUNCTIONS_ENV "PN_TT_ELTWISE_SPLIT_FUNCTIONS"
/** Is `function` one the split pays for? See the table above.
*
* READ ONCE. A `getenv` here is 136 calls a token on this family alone, and the knob cannot
* change inside a run. The names are compared WHOLE against
* `ft_tt_op_vector_function_name()`, which returns "SILU_MUL"/"SIGMOID"/"ADD"/"MUL" —
* UPPERCASE, so a lowercase list matches nothing, splits nothing, and reads exactly like a
* knob that is off. */
static bool eltwise_split_covers(const char *function)
{
static const char *list;
static bool cached;
if (!cached) {
cached = true;
list = getenv(PN_TT_ELTWISE_SPLIT_FUNCTIONS_ENV);
}
return pn_tt_name_list_contains(list, function);
}
uint32_t pn_tt_eltwise_group_members(const pn_tt_device *device)
{
if (!device || device->eltwise_members == 0u)
return 1u;
return device->eltwise_members;
}
bool pn_tt_device_eltwise_member_stat(const pn_tt_device *device, unsigned index,
uint32_t *out_core, uint64_t *out_ops, double *out_seconds)
{
const pn_tt_residency *member;
if (!device || index >= pn_tt_eltwise_group_members(device))
return false;
member = index == 0u ? &device->eltwise : &device->eltwise_member[index - 1u];
if (out_core)
*out_core = member->core_index;
if (out_ops)
*out_ops = member->ops;
if (out_seconds)
*out_seconds = device->eltwise_member_seconds[index];
return true;
}
static void eltwise_group_arm_wide(pn_tt_device *device,
const ft_tt_op_eltwise_request *placement);
/** The widest width any descriptor of this residency carries. */
static uint32_t eltwise_widest(const uint32_t *tile_counts, uint32_t count)
{
uint32_t widest = 0u, index;
for (index = 0; index < count; index++) {
if (tile_counts[index] > widest)
widest = tile_counts[index];
}
return widest;
}
void pn_tt_eltwise_group_plan(const pn_tt_device *device, const uint32_t *tile_counts,
uint32_t count, uint32_t *out_members, uint32_t *out_widest,
uint32_t *out_depth)
{
uint32_t sliced[PN_TT_ONCARD_MAX_ELTWISE_SHAPES];
uint32_t requested, members, cap, depth = 0u, index;
if (!out_members || !out_widest || !out_depth)
return;
*out_members = 1u;
*out_widest = 0u;
*out_depth = 0u;
if (!device || !tile_counts || count == 0u || count > PN_TT_ONCARD_MAX_ELTWISE_SHAPES)
return;
cap = pn_tt_env_u32(PN_TT_ELTWISE_DEPTH_ENV, PN_TT_ONCARD_MAX_DEPTH);
if (cap == 0u || cap > PN_TT_ONCARD_MAX_DEPTH)
cap = PN_TT_ONCARD_MAX_DEPTH;
requested = pn_tt_env_u32(PN_TT_ELTWISE_CORES_ENV, PN_TT_ELTWISE_CORES_DEFAULT);
/* THE TWO INSTRUMENTS DO NOT COMPOSE, AND THE SPLIT IS THE ONE THAT YIELDS. The
* queue-depth ladder counts K copies of one descriptor through
* `pn_tt_residency_submit()`; the split enqueues ONE descriptor per member directly, so
* `PN_TT_ONCARD_BATCH` would be inert here and its ladder would print a flat line
* produced by this file. Refusing names that. */
if (requested > 1u && pn_tt_env_u32("PN_TT_ONCARD_BATCH", 0u) > 1u &&
(!getenv("PN_TT_ONCARD_BATCH_FAMILY") ||
pn_tt_name_list_contains(getenv("PN_TT_ONCARD_BATCH_FAMILY"), "eltwise"))) {
fprintf(stderr, "pn_tt: REFUSING the eltwise split — PN_TT_ONCARD_BATCH is armed for "
"this family and the split does not go through the batched submit, so "
"the ladder would measure nothing. Run one instrument at a time.\n");
requested = 1u;
}
members = pn_tt_eltwise_split_member_limit(requested, tile_counts, count,
PN_TT_ELTWISE_MAX_MEMBERS);
/* THE DEPTH IS PART OF WHETHER A MEMBER COUNT IS LEGAL, not a consequence of it. A split
* whose slices no block depth serves is a descriptor the kernel refuses, so it shrinks
* here rather than on the card. */
while (members > 1u) {
for (index = 0; index < count; index++)
sliced[index] = tile_counts[index] / members;
depth = pn_tt_common_depth(sliced, count, cap);
if (depth != 0u)
break;
members = pn_tt_eltwise_split_member_limit(members - 1u, tile_counts, count,
PN_TT_ELTWISE_MAX_MEMBERS);
}
if (members > 1u) {
*out_widest = eltwise_widest(sliced, count);
} else {
depth = pn_tt_common_depth(tile_counts, count, cap);
*out_widest = eltwise_widest(tile_counts, count);
}
if (requested > 1u && members != requested)
fprintf(stderr, "pn_tt: eltwise SPLIT — %u members was asked for; %u is the largest "
"that divides every width at a legal block depth\n",
requested, members);
*out_members = members;
*out_depth = depth;
}
/** The placement every member of this device was armed at, rebuilt from what the device kept.
* It is DERIVED rather than stored because a stored copy is a second thing that must agree
* with `eltwise_depth` and the shape table, and this backend has already paid for a field
* that could disagree with the one beside it. */
static ft_tt_op_eltwise_request eltwise_placement(const pn_tt_device *device)
{
ft_tt_op_eltwise_request placement;
memset(&placement, 0, sizeof(placement));
placement.function = FT_TT_OP_VECTOR_FN_SILU_MUL; /* the widest of the six */
/* THE DECODE SPLIT'S slice, not the whole set's: the arm placed every member (wide ones
* included) for that width, and the block depth is derived from it — dividing by the
* W1a set's 15 placed a re-armed member at another depth, and every descriptor after an
* idle re-arm was refused (run_w1a_eltwise_wide.sh round 1). */
placement.tile_count = eltwise_widest(device->oncard_eltwise_tiles,
device->oncard_eltwise_count) /
(device->eltwise_decode_members ? device->eltwise_decode_members
: pn_tt_eltwise_group_members(device));
placement.prefetch_tiles = device->eltwise_depth;
placement.input_a.tile_bytes = PN_TT_TILE_BF16;
placement.input_b.tile_bytes = PN_TT_TILE_BF16;
placement.output.tile_bytes = PN_TT_TILE_BF16;
return placement;
}
void pn_tt_eltwise_group_arm(pn_tt_device *device, uint32_t members,
const ft_tt_op_eltwise_request *placement)
{
uint32_t first, index;
if (!device || !placement)
return;
pn_tt_eltwise_group_release(device);
memset(device->eltwise_member_seconds, 0, sizeof(device->eltwise_member_seconds));
device->eltwise_members = 1u;
device->eltwise_decode_members = 1u;
if (members < 2u) {
eltwise_group_arm_wide(device, placement);
return;
}
first = pn_tt_env_u32(PN_TT_ELTWISE_MEMBER_CORE_ENV, PN_TT_ELTWISE_MEMBER_CORE_DEFAULT);
for (index = 0; index + 1u < members; index++) {
pn_tt_residency *member = &device->eltwise_member[index];
ft_tt_launch_cb_entry placed[FT_TT_OP_ELTWISE_CB_COUNT];
if (!pn_tt_prepare_family_on_core(device, member, PN_TT_ELTWISE_PROGRAM,
"eltwise-member", PN_TT_ELTWISE_MEMBER_CORE_ENV,
first + index, PN_TT_ONCARD_TRIAD_MASK))
break;
if (ft_tt_op_eltwise_place_cbs(placement, placed, member->cb_buffer,
sizeof(member->cb_buffer), &member->cb_mask) !=
FT_TT_OK ||
!pn_tt_start_family(member)) {
pn_tt_residency_release(member);
break;
}
device->eltwise_members++;
}
/* COLLAPSE, DO NOT SHRINK. The header has the argument: a partial set is a member count
* the widths no longer divide, which is a ragged slice and a refused descriptor. */
if (device->eltwise_members != members) {
fprintf(stderr, "pn_tt: eltwise SPLIT — asked for %u members and %u armed; COLLAPSING "
"to one core, because a partial split is a width no member divides\n",
members, device->eltwise_members);
pn_tt_eltwise_group_release(device);
eltwise_group_arm_wide(device, placement);
return;
}
fprintf(stderr, "pn_tt: eltwise SPLIT — every descriptor over %u members, core %u then "
"%u..%u, at block depth %u\n",
device->eltwise_members, device->eltwise.core_index, first,
first + device->eltwise_members - 2u, device->eltwise_depth);
eltwise_group_arm_wide(device, placement);
}
/** W1a: arm the WIDE members after the decode split. A member that fails to arm stops the
* set where it is — the wide split is ragged by blocks, so any count is legal. */
static void eltwise_group_arm_wide(pn_tt_device *device, const ft_tt_op_eltwise_request *placement)
{
/* DEFAULT 13 (W1a, run_w1a_eltwise_wide.sh 2026-10-08): serve 1,630 TTFT 4.02 -> 3.25 s
* with the reader thread, decode flat; `PN_TT_ELTWISE_WIDE_CORES=0` off. */
const uint32_t want = pn_tt_env_u32(PN_TT_ELTWISE_WIDE_CORES_ENV, 13u);
const uint32_t first = pn_tt_env_u32(PN_TT_ELTWISE_WIDE_CORE_ENV,
PN_TT_ELTWISE_WIDE_CORE_DEFAULT);
uint32_t index;
device->eltwise_decode_members = device->eltwise_members;
for (index = 0; index < want && device->eltwise_members < PN_TT_ELTWISE_MAX_MEMBERS;
index++) {
pn_tt_residency *member = &device->eltwise_member[device->eltwise_members - 1u];
ft_tt_launch_cb_entry placed[FT_TT_OP_ELTWISE_CB_COUNT];
if (!pn_tt_prepare_family_on_core(device, member, PN_TT_ELTWISE_PROGRAM,
"eltwise-member", PN_TT_ELTWISE_WIDE_CORE_ENV,
first + index, PN_TT_ONCARD_TRIAD_MASK))
break;
if (ft_tt_op_eltwise_place_cbs(placement, placed, member->cb_buffer,
sizeof(member->cb_buffer), &member->cb_mask) !=
FT_TT_OK ||
!pn_tt_start_family(member)) {
pn_tt_residency_release(member);
break;
}
device->eltwise_members++;
}
if (want)
fprintf(stderr, "pn_tt: eltwise WIDE — %u of %u extra members armed from core %u; "
"descriptors of %u+ tiles run over all %u (block %u)\n",
device->eltwise_members - device->eltwise_decode_members, want, first,
pn_tt_env_u32(PN_TT_ELTWISE_WIDE_MIN_TILES_ENV,
PN_TT_ELTWISE_WIDE_MIN_TILES_DEFAULT),
device->eltwise_members, 2u * device->eltwise_depth);
}
uint32_t pn_tt_eltwise_group_members_for(const pn_tt_device *device, uint32_t tile_count,
uint32_t function)
{
static uint32_t min_wide;
uint32_t min_tiles;
if (!device)
return 1u;
if (min_wide == 0u)
min_wide = pn_tt_env_u32(PN_TT_ELTWISE_WIDE_MIN_TILES_ENV,
PN_TT_ELTWISE_WIDE_MIN_TILES_DEFAULT);
/* THE SFPU FUNCTIONS cost ~0.8 us a tile a member; ADD/MUL/SUB ~0.06 — a wide split of
* those pays its ~45 us a member enqueue only 13x wider (the residual ADD at Wt=6528 is
* 390 us on two). */
min_tiles = (function == FT_TT_OP_VECTOR_FN_SILU_MUL || function == FT_TT_OP_VECTOR_FN_SILU ||
function == FT_TT_OP_VECTOR_FN_SIGMOID)
? min_wide : 16u * min_wide;
if (device->eltwise_decode_members == 0u ||
device->eltwise_members <= device->eltwise_decode_members || tile_count < min_tiles ||
device->eltwise_depth == 0u ||
tile_count / (2u * device->eltwise_depth) < device->eltwise_members ||
(tile_count % (2u * device->eltwise_depth)) != 0u)
return device->eltwise_decode_members ? device->eltwise_decode_members
: pn_tt_eltwise_group_members(device);
return device->eltwise_members;
}
bool pn_tt_eltwise_group_slice(const pn_tt_device *device, uint32_t tile_count,
uint32_t members, uint32_t member, uint32_t tile_bytes,
pn_tt_eltwise_slice *out_slice)
{
if (members > 1u && members == device->eltwise_members &&
members > device->eltwise_decode_members && device->eltwise_decode_members != 0u) {
/* RED `eltwise_wide_shift`: a wide member's slice starts one block late — the slices
* then overlap and leave the last block unwritten; must DIFFER. */
if (!pn_tt_eltwise_split_slice_blocks(tile_count, members, member,
2u * device->eltwise_depth, tile_bytes, out_slice))
return false;
if (member > 0u && out_slice->tile_count && pn_tt_red("eltwise_wide_shift") &&
out_slice->first_tile + out_slice->tile_count < tile_count) {
out_slice->first_tile += 2u * device->eltwise_depth;
out_slice->byte_offset = out_slice->first_tile * tile_bytes;
}
return true;
}
return pn_tt_eltwise_split_slice(tile_count, members, member, tile_bytes, out_slice);
}
void pn_tt_eltwise_group_release(pn_tt_device *device)
{
uint32_t index;
if (!device)
return;
for (index = 1u; index < device->eltwise_members; index++)
pn_tt_residency_release(&device->eltwise_member[index - 1u]);
device->eltwise_members = device->eltwise.running ? 1u : 0u;
device->eltwise_decode_members = device->eltwise_members;
}
/**
* Retire ONE member of the set and arm a fresh one ON THE CORE IT ALREADY HELD.
*
* The core is captured BEFORE the release, because re-reading `core_index` off a released
* handle is how a member gets armed on core 0. Member 0 goes through the same code against
* its own env name — the alternative, `pn_tt_oncard_rearm()`, re-places the WHOLE set from
* scratch, which is the wrong operation in the middle of a descriptor whose other members
* have already retired.
*
* @return true when a fresh residency is live on that core and the slice may be replayed
*/
static bool eltwise_member_rearm(pn_tt_device *device, uint32_t member_index,
const ft_tt_op_eltwise_request *placement)
{
ft_tt_launch_cb_entry placed[FT_TT_OP_ELTWISE_CB_COUNT];
const char *core_env = member_index == 0u ? PN_TT_CORE_ELTWISE_ENV
: PN_TT_ELTWISE_MEMBER_CORE_ENV;
const char *name = member_index == 0u ? "eltwise" : "eltwise-member";
pn_tt_residency *member;
uint32_t core;
if (member_index >= pn_tt_eltwise_group_members(device))
return false;
member = member_index == 0u ? &device->eltwise : &device->eltwise_member[member_index - 1u];
core = member->core_index;
fprintf(stderr, "pn_tt: eltwise member %u refused a descriptor — retiring it by ASKING and "
"arming a fresh one on core %u\n", member_index, core);
pn_tt_residency_stop(member);
pn_tt_residency_release(member);
device->oncard_restarts++;
/* A core whose compute thread is still spinning out its bound is a WEDGE, not a re-arm:
* booting firmware onto it leaves `instrn_buf_status = 0x66` and needs a rung-3 recovery
* and sudo. Every other family here refuses on the same reading. */
if (!pn_tt_residency_retired(member)) {
fprintf(stderr, "pn_tt: eltwise member %u did not retire — REFUSING to arm a fresh one "
"on core %u\n", member_index, core);
return false;
}
if (!pn_tt_prepare_family_on_core(device, member, PN_TT_ELTWISE_PROGRAM, name, core_env,
core, PN_TT_ONCARD_TRIAD_MASK))
return false;
if (ft_tt_op_eltwise_place_cbs(placement, placed, member->cb_buffer,
sizeof(member->cb_buffer), &member->cb_mask) != FT_TT_OK) {
pn_tt_residency_release(member);
return false;
}
return pn_tt_start_family(member);
}
/** Advance one operand's base to `slice`. An UNSET operand — a unary descriptor's B, which
* the encoder zeroes and the kernel never reads — is left alone, because offsetting it
* would build an address out of nothing. */
static void eltwise_slice_operand(ft_tt_op_operand *operand, const pn_tt_eltwise_slice *slice)
{
if (operand->tile_bytes == 0u)
return;
operand->address += slice->first_tile * operand->tile_bytes;
}
/** Encode member `index`'s slice of `request`. @return false with the reason on stderr. */
static bool eltwise_slice_words(const pn_tt_device *device,
const ft_tt_op_eltwise_request *request, uint32_t members,
uint32_t index, uint32_t *out_words)
{
ft_tt_op_eltwise_request sliced = *request;
pn_tt_eltwise_slice slice;
if (!pn_tt_eltwise_group_slice(device, request->tile_count, members, index,
PN_TT_TILE_BF16, &slice) || slice.tile_count == 0u) {
fprintf(stderr, "pn_tt: %u eltwise tiles do not divide over %u members\n",
request->tile_count, members);
return false;
}
sliced.tile_count = slice.tile_count;
sliced.prefetch_tiles = device->eltwise_depth;
eltwise_slice_operand(&sliced.input_a, &slice);
if (ft_tt_op_vector_function_is_binary(request->function))
eltwise_slice_operand(&sliced.input_b, &slice);
eltwise_slice_operand(&sliced.output, &slice);
if (ft_tt_op_eltwise_parameters(&sliced, out_words) != FT_TT_OK) {
fprintf(stderr, "pn_tt: the eltwise slice [%u, %u) is one the family refuses\n",
slice.first_tile, slice.first_tile + slice.tile_count);
return false;
}
pn_tt_vec_probe_apply(out_words);
return true;
}
/** The member residency `index` of the set (member 0 is `eltwise` itself). */
static pn_tt_residency *eltwise_member_residency(pn_tt_device *device, uint32_t index)
{
return index == 0u ? &device->eltwise : &device->eltwise_member[index - 1u];
}
/**
* Await every member of an enqueued descriptor, then re-arm and replay any member that was
* refused, then charge the set. `words[index]` is member `index`'s slice.
*
* PER-MEMBER RE-ARM AND REPLAY. Replaying one member's slice is the SAME COMPUTATION and not
* merely a safe one: an elementwise descriptor is a pure function of operands it does not
* modify, and the slices are disjoint, so a member that succeeded is neither re-run nor read.
* A refused descriptor also leaves no circular-buffer traffic behind it — the reader aborts
* before the rebase. (A deferred descriptor keeps this property only while its operands are
* untouched: the caller settles it before anything rewrites them.)
*/
static int eltwise_group_complete(pn_tt_device *device, const ft_tt_op_eltwise_request *request,
uint32_t (*words)[FT_TT_OP_VECTOR_WORDS], uint32_t members,
bool *bad, double started)
{
uint32_t index;
int failed = 0;
/* EVERY member is awaited even after one has failed. An enqueue that got as far as the
* ring has a descriptor the card will run whatever this host does next, and leaving it
* un-awaited desynchronises that member's cursor for the rest of the run. */
for (index = 0; index < members; index++) {
pn_tt_residency *member = eltwise_member_residency(device, index);
const double member_started = pn_tt_now_seconds();
double member_elapsed;
if (pn_tt_residency_await(member, PN_TT_OP_TIMEOUT_MS) != 0)
bad[index] = true;
failed |= bad[index];
member_elapsed = pn_tt_now_seconds() - member_started;
device->eltwise_member_seconds[index] += member_elapsed;
member->seconds += member_elapsed;
}
if (failed) {
const ft_tt_op_eltwise_request placement = eltwise_placement(device);
failed = 0;
for (index = 0; index < members; index++) {
if (!bad[index])
continue;
if (!eltwise_member_rearm(device, index, &placement) ||
pn_tt_residency_submit(eltwise_member_residency(device, index),
FT_TT_QUEUE_OP_ELTWISE, words[index],
PN_TT_OP_TIMEOUT_MS) != 0)
failed = 1;
}
}
/* Charged to the SET, once, at the WHOLE descriptor's tile count — so the census key is
* the same row at every member count and its fit is comparable across the ladder. */
pn_tt_oncard_account(device, "eltwise", ft_tt_op_vector_function_name(request->function),
request->tile_count, pn_tt_now_seconds() - started);
return failed ? -1 : 0;
}
void pn_tt_eltwise_group_defer_next(pn_tt_device *device)
{
const char *env = getenv("PN_TT_ELTWISE_DEFER");
if (device && !(env && *env && atoi(env) == 0))
device->eltwise_defer_next = true;
}
uint64_t pn_tt_eltwise_group_deferred(const pn_tt_device *device)
{
return device ? device->eltwise_deferred : 0u;
}
int pn_tt_eltwise_group_settle(pn_tt_device *device)
{
bool bad[PN_TT_ELTWISE_MAX_MEMBERS];
uint32_t members;
if (!device || device->eltwise_pending_members == 0u)
return 0;
members = device->eltwise_pending_members;
device->eltwise_pending_members = 0u;
memset(bad, 0, sizeof(bad));
return eltwise_group_complete(device, &device->eltwise_pending_request,
device->eltwise_pending_words, members, bad,
device->eltwise_pending_started);
}
int pn_tt_eltwise_group_submit(pn_tt_device *device, const ft_tt_op_eltwise_request *request)
{
uint32_t words[PN_TT_ELTWISE_MAX_MEMBERS][FT_TT_OP_VECTOR_WORDS];
bool bad[PN_TT_ELTWISE_MAX_MEMBERS];
uint32_t members, index;
double started;
bool wide, defer, enqueue_failed = false;
if (!device || !request)
return -1;
defer = device->eltwise_defer_next;
device->eltwise_defer_next = false;
/* The ring is FIFO, but the bookkeeping is not: a pending descriptor is awaited first. */
if (pn_tt_eltwise_group_settle(device) != 0)
return -1;
members = pn_tt_eltwise_group_members_for(device, request->tile_count,
(uint32_t)request->function);
wide = members > device->eltwise_decode_members && device->eltwise_decode_members != 0u;
/* A WIDTH THE SET DOES NOT DIVIDE RUNS WHOLE ON MEMBER 0 rather than failing. The arm
* planned the split over every width the residency DECLARED, and a caller that reaches
* here with another one has already passed the depth guard — so member 0's placement
* serves it, and one core is slower rather than wrong. Failing instead would turn a new
* call site into a dropped descriptor, which on this family is a wrong answer with a
* return code nobody reads. Said once, because 136 descriptors a token is a flood. */
if (!wide && (request->tile_count % members) != 0u) {
static bool narrowed;
if (!narrowed) {
narrowed = true;
fprintf(stderr, "pn_tt: %u eltwise tiles do not divide over %u members — this "
"width runs WHOLE on member 0; the split was planned for the "
"widths the residency was armed with\n",
request->tile_count, members);
}
members = 1u;
}
/* AND A SHAPE BELOW THE BREAK-EVEN RUNS WHOLE ON MEMBER 0 TOO. The knee is a descriptor
* TIME of ~87 us and not a width — the residual ADD (77.5 us) and the attention MUL
* (69.0 us) are made SLOWER by a member, at the same Wt=128 where SILU_MUL and SIGMOID
* are made 1.46x and 1.41x faster. See the table beside
* PN_TT_ELTWISE_SPLIT_FUNCTIONS_ENV. Member 0's placement serves the whole width, so
* this is one core rather than wrong. */
if (members > 1u && !eltwise_split_covers(ft_tt_op_vector_function_name(request->function)))
members = 1u;
memset(bad, 0, sizeof(bad));
for (index = 0; index < members; index++)
if (!eltwise_slice_words(device, request, members, index, words[index]))
return -1;
/* RED `eltwise_defer_drop`: a deferred descriptor is never enqueued — its output is
* whatever the strip held. Must DIFFER: proves the gate scores the deferred op. */
if (defer && pn_tt_red("eltwise_defer_drop"))
return 0;
started = pn_tt_now_seconds();
for (index = 0; index < members; index++)
if (pn_tt_residency_enqueue(eltwise_member_residency(device, index),
FT_TT_QUEUE_OP_ELTWISE, words[index], 1u) != 0)
enqueue_failed = bad[index] = true;
/* THE RESIDENCY CLOCKS RUN ON THIS PATH TOO (class 28: `ops != 0 && seconds == 0` read
* as a free family). The enqueue is booked on member 0 and each member's serial await on
* that member, so the rows SUM to the set's wall and no time is booked twice. */
device->eltwise.seconds += pn_tt_now_seconds() - started;
/* PM 7 DEFERRED: the awaits wait for pn_tt_eltwise_group_settle(). A failed enqueue is
* completed (replayed) now instead, while the caller is still at this descriptor. */
if (defer && !enqueue_failed) {
memcpy(device->eltwise_pending_words, words, sizeof(words[0]) * members);
device->eltwise_pending_request = *request;
device->eltwise_pending_started = started;
device->eltwise_pending_members = members;
device->eltwise_deferred++;
return 0;
}
return eltwise_group_complete(device, request, words, members, bad, started);
}
#endif /* USE_TT */

243
src/accel/tt/pn_tt_eltwise_group.h Normal file
View File

@ -0,0 +1,243 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_eltwise_group.h
* @brief SPLIT ONE ELEMENTWISE DESCRIPTOR ACROSS N TENSIX. Arms `PN_TT_ELTWISE_CORES - 1`
* extra `eltwise` residencies and gives each a contiguous, disjoint slice of every
* descriptor's tile stream. `pn_tt_eltwise_split.h` holds the arithmetic.
*
* ---------------------------------------------------------------------------
* WHY: FOUR FAMILIES, ONE CEILING, AND IT IS ONE CORE
* ---------------------------------------------------------------------------
* Measured on silicon 2026-08-24, with the host out of the token: eltwise **4.43 GB/s**,
* render 3.65, reduce ~3.8, topk 6.09 — against the **28.3 GB/s a single core's own
* drain-only arm measured**, and 510.8 for the card. Four independent families sitting 7x
* below ONE core's transaction wall and 120x below the part. The bind is the per-tile path
* on one Tensix; it is neither the byte count nor the descriptor count.
*
* Meanwhile the matmul runs 8 members and the GDN head split runs 4 (measured 2.70x at 4).
* The elementwise family ran ONE — 136 descriptors a token, 27,648 tiles, 0.0374 s, **19.7%
* of a 0.197 s token** — and nothing had ever been split across cores here. It is the
* cleanest parallelism on the card: `y = f(a, b)` per tile, no reduction, no ordering.
*
* ---------------------------------------------------------------------------
* WHAT A MEMBER IS, AND WHAT IT IS NOT
* ---------------------------------------------------------------------------
* A member is a WHOLE `eltwise` residency on its own Tensix, running the same program at the
* same circular-buffer placement, and its descriptor is the caller's with three bases
* advanced and the tile count divided. That means:
*
* - **NO BUFFER IS SHARED.** Every member has its own L1 and its own CBs. The one-producer
* rule (`resident_queue_protocol.h` rule 1) is about a CB with two writers; there is no
* such CB here, because there is no such CB between any two cores.
* - **NO DRAM BYTE IS SHARED.** Member `m` reads `[m*W, (m+1)*W)` of A and of B and writes
* the same range of the output, for `W = tile_count / N`. Pairwise disjoint by
* construction and asserted off-hardware in `tests/test_tt_eltwise_split.c`.
* - **NO KERNEL CHANGES AND NO NEW OPCODE.** The descriptor is the shipped
* `FT_TT_QUEUE_OP_ELTWISE` with different words. `resident_queue_protocol.h` is a
* content-provenance input to EVERY embedded image, so an opcode registered there would
* make all 23 programs stale; nothing is registered there.
*
* ---------------------------------------------------------------------------
* THE CORES, AND WHY THIS SET CANNOT COLLIDE
* ---------------------------------------------------------------------------
* Member 0 is `device->eltwise` on `PN_TT_CORE_ELTWISE` (73). Members 1..N-1 run upward from
* `PN_TT_ELTWISE_MEMBER_CORE_FIRST`, default **106**, and 106 is chosen as the first index
* ABOVE EVERY CORE ANY FAMILY CAN REACH AT ANY LEGAL SETTING OF ANY KNOB:
*
* 71 matmul member 0 72/73/74 reduce / eltwise / topk
* 75..105 matmul members 1..31 82/83/84 rope / memory / attention
* (PN_TT_MATMUL_MAX_CORES = 32, first = 75)
* 85/86 render widen / narrow 90 gdn member 0
* 91, 103 the two head-norm slots 92..99 gdn head-split members (max 8)
* 95 gdnc (chunkwise) 100..102 the rectangle arm's control
* ---------------------------------------------------------------------
* 106..121 ELTWISE members 1..15 122..139 free
*
* That is a claim about the WHOLE map and not about today's defaults, which is the lesson
* the GDN family paid for: its 75 was clear of every other FAMILY and collided with matmul
* member 1 the day `PN_TT_MATMUL_CORES` moved, for 1560 mutual re-arms over 65 forwards with
* a healthy "residency up" logged each time. **A contested core does not fail; it computes
* badly** — 1.0e-2 contested against 6.9e-4 clean on the same descriptor.
* `pn_tt_prepare_family_on_core()` refuses a collision BY NAME and is still the guard; this
* default is only the first place to look, and `pn_tt_core_holder()` now walks these members
* too, so a later family that lands on one is told.
*
* ---------------------------------------------------------------------------
* THE KNOBS
* ---------------------------------------------------------------------------
* `PN_TT_ELTWISE_CORES=N` members, 1 = the pre-split path byte for byte
* `PN_TT_ELTWISE_MEMBER_CORE_FIRST=C` where members 1..N-1 start (106)
* `PN_TT_ELTWISE_DEPTH=D` CAP on the block depth, for the knee control below
*
* `PN_TT_ELTWISE_DEPTH` exists because **the depth MOVES WITH THE MEMBER COUNT and would
* otherwise be a confound in the sweep.** A residency is placed for one depth that must
* satisfy `2 * depth | tile_count` for every width it serves; Ornith's widths are 128 / 256 /
* 384, so unsplit it is 32 and at 8 members the slices are 16 / 32 / 48 and it is 8. A
* member-count ladder therefore also walks a depth ladder, and "it stopped at N" would not
* say which one stopped. `PN_TT_ELTWISE_DEPTH=8` at `PN_TT_ELTWISE_CORES=1` is the
* discriminating arm: one core, the split's depth, no split.
*
* ---------------------------------------------------------------------------
* THE ONE THING THE SPLIT BREAKS, AND IT IS NOT THE ARITHMETIC
* ---------------------------------------------------------------------------
* **A SLICE MOVES `C_BASE`, AND THE RESIDENT REPLICA TABLE IS KEYED ON THE EXACT TRIPLE.**
* `resident_vector_writer.c` mirrors an output strip by looking `(kVecCX, kVecCY, kVecCBase)`
* up in a table, and `ft_tt_op_replicas.h` says a miss "is the shipped path word for word".
* So member 0 hit the table and mirrored its slice, and every member above it MISSED and
* left its slice stale in seven banks — which the matmul members read. Every kernel retired
* clean and nothing refused, exactly as that header predicts a bad map behaves.
*
* Measured, and settled by a DISCRIMINATING NULL rather than by reading the code: with
* mirrors ON the split is `cmp` IDENTICAL at one member and 204 B / 8 B / 119 B / 57 B at
* 2 / 4 / 8 / 16 — wrong in proportion to the `(N-1)/N` of the strip that misses. With
* `PN_TT_ACTIVATION_REPLICAS=1` and no table to miss it is **IDENTICAL at 1, 2 AND 4**. The
* slice arithmetic was never the problem.
*
* `pn_tt_publish_replica_map()` now takes the split family's output strip and registers ONE
* SET PER SLICE, and publishes to the members as well — they were being armed with no table
* at all, which is not "no mirroring" but "whatever that L1 region held".
*
* @status SHIPPED ON at `PN_TT_ELTWISE_CORES=2`, the measured knee: 5.01 -> 5.17 tok/s
* against a control re-taken the same hour. `pn_tt_eltwise_group.c` carries the
* ladder, the fit and why 2 rather than 8.
* @issues None
* @todo None
*/
#ifndef PN_TT_ELTWISE_GROUP_H
#define PN_TT_ELTWISE_GROUP_H
#ifdef USE_TT
#include <stdbool.h>
#include <stdint.h>
#include "tt/ft_tt_op_vector.h"
#include "pn_tt_eltwise_split.h"
typedef struct pn_tt_device pn_tt_device;
/** Elementwise residencies one split may hold, member 0 included. The device's array holds
* one fewer, exactly as the GDN head split's does. */
#define PN_TT_ELTWISE_MAX_MEMBERS PN_TT_ELTWISE_SPLIT_MAX_MEMBERS
/** THE FAMILY'S PLACEMENT, in one place because two files now arm it. `pn_tt_oncard.c` arms
* member 0 and this module arms the rest, and both re-arm on a refusal — a second spelling
* of the program name or the core default would be two things that must agree. */
#define PN_TT_ELTWISE_PROGRAM "eltwise"
#define PN_TT_CORE_ELTWISE_ENV "PN_TT_CORE_ELTWISE"
#define PN_TT_CORE_ELTWISE_DEFAULT 73u
/**
* Decide the split for `tile_counts` BEFORE anything is placed, and report it.
*
* It runs before the placement rather than after because **the placement is sized from the
* SLICED width**: the input CBs are two blocks of the effective depth, and the depth is
* chosen from the tile counts descriptors will actually carry. Deciding the member count
* afterwards would place member 0 for a width no descriptor has.
*
* The member count is shrunk until it divides every width AND the sliced widths admit a
* common block depth. A shrink is reported, because a silent one shows up only as a number
* that did not move.
*
* @param device for the env knobs only; nothing is armed here
* @param tile_counts every width a descriptor of this residency will carry
* @param count how many of them
* @param out_members receives 1..PN_TT_ELTWISE_MAX_MEMBERS; never NULL
* @param out_widest receives the widest SLICE, for `place_cbs`; never NULL
* @param out_depth receives the ONE block depth every member runs; 0 when none serves
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
void pn_tt_eltwise_group_plan(const pn_tt_device *device, const uint32_t *tile_counts,
uint32_t count, uint32_t *out_members, uint32_t *out_widest,
uint32_t *out_depth);
/**
* Arm members 1..`members`-1 on their own cores, with member 0's placement.
*
* They take the IDENTICAL circular-buffer placement member 0 has, and that is a property of
* the family rather than a convenience: `ft_tt_op_eltwise_place_cbs()` reads the effective
* depth and the three page sizes and nothing else, so a member owning 16 tiles places
* exactly what a member owning 128 does.
*
* **A MEMBER THAT FAILS TO ARM COLLAPSES THE SPLIT TO ONE**, loudly, rather than shrinking
* it the way the GDN head split does. A shrink there is safe because any head count divides;
* here the widths must divide the member count exactly, and 8 members shrinking to 5 is a
* ragged slice on a 128-tile residual — a descriptor the kernel refuses. Unsplit is always
* correct and always available: member 0's placement serves the whole width too, since
* `2 * depth` divides a slice and the slice divides the width.
*
* @param device the device, with `device->eltwise` already started
* @param members what `pn_tt_eltwise_group_plan()` returned
* @param placement the request member 0 was placed with
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
void pn_tt_eltwise_group_arm(pn_tt_device *device, uint32_t members,
const ft_tt_op_eltwise_request *placement);
/**
* Run ONE elementwise descriptor across the whole set: ALL ENQUEUED BEFORE ANY IS AWAITED.
*
* The two loops being separate is the measurement and not a style. Enqueue-wait per member
* would run `members` SERIAL descriptors and print the flat line "it does not parallelise",
* produced by this function rather than by the card — the same trap `gdn_submit_delta_split()`
* and the core-scaling probe both call out.
*
* Time is charged to the SET, once, at the WHOLE descriptor's tile count, so the census key
* `(eltwise, function, tile_count)` is the same row at every member count and the
* `fixed + per_tile * tiles` fit is directly comparable across the ladder.
*
* @param device the device
* @param request the WHOLE descriptor, already validated by the caller
* @return 0, or -1 with the reason on stderr
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
int pn_tt_eltwise_group_submit(pn_tt_device *device, const ft_tt_op_eltwise_request *request);
/**
* PM 7: the NEXT pn_tt_eltwise_group_submit() enqueues and returns WITHOUT awaiting; the
* descriptor stays pending until pn_tt_eltwise_group_settle() or the next eltwise descriptor.
* Streamed (the prefill band loop), the puller's AWAIT then lands where the settle is, so the
* chain's next ops run under it. The caller owns the hazard: settle before anything reads the
* output or rewrites an operand. `PN_TT_ELTWISE_DEFER=0` makes this a no-op.
*/
void pn_tt_eltwise_group_defer_next(pn_tt_device *device);
/** Descriptors deferred so far (the report's proof the deferral is live). */
uint64_t pn_tt_eltwise_group_deferred(const pn_tt_device *device);
/** Await (and replay, if refused) the pending deferred descriptor. @return 0 when none. */
int pn_tt_eltwise_group_settle(pn_tt_device *device);
/** Release members 1..N-1 and set the count back to 1. Safe on a device that never split.
*
* A LEAKED TRIAD IS A WEDGED TENSIX for the next process to ask for that core, which is
* what the head-norm slots cost when `pn_tt_device_close()` released named fields and
* missed a set. This is a set; it is released by name from that function. */
void pn_tt_eltwise_group_release(pn_tt_device *device);
/** Live members, member 0 included. 1 when unsplit.
*
* The per-member ledger the report reads is `pn_tt_device_eltwise_member_stat()` in
* `pn_tt_device.h` — public, because the model's report is what consumes it. */
uint32_t pn_tt_eltwise_group_members(const pn_tt_device *device);
/** W1a: how many members a descriptor of `tile_count` tiles runs over — every member (decode
* + wide) for a wide descriptor, the decode split otherwise. */
uint32_t pn_tt_eltwise_group_members_for(const pn_tt_device *device, uint32_t tile_count,
uint32_t function);
/** Member `member`'s slice of a `tile_count`-tile descriptor run over `members` — the block
* split when `members` is the full wide set, the equal split otherwise. The replica map
* keys each member's mirrors from this, so the writer and the map cannot disagree. */
bool pn_tt_eltwise_group_slice(const pn_tt_device *device, uint32_t tile_count,
uint32_t members, uint32_t member, uint32_t tile_bytes,
pn_tt_eltwise_slice *out_slice);
#endif /* USE_TT */
#endif /* PN_TT_ELTWISE_GROUP_H */

81
src/accel/tt/pn_tt_eltwise_split.c Normal file
View File

@ -0,0 +1,81 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_eltwise_split.c
* @brief Implementation of the elementwise split arithmetic — member count and tile range.
*
* No device, no allocation, no I/O, and deliberately so: the header states why an address
* is the one thing here that fails silently, and this file is what a test with no card can
* hold to that.
*
* @status 0 checks failed off-hardware, `tests/test_tt_eltwise_split.c`.
* @issues None
* @todo None
*/
#include "pn_tt_eltwise_split.h"
uint32_t pn_tt_eltwise_split_member_limit(uint32_t requested, const uint32_t *tile_counts,
uint32_t count, uint32_t max_members)
{
uint32_t members;
if (!tile_counts || count == 0u || requested == 0u || max_members == 0u)
return 1u;
if (max_members > PN_TT_ELTWISE_SPLIT_MAX_MEMBERS)
max_members = PN_TT_ELTWISE_SPLIT_MAX_MEMBERS;
members = requested < max_members ? requested : max_members;
for (; members > 1u; members--) {
uint32_t index;
bool divides = true;
for (index = 0; index < count && divides; index++) {
/* A zero width is a caller defect and cannot be split at all. Refusing the whole
* split rather than skipping the row keeps "every width this residency serves"
* literally true of the answer. */
if (tile_counts[index] == 0u || (tile_counts[index] % members) != 0u)
divides = false;
}
if (divides)
return members;
}
return 1u;
}
bool pn_tt_eltwise_split_slice(uint32_t tile_count, uint32_t members, uint32_t member,
uint32_t tile_bytes, pn_tt_eltwise_slice *out_slice)
{
uint32_t per_member;
if (!out_slice || members == 0u || members > PN_TT_ELTWISE_SPLIT_MAX_MEMBERS ||
member >= members || tile_count == 0u || tile_bytes == 0u)
return false;
if ((tile_count % members) != 0u)
return false;
per_member = tile_count / members;
out_slice->first_tile = member * per_member;
out_slice->tile_count = per_member;
out_slice->byte_offset = out_slice->first_tile * tile_bytes;
return true;
}
bool pn_tt_eltwise_split_slice_blocks(uint32_t tile_count, uint32_t members, uint32_t member,
uint32_t block_tiles, uint32_t tile_bytes,
pn_tt_eltwise_slice *out_slice)
{
uint32_t blocks, base, extra, first_block, own;
if (!out_slice || members == 0u || members > PN_TT_ELTWISE_SPLIT_MAX_MEMBERS ||
member >= members || tile_count == 0u || tile_bytes == 0u || block_tiles == 0u ||
(tile_count % block_tiles) != 0u)
return false;
blocks = tile_count / block_tiles;
base = blocks / members;
extra = blocks % members;
first_block = member * base + (member < extra ? member : extra);
own = base + (member < extra ? 1u : 0u);
out_slice->first_tile = first_block * block_tiles;
out_slice->tile_count = own * block_tiles;
out_slice->byte_offset = out_slice->first_tile * tile_bytes;
return true;
}

129
src/accel/tt/pn_tt_eltwise_split.h Normal file
View File

@ -0,0 +1,129 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_eltwise_split.h
* @brief The ARITHMETIC of splitting one elementwise descriptor across N Tensix: how many
* members a set of widths admits, and which contiguous tile range each member owns.
*
* It is a separate module from `pn_tt_eltwise_group.h` for the reason `pn_tt_gdn_layout.h`
* is separate from `pn_tt_gdn.c`: everything here is pure integer arithmetic over a tile
* count, so it is provable with no card, no device and no `USE_TT` — and the one thing a
* split can get wrong that no error will ever report is an address.
*
* ---------------------------------------------------------------------------
* WHY AN ELEMENTWISE OP IS THE CLEANEST SPLIT ON THE CARD
* ---------------------------------------------------------------------------
* `resident_eltwise_reader.c` reads tile `t` of a stream at `base + t * TILE_BYTES`,
* unconditionally and in order — its own header calls this "the easy case", against the
* matmul reader, which had to argue that a block's B tiles are consecutive only under an
* N-major weight grid. So member `m` of an N-way split runs the SAME descriptor with
* `tile_count / N` and every base advanced by `m * (tile_count / N) * tile_bytes`, and the
* members touch pairwise-disjoint byte ranges of A, of B and of the output. There is no
* shared operand, no reduction, no ordering: `y = f(a, b)` per tile.
*
* ---------------------------------------------------------------------------
* WHY THE SPLIT MUST DIVIDE EXACTLY, WHERE THE MATMUL GROUP'S MAY BE RAGGED
* ---------------------------------------------------------------------------
* `pn_tt_matmul_group_place_weight_columns()` hands the remainder out one column at a time
* and carries a RUNNING SUM, because its members' widths only have to concatenate. This
* family cannot do that, and the reason is the circular buffers rather than the answer:
*
* 1. **The block-depth invariant is per descriptor.** A residency is placed for ONE depth,
* and every descriptor it will ever run must satisfy `2 * depth | tile_count` — the FIFO
* write pointer wraps only on an exact hit of the limit, so a descriptor that leaves it
* elsewhere walks the NEXT one up L1 with no error (`pn_tt_common_depth()`, and the top-k
* reader that landed on the queue control block at 0x170000 without faulting). A ragged
* slice is a tile count off that grid on exactly one member.
* 2. **A set is as slow as its longest member**, and every member is a whole residency
* paying a whole descriptor's fixed cost. An uneven split buys the biggest slice's time
* and wastes the rest — the same argument `gdn_head_member_count()` makes about v-heads.
*
* So the member count is clamped to a COMMON DIVISOR of every width the residency was armed
* for, and a width that does not divide shrinks the split for all of them rather than
* splitting some descriptors and not others. One depth, one placement, every member.
*
* @status Off-hardware only so far: `tests/test_tt_eltwise_split.c`. The card arm that
* consumes it is `pn_tt_eltwise_group.h`.
* @issues None
* @todo None
*/
#ifndef PN_TT_ELTWISE_SPLIT_H
#define PN_TT_ELTWISE_SPLIT_H
#include <stdbool.h>
#include <stdint.h>
/** Elementwise residencies one split may hold, member 0 included.
*
* SIXTEEN because that is what Ornith's narrowest elementwise width admits and not because
* the card stops there: the residual adds are 128 tiles, and 128/16 = 8 tiles is still two
* blocks of depth 4. Thirty-two would leave 4 tiles a member at depth 2, which is a
* descriptor that is nearly all fixed cost. Raising it is a measurement and sixteen more
* Tensix, not an edit. */
#define PN_TT_ELTWISE_SPLIT_MAX_MEMBERS 16u
/** One member's share of a descriptor: a contiguous tile range and the byte offset every
* operand base advances by to reach it. */
typedef struct pn_tt_eltwise_slice {
uint32_t first_tile;
uint32_t tile_count;
uint32_t byte_offset; /**< `first_tile * tile_bytes`, for the operand asked about */
} pn_tt_eltwise_slice;
/**
* The largest member count at or below `requested` that divides EVERY width exactly.
*
* Walks down from the clamp rather than taking a GCD, because the answer wanted is "the
* biggest split this residency can run", not "the biggest split that exists": `requested` is
* an operator's knob and the clamp is the array's size.
*
* @param requested what the run asked for; 0 or 1 means no split
* @param tile_counts every width a descriptor of this residency will carry
* @param count how many of them; 0 refuses
* @param max_members the caller's array bound, itself clamped to the module's
* @return 1..max_members; 1 whenever no larger count divides every width
*
* Thread-safe: yes Reentrant: yes Signal-safe: yes
*/
uint32_t pn_tt_eltwise_split_member_limit(uint32_t requested, const uint32_t *tile_counts,
uint32_t count, uint32_t max_members);
/**
* Which tiles member `member` of an `members`-way split owns, and at what byte offset.
*
* REFUSES rather than rounding when the split does not divide, because the caller that
* rounds here writes one member's answer over another's — a plausible wrong answer with no
* error anywhere, which is the failure this whole module exists to make unreachable.
*
* @param tile_count the WHOLE descriptor's tile count
* @param members 1..PN_TT_ELTWISE_SPLIT_MAX_MEMBERS
* @param member 0..members-1
* @param tile_bytes the operand's page size, for the byte offset
* @param out_slice receives the range; untouched on a refusal
* @return true, or false when an argument is out of range or the split is ragged
*
* Thread-safe: yes Reentrant: yes Signal-safe: yes
*/
bool pn_tt_eltwise_split_slice(uint32_t tile_count, uint32_t members, uint32_t member,
uint32_t tile_bytes, pn_tt_eltwise_slice *out_slice);
/**
* PHASE W1a — the WIDE split: member `member`'s share of `tile_count` dealt out in whole
* BLOCKS of `block_tiles` (the residency's `2 * depth`), the first `blocks % members` members
* one block more. Ragged by blocks and never by tiles, so every slice stays on the CB grid the
* depth rule protects whatever the member count — which lets a prefill-wide descriptor
* (Mt x 384 tiles, Mt odd) spread over members the width does not divide.
*
* A member past the last block gets an EMPTY slice (tile_count 0) at the end of the
* descriptor; the caller skips it. The slices tile [0, tile_count) exactly.
*
* @return true, or false when an argument is out of range or `tile_count` is not whole blocks
*
* Thread-safe: yes Reentrant: yes Signal-safe: yes
*/
bool pn_tt_eltwise_split_slice_blocks(uint32_t tile_count, uint32_t members, uint32_t member,
uint32_t block_tiles, uint32_t tile_bytes,
pn_tt_eltwise_slice *out_slice);
#endif /* PN_TT_ELTWISE_SPLIT_H */

132
src/accel/tt/pn_tt_fp32_strip.c Normal file
View File

@ -0,0 +1,132 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_fp32_strip.c
* @brief Implementation of the Float32 activation strip. See pn_tt_fp32_strip.h for why it
* is a second type and not a flag on `pn_tt_activation`.
*
* @status None
* @issues None
* @todo None
*/
#ifdef USE_TT
#include "pn_tt_fp32_strip.h"
#include <stdio.h>
#include "pn_tt_device_internal.h"
#include "pn_tt_gdn_layout.h"
int pn_tt_fp32_row0_move(pn_tt_device *device, uint32_t bank, uint32_t address, uint32_t tiles,
float *values, bool reading)
{
return pn_tt_fp32_row_move(device, bank, address, 0u, tiles, values, reading);
}
int pn_tt_fp32_row_move(pn_tt_device *device, uint32_t bank, uint32_t address, uint32_t row,
uint32_t tiles, float *values, bool reading)
{
const ft_tt_noc_access *bulk;
uint16_t x;
uint16_t y;
uint32_t tile;
uint32_t chunk;
if (!device || !values || bank >= device->dram_bank_count ||
row >= PN_TT_GDN_LAYOUT_TILE_DIM)
return -1;
/* Chained projections are awaited HERE — pn_tt_device.h, pn_tt_matvec_enqueue. */
if (pn_tt_matvec_flush(device) != 0)
return -1;
bulk = &device->runtime.bulk_dram;
x = (uint16_t)device->dram[bank].x;
y = (uint16_t)device->dram[bank].y;
for (tile = 0; tile < tiles; tile++) {
for (chunk = 0; chunk < PN_TT_GDN_LAYOUT_ROW0_CHUNKS; chunk++) {
float *host = values + (size_t)tile * PN_TT_TILE_DIM +
(size_t)chunk * PN_TT_GDN_LAYOUT_ROW0_CHUNK_ELEMS;
const uint64_t at = (uint64_t)address + pn_tt_gdn_row_offset(tile, row, chunk);
const int rc =
reading ? bulk->read(bulk->user, x, y, at, host,
PN_TT_GDN_LAYOUT_ROW0_CHUNK_BYTES)
: bulk->write(bulk->user, x, y, at, host,
PN_TT_GDN_LAYOUT_ROW0_CHUNK_BYTES);
if (rc != FT_TT_OK)
return -1;
}
}
if (reading)
device->bytes_read +=
(size_t)tiles * PN_TT_GDN_LAYOUT_ROW0_CHUNKS * PN_TT_GDN_LAYOUT_ROW0_CHUNK_BYTES;
else
device->bytes_written +=
(size_t)tiles * PN_TT_GDN_LAYOUT_ROW0_CHUNKS * PN_TT_GDN_LAYOUT_ROW0_CHUNK_BYTES;
return 0;
}
int pn_tt_fp32_strip_alloc(pn_tt_device *device, uint32_t tile_count,
pn_tt_fp32_strip *out_strip)
{
const size_t bytes = (size_t)tile_count * PN_TT_TILE_FP32;
ft_tt_buffer buffer;
uint32_t bank;
if (!device || !out_strip || tile_count == 0u || device->dram_bank_count == 0u)
return -1;
bank = device->strip_bank_cursor++ % device->dram_bank_count;
if (ft_tt_buffer_alloc(&device->allocator, FT_TT_MEM_SPACE_DRAM, bank, bytes, &buffer) !=
FT_TT_OK) {
fprintf(stderr, "pn_tt: DRAM full allocating a %u-tile Float32 strip\n", tile_count);
return -1;
}
out_strip->bank = buffer.logical_bank;
out_strip->address = (uint32_t)buffer.address;
out_strip->tile_count = tile_count;
/* THE PRECONDITION every row-0 write depends on and none of them can check. */
if (pn_tt_zero_dram(device, out_strip->bank, out_strip->address, bytes) != 0) {
fprintf(stderr, "pn_tt: could not stage a %u-tile Float32 strip zeroed\n", tile_count);
return -1;
}
return 0;
}
int pn_tt_fp32_strip_upload(pn_tt_device *device, const pn_tt_fp32_strip *strip,
const float *values, uint32_t count)
{
if (!strip || !values || count == 0u || (count % PN_TT_TILE_DIM) != 0u)
return -1;
if (count / PN_TT_TILE_DIM > strip->tile_count)
return -1;
/* The mover never writes through `values` on the outbound leg; the cast is the price of
* one function serving both directions. */
return pn_tt_fp32_row0_move(device, strip->bank, strip->address,
count / PN_TT_TILE_DIM, (float *)values, false);
}
int pn_tt_fp32_strip_download(pn_tt_device *device, const pn_tt_fp32_strip *strip,
float *out_values, uint32_t count)
{
if (!strip || !out_values || count == 0u || (count % PN_TT_TILE_DIM) != 0u)
return -1;
if (count / PN_TT_TILE_DIM > strip->tile_count)
return -1;
return pn_tt_fp32_row0_move(device, strip->bank, strip->address,
count / PN_TT_TILE_DIM, out_values, true);
}
ft_tt_op_operand pn_tt_fp32_strip_operand(const pn_tt_device *device,
const pn_tt_fp32_strip *strip)
{
ft_tt_op_operand operand;
operand.x = (uint16_t)device->dram[strip->bank].x;
operand.y = (uint16_t)device->dram[strip->bank].y;
operand.address = strip->address;
operand.tile_bytes = PN_TT_TILE_FP32;
return operand;
}
#endif /* USE_TT */

116
src/accel/tt/pn_tt_fp32_strip.h Normal file
View File

@ -0,0 +1,116 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_fp32_strip.h
* @brief A `[1, 32*tile_count]` activation strip of **Float32** tiles — what a program
* placed at a 4096-byte page reads and writes, where `pn_tt_activation` is the
* 2048-byte bf16 one.
*
* ---------------------------------------------------------------------------
* WHY A SECOND STRIP TYPE RATHER THAN A FLAG ON THE FIRST
* ---------------------------------------------------------------------------
* `pn_tt_activation` is bf16 in three separate places — the allocation size, the row-0
* chunk geometry (`ft_tt_tile_row0.h` reads 64 bytes and discards 32) and the operand's
* `tile_bytes` — and EVERY op in the backend but one takes it. Adding a page size to it
* would put a runtime branch on the shipped decode path to serve one probe and one
* residency, and a strip whose format is a field is a strip that can be handed to the wrong
* program with no refusal anywhere. Two types cannot be confused by the compiler.
*
* **In fp32 logical row 0 is two EXACTLY 64-byte pieces**, which is the DRAM read
* granularity — the bf16 path has to read 64 and discard 32. So this strip is cheaper per
* element to move than the bf16 one, not more expensive.
*
* The address arithmetic is `pn_tt_gdn_row0_offset()`. It is named for the family that
* needed it first, but it is the generic fp32-tile row-0 layout and it is the one piece of
* this file that is separately proven (`tests/test_gdn_layout.c`).
*
* @status Used by the per-head reduce residency (`pn_tt_head_norm.c`) and by the GDN family,
* which delegates its own row-0 mover here.
* @issues None
* @todo None
*/
#ifndef PN_TT_FP32_STRIP_H
#define PN_TT_FP32_STRIP_H
#ifdef USE_TT
#include <stdbool.h>
#include <stdint.h>
#include "pn_tt_device.h"
#include "tt/ft_tt_op_dispatch.h"
/** One strip of consecutive Float32 tiles in card DRAM. */
typedef struct pn_tt_fp32_strip {
uint32_t bank; /**< logical DRAM bank */
uint32_t address; /**< byte offset within that bank */
uint32_t tile_count; /**< features / 32 */
} pn_tt_fp32_strip;
/**
* Carve a `tile_count`-tile Float32 strip and stage it ZEROED.
*
* The zeroing is the precondition every row-0 write depends on and none of them can check:
* only row 0 is ever written, so rows 1-31 have to start zero and stay zero for a whole-tile
* reduce over them to contribute nothing.
*
* @return 0, or -1 with the reason on stderr
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
int pn_tt_fp32_strip_alloc(pn_tt_device *device, uint32_t tile_count,
pn_tt_fp32_strip *out_strip);
/** Write `count` floats into logical row 0 of a strip, with no conversion at all. */
int pn_tt_fp32_strip_upload(pn_tt_device *device, const pn_tt_fp32_strip *strip,
const float *values, uint32_t count);
/** Read logical row 0 of a strip back as `count` floats. */
int pn_tt_fp32_strip_download(pn_tt_device *device, const pn_tt_fp32_strip *strip,
float *out_values, uint32_t count);
/** An operand descriptor for a Float32 strip: its endpoint and the 4096-byte page. */
ft_tt_op_operand pn_tt_fp32_strip_operand(const pn_tt_device *device,
const pn_tt_fp32_strip *strip);
/** The same for an ordinary bf16 activation strip: its endpoint and the 2048-byte page.
*
* It lives beside the fp32 one because the two are read TOGETHER now — the strip bridge and
* the per-head norms take raw operands so one body can serve both pages, and a caller
* outside `src/accel/tt/` needs to name both sides of the seam. */
ft_tt_op_operand pn_tt_operand(const pn_tt_device *device, const pn_tt_activation *activation);
/**
* Move logical row 0 of `tiles` consecutive fp32 tiles at a DRAM address, in either
* direction. The primitive both this file's upload/download and the GDN stager use.
*
* `values` is `tiles * 32` floats and the two pieces of each row are ADJACENT in it, so no
* host staging buffer is needed in either direction.
*
* @param reading true reads the card into `values`, false writes `values` to the card
* @return 0, or -1
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
int pn_tt_fp32_row0_move(pn_tt_device *device, uint32_t bank, uint32_t address, uint32_t tiles,
float *values, bool reading);
/**
* The same move for ANY logical row of the tiles. `pn_tt_fp32_row0_move()` is `row = 0`.
*
* Only the chunk phase-isolation arm uses a non-zero row: the decode GDN family works in row
* 0 and the chunkwise family works in rows 0..C-1 of the same buffers, so isolating one of
* the two chunk kernels means shuttling row `t` to row 0 and the answer back. Every move is
* an exact fp32 copy — the arm changes WHICH KERNEL runs and nothing about the arithmetic.
*
* @param row 0..31
* @return 0, or -1
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
int pn_tt_fp32_row_move(pn_tt_device *device, uint32_t bank, uint32_t address, uint32_t row,
uint32_t tiles, float *values, bool reading);
#endif /* USE_TT */
#endif /* PN_TT_FP32_STRIP_H */

271
src/accel/tt/pn_tt_gate.c Normal file
View File

@ -0,0 +1,271 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_gate.c
* @brief Implementation of the merged GDN output gate. See pn_tt_gate.h — the argument for why
* a MIXED CB table is correct HERE and wrong in `gdn_float32.cbtable` lives there.
*
* @status None
* @issues None
* @todo None
*/
#ifdef USE_TT
#include "pn_tt_gate.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "pn_tt_device_internal.h"
#include "pn_tt_replica_map.h"
/**
* Which enabled Tensix the merged gate drives.
*
* 87 is free in the map `pn_tt_render.c` documents: the render pair holds 85/86, the GDN
* residency 90, rope/memory/attention 82/83/84, and eight matmul members stop at 82. It sits
* inside the range sixteen matmul members would reach, which is not an oversight —
* `pn_tt_prepare_family()` refuses a collision BY NAME against every live residency, so a run
* that wants both is told rather than silently sharing a core.
*/
#define PN_TT_CORE_GATE_NARROW_ENV "PN_TT_CORE_GATE_NARROW"
#define PN_TT_CORE_GATE_NARROW_DEFAULT 87u
#define PN_TT_GATE_NARROW_PROGRAM "gate_narrow"
#define PN_TT_GATE_NARROW_NAME "gate-narrow"
/**
* Whether the merged gate runs. **DEFAULT OFF — it was flipped ON on 2026-08-25 and flipped
* back the same day, because it is right in DECODE and it BREAKS THE CHUNKWISE PREFILL.**
*
* WHAT IT IS WORTH, MEASURED. Seven decode arms, one model load each, `company_after = 0`
* throughout: `cmp` **IDENTICAL** (189 B, md5 80fc6259), **20,448 -> 19,296 descriptors a
* token** (exactly 24 a token over 48 tokens, and -25,165,824 B), on-card **0.114 -> 0.112
* s/token** against a control that read 0.114 TWICE, wall 5.17 / 5.20 control against 5.22.
* The predicted ~7.5 ms was 6x optimistic and the census says why: the merged `silu_mul` costs
* **267.9 us a descriptor against the plain one's ~170** (2.097 us/tile against 0.998), because
* the fp32 unpack of a 4,096-byte operand is real work the bf16 unpack it replaces was not
* doing. Removing `render_narrow` is worth 0.0026 s/token and the heavier multiply gives back
* ~0.0024. Red arm `PN_TT_RED=gate_narrow_swap` DIFFERS at 190 B; both collision refusals fire.
*
* ---------------------------------------------------------------------------
* WHY IT IS OFF: IT BREAKS `gdnc`, ON TWO ARMS, ABSENT FROM THE THIRD
* ---------------------------------------------------------------------------
* With the chunkwise prefill armed (`PN_TT_PREFILL=1 PN_TT_PREFILL_GDNC=1`) and this default
* ON, both a C = 32 arm and a C = 16 arm produce:
*
* pn_tt: gdnc descriptor 4 did not complete: error
* pn_tt: render-narrow descriptor 1 did not complete: error
* pn_tt: the gdnc residency refused a descriptor — retiring it and arming a fresh one on 95
*
* after which documents 1 and 2 **DECLINE the batched prefill and run per token**. The same
* two arms with `PN_TT_GATE_NARROW=0` are clean, and one of them came back BYTE-IDENTICAL to a
* dump recorded before this residency existed — so the variable is this default and nothing
* else in the session. **The two-pass gate's `--require-prefill` refused the dump rather than
* scoring a sequential arm under a chunkwise label; that guard is the only reason this was not
* 339 records of the wrong thing.**
*
* NOT DIAGNOSED, AND DO NOT ASSUME THE CORE MAP. `gate_narrow` sits on 87 and `gdnc` on 95, so
* it is not a core collision, and `pn_tt_core_holder()` refuses those by name anyway. The open
* suspicion is arming ORDER: this residency is armed lazily, and the one thing already on
* record about lazy arming in this family is that **a residency armed after
* `pn_tt_publish_replica_map()` reads whatever was at that fixed L1 address** — the note beside
* the render pair says both of those still have that gap, and `render_narrow` is the other
* family erroring here. That is a hypothesis with an obvious first arm and no measurement yet.
*/
static bool gate_narrow_requested(void)
{
const char *want = getenv(PN_TT_GATE_NARROW_ENV);
return want && *want && atoi(want) != 0;
}
int pn_tt_device_arm_gate_narrow(pn_tt_device *device, uint32_t tiles)
{
pn_tt_gate_residency *slot;
ft_tt_op_eltwise_request request;
ft_tt_launch_cb_entry placed[FT_TT_OP_ELTWISE_CB_COUNT];
/* The depth search must not write into the residency's own CB buffer: `prepare_family` has
* not run yet, and it zeroes the whole handle. */
unsigned char scratch_cb[sizeof(device->gate.residency.cb_buffer)];
uint64_t scratch_mask;
uint32_t depth;
if (!device)
return -1;
if (tiles == 0u || device->gate.residency.running)
return 0;
if (!gate_narrow_requested())
return 0;
if (!pn_tt_program_is_present(PN_TT_GATE_NARROW_PROGRAM)) {
fprintf(stderr, "pn_tt: no `%s` program — the GDN gate keeps the render_narrow + "
"silu_mul pair\n", PN_TT_GATE_NARROW_PROGRAM);
return 0;
}
/* THE DEEPEST BLOCK THAT BOTH DIVIDES THE RUN AND FITS THE ARENA. The B stream is 4096 B
* a page here while A, the scratch and the output are 2048, so the arena question is not
* the shipped elementwise one and `place_cbs` is the only thing that can answer it.
* Walking down from the divisor rather than refusing is what stops a page size from
* silently costing a whole residency. */
for (depth = pn_tt_common_depth(&tiles, 1u, PN_TT_ONCARD_MAX_DEPTH); depth > 0u;
depth = pn_tt_common_depth(&tiles, 1u, depth - 1u)) {
memset(&request, 0, sizeof(request));
request.function = FT_TT_OP_VECTOR_FN_SILU_MUL;
request.tile_count = tiles;
request.prefetch_tiles = depth;
request.input_a.tile_bytes = PN_TT_TILE_BF16;
request.input_b.tile_bytes = PN_TT_TILE_FP32;
request.output.tile_bytes = PN_TT_TILE_BF16;
if (ft_tt_op_eltwise_place_cbs(&request, placed, scratch_cb, sizeof(scratch_cb),
&scratch_mask) == FT_TT_OK)
break;
}
if (depth == 0u) {
fprintf(stderr, "pn_tt: no block depth both divides a %u-tile gate and fits the L1 "
"arena at %u B / %u B input pages\n", tiles, PN_TT_TILE_BF16,
PN_TT_TILE_FP32);
return 0;
}
slot = &device->gate;
if (!pn_tt_prepare_family(device, &slot->residency, PN_TT_GATE_NARROW_PROGRAM,
PN_TT_GATE_NARROW_NAME, PN_TT_CORE_GATE_NARROW_ENV,
PN_TT_CORE_GATE_NARROW_DEFAULT, PN_TT_ONCARD_TRIAD_MASK))
return 0;
memset(&request, 0, sizeof(request));
request.function = FT_TT_OP_VECTOR_FN_SILU_MUL;
request.tile_count = tiles;
request.prefetch_tiles = depth;
request.input_a.tile_bytes = PN_TT_TILE_BF16;
request.input_b.tile_bytes = PN_TT_TILE_FP32;
request.output.tile_bytes = PN_TT_TILE_BF16;
if (ft_tt_op_eltwise_place_cbs(&request, placed, slot->residency.cb_buffer,
sizeof(slot->residency.cb_buffer),
&slot->residency.cb_mask) != FT_TT_OK) {
fprintf(stderr, "pn_tt: the gate-narrow circular buffers do not fit at %u tiles, "
"depth %u\n", tiles, depth);
pn_tt_residency_release(&slot->residency);
return 0;
}
if (!pn_tt_start_family(&slot->residency))
return 0;
/* THE TABLE THIS CORE READS MUST BE A WRITTEN FACT. This residency runs
* `resident_vector_writer.c` and it arms AFTER `pn_tt_publish_replica_map()` has already
* gone out, so without this line it reads whatever is at the fixed L1 replica-table
* address — which is not "no mirroring" but "whatever a previous process left there", the
* state that cost the elementwise split its `cmp`. `ssm_gated` is not a mirrored strip, so
* EMPTY is the correct table and not a stub. */
if (pn_tt_publish_empty_replica_map(device, &slot->residency, "the gate-narrow residency")
!= 0) {
pn_tt_residency_stop(&slot->residency);
pn_tt_residency_release(&slot->residency);
return 0;
}
slot->tiles = tiles;
slot->depth = depth;
fprintf(stderr, "pn_tt: gate-narrow is up on core %u — %u tiles, silu(bf16) * fp32 -> "
"bf16, depth %u. render_narrow leaves the decode token.\n",
slot->residency.core_index, tiles, depth);
return 0;
}
bool pn_tt_gate_narrow_is_armed(const pn_tt_device *device)
{
return device && device->gate.residency.running;
}
int pn_tt_gate_narrow(pn_tt_device *device, ft_tt_op_operand a_bf16, ft_tt_op_operand b_fp32,
ft_tt_op_operand out_bf16, uint32_t tiles)
{
pn_tt_gate_residency *slot;
ft_tt_op_eltwise_request request;
uint32_t words[FT_TT_OP_VECTOR_WORDS];
double started;
if (!device)
return -1;
slot = &device->gate;
if (!slot->residency.running)
return -1;
/* Chained projections are awaited HERE — pn_tt_device.h, pn_tt_matvec_enqueue. A gate whose
* `z` a queued matvec is still writing reads the PREVIOUS token's activation and nothing
* anywhere refuses it. */
if (pn_tt_matvec_flush(device) != 0)
return -1;
/* THE PAGE SIZE IS A THREE-SIDED CONTRACT WITH NO CARD-SIDE CHECK: the DRAM operand, the L1
* circular buffer and the reader's compile-time TILE_BYTES / TILE_BYTES_B all have to
* agree, and the device can see none of it. This is the only place that can. */
if (a_bf16.tile_bytes != PN_TT_TILE_BF16 || b_fp32.tile_bytes != PN_TT_TILE_FP32 ||
out_bf16.tile_bytes != PN_TT_TILE_BF16) {
fprintf(stderr, "pn_tt: gate-narrow wants %u/%u -> %u B operands and was handed "
"%u/%u -> %u\n", PN_TT_TILE_BF16, PN_TT_TILE_FP32, PN_TT_TILE_BF16,
a_bf16.tile_bytes, b_fp32.tile_bytes, out_bf16.tile_bytes);
return -1;
}
if (tiles == 0u || tiles > slot->tiles ||
ft_tt_op_block_depth(tiles, slot->depth) != slot->depth ||
(tiles % (2u * slot->depth)) != 0u) {
fprintf(stderr, "pn_tt: %u tiles is not a whole number of depth-%u blocks the "
"gate-narrow residency was placed for (%u)\n", tiles, slot->depth,
slot->tiles);
return -1;
}
memset(&request, 0, sizeof(request));
/* RED PROOF: the two operands of the gate swap. `silu(y) * z` is not `silu(z) * y`, both
* strips are the same width, and no shape check anywhere can see the difference — the same
* mutation `pn_tt_silu_mul()` carries, on the descriptor that replaces it. */
if (pn_tt_red("gate_narrow_swap")) {
request.function = FT_TT_OP_VECTOR_FN_SILU_MUL;
request.input_a = b_fp32;
request.input_b = a_bf16;
} else {
request.function = FT_TT_OP_VECTOR_FN_SILU_MUL;
request.input_a = a_bf16;
request.input_b = b_fp32;
}
request.tile_count = tiles;
request.prefetch_tiles = slot->depth;
request.output = out_bf16;
if (ft_tt_op_eltwise_parameters(&request, words) != FT_TT_OK)
return -1;
pn_tt_vec_probe_apply(words);
started = pn_tt_now_seconds();
if (pn_tt_residency_submit(&slot->residency, FT_TT_QUEUE_OP_ELTWISE, words,
PN_TT_OP_TIMEOUT_MS) != 0) {
/* The words address DRAM only, so the same descriptor is valid for a fresh residency —
* the re-arm argument every other family here makes. */
fprintf(stderr, "pn_tt: the gate-narrow residency refused a descriptor — retiring it "
"by ASKING and arming a fresh one\n");
pn_tt_residency_stop(&slot->residency);
pn_tt_residency_release(&slot->residency);
device->oncard_restarts++;
if (!pn_tt_prepare_family(device, &slot->residency, PN_TT_GATE_NARROW_PROGRAM,
PN_TT_GATE_NARROW_NAME, PN_TT_CORE_GATE_NARROW_ENV,
PN_TT_CORE_GATE_NARROW_DEFAULT, PN_TT_ONCARD_TRIAD_MASK) ||
!pn_tt_start_family(&slot->residency) ||
pn_tt_residency_submit(&slot->residency, FT_TT_QUEUE_OP_ELTWISE, words,
PN_TT_OP_TIMEOUT_MS) != 0)
return -1;
}
pn_tt_oncard_account(device, PN_TT_GATE_NARROW_NAME, "silu_mul", tiles,
pn_tt_now_seconds() - started);
return 0;
}
void pn_tt_gate_release(pn_tt_device *device)
{
if (!device)
return;
if (device->gate.residency.running) {
pn_tt_residency_stop(&device->gate.residency);
pn_tt_residency_release(&device->gate.residency);
}
memset(&device->gate, 0, sizeof(device->gate));
}
#endif /* USE_TT */

118
src/accel/tt/pn_tt_gate.h Normal file
View File

@ -0,0 +1,118 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_gate.h
* @brief THE GDN OUTPUT GATE WITH THE fp32->bf16 SEAM FOLDED INTO IT — one descriptor that
* replaces `render_narrow` + `silu_mul`, reading the fp32 `y_norm` strip DIRECTLY.
*
* ---------------------------------------------------------------------------
* WHAT IT DELETES, AND WHY THAT IS FREE
* ---------------------------------------------------------------------------
* Today the GDN block's last two steps are:
*
* render_narrow(y_norm_fp32) -> y_bridge_bf16 24 descriptors/token
* silu_mul(z_bf16, y_bridge_bf16) -> gated_bf16 24 descriptors/token
*
* The first exists ONLY to change format. It reads 128 fp32 tiles, reads 128 MORE fp32 tiles
* of ZEROS (the family has no identity, so a copy is `x + 0`), and writes 128 bf16 tiles —
* 1,310,720 B a layer, 31,457,280 B a token, to move no information at all.
*
* `gate_narrow` is the SECOND descriptor with a MIXED CB table: cb 1 is Float32 while cb 0,
* cb 2 and cb 16 stay Float16_b. `conditional_destination_format()` then gives cb 1
* `unpack_src Float32 / unpack_dst Float16_b / tile_size 4096` — verified off-hardware to be
* the SAME descriptor triple `render_narrow.cbtable` gives its own fp32 input. So the
* unpacker performs the identical truncating fp32->bf16 conversion, on the identical bits, one
* descriptor and one DRAM round trip earlier than the multiply that consumes it. **The
* arithmetic is the same and the operand the multiplier sees is the same bits.** That is why
* this is expected to be `cmp` IDENTICAL and why anything else is a finding.
*
* The one thing that had to change is `resident_eltwise_reader.c`, which carried ONE
* `TILE_BYTES` for both input streams. It now takes a fifth compile-time arg for B, refuses a
* zero on a binary descriptor rather than reading 0 bytes a tile, and every pre-existing
* recipe passes its own A size — so `eltwise`, `render_widen` and `render_narrow` rebuild
* byte-identical, which is the control that says the new argument is inert when unused.
*
* ---------------------------------------------------------------------------
* WHY NOT ON THE SHARED ELEMENTWISE RESIDENCY
* ---------------------------------------------------------------------------
* A residency runs ONE program, and a program's CB formats are baked into its compute images.
* The shipped `eltwise` residency serves the residual ADDs, the SwiGLU and attention's
* SIGMOID+MUL, all of which are bf16 on both inputs; giving it a Float32 cb 1 would make every
* one of them read a bf16 strip as fp32. So the gate gets its own core, and the descriptor
* COUNT does not change by moving it — 24 gate descriptors were already being issued, they
* are simply issued to a different residency. What goes is the 24 renders.
*
* Its width is Wt=128, which is BELOW the measured elementwise split break-even (~143 tiles at
* the first doubling; a 128-tile MUL measured 101.7 -> 146.9 us, WORSE, at two members). So a
* solo residency is not a compromise here, it is the right side of that knee.
*
* ---------------------------------------------------------------------------
* WHY NOT THE GDN WRITER — REFUTED, DO NOT RETRY
* ---------------------------------------------------------------------------
* `gdn_float32.cbtable`'s cb 16 carries the UPDATED RECURRENT STATE as well as `y`, interleaved
* per tile-row. One CB, one page size, two sinks: narrowing it to bf16 narrows the state the
* next token reads. The seam cannot move upstream of the gated RMS norm.
*
* @status DEFAULT OFF. Flipped ON 2026-08-25 and flipped BACK the same day. In DECODE it is
* right and cheap — `cmp` IDENTICAL, 20,448 -> 19,296 descriptors a token, -25,165,824 B,
* on-card 0.114 -> 0.112 s/token against a control that read 0.114 twice. With the CHUNKWISE
* PREFILL armed it puts `gdnc` and `render-narrow` into `did not complete: error` and the
* prefill silently declines to the per-token path. See pn_tt_gate.c for the two arms that
* show it and the one that does not.
* @issues Breaks `gdnc` when the chunkwise prefill is armed. Undiagnosed; not a core collision.
* @issues None
* @todo None
*/
#ifndef PN_TT_GATE_H
#define PN_TT_GATE_H
#ifdef USE_TT
#include <stdbool.h>
#include <stdint.h>
#include "pn_tt_device.h"
#include "tt/ft_tt_op_dispatch.h"
/** The flag. Default OFF; `1` arms the merged gate and takes `render_narrow` out of decode.
* DO NOT flip the default again without a chunkwise-prefill arm beside the decode one. */
#define PN_TT_GATE_NARROW_ENV "PN_TT_GATE_NARROW"
/**
* Arm the merged gate for the widest `gated` strip it will be asked for.
*
* Never fatal: without the `gate_narrow` program, or with the flag unset, the caller keeps the
* render + silu_mul pair it had. The core is `PN_TT_CORE_GATE_NARROW` (87 by default — free
* between the render pair at 85/86 and the GDN residency at 90) and is refused BY NAME against
* any live family, exactly like every other residency here.
*
* @param tiles the widest bf16 output strip, in tiles; 0 skips
* @return 0 (even when nothing armed), -1 on a bad argument
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
int pn_tt_device_arm_gate_narrow(pn_tt_device *device, uint32_t tiles);
/** Is the merged gate live? A caller that gets `false` must use render_narrow + silu_mul. */
bool pn_tt_gate_narrow_is_armed(const pn_tt_device *device);
/**
* `out = silu(a) * b`, with `a` and `out` bf16 tiles and `b` FP32 tiles.
*
* The operands are raw for the same reason `pn_tt_render`'s are: the three buffers this joins
* are three different C types differing only in page size. Each side's page size is checked
* here, because a 2048-byte operand under a 4096-byte stream is the failure mode with no
* card-side refusal anywhere (measured on the reduce family: constants built at 2048 under a
* 4096 program score 1.0e+00 with nothing refusing).
*
* @return 0, or -1 with the reason on stderr
*/
int pn_tt_gate_narrow(pn_tt_device *device, ft_tt_op_operand a_bf16, ft_tt_op_operand b_fp32,
ft_tt_op_operand out_bf16, uint32_t tiles);
/** Release the residency. */
void pn_tt_gate_release(pn_tt_device *device);
#endif /* USE_TT */
#endif /* PN_TT_GATE_H */

2703
src/accel/tt/pn_tt_gdn.c Normal file
View File

File diff suppressed because it is too large Load Diff

319
src/accel/tt/pn_tt_gdn.h Normal file
View File

@ -0,0 +1,319 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_gdn.h
* @brief The GATED-DELTA-NET block on the card: one resident core running
* `foundry/tt/ft_tt_op_gdn.h`'s two phases over per-layer state that lives in card
* DRAM and is never downloaded.
*
* Model-agnostic on purpose. It knows S, the head counts, the conv width and the tap
* count; it does not know what a Qwen3.5 layer is. `src/models/qwen35moe_gdn_tt.c`
* is the caller and owns the host arithmetic this op deliberately does not do.
*
* ---------------------------------------------------------------------------
* WHAT THIS OWNS AND WHAT IT DOES NOT
* ---------------------------------------------------------------------------
* The op is TWO descriptors a layer — `conv1d + SiLU` with the rolling window, then the
* delta rule over every v-head — and between them sit two per-head NORMS that are NOT in it:
* the q/k L2 norm, and the output gated norm. Those are ordinary reduces at a SECOND width
* (`S/32` = 4 tile columns against the model's `n_embd/32` = 128) and they live on the narrow
* `reduce_fp32` residency `pn_tt_head_norm.h` places.
*
* **THERE ARE TWO CALLERS AND THEY ARE NOT THE SAME ARM.**
*
* HOST-MEDIATED `pn_tt_gdn_conv()` / `pn_tt_gdn_delta()` take and return host float
* arrays, so the two norms between them run on the CPU. That arm is SLOWER
* than the host recurrence and it is arithmetic, not a defect: the host
* still pulls the 49,408 B of projections AND now pulls `conv_out` (32,768 B)
* and `y` (16,384 B) a layer. It is the CORRECTNESS vehicle — the thing the
* on-card gate scores against the CPU oracle — and nothing else.
* THE STRIP `pn_tt_gdn_conv_oncard()` / `pn_tt_gdn_delta_oncard()` take nothing and
* BRIDGE return nothing: every operand is already in card DRAM, the norms run on
* the card between them, and the only bytes that cross are `alpha`/`beta`
* down and 256 B of gate block up. `qwen35moe_gdn_strip_tt.c` is the caller
* and readback goes 1478.09 -> 326.09 KiB/token.
*
* ---------------------------------------------------------------------------
* WHAT LIVES ON THE CARD, PER SSM LAYER, AND WHAT IT COSTS
* ---------------------------------------------------------------------------
* Every size comes from `ft_tt_op_gdn_*_bytes()` so the capacity figure and the allocation
* are one expression. At Ornith's shape (S=128, 32 v-heads, 16 k-heads, conv_dim 8192,
* 4 taps):
*
* the recurrent state 2 MiB [v_heads, S, S] fp32, oracle-transposed, READ AND WRITTEN
* the conv window 3 MiB taps-1 TAP-MAJOR PLANES of conv_dim/32 tiles, R/W
* the conv weight 4 MiB taps planes, a CONSTANT staged once, TRANSPOSED at load
* -----
* 9 MiB a layer, 216 MiB over 24 SSM layers
*
* The 50.25 MiB the capacity plan reserves counted the window as `conv_dim * (taps-1)` fp32
* ELEMENTS. A plane is a one-row tilized strip and the tile quantum is 32 rows, so the window
* is 32x that; carried state alone is 120 MiB. See `ft_tt_op_gdn.h`.
*
* Plus ONE shared set of per-token staging buffers — the qkv stream in, the stream out (which
* is CONV's output AND DELTA's input, one buffer since both are the same row-0 strip), the
* gate block and `y` — 2.6 MiB total, because a token runs one layer at a time and nothing
* carries between them.
*
* @status SHIPS ON. On silicon (p150a, 2026-08-21) the strip-bridge arm is `cmp` IDENTICAL
* to the 48-token oracle at **326.09 KiB/token and 3.79 tok/s**, against the control's
* 1478.09 and 2.20; the host-mediated arm beside it is IDENTICAL too, at 2630.09 and
* 1.43. `PN_TT_ONCARD=all` selects the family now — but the block WITHOUT the bridge
* is a wall regression, so a DEFAULT arm that loses the bridge gives the recurrence
* back to the host and a NAMED one keeps it.
* @issues **EVERY NUMBER MEASURED ON A CONTESTED CORE IS VOID.** Core 75 is a matmul group
* member at `PN_TT_MATMUL_CORES=4` and members run upward from it, so the contested
* set MOVES with the count. While two residencies were retiring each other the same
* CONV descriptor scored 1.0e-2 max-norm with 1957 of 8192 channels off; on an
* uncontested core it is 6.872e-04 with 3. A contested core does not fail, it
* computes badly, and both arms log a healthy "residency up".
*
* The carried state is reset when the forward is asked for position 0. A caller that
* starts mid-sequence from a prefix cache would inherit the previous sequence's
* state; the host arm has the same shape of problem and solves it by owning the
* state per sequence, which this cannot while there is one region set per model.
* @todo
* - [ ] An all-Float32 elementwise program, so the strip bridge's `silu(z) *` runs in fp32
* and only the final `gated` is narrowed. Modelled at 1.909e-03 against 1.023e-02.
*/
#ifndef PN_TT_GDN_H
#define PN_TT_GDN_H
#ifdef USE_TT
#include <stdint.h>
#include "pn_tt_device.h"
#include "tt/ft_tt_op_dispatch.h"
/** The shape one model's GDN block has. Every field is read off the model's config, never
* assumed — `k_heads` must divide `v_heads` because the GQA repeat is `vh % k_heads`. */
typedef struct pn_tt_gdn_shape {
uint32_t state_size; /**< S; head_k_dim == head_v_dim, a multiple of 32 */
uint32_t v_heads;
uint32_t k_heads;
uint32_t conv_dim; /**< 2*key_dim + val_dim, a multiple of 32 */
uint32_t conv_taps; /**< ssm_conv; the window is taps-1 deep */
uint32_t layer_count; /**< SSM layers, each with its own carried state */
} pn_tt_gdn_shape;
/**
* Arm the GDN residency (one more core) and allocate every layer's carried state.
*
* Never fatal on its own — a card without the `gdn` program leaves the recurrence on the
* host — but `PN_TT_ONCARD=gdn` makes a MISSING program a refusal, exactly like the other
* families (pn_tt_device_arm_oncard).
*
* The conv WEIGHT is allocated here and left uninitialised: it is a per-layer constant and
* the caller uploads it with `pn_tt_gdn_conv_weight_upload()` once, at load.
*
* @return 0 (even when nothing armed), -1 on a bad argument
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
int pn_tt_device_arm_gdn(pn_tt_device *device, const pn_tt_gdn_shape *shape);
/** Did the GDN family actually arm for this run? */
bool pn_tt_gdn_is_armed(const pn_tt_device *device);
/**
* TRANSPOSE one layer's conv1d weight into tap-major planes and upload it. Once, at load.
*
* `weight` is the oracle's own `layer->ssm_conv1d`: ROW-MAJOR `[conv_dim, taps]`, i.e.
* `w[c*taps + t]`. The card wants `taps` PLANES, plane `t` holding `w[.][t]` laid out exactly
* like the qkv stream. **That is a transpose and not a reshape** — a reshape gives a wrong
* answer with no error anywhere, which is why this is a named function taking the oracle's
* order rather than a memcpy at the call site.
*
* @param layer 0..layer_count-1, the SSM-layer ordinal the two calls below also take
* @param weight `conv_dim * taps` floats in the oracle's order
* @return 0, or -1 with the reason on stderr
*/
int pn_tt_gdn_conv_weight_upload(pn_tt_device *device, uint32_t layer, const float *weight);
/** Zero every layer's carried state — the conv window and the recurrent state. The start of
* a sequence, and the state this op is armed with. */
int pn_tt_gdn_state_reset(pn_tt_device *device);
/** STEP 4 S2 — the verify's per-row states, copy-free. Reserve `slots` snapshot slots (every
* layer's window + state, on the live region's bank); route the decode CONV/DELTA writes of
* the state to slot `slot` (-1: back in place); trade layer `layer`'s live window + state with
* slot `slot`'s. @return 0 or -1 */
int pn_tt_gdn_snapshot_reserve(pn_tt_device *device, uint32_t slots);
int pn_tt_gdn_state_out(pn_tt_device *device, int slot);
int pn_tt_gdn_layer_swap(pn_tt_device *device, uint32_t layer, uint32_t slot);
int pn_tt_gdn_layer_swap_part(pn_tt_device *device, uint32_t layer, uint32_t slot, bool window,
bool state);
/** S2c: the row select for the next CONV / DELTA descriptors (0 = the decode row-0 walk). */
int pn_tt_gdn_set_row(pn_tt_device *device, uint32_t row);
/** S2c: queue split CONV/DELTA descriptors without awaiting (per-row gate blocks); await them
* all with pn_tt_gdn_await_deferred() before anything reads their output. */
int pn_tt_gdn_defer(pn_tt_device *device, bool defer);
int pn_tt_gdn_await_deferred(pn_tt_device *device);
/** S2d: every row's gate block (decay[rows][H], beta[rows][H]) in ONE write, then DELTA
* descriptors that use the staged blocks (no per-row upload). */
int pn_tt_gdn_gate_upload_rows(pn_tt_device *device, uint32_t rows, const float *decay,
const float *beta);
int pn_tt_gdn_delta_descriptor_only(pn_tt_device *device, uint32_t layer);
/** MULTI-ROW DELTA: verify rows 0..rows-1 in one descriptor per member (see pn_tt_gdn.c). */
int pn_tt_gdn_delta_rows_oncard(pn_tt_device *device, uint32_t layer, uint32_t rows);
/** PM 6 MULTI-ROW CONV: the CONV of verify rows [0, rows) in ONE descriptor per member (strip
* mode): row r reads the window its one-row descriptor would have, writes its output to row r
* and its rolled window to snapshot slot r; then the window swaps in row order (live = after
* the last row, slot r = before row r). -2 when refused before anything ran (the caller may
* take the per-row walk), -1 on a card failure. */
int pn_tt_gdn_conv_rows_oncard(pn_tt_device *device, uint32_t layer, uint32_t rows);
/**
* The per-phase clean/replayed figures on stderr, ONCE, at release.
*
* NOT a duplicate of the `qwen35_tt: gdn phase` report line. That one runs under
* `tokens % PN_TT_REPORT == 0`, so an arm that CORRUPTS the model's output — which every
* data arm on this op does by construction — ends the run below the threshold and the
* instrument that would have scored the arm does not print at all. Measured 2026-08-27.
* Silent when the family ran nothing.
*/
void pn_tt_gdn_report_phases(const pn_tt_device *device);
/**
* PHASE CONV: `conv_out = SiLU(conv1d(qkv))`, with the window rolled ON THE CARD.
*
* @param qkv `conv_dim` floats, the fused projection
* @param conv_out `conv_dim` floats, written
* @return 0, or -1 with the reason on stderr
*/
int pn_tt_gdn_conv(pn_tt_device *device, uint32_t layer, const float *qkv, float *conv_out);
/**
* PHASE DELTA: the four-pass delta rule over every v-head, with the `[S,S]` state read,
* updated and written back ON THE CARD.
*
* **`q` and `k` are PER K-HEAD and must NOT be GQA-repeated.** The reader does the repeat as
* one modulo; staging the repeated 32-head form doubles them and is the 1.33x-worse row of
* the split-point table.
*
* @param q `k_heads * state_size` floats, L2-normalised, un-repeated
* @param k likewise
* @param v `v_heads * state_size` floats
* @param decay `v_heads` floats — `exp(g_h)`, already through the host `softplus` and `exp`
* because there is no SFPU `log`
* @param beta `v_heads` floats, already through the sigmoid
* @param y `v_heads * state_size` floats, written
* @return 0, or -1 with the reason on stderr
*/
int pn_tt_gdn_delta(pn_tt_device *device, uint32_t layer, const float *q, const float *k,
const float *v, const float *decay, const float *beta, float *y);
/* ============================================================================
* THE ON-CARD CHAIN — the same two descriptors with no host array anywhere
* ============================================================================
* The five calls below are the whole of A2. The two phases become
*
* widen(ssm_qkv bf16) -> gdn_stream_in
* pn_tt_gdn_conv_oncard() -> the strip at pn_tt_gdn_stream_operand()
* an L2 head norm over its first pn_tt_gdn_key_tiles() tiles, IN PLACE
* pn_tt_gdn_delta_oncard() -> the strip at pn_tt_gdn_y_operand()
* an RMS head norm, narrow(), then silu(z) * on the elementwise residency
*
* and the only bytes that cross the link for the block are `alpha`/`beta` down and the
* 256-byte gate block up. The activations do not: readback 1478.09 -> 326.09 KiB/token.
*
* The L2 norm runs IN PLACE over the conv output because PHASE CONV's output buffer IS PHASE
* DELTA's input buffer — the two are the same 256-tile row-0 strip and q, k and v are already
* where `ftg_delta_stream_base()` reads them. In place is safe here for a pipeline reason and
* not a hopeful one: the reduce reader prefetches ONE row block, so it has read block `b`
* before the writer can have written block `b`. There is a discriminating arm for it.
* ============================================================================ */
/** PHASE CONV over whatever is in the stream-in buffer, writing the stream buffer. */
int pn_tt_gdn_conv_oncard(pn_tt_device *device, uint32_t layer);
/** PHASE DELTA over whatever is in the stream buffer, writing `y`. Uploads the 256-byte gate
* block and nothing else. */
int pn_tt_gdn_delta_oncard(pn_tt_device *device, uint32_t layer, const float *decay,
const float *beta);
/** PM 7 ITEM 6: where the next DELTA reads its gate — `rows` rows (1 = the single gate, or a
* deferred row's block) — so the card can write it there (gdngate FLAT). @return 0 or -1. */
int pn_tt_gdn_gate_target(pn_tt_device *device, uint32_t rows, uint32_t *bank,
uint32_t *address, uint32_t *row_stride);
/** `decay[v_heads]` then `beta[v_heads]`, flat fp32, into the gate buffer. Public because the
* on-card path uploads it without staging anything else. */
int pn_tt_gdn_gate_upload(pn_tt_device *device, const float *decay, const float *beta);
/** PHASE CONV's INPUT — where the widened `ssm_qkv` strip has to land. */
ft_tt_op_operand pn_tt_gdn_stream_in_operand(const pn_tt_device *device);
/** PHASE CONV's OUTPUT and PHASE DELTA's INPUT, which are ONE buffer. */
ft_tt_op_operand pn_tt_gdn_stream_operand(const pn_tt_device *device);
/** PHASE DELTA's OUTPUT. */
ft_tt_op_operand pn_tt_gdn_y_operand(const pn_tt_device *device);
/**
* Read logical row 0 of `tiles` tiles of the stream buffer, starting at `first_tile`.
*
* A PROBE, and the only reader of it: on the shipped on-card path nothing downloads this
* buffer at all. It exists so the in-place q/k L2 norm can be scored against its own preimage
* rather than asserted — see `qwen35moe_gdn_strip_tt.c`.
*
* @param out `tiles * 32` floats
* @return 0, or -1
*/
int pn_tt_gdn_stream_read(pn_tt_device *device, uint32_t first_tile, uint32_t tiles,
float *out);
/* ============================================================================
* THE PHASE-ISOLATION SEAM — for attributing a chunkwise divergence, and nothing else
* ============================================================================
* The chunkwise family (`pn_tt_gdnc.h`) writes rows 0..C-1 of these same three buffers; the
* decode family writes row 0 and ZERO ROWS 1..31, because it computes whole tiles over an
* input whose rows 1..31 are padding. So a CHUNK descriptor's answer cannot simply be handed
* to the next DECODE descriptor and back: each decode descriptor would wipe the other rows.
*
* The two calls below are what lets one phase run per position over a chunk's rows while the
* other stays chunkwise: shuttle row `t` to row 0, run the decode descriptor, take row 0
* away again, and restore the decode family's zero-rows invariant in between. Every move is
* an exact fp32 copy over the link, so a hybrid arm's divergence from the sequential path is
* attributable to the ONE chunk kernel it still runs.
*
* SLOW ON PURPOSE — this is a quality instrument, not a speed path, and it is only ever
* reached from `PN_TT_PREFILL_GDNC_PHASE`.
* ============================================================================ */
/** Which of the family's three row-major fp32 strips a row move addresses. */
typedef enum {
PN_TT_GDN_STRIP_STREAM_IN = 0, /**< PHASE CONV's input */
PN_TT_GDN_STRIP_STREAM_OUT, /**< PHASE CONV's output, PHASE DELTA's input */
PN_TT_GDN_STRIP_Y /**< PHASE DELTA's output */
} pn_tt_gdn_strip_id;
/**
* Move logical row `row` of the first `tiles` tiles of one strip, in either direction.
*
* @param values `tiles * 32` floats
* @return 0, or -1 on an unarmed family, a bad strip or a row past 31
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
int pn_tt_gdn_strip_row_move(pn_tt_device *device, pn_tt_gdn_strip_id strip, uint32_t row,
uint32_t tiles, float *values, bool reading);
/** Zero one whole strip — the decode family's precondition, which a chunk descriptor's
* whole-tile write is exactly what breaks. */
int pn_tt_gdn_strip_zero(pn_tt_device *device, pn_tt_gdn_strip_id strip);
/** `conv_dim/32` — the whole stream. */
uint32_t pn_tt_gdn_stream_tiles(const pn_tt_device *device);
/** `2 * k_heads * state_size/32` — q then k, the prefix the L2 norm covers. */
uint32_t pn_tt_gdn_key_tiles(const pn_tt_device *device);
/** `v_heads * state_size/32` — v, and the same count as `y`. */
uint32_t pn_tt_gdn_value_tiles(const pn_tt_device *device);
#endif /* USE_TT */
#endif /* PN_TT_GDN_H */

177
src/accel/tt/pn_tt_gdn_checkpoint.c Normal file
View File

@ -0,0 +1,177 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_gdn_checkpoint.c
* @brief The multi-turn checkpoint of the carried GDN state. See pn_tt_gdn_checkpoint.h.
*
* @status Per-layer capture / restore through the puller's COPY; sampled verify.
* @issues None
* @todo None
*/
#ifdef USE_TT
#include "pn_tt_gdn_checkpoint.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "pn_tt_device_internal.h"
#include "pn_tt_dispatch.h"
#include "pn_tt_gdn.h"
#include "../../core/bench.h"
/** The bytes each end of a verified buffer is compared over. */
#define CKPT_VERIFY_BYTES (64u * 1024u)
/** RED `ckpt_copy_short`: the last 4 KiB of every buffer is left. */
#define CKPT_RED_SHORT_BYTES 4096u
static int ckpt_reserve(pn_tt_device *device)
{
const uint32_t layers = device->gdn_shape.layer_count;
pn_tt_gdn_region *set;
uint32_t layer;
if (device->gdn_checkpoint)
return 0;
set = calloc(layers, sizeof(*set));
if (!set)
return -1;
for (layer = 0; layer < layers; layer++)
if (pn_tt_gdn_alloc_twin(device, layer, &set[layer]) != 0) {
/* The DRAM already taken stays with the allocator until close; only the host
* array is dropped, and a later capture tries again. */
free(set);
return -1;
}
device->gdn_checkpoint = set;
return 0;
}
/** Copy one buffer card side. Both ends are exactly `bytes` long and 64 B aligned. */
static int ckpt_copy(pn_tt_device *device, const pn_tt_gdn_buffer *from,
const pn_tt_gdn_buffer *to, size_t bytes)
{
if (pn_tt_red("ckpt_copy_short") && bytes > CKPT_RED_SHORT_BYTES)
bytes -= CKPT_RED_SHORT_BYTES;
return pn_tt_copy_dram_oncard(device, from->bank, from->address, to->bank, to->address,
bytes);
}
/** Copy one layer's window + state, `to_checkpoint` choosing the direction. */
static int ckpt_copy_layer(pn_tt_device *device, uint32_t layer, bool to_checkpoint)
{
const pn_tt_gdn_region *live = &device->gdn_region[layer];
const pn_tt_gdn_region *saved = &device->gdn_checkpoint[layer];
const pn_tt_gdn_region *from = to_checkpoint ? live : saved;
const pn_tt_gdn_region *to = to_checkpoint ? saved : live;
size_t window, state;
pn_tt_gdn_carried_bytes(device, &window, &state);
if ((window & 63u) != 0u || (state & 63u) != 0u) {
fprintf(stderr, "pn_tt: gdn checkpoint — the carried buffers (%zu / %zu B) are not "
"64 B multiples; the on-card copy cannot move them\n", window, state);
return -1;
}
if (ckpt_copy(device, &from->window, &to->window, window) != 0 ||
ckpt_copy(device, &from->state, &to->state, state) != 0)
return -1;
return 0;
}
int pn_tt_gdn_checkpoint_capture_layer(pn_tt_device *device, uint32_t layer)
{
if (!pn_tt_gdn_is_armed(device) || layer >= device->gdn_shape.layer_count ||
ckpt_reserve(device) != 0)
return -1;
if (ckpt_copy_layer(device, layer, true) != 0) {
fprintf(stderr, "pn_tt: gdn checkpoint — the capture copy of layer %u failed\n", layer);
return -1;
}
return 0;
}
int pn_tt_gdn_checkpoint_restore(pn_tt_device *device)
{
const double started = pn_time_ms();
uint32_t layer;
if (!pn_tt_gdn_is_armed(device) || !device->gdn_checkpoint)
return -1;
/* Positions move backwards: the on-card dispatch model re-learns, as at a sequence start. */
pn_tt_dispatch_sequence_reset();
if (pn_tt_red("ckpt_restore_skip")) {
fprintf(stderr, "pn_tt: gdn checkpoint — RED ckpt_restore_skip: nothing restored\n");
return 0;
}
for (layer = 0; layer < device->gdn_shape.layer_count; layer++)
if (ckpt_copy_layer(device, layer, false) != 0) {
fprintf(stderr, "pn_tt: gdn checkpoint — the restore copy failed at layer %u; the "
"live state is undefined\n", layer);
return -1;
}
fprintf(stderr, "pn_tt: gdn checkpoint — restored %u layers in %.1f ms\n",
device->gdn_shape.layer_count, pn_time_ms() - started);
return 0;
}
bool pn_tt_gdn_checkpoint_verify_wanted(void)
{
return pn_tt_env_u32("PN_TT_CKPT_VERIFY", 0u) != 0u;
}
/** Count differing words between the first and last CKPT_VERIFY_BYTES of two buffers. */
static long long ckpt_verify_buffer(pn_tt_device *device, const pn_tt_gdn_buffer *a,
const pn_tt_gdn_buffer *b, size_t bytes, uint32_t *ha,
uint32_t *hb, unsigned long long *words)
{
const size_t span = bytes < CKPT_VERIFY_BYTES ? bytes : CKPT_VERIFY_BYTES;
const size_t offsets[2] = { 0u, bytes - span };
long long differ = 0;
size_t end, i;
for (end = 0; end < 2u; end++) {
const uint32_t at = (uint32_t)offsets[end];
if (pn_tt_read_dram(device, a->bank, a->address + at, ha, span) != 0 ||
pn_tt_read_dram(device, b->bank, b->address + at, hb, span) != 0)
return -1;
for (i = 0; i < span / 4u; i++)
differ += ha[i] != hb[i];
*words += span / 4u;
}
return differ;
}
long long pn_tt_gdn_checkpoint_verify_layer(pn_tt_device *device, uint32_t layer,
unsigned long long *words)
{
uint32_t *ha, *hb;
long long window_differ, state_differ;
size_t window, state;
if (!pn_tt_gdn_checkpoint_verify_wanted() || !device || !device->gdn_checkpoint ||
layer >= device->gdn_shape.layer_count)
return 0;
ha = malloc(CKPT_VERIFY_BYTES);
hb = malloc(CKPT_VERIFY_BYTES);
if (!ha || !hb) {
free(ha);
free(hb);
return -1;
}
pn_tt_gdn_carried_bytes(device, &window, &state);
window_differ = ckpt_verify_buffer(device, &device->gdn_region[layer].window,
&device->gdn_checkpoint[layer].window, window, ha, hb,
words);
state_differ = ckpt_verify_buffer(device, &device->gdn_region[layer].state,
&device->gdn_checkpoint[layer].state, state, ha, hb,
words);
free(ha);
free(hb);
return window_differ < 0 || state_differ < 0 ? -1 : window_differ + state_differ;
}
#endif /* USE_TT */

64
src/accel/tt/pn_tt_gdn_checkpoint.h Normal file
View File

@ -0,0 +1,64 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_gdn_checkpoint.h
* @brief The multi-turn checkpoint of the carried GDN state: every SSM layer's conv window and
* recurrent state copied, ON THE CARD, to a second region set and back.
*
* A hybrid model cannot rewind by position: the recurrent state is a fold of the whole prefix.
* The model layer takes a checkpoint at a prefill PASS boundary (`qwen35moe_prefill_tt.c`) and
* a later turn restores it and batch-prefills from there with the same passes a cold run would
* take, so its output is the cold run's. The attention K/V needs no copy: rows below the
* checkpoint are never rewritten while it is valid (the model layer enforces that).
*
* The copies are the puller's COPY op (`resident_puller.c` kOpCopy): ~120 MB card side instead
* of 22 s over a 5.3 MB/s readback. The checkpoint stays in card DRAM; nothing is written to
* the host's disk (bank-never-solos: a blob for the user's machine is a separate export).
*
* status: per-layer capture, restore and a sampled byte verify.
* issues: one checkpoint a device (one sequence on the card at a time).
* todo: none.
*/
#ifndef PN_TT_GDN_CHECKPOINT_H
#define PN_TT_GDN_CHECKPOINT_H
#ifdef USE_TT
#include "pn_tt_device.h"
/**
* Copy SSM layer `layer`'s live window + state into its checkpoint twin (the set is allocated
* on first use). Called between two 32-row bands of a prefill pass, when that layer's state is
* the state at the checkpoint position. Flushes the card first. RED `ckpt_copy_short` leaves
* each buffer's last 4 KiB uncopied.
*
* @return 0, or -1 (the checkpoint must then be treated as absent)
*/
int pn_tt_gdn_checkpoint_capture_layer(pn_tt_device *device, uint32_t layer);
/**
* Copy every layer's checkpoint twin back over the live window + state, and restart the
* dispatch model's learning (positions move backwards). RED `ckpt_restore_skip` reports
* success and copies nothing: the live state stays the turn's end and the output must DIFFER.
*
* @return 0, or -1 (nothing captured, or a copy failed: the live state is then UNDEFINED and
* the caller must start the sequence over)
*/
int pn_tt_gdn_checkpoint_restore(pn_tt_device *device);
/**
* `PN_TT_CKPT_VERIFY=1`: compare the first and last 64 KiB of layer `layer`'s live window and
* state with its checkpoint twin over the link. Call right after that layer's capture, or after
* a restore. The gate of the copy is bytes, not an argmax.
*
* @return the 32-bit words that differ (0 when the knob is off), or -1 when a read failed;
* `*words` grows by the words compared
*/
long long pn_tt_gdn_checkpoint_verify_layer(pn_tt_device *device, uint32_t layer,
unsigned long long *words);
/** Is `PN_TT_CKPT_VERIFY` set? */
bool pn_tt_gdn_checkpoint_verify_wanted(void);
#endif /* USE_TT */
#endif /* PN_TT_GDN_CHECKPOINT_H */

218
src/accel/tt/pn_tt_gdn_core_scaling.c Normal file
View File

@ -0,0 +1,218 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_gdn_core_scaling.c
* @brief Implementation of the GDN multi-core probe. See pn_tt_gdn_core_scaling.h.
*
* @status None
* @issues None
* @todo None
*/
#ifdef USE_TT
#include "pn_tt_gdn_core_scaling.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include "pn_tt_device_internal.h"
#include "pn_tt_gdn.h"
/** Extra cores this probe will ever hold. 90 is the GDN's own and 91 is the head norm's, so
* the extras start after them and eight of them clears every other residency's default. */
#define PN_TT_GDN_PROBE_MAX_EXTRA 7u
/** First Tensix the extras take. A default, not a constant — `PN_TT_GDN_PROBE_CORE_FIRST`
* moves it, and `pn_tt_prepare_family_on_core()` still refuses a matmul member by name. */
#define PN_TT_GDN_PROBE_CORE_FIRST_ENV "PN_TT_GDN_PROBE_CORE_FIRST"
#define PN_TT_GDN_PROBE_CORE_FIRST_DEFAULT 92u
/** Descriptor sets timed at each width. Enough that the spread is visible and few enough
* that the probe is over in under a second. */
#define PN_TT_GDN_PROBE_ROUNDS 16u
static double probe_now(void)
{
struct timespec ts;
if (clock_gettime(CLOCK_MONOTONIC, &ts) != 0)
return 0.0;
return (double)ts.tv_sec + (double)ts.tv_nsec * 1e-9;
}
static uint32_t probe_env_u32(const char *name, uint32_t fallback)
{
const char *value = getenv(name);
long parsed;
if (!value || !*value)
return fallback;
parsed = strtol(value, NULL, 10);
return parsed > 0 ? (uint32_t)parsed : fallback;
}
/** An fp32 operand descriptor for one of this family's DRAM buffers. */
static ft_tt_op_operand probe_operand(const pn_tt_device *device,
const pn_tt_gdn_buffer *buffer)
{
ft_tt_op_operand operand;
operand.x = (uint16_t)device->dram[buffer->bank].x;
operand.y = (uint16_t)device->dram[buffer->bank].y;
operand.address = buffer->address;
operand.tile_bytes = FT_TT_OP_GDN_FP32_TILE_BYTES;
return operand;
}
/**
* Arm one extra GDN residency on `core`, with the SAME circular-buffer placement the shipped
* one has.
*
* The placement comes from a CONV-shaped request even though the probe submits DELTA
* descriptors, because one placement serves both phases — that is a property of this family
* and `pn_tt_gdn.c` arms the real residency exactly the same way. Arming it any other way
* would be measuring a different residency from the one the model runs.
*/
static bool probe_arm_one(pn_tt_device *device, pn_tt_residency *residency, uint32_t core,
const ft_tt_op_gdn_request *shape)
{
ft_tt_launch_cb_entry placed[FT_TT_OP_GDN_CB_COUNT];
ft_tt_op_gdn_request request = *shape;
request.phase = FT_TT_OP_GDN_PHASE_CONV;
request.activation_in.tile_bytes = FT_TT_OP_GDN_FP32_TILE_BYTES;
request.activation_out.tile_bytes = FT_TT_OP_GDN_FP32_TILE_BYTES;
if (!pn_tt_prepare_family_on_core(device, residency, PN_TT_GDN_PROGRAM, "gdn-probe",
PN_TT_GDN_PROBE_CORE_FIRST_ENV, core,
PN_TT_ONCARD_TRIAD_MASK))
return false;
if (ft_tt_op_gdn_place_cbs(&request, placed, residency->cb_buffer,
sizeof(residency->cb_buffer), &residency->cb_mask) != FT_TT_OK) {
pn_tt_residency_release(residency);
return false;
}
if (!pn_tt_start_family(residency)) {
pn_tt_residency_release(residency);
return false;
}
return true;
}
/**
* One timed set: `width` descriptors started before any is waited on, `rounds` times.
*
* THE ENQUEUES COME FIRST AND THE WAITS COME SECOND. Enqueue-wait per core would measure
* `width` SERIAL descriptors and print a flat line at every width — the shape of "it does
* not parallelise" produced by the harness rather than by the card, which is the failure this
* whole project keeps finding. The two loops being separate is the measurement.
*/
static double probe_time_width(pn_tt_device *device, pn_tt_residency **residency,
uint32_t width, uint32_t rounds,
const ft_tt_op_gdn_request *template_request, bool *out_ok)
{
uint32_t words[PN_TT_GDN_PROBE_MAX_EXTRA + 1u][FT_TT_OP_GDN_WORDS];
double started;
uint32_t round;
uint32_t index;
/* Each core gets its OWN layer's carried state, so no two descriptors of a set touch the
* same DRAM and the timing is not a serialisation on one address. */
for (index = 0; index < width; index++) {
ft_tt_op_gdn_request request = *template_request;
request.state_recurrent = probe_operand(device, &device->gdn_region[index].state);
if (ft_tt_op_gdn_parameters(&request, words[index]) != FT_TT_OK) {
*out_ok = false;
return 0.0;
}
}
started = probe_now();
for (round = 0; round < rounds; round++) {
for (index = 0; index < width; index++)
if (pn_tt_residency_enqueue(residency[index], FT_TT_QUEUE_OP_GDN, words[index],
1u) != 0) {
*out_ok = false;
return 0.0;
}
for (index = 0; index < width; index++)
if (pn_tt_residency_await(residency[index], PN_TT_OP_TIMEOUT_MS) != 0) {
*out_ok = false;
return 0.0;
}
}
*out_ok = true;
return probe_now() - started;
}
void pn_tt_gdn_core_scaling_probe(pn_tt_device *device,
const ft_tt_op_gdn_request *template_request)
{
const uint32_t asked = probe_env_u32("PN_TT_GDN_CORES", 0u);
const uint32_t first = probe_env_u32(PN_TT_GDN_PROBE_CORE_FIRST_ENV,
PN_TT_GDN_PROBE_CORE_FIRST_DEFAULT);
pn_tt_residency extra[PN_TT_GDN_PROBE_MAX_EXTRA];
pn_tt_residency *set[PN_TT_GDN_PROBE_MAX_EXTRA + 1u];
double one_core = 0.0;
uint32_t armed = 0;
uint32_t width;
uint32_t index;
if (!device || !template_request || asked < 2u || device->gdn_probe_done)
return;
device->gdn_probe_done = true;
if (!pn_tt_gdn_is_armed(device))
return;
width = asked > PN_TT_GDN_PROBE_MAX_EXTRA + 1u ? PN_TT_GDN_PROBE_MAX_EXTRA + 1u : asked;
/* A width past the layer count would give two cores the same state region, which is the
* one thing this probe promises not to do. */
if (width > device->gdn_shape.layer_count)
width = device->gdn_shape.layer_count;
memset(extra, 0, sizeof(extra));
set[0] = &device->gdn;
for (index = 0; index + 1u < width; index++) {
if (!probe_arm_one(device, &extra[index], first + index, template_request))
break;
set[index + 1u] = &extra[index];
armed++;
}
if (armed + 1u < width) {
fprintf(stderr, "pn_tt: gdn CORE SCALING — only %u of the %u cores armed; the sweep "
"stops there\n", armed + 1u, width);
width = armed + 1u;
}
fprintf(stderr, "pn_tt: gdn CORE SCALING — %u cores from %u, %u rounds each. THIS WRITES "
"THE CARRIED STATE OF %u LAYERS: the run's text is not evidence.\n",
width, first, PN_TT_GDN_PROBE_ROUNDS, width);
for (index = 1u; index <= width; index <<= 1) {
bool ok = false;
const double seconds =
probe_time_width(device, set, index, PN_TT_GDN_PROBE_ROUNDS, template_request, &ok);
const double per_set = seconds / (double)PN_TT_GDN_PROBE_ROUNDS;
const double per_descriptor = per_set / (double)index;
if (!ok) {
fprintf(stderr, "pn_tt: gdn CORE SCALING — width %u FAILED; the residencies are "
"stranded and the sweep stops\n", index);
break;
}
if (index == 1u)
one_core = per_set;
fprintf(stderr, "pn_tt: gdn CORE SCALING width %2u — %.4f s a set, %.1f us a "
"descriptor, %.2fx the throughput of one core\n",
index, per_set, 1e6 * per_descriptor,
per_set > 0.0 ? one_core * (double)index / per_set : 0.0);
}
for (index = 0; index < armed; index++)
pn_tt_residency_release(&extra[index]);
fprintf(stderr, "pn_tt: gdn CORE SCALING — %u extra cores given back\n", armed);
}
#endif /* USE_TT */

71
src/accel/tt/pn_tt_gdn_core_scaling.h Normal file
View File

@ -0,0 +1,71 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_gdn_core_scaling.h
* @brief Does the GDN block go faster on more than one Tensix? An INSTRUMENT, armed only by
* `PN_TT_GDN_CORES` and never by the shipped path.
*
* ---------------------------------------------------------------------------
* WHY THE QUESTION IS OPEN AND WHY IT NEEDS SILICON
* ---------------------------------------------------------------------------
* The GDN op runs on ONE core. Its outer loop is over v-heads, and the heads are independent
* — each carries its own [S,S] state, reads its own v/decay/beta and writes its own row of y
* — so the arithmetic parallelises trivially. What is NOT known is whether the CARD does:
* the descriptor moves the carried state through DRAM, and Phase 1's core-scaling knee at
* N=4 was measured on the MATMUL path, whose traffic is a broadcast weight strip and nothing
* like this one. A knee measured on one op is not a property of the machine.
*
* So this probe asks the machine directly: run N copies of the SAME descriptor on N cores at
* once, each against a DIFFERENT SSM layer's carried state so no two touch the same DRAM,
* and compare the wall time of the set against one core's. Perfect scaling is N descriptors
* in one descriptor's time. Anything less is a shared resource, and the ratio prices it.
*
* **Full-width copies rather than a head split, deliberately.** Splitting 32 heads over N
* cores would measure the same thing and would also need N sets of per-core operand offsets
* — new address arithmetic, in a probe, whose bugs would look exactly like a scaling result.
* N full copies need no new addresses at all: the only word that changes per core is the
* state operand, and it is a region this device already allocated. If N copies cost one
* copy's time, then N head-slices certainly do.
*
* ---------------------------------------------------------------------------
* WHAT IT COSTS, WHICH IS NOT NOTHING
* ---------------------------------------------------------------------------
* The probe WRITES the carried state of the layers it borrows. It runs once, at the first
* DELTA descriptor of a run, where the state is still the zero the sequence starts from —
* but it leaves those layers holding a recurrence step the model did not ask for. **A run
* with this knob set produces text that is not evidence of anything.** It is a stopwatch.
*
* It also holds N-1 extra Tensix for the length of the sweep and then gives them back. A
* GDN triad that exits on a circular-buffer spin bound wedges its core until a rung-3 reset,
* so the extra residencies are retired by ASKING, the same way every other one is.
*
* @status None
* @issues None
* @todo None
*/
#ifndef PN_TT_GDN_CORE_SCALING_H
#define PN_TT_GDN_CORE_SCALING_H
#ifdef USE_TT
#include "pn_tt_device.h"
#include "tt/ft_tt_op_gdn.h"
/**
* Run the sweep, print the table on stderr, and give the extra cores back.
*
* Does nothing at all unless `PN_TT_GDN_CORES` is set to 2 or more. Safe to call on every
* DELTA descriptor — it runs once per device and remembers that it has.
*
* @param device an armed GDN device (`pn_tt_gdn_is_armed`)
* @param template a fully built DELTA request; the probe changes only its state operand
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
void pn_tt_gdn_core_scaling_probe(pn_tt_device *device,
const ft_tt_op_gdn_request *template_request);
#endif /* USE_TT */
#endif /* PN_TT_GDN_CORE_SCALING_H */

60
src/accel/tt/pn_tt_gdn_layout.c Normal file
View File

@ -0,0 +1,60 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_gdn_layout.c
* @brief Implementation of the GDN stager's address arithmetic. See pn_tt_gdn_layout.h.
*
* @status Pure; `tests/test_gdn_layout.c` proves it against literals derived from the
* protocol header and the CPU oracle rather than from these functions.
* @issues None
* @todo None
*/
#ifdef USE_TT
#include "pn_tt_gdn_layout.h"
#include <stddef.h>
#include "tt/ft_tt_tilize.h"
uint32_t pn_tt_gdn_head_tile(uint32_t head, uint32_t state)
{
return head * (state / PN_TT_GDN_LAYOUT_TILE_DIM);
}
uint32_t pn_tt_gdn_row_offset(uint32_t tile, uint32_t row, uint32_t chunk)
{
/* Four 16x16 faces in order, so the face holding `[row][chunk*16 .. chunk*16+15]` is
* `(row/16)*2 + chunk`, and each face is `TILE_BYTES/4` — 1024 B in fp32. Inside the face
* the row is `row % 16` rows of 16 fp32 down. */
const uint32_t face = (row / 16u) * 2u + chunk;
return tile * PN_TT_GDN_LAYOUT_TILE_BYTES + face * (PN_TT_GDN_LAYOUT_TILE_BYTES / 4u) +
(row % 16u) * PN_TT_GDN_LAYOUT_ROW0_CHUNK_BYTES;
}
uint32_t pn_tt_gdn_row0_offset(uint32_t tile, uint32_t chunk)
{
/* Face 0 is elements [0,256) of the tile and face 1 is [256,512), so the two halves of
* row 0 are `TILE_BYTES/4` apart — 1024 B in fp32. */
return pn_tt_gdn_row_offset(tile, 0u, chunk);
}
int pn_tt_gdn_stage_plane(const float *source, uint32_t first, uint32_t stride,
uint32_t conv_dim, float *scratch, unsigned char *tiles_out)
{
uint32_t channel;
if (!source || !scratch || !tiles_out || conv_dim == 0u || stride == 0u ||
(conv_dim % PN_TT_GDN_LAYOUT_TILE_DIM) != 0u)
return -1;
for (channel = 0; channel < conv_dim; channel++)
scratch[channel] = source[(size_t)first + (size_t)channel * stride];
if (ft_tt_tilize_strip(scratch, sizeof(*scratch), 1u, conv_dim,
(size_t)conv_dim * sizeof(*scratch), tiles_out) != FT_TT_TILIZE_OK)
return -1;
return 0;
}
#endif /* USE_TT */

130
src/accel/tt/pn_tt_gdn_layout.h Normal file
View File

@ -0,0 +1,130 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_gdn_layout.h
* @brief The GDN stager's ADDRESS ARITHMETIC, with no device in it — the four conversions
* between a host float array and the DRAM layouts
* `foundry/tt/kernels/resident_gdn_protocol.h` settles.
*
* Separate from `pn_tt_gdn.c` for one reason: these are the functions a wrong answer
* hides in, and split out they are provable with no card in the machine
* (`tests/test_gdn_layout.c`). Everything left in `pn_tt_gdn.c` is allocation,
* descriptor encoding and NoC calls — things a test cannot reach anyway.
*
* ---------------------------------------------------------------------------
* WHAT "TILIZED" MEANS HERE, ONCE
* ---------------------------------------------------------------------------
* What `ft_tt_tilize_strip()` produces: `ceil(rows/32)` row blocks in order, each
* `ceil(cols/32)` FACE-ORDERED 32x32 tiles in order, so element `[r][c]` lands at tile
* `(r/32)*ceil(cols/32) + c/32`, position `[r%32][c%32]`. Every operand of this family is
* fp32, so a tile is 4096 bytes.
*
* @status Pure C, no device, no allocation beyond what the caller hands in.
* @issues None
* @todo None
*/
#ifndef PN_TT_GDN_LAYOUT_H
#define PN_TT_GDN_LAYOUT_H
#ifdef USE_TT
#include <stdint.h>
/** Bytes of one fp32 tile, and the elements along one of its edges. */
#define PN_TT_GDN_LAYOUT_TILE_BYTES 4096u
#define PN_TT_GDN_LAYOUT_TILE_DIM 32u
/** The two 64-byte pieces logical row 0 of an fp32 tile is split across — columns 0-15 at the
* top of face 0, columns 16-31 at the top of face 1. */
#define PN_TT_GDN_LAYOUT_ROW0_CHUNKS 2u
#define PN_TT_GDN_LAYOUT_ROW0_CHUNK_ELEMS 16u
#define PN_TT_GDN_LAYOUT_ROW0_CHUNK_BYTES (PN_TT_GDN_LAYOUT_ROW0_CHUNK_ELEMS * 4u)
/**
* Byte offset, within a strip of consecutive fp32 tiles, of one of logical row 0's two
* pieces.
*
* The ONE place that layout is expressed, and separate for the reason `ft_tt_tile_row0.h`
* gives about its own: the two mutations that matter — a wrong face stride, and the two
* chunks swapped — are wrong values out of exactly this function.
*
* **In fp32 both pieces are exactly 64 bytes, which is the DRAM READ alignment.** The bf16
* row-0 path has to read 64 and discard 32; this one wastes nothing, which is why the conv
* stream crosses the link as row 0 rather than as whole tiles.
*
* @param tile which tile of the strip
* @param chunk 0 for columns 0-15, 1 for columns 16-31
* @return byte offset from the strip's base
*
* Thread-safe: yes Reentrant: yes Signal-safe: yes
*/
uint32_t pn_tt_gdn_row0_offset(uint32_t tile, uint32_t chunk);
/**
* The same offset for ANY logical row of the tile, not just row 0.
*
* An fp32 tile is four FACES of 16x16, face-ordered: face `(row/16)*2 + (col/16)` sits at
* `face * TILE_BYTES/4`, and inside a face row `row%16` sits at `(row%16) * 64`. Row 0 is the
* `row = 0` case and nothing else, so `pn_tt_gdn_row0_offset()` is written in terms of THIS
* function and the layout is still expressed once.
*
* **IT EXISTS FOR THE CHUNK PHASE-ISOLATION ARM AND NOTHING ELSE.** The decode GDN family
* reads and writes logical row 0; the chunkwise family reads and writes rows 0..C-1 of the
* SAME buffers. Attributing a divergence to one of the two chunk kernels means running the
* OTHER phase per position across a chunk's rows, which needs row `t` moved to row 0 and the
* answer moved back — exact fp32 copies, so the arm adds no arithmetic of its own.
*
* @param tile which tile of the strip
* @param row 0..31; anything else is clamped by the caller, not here
* @param chunk 0 for columns 0-15, 1 for columns 16-31
* @return byte offset from the strip's base
*
* Thread-safe: yes Reentrant: yes Signal-safe: yes
*/
uint32_t pn_tt_gdn_row_offset(uint32_t tile, uint32_t row, uint32_t chunk);
/**
* Gather `conv_dim` channels at STRIDE `stride` starting at `first`, and tilize them as a
* ONE-ROW strip of `conv_dim/32` fp32 tiles.
*
* **THE CONV WEIGHT'S TAP PLANE `t` IS `(first = t, stride = taps)`, AND THAT IS THE WHOLE
* TRANSPOSE.** The oracle stores the weight row-major `[conv_dim, taps]` — `w[c*taps + t]` —
* and the card wants `taps` planes, plane `t` holding `w[.][t]` laid out exactly like the qkv
* stream. The plausible wrong answer is a RESHAPE, `(first = t*conv_dim, stride = 1)`: the
* same bytes, the same length, a different answer, and nothing on the card can see the
* difference. Spelling both as two arguments of one function is what makes the difference a
* thing a test can state.
*
* One row is `valid_rows = 1`: rows 1-31 are the tile quantum and the tilizer zeroes them,
* which is correct because the compute thread's whole-tile multiply against a zero stream row
* is zero and `SiLU(0) = 0`.
*
* @param source at least `first + (conv_dim-1)*stride + 1` floats
* @param scratch `conv_dim` floats of caller-owned scratch
* @param tiles_out `conv_dim/32 * 4096` bytes
* @return 0, or -1 on a bad argument
*
* Thread-safe: yes (caller owns both buffers) Reentrant: yes Signal-safe: no
*/
int pn_tt_gdn_stage_plane(const float *source, uint32_t first, uint32_t stride,
uint32_t conv_dim, float *scratch, unsigned char *tiles_out);
/**
* The first tile of head `head`'s `state` values inside a HEAD-MAJOR ROW-0 STRIP.
*
* `q`, `k`, `v` at `kGdnActivationIn` and `y` at `kGdnActivationOut` are all row-0 strips now:
* a head's `state` values are `state/32` CONSECUTIVE tiles, all of them in row 0, exactly the
* geometry `pn_tt_gdn_row0_offset()` addresses and exactly the geometry PHASE CONV's output
* already has. They used to be tilized `[heads, state]` matrices with one row per head, which
* is 32x denser in DRAM and forced the host to de-tilize, L2-normalise and re-tilize between
* the two phases — 49,408 B a layer over PCIe.
*
* There is no `pn_tt_gdn_stage_stream()` any more because there is nothing left to stage: a
* row-0 strip is written and read by `pn_tt_fp32_row0_move()`, the same primitive the conv
* stream and the per-head reduce residency's strips already use.
*/
uint32_t pn_tt_gdn_head_tile(uint32_t head, uint32_t state);
#endif /* USE_TT */
#endif /* PN_TT_GDN_LAYOUT_H */

2119
src/accel/tt/pn_tt_gdnc.c Normal file
View File

File diff suppressed because it is too large Load Diff

208
src/accel/tt/pn_tt_gdnc.h Normal file
View File

@ -0,0 +1,208 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_gdnc.h
* @brief The CHUNKWISE Gated-Delta-Net block on the card: the same two phases the decode
* family runs, over C <= 32 POSITIONS AT ONCE, on `foundry/tt/ft_tt_op_gdnc.h`.
*
* ---------------------------------------------------------------------------
* IT BORROWS THE DECODE FAMILY'S CARRIED STATE, BYTE FOR BYTE
* ---------------------------------------------------------------------------
* This is the single most important property of this file and it was designed in rather than
* discovered: the rolling conv window and the `[v_heads, S, S]` recurrent state the chunkwise
* kernels read and write are **the buffers `pn_tt_gdn.c` allocated**, at the same addresses,
* in the same layout (`ftgc_plane_tile()` is `ftg_conv_plane_tile()`, `ftgc_state_tile()` is
* `ftg_state_tile()`). So a prompt that prefills in chunks and then decodes one token at a
* time is ONE sequence with no transcription at the seam, and the `1..taps-2` tail a chunk
* cannot carry is run through the decode family on the same bytes.
*
* The consequence for arming: **this family REQUIRES the `gdn` family to be armed** and
* refuses otherwise. It owns no carried state of its own.
*
* ---------------------------------------------------------------------------
* WHAT IT DOES OWN, AND WHY EACH IS A SECOND BUFFER
* ---------------------------------------------------------------------------
* the CONV constant PER LAYER. `taps` ROW-REPLICATED tap-major planes plus the `2*taps`
* band tiles riding at the end of them. The decode family's weight
* carries `w[c][tap]` in ROW 0 with zeros below; a chunk needs it at
* every row, because the tap coefficient is constant across positions
* and the broadcast surface is BROADCAST_COL only. Same tile count,
* same addresses, different `valid_rows` — and a SEPARATE buffer,
* because the shipped decode path's byte-identity is a hard gate and
* is not worth re-deriving through a shared operand.
* 4.03 MiB a layer at Ornith, 96.8 MiB over 24.
* the DELTA constant ONE for the model: the four program constants (`I`, `2I`, the
* strictly-lower mask, the `scale` column) and one decay-ratio tile per
* v-head. 147 KiB. The decay tiles are rebuilt every layer of every
* chunk; the four constants are written once.
* the GATE block ONE for the model, `v_heads * 160` floats, and it MUST sit in the
* DELTA constant's bank — it shares that operand's packed NoC
* coordinate word and nothing on the device can catch a violation.
*
* ---------------------------------------------------------------------------
* THE DECAY-RATIO MATRIX IS BUILT ON THE HOST, DELIBERATELY
* ---------------------------------------------------------------------------
* `ft_tt_op_gdnc_build_decay()` is a product recurrence: every entry in (0,1], no cumulative
* reciprocal, no logarithm. Both alternatives were modelled through this card's Tf32
* truncating unpack and both are worse — an on-card doubling construction costs the whole
* precision advantage of going chunkwise (y 8.6e-03 -> 1.7e-02), and `exp(log P_t - log P_s)`
* is safe against OVERFLOW and not against the UNPACK, because cumulative logs reach -50 at
* Ornith's decays and a Tf32 truncation of -50 is 2.4% error in D before the exp is reached.
* **It costs 128 KiB a layer a chunk on the link, which is 0.33 ms a position at 32 rows, and
* that is the price of the accuracy. Do not move it to the card.**
*
* ---------------------------------------------------------------------------
* WHAT THE CALLER OWES
* ---------------------------------------------------------------------------
* 1. `chunk_rows >= conv_taps - 1` for PHASE CONV. Below that the chunk cannot REPLACE the
* window, only shift it, and both the host encoder and the device refuse. The caller
* routes a shorter tail through `pn_tt_gdn_conv_oncard()` / `pn_tt_gdn_delta_oncard()`,
* which is the same arithmetic on the same bytes. `pn_tt_gdnc_min_chunk_rows()` is that
* floor. PHASE DELTA has no floor.
* 2. **ROWS `chunk_rows..31` OF THE STREAM MUST BE ZERO.** They are computed and discarded,
* and `SiLU(0) = 0` cannot surprise anyone — a stale value there would propagate through
* arithmetic nobody inspects. In `qwen35moe_prefill_tt.c` this holds for free and by
* construction: `pn_tt_activation_upload_rows()` tilizes at `valid_rows = rows`, so the
* pass's `hidden` strip is zero above the chunk, an RMS norm of a zero row is zero, and
* an `Mt=1` matvec of a zero row is zero — so every projection is zero above the chunk.
*
* @status NEW, default OFF behind `PN_TT_PREFILL_GDNC`. The three kernels are green on
* silicon (`foundry/tt/tests/test_phase6_gdnc.c`): PHASE CONV 681.4 us a descriptor
* flat in C, PHASE DELTA 435.5 us at 32 v-heads, both parity-checked over 103
* positions as 32+32+32+7.
* @issues The `gdnc` residency is ONE core and PHASE DELTA is not head-split here. The
* decode family measured its DELTA embarrassingly parallel in the head and the
* chunkwise one fits `13.8 us + N x 13.18`, so the split is available and unbuilt.
* @todo None
*/
#ifndef PN_TT_GDNC_H
#define PN_TT_GDNC_H
#ifdef USE_TT
#include <stdbool.h>
#include <stdint.h>
#include "pn_tt_device.h"
/**
* Arm the chunkwise residency and allocate the three buffers above.
*
* Reads the shape off the ALREADY-ARMED `gdn` family, so the two cannot disagree about
* `conv_dim`, the tap count or the head counts — which is what makes sharing the window and
* the recurrent state safe.
*
* Never fatal on its own: a card without the `gdnc` program, a core already held, or DRAM
* that will not take the constants all leave the family unarmed and the caller on the
* per-position path.
*
* @return 0 (even when nothing armed), -1 on a bad argument or a failed allocation
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
int pn_tt_device_arm_gdnc(pn_tt_device *device);
/** Did the chunkwise family actually arm for this run? */
bool pn_tt_gdnc_is_armed(const pn_tt_device *device);
/** The host's 1/sqrt(S) word, as the COMPACT block's header carries it. */
uint32_t pn_tt_gdnc_compact_scale_bits(const pn_tt_device *device);
/** PM 7 ITEM 5: one DELTA descriptor whose COMPACT gate block the card already computed (the
* GDN GATE family) at `gate_address`, in the DELTA constant bank. Nothing is staged. */
int pn_tt_gdnc_delta_oncard_card_gate(pn_tt_device *device, uint32_t layer, uint32_t first_row,
uint32_t chunk_rows, uint32_t gate_address);
/** The shortest chunk PHASE CONV will run — `conv_taps - 1`. Shorter tails go through the
* decode family. 0 when the family is not armed. */
uint32_t pn_tt_gdnc_min_chunk_rows(const pn_tt_device *device);
/**
* TRANSPOSE one layer's conv1d weight into ROW-REPLICATED tap-major planes and upload it,
* then write the `2*taps` band tiles behind them. Once, at load.
*
* `weight` is the oracle's own `layer->ssm_conv1d`: ROW-MAJOR `[conv_dim, taps]`. Same
* transpose `pn_tt_gdn_conv_weight_upload()` does, with `valid_rows` 1 -> 32.
*
* @param layer 0..layer_count-1, the SSM-layer ordinal
* @return 0, or -1 with the reason on stderr
*/
int pn_tt_gdnc_conv_weight_upload(pn_tt_device *device, uint32_t layer, const float *weight);
/**
* PHASE CONV over `chunk_rows` positions of the GDN stream-in buffer, writing the stream
* buffer and rolling the shared window ONCE, to the chunk's last `taps-1` positions.
*
* @param chunk_rows `conv_taps-1` .. 32
* @return 0, or -1 with the reason on stderr
*/
int pn_tt_gdnc_conv_oncard(pn_tt_device *device, uint32_t layer, uint32_t chunk_rows);
/**
* PHASE DELTA over the same `chunk_rows` positions, reading q|k|v out of the stream buffer
* and writing `y`, with the shared recurrent state advanced ONCE across the whole chunk.
*
* Uploads the per-head decay-ratio tiles (128 KiB) and the gate block (20 KiB) and nothing
* else.
*
* @param decay `chunk_rows * v_heads` floats, ROW-MAJOR by position — `exp(g)`, already
* through the host `softplus` and `exp` because there is no SFPU `log`
* @param beta likewise, already through the sigmoid
* @return 0, or -1 with the reason on stderr
*/
int pn_tt_gdnc_delta_oncard(pn_tt_device *device, uint32_t layer, uint32_t chunk_rows,
const float *decay, const float *beta);
/**
* PHASE DELTA over rows `[first_row, first_row + chunk_rows)` of the SAME 32-row strip — the
* IN-PLACE SUB-CHUNK, and the reason a shorter chunk costs one extra descriptor instead of a
* row shuttle.
*
* `first_row = 0` IS `pn_tt_gdnc_delta_oncard()` — same operands, same encoding, same staged
* bytes — so a caller that never sub-chunks is unchanged by construction.
*
* WHY IT EXISTS. PHASE DELTA owns 100% of the chunkwise prefill's quality variance and the
* chunk length is the whole of it: six arms against one oracle, C = 16 passing all six
* criteria with the best figures of the ladder and C = 32 failing worst-document. Everything
* else in the SSM layer batches at 32 and is unaffected by C, so the arm that keeps the speed
* and takes the quality is ONE PHASE CONV over the whole pass, ONE L2 norm over the whole
* pass, and `ceil(rows / C)` of these.
*
* @param first_row a multiple of 16, with `first_row + chunk_rows <= 32`
* @param decay,beta `chunk_rows * v_heads` floats for THIS sub-chunk's rows, row-major by
* position. The caller slices; nothing here reads outside the sub-chunk
* @return 0, or -1 with the reason on stderr
*/
int pn_tt_gdnc_delta_oncard_at(pn_tt_device *device, uint32_t layer, uint32_t first_row,
uint32_t chunk_rows, const float *decay, const float *beta);
/**
* Stage `(layer, first_row, chunk_rows)`'s decay tiles and gate block into that sub-chunk's
* slot AHEAD of its descriptor, which then uses them instead of staging. Meant to run from the
* overlap hook, i.e. while the card is busy with something that does not read that slot.
*
* @return 0, or -1 (nothing is marked staged)
*/
int pn_tt_gdnc_delta_prestage_at(pn_tt_device *device, uint32_t layer, uint32_t first_row,
uint32_t chunk_rows, const float *decay, const float *beta);
/**
* ONE-SHOT: run `fn(context)` inside the next gdnc descriptor, between its enqueue and its
* await — host work hidden under card time. Its seconds are booked to the stage line, not the
* phase. A failing `fn` fails that descriptor's call.
*/
void pn_tt_gdnc_set_overlap(pn_tt_device *device, int (*fn)(void *context), void *context);
/**
* ONE-SHOT, a second slot: `fn(context)` runs in the next gdnc window AFTER the first slot's
* hook. Its result is ignored (its owner redoes the work inline when it did not happen), and it
* may set this slot again for the window after. NULL clears it.
*/
void pn_tt_gdnc_set_overlap_aux(pn_tt_device *device, int (*fn)(void *context), void *context);
/** Retire the residency and free what arming allocated. Safe on a device that never armed. */
void pn_tt_gdnc_release(pn_tt_device *device);
#endif /* USE_TT */
#endif /* PN_TT_GDNC_H */

526
src/accel/tt/pn_tt_gdngate.c Normal file
View File

@ -0,0 +1,526 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_gdngate.c
* @brief The GDN GATE family host half. See pn_tt_gdngate.h for what and why.
*
* @status Written PM 7 (2026-10-08). Unexecuted.
* @issues None
* @todo None
*/
#ifdef USE_TT
#include "pn_tt_gdngate.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "pn_tt_device_internal.h"
#include "tt/ft_tt_op_dispatch.h"
#include "tt/kernels/resident_gdngate_protocol.h"
#define PN_TT_GDNGATE_PROGRAM "gdngate"
#define PN_TT_GDNGATE_CORE_ENV "PN_TT_GDNGATE_CORE_LIST"
/** The cores no family holds with every decode family and the 48-core prefill grid (0-47)
* armed — read off a full arm's `residency up on core` census (PM 7, 2026-10-08). */
#define PN_TT_GDNGATE_CORE_DEFAULT "121,126,127,128"
/** The two CBs: the reader's stage (two tiles + the constants, addressed) and the blocks. */
#define PN_TT_GDNGATE_CB_STAGE 0u
#define PN_TT_GDNGATE_CB_OUT 16u
#define PN_TT_GDNGATE_CB_OUT_PAGES 2u
/** `PN_TT_GDNGATE_CORE_LIST` (comma-separated core indices) into `cores`. @return count. */
static uint32_t gdngate_core_list(uint32_t *cores)
{
const char *env = getenv(PN_TT_GDNGATE_CORE_ENV);
const char *at = (env && *env) ? env : PN_TT_GDNGATE_CORE_DEFAULT;
uint32_t count = 0u;
char *end;
while (*at && count < PN_TT_GDNGATE_MAX_MEMBERS) {
const unsigned long core = strtoul(at, &end, 10);
if (end == at)
break;
cores[count++] = (uint32_t)core;
at = *end == ',' ? end + 1 : end;
}
return count;
}
static uint32_t gdngate_xy(const pn_tt_device *device, uint32_t bank)
{
return (uint32_t)device->dram[bank].x | ((uint32_t)device->dram[bank].y << 16);
}
static uint32_t gdngate_align64(uint32_t bytes)
{
return (bytes + 63u) & ~63u;
}
/** Band 0's DRAM address of a strip, whatever band it is seeked to. */
static uint32_t gdngate_band0_address(const pn_tt_activation *strip)
{
return strip->address - strip->row_offset * strip->tile_count * PN_TT_TILE_BF16;
}
/** Prepare, place and start member `index` on `core`. */
static bool gdngate_arm_member(pn_tt_device *device, uint32_t index, uint32_t core)
{
pn_tt_residency *member = &device->gdngate[index];
ft_tt_launch_cb_entry placed[2];
ft_tt_op_cb_request requests[2];
if (!pn_tt_prepare_family_on_core(device, member, PN_TT_GDNGATE_PROGRAM, "gdngate",
PN_TT_GDNGATE_CORE_ENV, core, PN_TT_ONCARD_DM_MASK))
return false;
requests[0].index = PN_TT_GDNGATE_CB_STAGE;
requests[0].page_bytes = 2u * PN_TT_TILE_BF16 + kGgConstWords * 4u;
requests[0].page_count = 1u;
requests[1].index = PN_TT_GDNGATE_CB_OUT;
requests[1].page_bytes = gdngate_align64(ftgg_block_bytes(device->gdngate_heads));
requests[1].page_count = PN_TT_GDNGATE_CB_OUT_PAGES;
if (ft_tt_op_cb_place(requests, 2u, placed, member->cb_buffer, sizeof(member->cb_buffer),
&member->cb_mask) != FT_TT_OK) {
fprintf(stderr, "pn_tt: the gdngate circular buffers do not fit\n");
pn_tt_residency_release(member);
return false;
}
return pn_tt_start_family(member);
}
/** Upload every layer's constants block: ssm_a[32] | ssm_dt[32] (zero past `heads`). */
static int gdngate_upload_constants(pn_tt_device *device, const float *const *ssm_a,
const float *const *ssm_dt)
{
const size_t bytes = (size_t)device->gdngate_layers * kGgConstWords * 4u;
uint32_t *table = calloc(1, bytes);
ft_tt_buffer buffer;
int rc = -1;
if (!table)
return -1;
for (uint32_t layer = 0; layer < device->gdngate_layers; layer++) {
uint32_t *row = table + (size_t)layer * kGgConstWords;
memcpy(row, ssm_a[layer], device->gdngate_heads * sizeof(float));
memcpy(row + kGgMaxHeads, ssm_dt[layer], device->gdngate_heads * sizeof(float));
}
if (ft_tt_buffer_alloc(&device->allocator, FT_TT_MEM_SPACE_DRAM, 0u, bytes, &buffer) ==
FT_TT_OK) {
device->gdngate_constants.bank = buffer.logical_bank;
device->gdngate_constants.address = (uint32_t)buffer.address;
rc = pn_tt_write_dram(device, buffer.logical_bank, (uint32_t)buffer.address, table,
bytes);
}
free(table);
return rc;
}
int pn_tt_gdngate_arm(pn_tt_device *device, uint32_t layers, uint32_t heads,
const float *const *ssm_a, const float *const *ssm_dt, uint32_t max_rows,
uint32_t scale_bits)
{
const char *env = getenv("PN_TT_GDNGATE");
uint32_t want, index;
ft_tt_buffer buffer;
size_t array_bytes;
if (!device || (env && *env && atoi(env) == 0))
return -1;
if (layers == 0u || heads == 0u || heads > kGgMaxHeads || !ssm_a || !ssm_dt ||
max_rows == 0u || !pn_tt_program_is_present(PN_TT_GDNGATE_PROGRAM)) {
fprintf(stderr, "pn_tt: gdngate refused — %u layer(s), %u head(s), program %s\n",
layers, heads,
pn_tt_program_is_present(PN_TT_GDNGATE_PROGRAM) ? "present" : "ABSENT");
return -1;
}
want = gdngate_core_list(device->gdngate_cores);
if (want == 0u)
return -1;
device->gdngate_layers = layers;
device->gdngate_heads = heads;
device->gdngate_scale_bits = scale_bits;
device->gdngate_max_bands = (max_rows + kGgTileRows - 1u) / kGgTileRows;
device->gdngate_block_stride = gdngate_align64(ftgg_block_bytes(heads));
/* THE DELTA DESCRIPTOR READS ITS GATE AT ITS CONSTANT'S xy: the card-gate DELTA always
* uses constant slot 0 (pn_tt_gdnc_delta_oncard_card_gate), so the blocks go in its bank. */
array_bytes = (size_t)layers * 2u * device->gdngate_max_bands * device->gdngate_block_stride;
if (ft_tt_buffer_alloc(&device->allocator, FT_TT_MEM_SPACE_DRAM,
device->gdnc_delta_constant[0].bank, array_bytes, &buffer) !=
FT_TT_OK) {
fprintf(stderr, "pn_tt: gdngate — no room for %zu bytes of gate blocks\n", array_bytes);
return -1;
}
device->gdngate_blocks.bank = buffer.logical_bank;
device->gdngate_blocks.address = (uint32_t)buffer.address;
if (gdngate_upload_constants(device, ssm_a, ssm_dt) != 0) {
fprintf(stderr, "pn_tt: gdngate — the constants table did not upload\n");
return -1;
}
for (index = 0; index < want; index++) {
if (!gdngate_arm_member(device, index, device->gdngate_cores[index]))
break;
device->gdngate_members = index + 1u;
}
if (device->gdngate_members == 0u) {
fprintf(stderr, "pn_tt: gdngate — no member armed\n");
return -1;
}
fprintf(stderr, "pn_tt: gdngate — %u member(s), %u layer(s) x %u band(s) of gate blocks "
"(%zu bytes)\n",
device->gdngate_members, layers, device->gdngate_max_bands, array_bytes);
return 0;
}
bool pn_tt_gdngate_enabled(const pn_tt_device *device)
{
return device && device->gdngate_members > 0u;
}
uint32_t pn_tt_gdngate_block_address(const pn_tt_device *device, uint32_t layer, uint32_t band,
uint32_t sub)
{
const uint32_t per_layer = 2u * device->gdngate_max_bands * device->gdngate_block_stride;
return device->gdngate_blocks.address + layer * per_layer +
(2u * band + sub) * device->gdngate_block_stride;
}
/** The parameter words of one descriptor: bands [first, first + count) of `layer`. */
static void gdngate_words(const pn_tt_device *device, uint32_t layer,
const pn_tt_activation *alpha, const pn_tt_activation *beta,
uint32_t rows, uint32_t sub_rows, uint32_t first, uint32_t count,
uint32_t *words)
{
const uint32_t band_bytes = alpha->tile_count * PN_TT_TILE_BF16;
const uint32_t done = first * kGgTileRows;
const uint32_t take = rows - done < count * kGgTileRows ? rows - done : count * kGgTileRows;
memset(words, 0, kGgWords * sizeof(uint32_t));
words[kGgFlags] = pn_tt_red("gdngate_beta_as_decay") ? kGgFlagRedBetaAsDecay : 0u;
words[kGgRows] = take;
words[kGgBandCount] = count;
words[kGgSubRows] = sub_rows;
words[kGgHeads] = device->gdngate_heads;
words[kGgAlphaXY] = gdngate_xy(device, alpha->bank);
words[kGgAlphaBase] = gdngate_band0_address(alpha) + first * band_bytes;
words[kGgBetaXY] = gdngate_xy(device, beta->bank);
words[kGgBetaBase] = gdngate_band0_address(beta) + first * band_bytes;
words[kGgBandStride] = band_bytes;
words[kGgConstXY] = gdngate_xy(device, device->gdngate_constants.bank);
words[kGgConstBase] = device->gdngate_constants.address + layer * kGgConstWords * 4u;
words[kGgDstXY] = gdngate_xy(device, device->gdngate_blocks.bank);
words[kGgDstBase] = pn_tt_gdngate_block_address(device, layer, first, 0u);
words[kGgBlockStride] = device->gdngate_block_stride;
words[kGgScaleBits] = device->gdngate_scale_bits;
}
int pn_tt_gdngate_issue_bands(pn_tt_device *device, uint32_t layer,
const pn_tt_activation *alpha, const pn_tt_activation *beta,
uint32_t rows, uint32_t sub_rows, uint32_t first_band,
uint32_t band_count)
{
const uint32_t bands = (rows + kGgTileRows - 1u) / kGgTileRows;
uint32_t members, per_member, first = first_band, index;
uint32_t words[kGgWords];
if (!pn_tt_gdngate_enabled(device) || !alpha || !beta || rows == 0u || band_count == 0u ||
layer >= device->gdngate_layers || bands > device->gdngate_max_bands ||
first_band + band_count > bands || alpha->tile_count != 1u || beta->tile_count != 1u)
return -1;
members = device->gdngate_members < band_count ? device->gdngate_members : band_count;
per_member = (band_count + members - 1u) / members;
device->gdngate_last.layer = layer;
device->gdngate_last.alpha = *alpha;
device->gdngate_last.beta = *beta;
device->gdngate_last.rows = rows;
device->gdngate_last.sub_rows = sub_rows;
for (index = 0; index < members && first < first_band + band_count; index++) {
const uint32_t end = first_band + band_count;
const uint32_t count = end - first < per_member ? end - first : per_member;
/* Round-robin over issues too, so a super-band's few bands do not all land on 0. */
const uint32_t member = (device->gdngate_next_member + index) % device->gdngate_members;
gdngate_words(device, layer, alpha, beta, rows, sub_rows, first, count, words);
if (pn_tt_residency_enqueue(&device->gdngate[member], FT_TT_QUEUE_OP_GDN_GATE, words,
1u) != 0)
device->gdngate_failed = true;
device->gdngate_pending[member]++;
first += count;
}
device->gdngate_next_member = (device->gdngate_next_member + members) %
device->gdngate_members;
device->gdngate_issues++;
return 0;
}
int pn_tt_gdngate_issue(pn_tt_device *device, uint32_t layer, const pn_tt_activation *alpha,
const pn_tt_activation *beta, uint32_t rows, uint32_t sub_rows)
{
if (device)
device->gdngate_layer_issues++;
return pn_tt_gdngate_issue_bands(device, layer, alpha, beta, rows, sub_rows, 0u,
(rows + kGgTileRows - 1u) / kGgTileRows);
}
/** Re-arm every member and replay the last issue, awaited. The strips are untouched (the
* caller settles before its band loop rewrites anything), so the replay is the same op. */
static int gdngate_replay(pn_tt_device *device)
{
uint32_t index;
for (index = 0; index < device->gdngate_members; index++) {
const uint32_t core = device->gdngate_cores[index];
fprintf(stderr, "pn_tt: gdngate member %u refused a descriptor — retiring it by ASKING "
"and arming a fresh one on core %u\n", index, core);
pn_tt_residency_stop(&device->gdngate[index]);
pn_tt_residency_release(&device->gdngate[index]);
if (!gdngate_arm_member(device, index, core))
return -1;
device->gdngate_pending[index] = 0u;
}
device->gdngate_failed = false;
if (pn_tt_gdngate_issue(device, device->gdngate_last.layer, &device->gdngate_last.alpha,
&device->gdngate_last.beta, device->gdngate_last.rows,
device->gdngate_last.sub_rows) != 0)
return -1;
for (index = 0; index < device->gdngate_members; index++) {
if (device->gdngate_pending[index] != 0u &&
pn_tt_residency_await(&device->gdngate[index], PN_TT_OP_TIMEOUT_MS) != 0)
return -1;
device->gdngate_pending[index] = 0u;
}
return device->gdngate_failed ? -1 : 0;
}
int pn_tt_gdngate_settle(pn_tt_device *device)
{
const double started = pn_tt_now_seconds();
bool failed;
uint32_t index;
int rc;
if (!pn_tt_gdngate_enabled(device))
return -1;
failed = device->gdngate_failed;
for (index = 0; index < device->gdngate_members; index++) {
if (device->gdngate_pending[index] == 0u)
continue;
if (pn_tt_residency_await(&device->gdngate[index], PN_TT_OP_TIMEOUT_MS) != 0)
failed = true;
device->gdngate_pending[index] = 0u;
}
rc = failed ? gdngate_replay(device) : 0;
device->gdngate_settle_seconds += pn_tt_now_seconds() - started;
device->gdngate_settles++;
return rc;
}
void pn_tt_gdngate_report(const pn_tt_device *device)
{
if (!pn_tt_gdngate_enabled(device))
return;
fprintf(stderr, "pn_tt: gdngate — %llu issue(s) (%llu at a layer start), %llu settle(s) "
"waiting %.4f s, %llu FLAT (cumulative)\n",
(unsigned long long)device->gdngate_issues,
(unsigned long long)device->gdngate_layer_issues,
(unsigned long long)device->gdngate_settles, device->gdngate_settle_seconds,
(unsigned long long)device->gdngate_flats);
}
/** FLAT words for heads [first, first + count) of `heads_total`. */
static void gdngate_flat_words(const pn_tt_device *device, uint32_t layer,
const pn_tt_activation *alpha, const pn_tt_activation *beta,
uint32_t rows, uint32_t dst_bank, uint32_t dst_address,
uint32_t row_stride, uint32_t first, uint32_t count,
uint32_t *words)
{
gdngate_words(device, layer, alpha, beta, rows, 16u, 0u, 1u, words);
words[kGgFlags] |= kGgFlagFlat;
words[kGgSubRows] = first;
words[kGgHeads] = count;
words[kGgBandStride] = device->gdngate_heads;
words[kGgDstXY] = gdngate_xy(device, dst_bank);
words[kGgDstBase] = dst_address;
words[kGgBlockStride] = row_stride;
}
int pn_tt_gdngate_flat(pn_tt_device *device, uint32_t layer, const pn_tt_activation *alpha,
const pn_tt_activation *beta, uint32_t rows, uint32_t dst_bank,
uint32_t dst_address, uint32_t row_stride)
{
uint32_t words[PN_TT_GDNGATE_MAX_MEMBERS][kGgWords];
uint32_t members, per, index;
int failed = 0;
if (!pn_tt_gdngate_enabled(device) || !alpha || !beta || rows == 0u || rows > kGgTileRows ||
layer >= device->gdngate_layers || alpha->tile_count != 1u || beta->tile_count != 1u)
return -1;
/* THE HEADS SPLIT OVER THE MEMBERS in slices of a multiple of 4 (16-byte runs), each
* member writing only its slice's two runs a row; `PN_TT_GDNGATE_FLAT_SPLIT=0` = one. */
members = pn_tt_env_u32("PN_TT_GDNGATE_FLAT_SPLIT", 1u) ? device->gdngate_members : 1u;
while (members > 1u && (device->gdngate_heads % (members * 4u)) != 0u)
members--;
per = device->gdngate_heads / members;
for (index = 0; index < members; index++) {
gdngate_flat_words(device, layer, alpha, beta, rows, dst_bank, dst_address, row_stride,
index * per, per, words[index]);
if (pn_tt_residency_enqueue(&device->gdngate[index], FT_TT_QUEUE_OP_GDN_GATE,
words[index], 1u) != 0)
failed = 1;
}
device->gdngate_flats++;
/* Awaited here: the DELTA after it reads what they write (streamed: puller AWAITs). */
for (index = 0; index < members; index++) {
pn_tt_residency *member = &device->gdngate[index];
if (pn_tt_residency_await(member, PN_TT_OP_TIMEOUT_MS) == 0)
continue;
/* A refused member (idle retire): re-armed and its slice run again, awaited. */
fprintf(stderr, "pn_tt: gdngate member %u refused a FLAT descriptor — retiring it by "
"ASKING and arming a fresh one on core %u\n",
index, device->gdngate_cores[index]);
pn_tt_residency_stop(member);
pn_tt_residency_release(member);
if (!gdngate_arm_member(device, index, device->gdngate_cores[index]) ||
pn_tt_residency_submit(member, FT_TT_QUEUE_OP_GDN_GATE, words[index],
PN_TT_OP_TIMEOUT_MS) != 0)
failed = 1;
}
return failed ? -1 : 0;
}
long pn_tt_gdngate_verify_layer(pn_tt_device *device, uint32_t layer,
const pn_tt_activation *alpha, const pn_tt_activation *beta,
uint32_t rows, uint32_t sub_rows)
{
const uint32_t bands = (rows + kGgTileRows - 1u) / kGgTileRows;
const uint32_t words = ftgg_block_bytes(device->gdngate_heads) / 4u;
uint32_t constants[kGgConstWords], expect[kGgBlockBytes / 4u], got[kGgBlockBytes / 4u];
uint16_t tiles[2][PN_TT_TILE_BF16 / 2u];
long bad = 0;
if (pn_tt_read_dram(device, device->gdngate_constants.bank,
device->gdngate_constants.address + layer * kGgConstWords * 4u,
constants, sizeof(constants)) != 0)
return -1;
for (uint32_t band = 0; band < bands; band++) {
const uint32_t band_rows = rows - band * kGgTileRows < kGgTileRows
? rows - band * kGgTileRows : kGgTileRows;
const uint32_t offset = band * alpha->tile_count * PN_TT_TILE_BF16;
if (pn_tt_read_dram(device, alpha->bank, gdngate_band0_address(alpha) + offset,
tiles[0], sizeof(tiles[0])) != 0 ||
pn_tt_read_dram(device, beta->bank, gdngate_band0_address(beta) + offset, tiles[1],
sizeof(tiles[1])) != 0)
return -1;
for (uint32_t sub = 0; sub < 2u; sub++) {
const uint32_t live = ftgg_sub_live_rows(band_rows, sub_rows, sub);
if (live == 0u)
continue;
ftgg_build_block(expect, device->gdngate_heads, device->gdngate_scale_bits,
tiles[0], tiles[1], constants, sub * sub_rows, live, 0);
if (pn_tt_read_dram(device, device->gdngate_blocks.bank,
pn_tt_gdngate_block_address(device, layer, band, sub), got,
words * 4u) != 0)
return -1;
for (uint32_t word = 0; word < words; word++)
bad += expect[word] != got[word];
}
}
return bad;
}
/** Fill `count` face-ordered bf16 tiles so that, over all of them, every bf16 bit pattern
* appears exactly once at each head column after rotation `rotation` (value v lands in
* band v / 1024, row (v / 32) % 32, head (v + rotation) % 32). Beta takes the reverse. */
static void gdngate_sweep_tiles(uint16_t *alpha, uint16_t *beta, uint32_t count,
uint32_t rotation)
{
const uint32_t per_tile = PN_TT_TILE_BF16 / 2u;
for (uint32_t value = 0; value < count * per_tile; value++) {
const uint32_t band = value / per_tile, row = (value / 32u) % 32u;
const uint32_t head = (value + rotation) % 32u;
const size_t at = (size_t)band * per_tile + ftgg_face_index(row, head);
alpha[at] = (uint16_t)value;
beta[at] = (uint16_t)(0xFFFFu - value);
}
}
long pn_tt_gdngate_sweep(pn_tt_device *device, uint32_t rotations)
{
const uint32_t bands = 64u, rows = bands * kGgTileRows;
const size_t bytes = (size_t)bands * PN_TT_TILE_BF16;
uint16_t *host = malloc(2u * bytes);
pn_tt_activation alpha, beta;
ft_tt_buffer a_buffer, b_buffer;
double compute_seconds = 0.0;
long bad = 0, part;
if (!host || bands > device->gdngate_max_bands ||
ft_tt_buffer_alloc(&device->allocator, FT_TT_MEM_SPACE_DRAM, 1u, bytes, &a_buffer) !=
FT_TT_OK ||
ft_tt_buffer_alloc(&device->allocator, FT_TT_MEM_SPACE_DRAM, 2u, bytes, &b_buffer) !=
FT_TT_OK) {
free(host);
return -1;
}
memset(&alpha, 0, sizeof(alpha));
alpha.bank = a_buffer.logical_bank;
alpha.address = (uint32_t)a_buffer.address;
alpha.tile_count = 1u;
alpha.replica_count = 1u;
alpha.tile_rows = bands;
alpha.active_rows = 1u;
beta = alpha;
beta.bank = b_buffer.logical_bank;
beta.address = (uint32_t)b_buffer.address;
for (uint32_t rotation = 0; rotation < rotations && bad >= 0; rotation++) {
gdngate_sweep_tiles(host, host + bytes / 2u, bands, rotation);
if (pn_tt_write_dram(device, alpha.bank, alpha.address, host, bytes) != 0 ||
pn_tt_write_dram(device, beta.bank, beta.address, host + bytes / 2u, bytes) != 0)
bad = -1;
for (uint32_t layer = 0; bad >= 0 && layer < device->gdngate_layers; layer++) {
const double started = pn_tt_now_seconds();
int rc = pn_tt_gdngate_issue(device, layer, &alpha, &beta, rows, 16u);
if (rc == 0)
rc = pn_tt_gdngate_settle(device);
compute_seconds += pn_tt_now_seconds() - started;
if (rc != 0 ||
(part = pn_tt_gdngate_verify_layer(device, layer, &alpha, &beta, rows, 16u)) < 0)
bad = -1;
else
bad += part;
}
}
free(host);
fprintf(stderr, "pn_tt: gdngate SWEEP — every bf16 alpha and beta x %u head rotation(s) x "
"%u layers' constants: %ld word(s) differ from the host build; card "
"%.3f s over %u member(s) = %.2f us a (beta, decay) pair a member\n",
rotations, device->gdngate_layers, bad, compute_seconds, device->gdngate_members,
compute_seconds * 1e6 * device->gdngate_members /
((double)rotations * device->gdngate_layers * rows * device->gdngate_heads));
return bad;
}
void pn_tt_gdngate_release(pn_tt_device *device)
{
uint32_t index;
if (!device)
return;
for (index = 0; index < device->gdngate_members; index++)
pn_tt_residency_release(&device->gdngate[index]);
device->gdngate_members = 0u;
}
#endif /* USE_TT */

93
src/accel/tt/pn_tt_gdngate.h Normal file
View File

@ -0,0 +1,93 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_gdngate.h
* @brief The GDN GATE family host half: the Gated DeltaNet's per-position decay and beta
* computed ON THE CARD from the alpha/beta projection strips, as the gdnc DELTA reader's
* COMPACT gate blocks, so the prefill never reads alpha/beta back to the host.
*
* WHY THE BITS ARE THE HOST'S: the card reader and this file compile the same integer-only
* `ft_det_math.h` (foundry/tt/kernels), through the same block builder
* (`ftgg_build_block()` in resident_gdngate_protocol.h).
*
* THE SHAPE. One member per core of `PN_TT_GDNGATE_CORE_LIST` (default 121,126,127,128 — the
* cores no family holds). Each SSM layer owns a gate array of
* 2 * `max_bands` blocks in the gdnc DELTA constant bank (the DELTA descriptor reads its gate
* at the constant word's xy); block (band, sub) is at `layer_base + (2 * band + sub) * stride`.
* A layer's bands are dealt to the members in contiguous ranges, issued WITHOUT awaiting, and
* settled once before the layer's band loop (a puller AWAIT when streamed).
*
* @status Written PM 7 (2026-10-08). Unexecuted.
* @issues None
* @todo None
*/
#ifndef PANTHEON_PN_TT_GDNGATE_H
#define PANTHEON_PN_TT_GDNGATE_H
#ifdef USE_TT
#include <stdbool.h>
#include <stdint.h>
#include "pn_tt_device.h"
/** Arm the members, upload every layer's constants (`ssm_a`, `ssm_dt` for each SSM layer, row
* `layer` = 32 + 32 fp32), and allocate the gate arrays for up to `max_rows` positions a
* pass. `scale_bits` is the host's 1/sqrt(S) word (header word 0). @return 0, or -1 (the
* family is then simply not used: the host gate path stays). */
int pn_tt_gdngate_arm(pn_tt_device *device, uint32_t layers, uint32_t heads,
const float *const *ssm_a, const float *const *ssm_dt, uint32_t max_rows,
uint32_t scale_bits);
/** Is the family armed and wanted (`PN_TT_GDNGATE=0` off)? */
bool pn_tt_gdngate_enabled(const pn_tt_device *device);
/** Issue layer `layer`'s blocks for rows [0, rows) of the alpha/beta strips (any seek; band 0
* is computed), with the DELTA sub-chunk length `sub_rows`. Not awaited. */
int pn_tt_gdngate_issue(pn_tt_device *device, uint32_t layer, const pn_tt_activation *alpha,
const pn_tt_activation *beta, uint32_t rows, uint32_t sub_rows);
/** The same for bands [first_band, first_band + band_count) of a `rows`-row layer — the W2
* wavefront issues the next layer's gates super-band by super-band. */
int pn_tt_gdngate_issue_bands(pn_tt_device *device, uint32_t layer,
const pn_tt_activation *alpha, const pn_tt_activation *beta,
uint32_t rows, uint32_t sub_rows, uint32_t first_band,
uint32_t band_count);
/** Await every issued descriptor (streamed: puller AWAITs). A refused member is re-armed and
* its descriptor replayed (the op is a pure function of strips nobody has rewritten yet). */
int pn_tt_gdngate_settle(pn_tt_device *device);
/** ITEM 6: the DECODE gate layout for rows [0, rows) of band 0 of the strips (flat
* decay[H] | beta[H] a row at `row_stride`), written at `dst_bank`/`dst_address` — where the
* gdn DELTA reads it (pn_tt_gdn_gate_target). Enqueued AND awaited. @return 0 or -1. */
int pn_tt_gdngate_flat(pn_tt_device *device, uint32_t layer, const pn_tt_activation *alpha,
const pn_tt_activation *beta, uint32_t rows, uint32_t dst_bank,
uint32_t dst_address, uint32_t row_stride);
/** The DRAM address of block (band, sub) of `layer`, in the gdnc DELTA constant bank. */
uint32_t pn_tt_gdngate_block_address(const pn_tt_device *device, uint32_t layer, uint32_t band,
uint32_t sub);
/** VERIFY (`PN_TT_GDNGATE_VERIFY=1`, gates only): read layer `layer`'s blocks for `rows` back
* and compare them word for word with the host build of the same block from host-read
* alpha/beta. @return mismatching words, or -1 on a read failure. */
long pn_tt_gdngate_verify_layer(pn_tt_device *device, uint32_t layer,
const pn_tt_activation *alpha, const pn_tt_activation *beta,
uint32_t rows, uint32_t sub_rows);
/** THE CARD == HOST SWEEP (`PN_TT_GDNGATE_SWEEP=<rotations>`, gates only): synthetic strips
* holding every bf16 bit pattern as alpha and as beta, rotated over the head columns, through
* the family with every layer's real constants, each block compared with the host build.
* @return mismatching words, or -1. Leaves the DRAM it used allocated (a gate arm). */
long pn_tt_gdngate_sweep(pn_tt_device *device, uint32_t rotations);
/** One stderr line: issues (and how many at a layer start), settles and their wait, FLATs. */
void pn_tt_gdngate_report(const pn_tt_device *device);
/** Retire and release every member (device close). */
void pn_tt_gdngate_release(pn_tt_device *device);
#endif /* USE_TT */
#endif /* PANTHEON_PN_TT_GDNGATE_H */

531
src/accel/tt/pn_tt_head_norm.c Normal file
View File

@ -0,0 +1,531 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_head_norm.c
* @brief Implementation of the per-head reduce residencies. See pn_tt_head_norm.h — the
* layout argument that makes this file short lives there, as does the reason there
* is a SET of these keyed by width rather than one.
*
* @status None
* @issues None
* @todo None
*/
#ifdef USE_TT
#include "pn_tt_head_norm.h"
#include <stdio.h>
#include <string.h>
#include "pn_tt_device_internal.h"
#include "pn_tt_fp32_strip.h"
/**
* Which enabled Tensix each slot drives.
*
* Defaults, not constants, and each must be a core no other residency holds AND that the
* matmul group cannot grow into: members run upward from `PN_TT_MATMUL_CORE_FIRST` = 96, so
* the shipped `PN_TT_MATMUL_CORES=16` stops at 110. The rest of the map, which is why the second
* slot is not 92:
*
* 71 matmul member 0 72/73/74 reduce / eltwise / sample
* 96..110 matmul members 1..15 82/83/84 rope / memory / attention
* 85/86 render-widen / -narrow 90 gdn member 0
* 91 head norm, slot 0 92..94 gdn head-split members (4)
* 111 head norm, slot 1 112 eltwise split member
* 114..120 gdnc members
*
* SLOT 0 IS WHOEVER ARMS FIRST, not whoever needs which width. With both families up the GDN
* block's Wt=4 takes core 91 and attention's Wt=8 takes 103; with attention alone, attention
* takes 91. Nothing downstream depends on the number — routing is by WIDTH — but a probe
* reading a core map does, which is why the order is written down rather than inferred.
*
* 85 and 86 are the strip bridge's two render residencies and they are inside the range the
* matmul group grows into: at `PN_TT_MATMUL_CORES=16` members reach 90 and would collide with
* both of them and with the GDN block. The refusal below NAMES that, but the shipped count is
* 8 (members 75..81) and there is headroom to 84 only.
*
* `pn_tt_prepare_family_on_core()` refuses a collision BY NAME against every live residency,
* and that refusal is the guard — a contested core does not fail, it computes badly.
*/
#define PN_TT_CORE_HEAD_NORM_ENV "PN_TT_CORE_HEAD_NORM"
#define PN_TT_CORE_HEAD_NORM_DEFAULT 91u
#define PN_TT_CORE_HEAD_NORM_2_ENV "PN_TT_CORE_HEAD_NORM_2"
/* 111 since the PX7 placement shipped (was 103, inside the 16-member matmul's 96..110). */
#define PN_TT_CORE_HEAD_NORM_2_DEFAULT 111u
/** THE HEAD SPLIT, per slot. Members 1..N-1 of slot 0 run on 130..136 and of slot 1 on
* 123..128 — the two free blocks above every other family's default (render members 122 and
* 129 sit between them). Each slot's count is capped to its block so no member index can
* leave the grid. A call whose heads the count does not divide runs whole on member 0. */
#define PN_TT_HEAD_NORM_CORES_ENV "PN_TT_HEAD_NORM_CORES"
#define PN_TT_HEAD_NORM_CORES_DEFAULT 4u /* 0.345 -> 0.150 s, prefill +3.2%, decode +3.3% */
#define PN_TT_HEAD_NORM_MEMBER_CORE_ENV "PN_TT_HEAD_NORM_MEMBER_CORE_FIRST"
static const uint32_t g_head_norm_member_first[PN_TT_HEAD_NORM_MAX_WIDTHS] = { 130u, 123u };
static const uint32_t g_head_norm_member_room[PN_TT_HEAD_NORM_MAX_WIDTHS] = { 7u, 6u };
/** Both members every slot serves. The placement is the UNION: RMS norm carries gamma,
* L2 norm does not, and a stream the residency does not carry is placed at one page — so
* arming for the union is what lets one core run both. */
static const ft_tt_op_reduce_function g_head_norm_members[] = { FT_TT_OP_REDUCE_FN_RMS_NORM,
FT_TT_OP_REDUCE_FN_L2_NORM };
#define PN_TT_HEAD_NORM_MEMBER_COUNT \
(sizeof(g_head_norm_members) / sizeof(g_head_norm_members[0]))
/** The core environment name and default for slot `index`. */
static void pn_tt_head_norm_core_for_slot(uint32_t index, const char **out_env,
uint32_t *out_default)
{
if (index == 0u) {
*out_env = PN_TT_CORE_HEAD_NORM_ENV;
*out_default = PN_TT_CORE_HEAD_NORM_DEFAULT;
return;
}
*out_env = PN_TT_CORE_HEAD_NORM_2_ENV;
*out_default = PN_TT_CORE_HEAD_NORM_2_DEFAULT;
}
/** The live slot placed for `tile_cols`, or NULL. Const and non-const forms, because the
* routing question is asked from both. */
static const pn_tt_head_norm_residency *pn_tt_head_norm_slot_const(const pn_tt_device *device,
uint32_t tile_cols)
{
uint32_t index;
for (index = 0; index < device->head_norm_count; index++) {
if (device->head_norm[index].tile_cols == tile_cols &&
device->head_norm[index].residency.running)
return &device->head_norm[index];
}
return NULL;
}
static pn_tt_head_norm_residency *pn_tt_head_norm_slot(pn_tt_device *device, uint32_t tile_cols)
{
return (pn_tt_head_norm_residency *)pn_tt_head_norm_slot_const(device, tile_cols);
}
/** Which reduce PROGRAM serves a page size. The two are the same choice: `reduce_fp32` IS
* the `TILE_BYTES=4096` build of the same three kernel sources. */
static const char *pn_tt_head_norm_program_for(uint32_t tile_bytes)
{
return tile_bytes == PN_TT_TILE_FP32 ? PN_TT_REDUCE_FP32_PROGRAM_NAME
: PN_TT_REDUCE_PROGRAM_NAME;
}
/** Arm, or re-arm after a refusal, the slot at `index` for the width and page it records. */
static bool pn_tt_arm_head_norm_slot(pn_tt_device *device, uint32_t index)
{
pn_tt_head_norm_residency *slot = &device->head_norm[index];
ft_tt_op_reduce_residency residency;
ft_tt_launch_cb_entry placed[FT_TT_OP_REDUCE_CB_COUNT];
const char *core_env;
uint32_t core_default;
pn_tt_head_norm_core_for_slot(index, &core_env, &core_default);
if (ft_tt_op_reduce_residency_for(g_head_norm_members, PN_TT_HEAD_NORM_MEMBER_COUNT,
slot->tile_cols, slot->tile_bytes, &residency) !=
FT_TT_OK) {
fprintf(stderr, "pn_tt: no reduce residency serves the head width Wt=%u — those "
"per-head norms stay on the host\n", slot->tile_cols);
return false;
}
if (!pn_tt_prepare_family(device, &slot->residency,
pn_tt_head_norm_program_for(slot->tile_bytes), "head-norm",
core_env, core_default, PN_TT_ONCARD_TRIAD_MASK))
return false;
if (ft_tt_op_reduce_place_cbs(&residency, placed, slot->residency.cb_buffer,
sizeof(slot->residency.cb_buffer),
&slot->residency.cb_mask) != FT_TT_OK) {
fprintf(stderr, "pn_tt: the per-head reduce circular buffers do not fit at Wt=%u\n",
slot->tile_cols);
pn_tt_residency_release(&slot->residency);
return false;
}
return pn_tt_start_family(&slot->residency);
}
/** Arm head-split member `member` (1-based) of slot `index` on `core`. */
static bool pn_tt_arm_head_norm_member(pn_tt_device *device, uint32_t index, uint32_t member,
uint32_t core)
{
pn_tt_head_norm_residency *slot = &device->head_norm[index];
pn_tt_residency *residency = &slot->member[member - 1u];
ft_tt_op_reduce_residency shape;
ft_tt_launch_cb_entry placed[FT_TT_OP_REDUCE_CB_COUNT];
if (ft_tt_op_reduce_residency_for(g_head_norm_members, PN_TT_HEAD_NORM_MEMBER_COUNT,
slot->tile_cols, slot->tile_bytes, &shape) != FT_TT_OK)
return false;
if (!pn_tt_prepare_family_on_core(device, residency,
pn_tt_head_norm_program_for(slot->tile_bytes),
"head-norm-member", PN_TT_HEAD_NORM_MEMBER_CORE_ENV,
core, PN_TT_ONCARD_TRIAD_MASK))
return false;
if (ft_tt_op_reduce_place_cbs(&shape, placed, residency->cb_buffer,
sizeof(residency->cb_buffer), &residency->cb_mask) !=
FT_TT_OK) {
pn_tt_residency_release(residency);
return false;
}
return pn_tt_start_family(residency);
}
/** Arm slot `index`'s head-split members; a member that will not come up shrinks the split. */
static void pn_tt_arm_head_norm_members(pn_tt_device *device, uint32_t index)
{
pn_tt_head_norm_residency *slot = &device->head_norm[index];
const uint32_t asked = pn_tt_env_u32(PN_TT_HEAD_NORM_CORES_ENV,
PN_TT_HEAD_NORM_CORES_DEFAULT);
const uint32_t first = g_head_norm_member_first[index];
const uint32_t room = g_head_norm_member_room[index] + 1u;
uint32_t want = asked < 1u ? 1u : (asked > room ? room : asked);
uint32_t up = 1u, member;
slot->members = 1u;
for (member = 1u; member < want; member++) {
if (!pn_tt_arm_head_norm_member(device, index, member, first + member - 1u))
break;
up++;
}
slot->members = up;
if (asked > 1u)
fprintf(stderr, "pn_tt: head-norm Wt=%u HEAD SPLIT — asked %u, running %u member(s) "
"from core %u (members 1.. on %u..)\n",
slot->tile_cols, asked, up, slot->residency.core_index, first);
}
/** Retire and re-arm member `member` (1-based) of slot `index` on its own core. */
static bool pn_tt_head_norm_member_rearm(pn_tt_device *device, uint32_t index, uint32_t member)
{
pn_tt_residency *residency = &device->head_norm[index].member[member - 1u];
const uint32_t core = residency->core_index;
fprintf(stderr, "pn_tt: head-norm member %u refused a descriptor — retiring it by ASKING "
"and arming a fresh one on core %u\n", member, core);
pn_tt_residency_stop(residency);
pn_tt_residency_release(residency);
device->oncard_restarts++;
if (!pn_tt_residency_retired(residency))
return false;
return pn_tt_arm_head_norm_member(device, index, member, core);
}
int pn_tt_device_arm_head_norm(pn_tt_device *device, uint32_t head_width)
{
return pn_tt_device_arm_head_norm_at(device, head_width, PN_TT_TILE_FP32);
}
int pn_tt_device_arm_head_norm_at(pn_tt_device *device, uint32_t head_width,
uint32_t tile_bytes)
{
const char *program = pn_tt_head_norm_program_for(tile_bytes);
uint32_t tile_cols;
uint32_t index;
if (!device || head_width == 0u || (head_width % PN_TT_TILE_DIM) != 0u)
return -1;
if (tile_bytes != PN_TT_TILE_FP32 && tile_bytes != PN_TT_TILE_BF16)
return -1;
tile_cols = head_width / PN_TT_TILE_DIM;
/* Idempotent per width: the second GDN layer to ask must not take a second core. A width
* already placed at ANOTHER page is a refusal rather than a second slot, because the
* routing downstream is by width and two slots at one width would be a coin toss. */
if (pn_tt_head_norm_slot(device, tile_cols) != NULL) {
if (pn_tt_head_norm_slot(device, tile_cols)->tile_bytes != tile_bytes) {
fprintf(stderr, "pn_tt: Wt=%u is already placed at a %u-byte page and was asked "
"for at %u — one width, one residency\n", tile_cols,
pn_tt_head_norm_slot(device, tile_cols)->tile_bytes, tile_bytes);
return -1;
}
return 0;
}
if (device->head_norm_count >= PN_TT_HEAD_NORM_MAX_WIDTHS) {
fprintf(stderr, "pn_tt: %u per-head reduce residencies are already placed (Wt",
device->head_norm_count);
for (index = 0; index < device->head_norm_count; index++)
fprintf(stderr, " %u", device->head_norm[index].tile_cols);
fprintf(stderr, ") — the Wt=%u norms stay on the host. Raise "
"PN_TT_HEAD_NORM_MAX_WIDTHS.\n", tile_cols);
return 0;
}
if (!pn_tt_program_is_present(program)) {
fprintf(stderr, "pn_tt: no `%s` program — the Wt=%u per-head norms stay on the "
"host\n", program, tile_cols);
return 0;
}
index = device->head_norm_count;
device->head_norm[index].tile_cols = tile_cols;
device->head_norm[index].tile_bytes = tile_bytes;
if (!pn_tt_arm_head_norm_slot(device, index)) {
device->head_norm[index].tile_cols = 0u;
device->head_norm[index].tile_bytes = 0u;
return 0;
}
device->head_norm_count++;
pn_tt_arm_head_norm_members(device, index);
return 0;
}
bool pn_tt_head_norm_is_armed(const pn_tt_device *device, uint32_t head_width)
{
if (!device)
return false;
if (head_width == 0u)
return device->head_norm_count != 0u &&
device->head_norm[0].residency.running;
if ((head_width % PN_TT_TILE_DIM) != 0u)
return false;
return pn_tt_head_norm_slot_const(device, head_width / PN_TT_TILE_DIM) != NULL;
}
int pn_tt_head_norm_constants_upload(pn_tt_device *device, ft_tt_op_reduce_function function,
const float *gamma, uint32_t width, float eps,
pn_tt_norm_constants *out_constants)
{
uint32_t page;
if (!device || !out_constants || width == 0u || (width % PN_TT_TILE_DIM) != 0u)
return -1;
if (pn_tt_head_norm_slot(device, width / PN_TT_TILE_DIM) == NULL) {
fprintf(stderr, "pn_tt: head-norm constants at Wt=%u, which no per-head reduce "
"residency was placed for\n", width / PN_TT_TILE_DIM);
return -1;
}
/* RED PROOF, on the card: build a Float32 residency's constants at the bf16 page under a
* program placed at 4096. Nothing refuses it — the reader simply reads 4096 bytes where
* 2048 were written — so the arm exists to show that the page size is load-bearing and
* that the probe can see it. `PN_TT_RED=head_norm_bf16_constants`. It is a NO-OP on a
* slot already at the bf16 page, which is stated rather than left to be discovered. */
page = pn_tt_head_norm_slot(device, width / PN_TT_TILE_DIM)->tile_bytes;
if (page == PN_TT_TILE_FP32 && pn_tt_red("head_norm_bf16_constants"))
page = PN_TT_TILE_BF16;
return pn_tt_reduce_constants_upload(device, function, gamma, width, eps, page,
out_constants);
}
/**
* THE HEAD SPLIT: member m reduces heads [m*per, (m+1)*per) — the same descriptor with
* `row_blocks = per` and the input and output bases moved by `m*per*Wt` tiles. Gamma is
* shared by every head, so the constants operand is not moved. Enqueue all, then await all;
* a member that fails is re-armed and its slice submitted alone, after every member was
* awaited (an enqueued descriptor is always awaited, or that member's cursor desynchronises).
*/
static int pn_tt_head_norm_submit_split(pn_tt_device *device, pn_tt_head_norm_residency *slot,
uint32_t index, const ft_tt_op_reduce_request *whole)
{
const uint32_t members = slot->members;
const uint32_t per = whole->row_blocks / members;
const uint32_t step = per * whole->tile_cols * slot->tile_bytes;
uint32_t words[PN_TT_HEAD_NORM_MAX_MEMBERS][FT_TT_OP_VECTOR_WORDS];
pn_tt_residency *set[PN_TT_HEAD_NORM_MAX_MEMBERS];
bool bad[PN_TT_HEAD_NORM_MAX_MEMBERS];
double started = pn_tt_now_seconds();
int failed = 0;
uint32_t m;
memset(bad, 0, sizeof(bad));
for (m = 0; m < members; m++) {
ft_tt_op_reduce_request sliced = *whole;
sliced.row_blocks = per;
sliced.input.address += m * step;
sliced.output.address += m * step;
if (ft_tt_op_reduce_parameters(&sliced, words[m]) != FT_TT_OK)
return -1;
pn_tt_vec_probe_apply(words[m]);
set[m] = m == 0u ? &slot->residency : &slot->member[m - 1u];
}
for (m = 0; m < members; m++)
if (pn_tt_residency_enqueue(set[m], FT_TT_QUEUE_OP_REDUCE, words[m], 1u) != 0)
failed = bad[m] = 1;
set[0]->seconds += pn_tt_now_seconds() - started;
for (m = 0; m < members; m++) {
const double at = pn_tt_now_seconds();
if (pn_tt_residency_await(set[m], PN_TT_OP_TIMEOUT_MS) != 0)
failed = bad[m] = 1;
set[m]->seconds += pn_tt_now_seconds() - at;
}
if (!failed)
return 0;
for (m = 0; m < members; m++) {
if (!bad[m])
continue;
if (m == 0u) {
pn_tt_residency_stop(&slot->residency);
pn_tt_residency_release(&slot->residency);
device->oncard_restarts++;
if (!pn_tt_arm_head_norm_slot(device, index))
return -1;
} else if (!pn_tt_head_norm_member_rearm(device, index, m)) {
return -1;
}
if (pn_tt_residency_submit(set[m], FT_TT_QUEUE_OP_REDUCE, words[m],
PN_TT_OP_TIMEOUT_MS) != 0)
return -1;
}
return 0;
}
/**
* The whole of a per-head reduce, over operands the caller has already described.
*
* Taking OPERANDS rather than a strip type is what lets one body serve both pages: an fp32
* strip and a bf16 activation differ in the page size and in nothing else this function
* needs, and the page it must agree with is the SLOT's — checked here, because a 2048-byte
* operand under a 4096-byte program is the failure with no card-side refusal anywhere.
*/
static int pn_tt_head_norm_submit(pn_tt_device *device, ft_tt_op_reduce_function function,
const pn_tt_norm_constants *constants, uint32_t heads,
ft_tt_op_operand input, ft_tt_op_operand output,
uint32_t input_tiles, uint32_t output_tiles)
{
ft_tt_op_reduce_request request;
uint32_t words[FT_TT_OP_VECTOR_WORDS];
pn_tt_head_norm_residency *slot;
uint32_t tiles;
uint32_t index;
double started;
if (!device || !constants || heads == 0u)
return -1;
/* THE CONSTANTS CHOOSE THE CORE. They were built for one width and carry it, so routing
* on them is what makes "the wrong residency" unrepresentable rather than merely
* unlikely — a reduce at another width would read this buffer's gamma at the wrong
* stride and return a smoothly wrong norm with nothing refusing. */
slot = pn_tt_head_norm_slot(device, constants->tile_cols);
if (!slot) {
fprintf(stderr, "pn_tt: no per-head reduce residency is placed for Wt=%u\n",
constants->tile_cols);
return -1;
}
/* Chained projections are awaited HERE — pn_tt_device.h, pn_tt_matvec_enqueue. */
if (pn_tt_matvec_flush(device) != 0)
return -1;
tiles = heads * constants->tile_cols;
if (input_tiles < tiles || output_tiles < tiles) {
fprintf(stderr, "pn_tt: a %u-head norm at Wt=%u needs %u tiles; the strips hold %u "
"and %u\n", heads, constants->tile_cols, tiles, input_tiles,
output_tiles);
return -1;
}
if (input.tile_bytes != slot->tile_bytes || output.tile_bytes != slot->tile_bytes) {
fprintf(stderr, "pn_tt: a Wt=%u head norm was handed %u/%u-byte operands under a "
"%u-byte program — the page size is a three-sided contract with no "
"card-side check\n", constants->tile_cols, input.tile_bytes,
output.tile_bytes, slot->tile_bytes);
return -1;
}
memset(&request, 0, sizeof(request));
request.function = function;
/* ONE ROW BLOCK PER HEAD. The reader reads row block `b` at tiles `b*tile_cols + wt`,
* so over a [1, heads*W] strip block `h` is head `h`'s W elements — all of them in
* ROW 0, where the projection that produced them left them. pn_tt_head_norm.h has the
* argument; this line is the whole of the implementation of it. */
request.row_blocks = heads;
request.tile_cols = constants->tile_cols;
request.prefetch_tiles = constants->tile_cols;
request.input = input;
request.constants.x = (uint16_t)device->dram[constants->bank].x;
request.constants.y = (uint16_t)device->dram[constants->bank].y;
request.constants.address = constants->address;
request.constants.tile_bytes = slot->tile_bytes;
request.output = output;
if (ft_tt_op_reduce_parameters(&request, words) != FT_TT_OK)
return -1;
pn_tt_vec_probe_apply(words);
started = pn_tt_now_seconds();
index = (uint32_t)(slot - &device->head_norm[0]);
if (slot->members > 1u && (heads % slot->members) == 0u) {
if (pn_tt_head_norm_submit_split(device, slot, index, &request) != 0)
return -1;
pn_tt_oncard_account(device, "head-norm",
function == FT_TT_OP_REDUCE_FN_L2_NORM ? "l2_norm" : "rms_norm",
slot->tile_cols, pn_tt_now_seconds() - started);
return 0;
}
if (pn_tt_residency_submit(&slot->residency, FT_TT_QUEUE_OP_REDUCE, words,
PN_TT_OP_TIMEOUT_MS) != 0) {
/* The words address DRAM only, so the same descriptor is valid for a fresh
* residency — the re-arm argument pn_tt_oncard.c makes for the other three. */
fprintf(stderr, "pn_tt: the Wt=%u head-norm residency refused a descriptor — "
"retiring it by ASKING and arming a fresh one on core %u\n",
slot->tile_cols, slot->residency.core_index);
pn_tt_residency_stop(&slot->residency);
pn_tt_residency_release(&slot->residency);
device->oncard_restarts++;
if (!pn_tt_arm_head_norm_slot(device, index) ||
pn_tt_residency_submit(&slot->residency, FT_TT_QUEUE_OP_REDUCE, words,
PN_TT_OP_TIMEOUT_MS) != 0)
return -1;
}
pn_tt_oncard_account(device, "head-norm",
function == FT_TT_OP_REDUCE_FN_L2_NORM ? "l2_norm" : "rms_norm",
slot->tile_cols, pn_tt_now_seconds() - started);
return 0;
}
int pn_tt_head_norm(pn_tt_device *device, ft_tt_op_reduce_function function,
const pn_tt_norm_constants *constants, uint32_t heads,
const pn_tt_fp32_strip *input, const pn_tt_fp32_strip *output)
{
if (!device || !input || !output)
return -1;
return pn_tt_head_norm_submit(device, function, constants, heads,
pn_tt_fp32_strip_operand(device, input),
pn_tt_fp32_strip_operand(device, output),
input->tile_count, output->tile_count);
}
int pn_tt_head_norm_bf16(pn_tt_device *device, ft_tt_op_reduce_function function,
const pn_tt_norm_constants *constants, uint32_t heads,
const pn_tt_activation *input, const pn_tt_activation *output)
{
if (!device || !input || !output)
return -1;
return pn_tt_head_norm_submit(device, function, constants, heads,
pn_tt_operand(device, input),
pn_tt_operand(device, output), input->tile_count,
output->tile_count);
}
int pn_tt_head_norm_operands(pn_tt_device *device, ft_tt_op_reduce_function function,
const pn_tt_norm_constants *constants, uint32_t heads,
ft_tt_op_operand input, ft_tt_op_operand output,
uint32_t input_tiles, uint32_t output_tiles)
{
return pn_tt_head_norm_submit(device, function, constants, heads, input, output,
input_tiles, output_tiles);
}
void pn_tt_head_norm_defer(pn_tt_device *device, uint32_t tile_cols, bool on)
{
pn_tt_head_norm_residency *slot = device ? pn_tt_head_norm_slot(device, tile_cols) : NULL;
if (!slot)
return;
slot->residency.defer_await = on;
for (uint32_t member = 1u; member < slot->members; member++)
slot->member[member - 1u].defer_await = on;
}
int pn_tt_head_norm_settle(pn_tt_device *device, uint32_t tile_cols)
{
pn_tt_head_norm_residency *slot = device ? pn_tt_head_norm_slot(device, tile_cols) : NULL;
int rc;
if (!slot)
return 0;
rc = pn_tt_residency_settle(&slot->residency, PN_TT_OP_TIMEOUT_MS);
for (uint32_t member = 1u; member < slot->members; member++)
if (pn_tt_residency_settle(&slot->member[member - 1u], PN_TT_OP_TIMEOUT_MS) != 0)
rc = -1;
return rc;
}
#endif /* USE_TT */

288
src/accel/tt/pn_tt_head_norm.h Normal file
View File

@ -0,0 +1,288 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_head_norm.h
* @brief PER-HEAD norms on EXTRA reduce residencies placed for the HEAD width — the norms
* the Gated-Delta-Net block and full attention have and the model's RMS norms do not.
*
* Model-agnostic: it knows a head count and a head width. It does not know what a
* Qwen3.5 layer is.
*
* ---------------------------------------------------------------------------
* ONE RESIDENCY PER WIDTH, AND A HYBRID MODEL NEEDS TWO
* ---------------------------------------------------------------------------
* A reduce residency is placed for ONE Wt and refuses any other (next section), so "the head
* width" is not one number for a GDN+attention hybrid. Ornith:
*
* GDN per-head norms S = 128 Wt 4 PN_TT_CORE_HEAD_NORM default 91
* attention QK-norms head_dim = 256 Wt 8 PN_TT_CORE_HEAD_NORM_2 default 103
* the model's own RMS n_embd = 4096 Wt 128 the `reduce` residency default 72
*
* So this is a SET keyed by width, not a singleton. `pn_tt_device_arm_head_norm()` is
* idempotent per width and takes one more Tensix per new one; every other entry point routes
* on the width, and the width comes from the CONSTANTS the caller passes rather than from an
* argument — a constants buffer is built FOR one width, and a reduce run at another width
* would read the wrong gamma out of it and return a smoothly wrong norm.
*
* ---------------------------------------------------------------------------
* WHY A SEPARATE RESIDENCY, AND WHY THAT IS THE WHOLE OF THE COST
* ---------------------------------------------------------------------------
* A reduce residency is PLACED for one Wt: `ft_tt_op_reduce_place_cbs()` sizes CB_IN,
* CB_MID, CB_GAMMA and CB_BETA at `max_tile_cols` pages, and `pn_tt_rms_norm()` refuses a
* descriptor whose Wt is not the one it was placed for. The model's norms are
* `n_embd/32 = 128` tile columns; the GDN's per-head norms are `S/32 = 4`. So they get
* their own core, and there are 140.
*
* The two widths are not merely different, they must not SHARE: `CB_GAMMA` is
* `max_tile_cols` pages and every descriptor pushes `tile_cols` of them in ONE go, so a
* Wt=4 descriptor leaves the write pointer at page 4 of 128 and the next Wt=128 descriptor
* straddles the limit. `ft_tensix_cb_push_grid_is_safe()` catches it and REFUSES — a
* re-arm, not a wrong answer — but the refusal would land on one descriptor in two.
*
* ---------------------------------------------------------------------------
* THE LAYOUT RESULT THIS FILE EXISTS TO USE, AND IT COST NOTHING
* ---------------------------------------------------------------------------
* **A per-head norm over a ROW-VECTOR activation strip needs no reshape at all.** The
* scoping for this item assumed the head norms wanted a `[heads, S]` matrix — one row per
* head — which every tensor around them is not: `pn_tt_activation_alloc()` strips are
* `[1, 32*tile_count]`, row 0 of each tile and 31 rows of padding.
*
* `resident_reduce_reader.c` reads row block `b` at tiles `b*tile_cols + wt`, consecutive,
* and `row_blocks = tile_count / tile_cols`. So over a `[1, heads*S]` strip with
* `tile_cols = S/32`:
*
* row block h == tiles [h*S/32, +S/32) == head h's S elements, all in ROW 0
*
* and each row block is reduced PER ROW. One descriptor, `heads` independent norms, the
* data exactly where the projection that produced it left it. The `[heads, S]` matrix
* layout would have needed a transpose of 32-element runs out of row 0 of `4*heads` tiles;
* this needs nothing.
*
* The consequence is that `gamma` is shared across heads, which is what the GDN's output
* gated norm wants — `ssm_norm` is `S` values applied identically to every v-head — and
* the q/k L2 norm has no gamma at all.
*
* ---------------------------------------------------------------------------
* THE TWO MEMBERS
* ---------------------------------------------------------------------------
* q/k L2 norm `FT_TT_OP_REDUCE_FN_L2_NORM`, no gamma, eps passed as `eps*eps`
* (foundry `ft_tt_op_vector.h` has why). Pantheon's oracle is
* `pn_qwen35moe_l2_norm()`.
* output gated `FT_TT_OP_REDUCE_FN_RMS_NORM` with gamma = `ssm_norm`, then
* `pn_tt_silu_mul(z, normed)` on the ELEMENTWISE residency. Pantheon's
* oracle is `pn_qwen35moe_gated_norm()`, which is exactly those two steps.
* attention QK `FT_TT_OP_REDUCE_FN_RMS_NORM` with gamma = `attn_q_norm` / `attn_k_norm`,
* at the ATTENTION head width, on the second slot. Pantheon's oracle is
* `qwen35moe_qk_norm_heads()` in `qwen35moe_attn.c`, which is
* `pn_vrms_norm_f32` per head with one shared weight — the same "gamma is
* shared across heads" the row-block layout gives for free.
*
* **THE QK MEMBER IS THE RMS MEMBER, AND THE RMS MEMBER IS THE ONE THAT MISSES THE GATE.**
* At Wt 4 it measured 3.165e-03 against 1e-3 (below), and the cause is the chain STEP COUNT
* rather than the width, so Wt 8 inherits it. That is a numerics change to attention's
* scores, not a rounding detail, and it is why on-card attention stays opt-in.
*
* ---------------------------------------------------------------------------
* THE RESIDENCY IS Float32, AND THAT IS WHY IT WORKS AT ALL
* ---------------------------------------------------------------------------
* The FIRST arming of this residency ran the shipped bf16 `reduce` program and MISSED THE
* GATE BY 9x: `PN_TT_GDN_NORMCHECK=1` on Ornith's real per-layer data scored the q/k L2 norm
* at **8.599e-03** max-norm relative per head and the output gated norm at **1.283e-02**
* against 1e-3, with 146,844 of 196,608 q/k elements out.
*
* **The error is per REDUCE, so it is the narrow norms that cannot afford it.** The model's
* own RMS norms average the same per-buffer rounding over 4096 elements and pass; a per-head
* norm has 128. And the lever is the CB TABLE and not the activation strip: priced apart
* off-hardware in `foundry/tt/tests/test_vector_ops.c` on this exact shape, the bf16 REDUCE
* alone is 6.126e-03 while rendering an fp32 operand into a bf16 strip adds only 1.0e-03, so
* an fp32 strip in front of a bf16 reduce would have bought nothing.
*
* So this residency arms `reduce_fp32` — `reduce_float32.cbtable` at `dst_accum_mode 1`, a
* `TILE_BYTES=4096` build of the SAME three kernel sources — and everything it touches is
* fp32 by consequence: `pn_tt_fp32_strip` in and out, and a constants buffer built at
* `PN_TT_TILE_FP32`. There is no fallback to the bf16 program: at this width that would be a
* wrong answer rather than a slow one.
*
* WHAT THAT DOES NOT BUY, AND IT IS NOT ENOUGH FOR ONE OF THE TWO MEMBERS. A Float32 CB is
* not fp32 arithmetic: the generated `chlkc_descriptors.h` for this program carries
* `unpack_src_format = 0` (Float32) beside `unpack_dst_format = 4`, and 4 is **Tf32** — E8M10,
* ten stored mantissa bits, TRUNCATED rather than rounded, ~4.9e-4 an operand and biased low.
* `math_fidelity` cannot recover it; the bits are gone before the FPU sees the operand.
*
* Modelled off-hardware at this shape (`foundry/tt/tests/test_vector_ops.c`) the two members
* land on OPPOSITE SIDES of the gate:
*
* q/k L2 bf16 7.143e-03 -> Float32 8.406e-04 CLEARS 1e-3
* RMS bf16 8.101e-03 -> Float32 1.566e-03 DOES NOT
*
* **The difference is the CHAIN LENGTH, not the member.** The L2 member's final chain is one
* step; the RMS member's is two — gamma, then scale — and every chain step is a circular-
* buffer round trip, so every step is one more Tf32 unpack of the value. The same program run
* with an fp32 unpack the hardware does not have scores 1.0e-07 and 2.1e-07, so the ENTIRE
* Float32 residue is truncation: no CB table moves it and the next lever is the chain's step
* count, which is `resident_reduce_compute.c` and not a table.
*
* ---------------------------------------------------------------------------
* WHAT IT MEASURED ON SILICON, AND NEITHER NORM CLEARS
* ---------------------------------------------------------------------------
* `PN_TT_GDN_NORMCHECK=1` on Ornith's real per-layer data, 48 SSM layers, twice, identical
* both times, 0 refusals:
*
* bf16 `reduce` Float32 `reduce_fp32` gate
* q/k L2 8.599e-03 1.737e-03 1e-3 4.95x better, 1.7x out
* RMS alone - 3.165e-03 1e-3 3.2x out
* gated 1.283e-02 1.023e-02 1e-3 1.25x better, 10x out
* q/k elements out 146,844/196,608 25,718/196,608 5.7x fewer
*
* **The Float32 program is a real 4.95x on the q/k norm and it is not enough.** Three things
* are named by those three rows and they are different problems:
*
* 1. **Tf32 is the floor on the reduce**, and the modelled 8.406e-04 / 1.566e-03 land about
* 2x optimistic against the card. The model uses libm for the `rsqrt`; the card uses the
* SFPU's polynomial, which is a SECOND approximation the model cannot have and the
* leading candidate for that 2x. At bf16 it was invisible under a rounding 8x bigger.
* 2. **The RMS member pays two more Tf32 unpacks than the L2 member** — 3.165e-03 against
* 1.737e-03 — because its chain has two steps and every step is a CB round trip.
* 3. **`gated` barely moved because the reduce is not what bounds it.** It is `silu(z) *`
* the RMS answer on the **bf16** ELEMENTWISE residency, so the fp32 answer is rendered
* into a bf16 strip before the gate sees it. Fixing the reduce cannot help `gated` until
* that bridge goes, which is the qkv/z strip item and not this one.
*
* @status RUN ON SILICON 2026-08-19, p150a, core 91 — armed, 96 descriptors over two
* tokens, 0 refusals, 0 restarts, core `instrn_buf_status` 0x77 after. **THE LAYOUT
* RESULT ABOVE IS CONFIRMED**: `row_blocks = heads` over a `[1, heads*S]` strip
* scores 8.6e-03 against the oracle, not the O(1) a wrong axis would give.
* @issues None
* @todo None
*/
#ifndef PN_TT_HEAD_NORM_H
#define PN_TT_HEAD_NORM_H
#ifdef USE_TT
#include <stdint.h>
#include "pn_tt_device.h"
#include "pn_tt_fp32_strip.h"
#include "tt/ft_tt_op_vector.h"
/**
* Arm a per-head reduce residency (one more core) placed for `head_width`.
*
* IDEMPOTENT PER WIDTH: asking twice for the same width is a no-op, and asking for a second
* width takes a second core. A model with two head widths calls this twice.
*
* Never fatal: a card without the `reduce_fp32` program, a core already held, or no slot
* left leaves that width's per-head norms on the host and
* `pn_tt_head_norm_is_armed(device, width)` false.
*
* @param head_width the head width in ELEMENTS, a multiple of 32
* @return 0 (even when nothing armed), -1 on a bad argument
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
int pn_tt_device_arm_head_norm(pn_tt_device *device, uint32_t head_width);
/**
* The same, naming the PAGE SIZE — which is also which reduce program arms.
*
* `FT_TT_OP_TILE_BYTES_FP32` is `reduce_fp32` and what `pn_tt_device_arm_head_norm()` asks
* for; `FT_TT_OP_TILE_BYTES_BF16` is the shipped `reduce`.
*
* **bf16 IS THE RIGHT ANSWER FOR ATTENTION'S QK-NORM AND THE WRONG ONE FOR THE GDN'S.** The
* GDN norms sit in the middle of an fp32 recurrence, so the Tf32 floor is the best available
* and bf16 misses the 1e-3 gate by 9x. Flash decode is bf16 END TO END by construction — the
* KV cache is bf16 (`ft_tt_op_memory.h` §1.6), the constants are bf16, the RoPE table is
* bf16 — so a Float32 QK-norm would need a bf16->fp32 render before it and an fp32->bf16
* render after, NEITHER OF WHICH EXISTS on the card, and the answer would then be rounded
* straight back to bf16 by the very next op. It would buy precision the next descriptor
* discards, at the cost of the strip bridge the GDN norms are already blocked on.
*
* A width already placed at the other page is a REFUSAL, not a second slot: routing is by
* width, and two slots at one width would be a coin toss.
*
* @param tile_bytes `FT_TT_OP_TILE_BYTES_FP32` or `FT_TT_OP_TILE_BYTES_BF16` — foundry's
* own names, because the page size is foundry's contract and a second
* spelling of it here would be a second thing to keep in step
* @return 0 (even when nothing armed), -1 on a bad argument or a page clash
*/
int pn_tt_device_arm_head_norm_at(pn_tt_device *device, uint32_t head_width,
uint32_t tile_bytes);
/** Did a per-head reduce residency for THIS width arm for this run?
* `head_width` in ELEMENTS; 0 asks whether ANY did. */
bool pn_tt_head_norm_is_armed(const pn_tt_device *device, uint32_t head_width);
/**
* Upload one member's constants buffer: the scaler tile, the eps tile, then gamma expanded
* to full tiles when the member takes one.
*
* @param function which member the constants are FOR — the scaler and the eps tile both
* depend on it, so a buffer built for one member is wrong for another
* @param gamma `width` values, or NULL for a member with no gamma
* @param width the head width in ELEMENTS, which must be the armed one
* @param eps what the member's reduce should see added; **L2 norm takes `eps*eps`**
* @return 0, or -1 with the reason on stderr
*/
int pn_tt_head_norm_constants_upload(pn_tt_device *device, ft_tt_op_reduce_function function,
const float *gamma, uint32_t width, float eps,
pn_tt_norm_constants *out_constants);
/**
* One per-head norm over a row-vector strip: `heads` independent reduces of `width`
* elements each, in ONE descriptor.
*
* `input` and `output` are **Float32** strips — this residency is placed by
* `reduce_float32.cbtable` and its reader issues 4096-byte reads. Each must hold at least
* `heads * width / 32` tiles, and they MAY be the same strip — the reader has read a row block before the writer can write it, and the
* blocks do not alias.
*
* Which residency runs it is chosen by `constants->tile_cols`, so a caller cannot point a
* width at the wrong core: the constants and the placement are the same width by
* construction, and a width nothing armed is a refusal rather than a wrong reduce.
*
* @param heads row blocks, i.e. how many heads this descriptor norms
* @return 0, or -1 with the reason on stderr
*/
int pn_tt_head_norm(pn_tt_device *device, ft_tt_op_reduce_function function,
const pn_tt_norm_constants *constants, uint32_t heads,
const pn_tt_fp32_strip *input, const pn_tt_fp32_strip *output);
/** The same over ordinary bf16 activation strips, for a slot armed at `PN_TT_TILE_BF16`.
* Handing either entry point the other page's operands is REFUSED — the page size is a
* three-sided contract and the card cannot see a violation of it. */
int pn_tt_head_norm_bf16(pn_tt_device *device, ft_tt_op_reduce_function function,
const pn_tt_norm_constants *constants, uint32_t heads,
const pn_tt_activation *input, const pn_tt_activation *output);
/**
* The same over RAW OPERANDS, for a caller whose buffers are neither of the two strip types.
*
* The GDN block's are: `conv_out` and `y` are allocated by the GDN family and addressed by its
* own `ft_tt_op_operand` accessors, and the whole point of the on-card chain is that the norms
* read them WHERE THEY LIE instead of through a host round trip. An fp32 strip and a GDN
* buffer differ in nothing this function needs except the address, and the page each must
* carry is still checked against the SLOT's.
*
* `input_tiles` / `output_tiles` are what the caller is willing to have read and written; the
* `heads * tile_cols` this descriptor actually touches must fit inside both.
*
* **INPUT AND OUTPUT MAY BE THE SAME BUFFER.** The GDN q/k norm runs IN PLACE over the first
* `2*k_heads*state_tiles` tiles of the conv output, and that is safe by pipeline order rather
* than by luck: the reader prefetches ONE row block, so block `b` has been read before the
* writer can write block `b`. Nothing here enforces it — it is stated so that an in-place
* caller is a decision rather than an accident.
*/
int pn_tt_head_norm_operands(pn_tt_device *device, ft_tt_op_reduce_function function,
const pn_tt_norm_constants *constants, uint32_t heads,
ft_tt_op_operand input, ft_tt_op_operand output,
uint32_t input_tiles, uint32_t output_tiles);
/** PM 7 ITEM 7: defer the awaits of the `tile_cols`-wide head-norm slot (`on`), and settle
* them. See pn_tt_residency_settle(). */
void pn_tt_head_norm_defer(pn_tt_device *device, uint32_t tile_cols, bool on);
int pn_tt_head_norm_settle(pn_tt_device *device, uint32_t tile_cols);
#endif /* USE_TT */
#endif /* PN_TT_HEAD_NORM_H */

114
src/accel/tt/pn_tt_issue.c Normal file
View File

@ -0,0 +1,114 @@
/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file pn_tt_issue.c
* @brief Implementation of the issue-shape selector (see pn_tt_issue.h for the whole
* argument, including the WC/UC store-ordering seam).
*
* @status Green.
* @issues None
* @todo None
*/
#ifdef USE_TT
#include "pn_tt_issue.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "pn_tt_name_list.h"
#define PN_TT_ISSUE_ENV "PN_TT_ISSUE"
/** Bytes one WORD32 access moves. The other unit is FT_TT_NOC_COPY_UNIT_BYTES. */
#define PN_TT_ISSUE_WORD_BYTES 4u
/** Bytes one BLOCK16 access moves — the same 16 the L1 alignment rule is written in. */
#define PN_TT_ISSUE_BLOCK_BYTES 16u
pn_tt_issue_shape pn_tt_issue_shape_selected(void)
{
static int cached = -1;
if (cached < 0) {
const char *value = getenv(PN_TT_ISSUE_ENV);
/* DEFAULT `wide` SINCE 2026-08-27, AND THE FLIP IS ITS OWN CONFIRMATION ROUND rather
* than an inference off the ladder that unblocked it. `wide` was a 5.7% REGRESSION
* while every issue paid a second TLB re-aim at 8.9 us against the 25 accesses' 0.8;
* with the window bank absorbing both aims it is a win, and the round that flipped it
* measured decode 0.1391 (word, three arms, all three to four decimals — a 0.00%
* control-vs-control drift) against 0.1381/0.1378 (wide): **+0.83%, -1.15 ms/token**.
* Prefill 165.19/165.27 -> 166.52 tok/s on the `batched prefill —` line's own
* denominator, **+0.80%** against a 0.048% drift, the gain in the `sequential` term.
* Every arm `cmp` IDENTICAL at 189 B / 80fc6259, and the two 256-token soaks are
* byte-identical to each other at 587 B / abca1fa9. The instrument that says the knob
* armed is the issue clock: 10.0-10.1 -> 5.8-6.1 us/op. See pn_tt_issue.h. */
cached = (int)PN_TT_ISSUE_WIDE;
if (value && *value) {
if (strcmp(value, "word") == 0)
cached = (int)PN_TT_ISSUE_WORD;
else if (strcmp(value, "wide") == 0)
cached = (int)PN_TT_ISSUE_WIDE;
else
fprintf(stderr,
"pn_tt: %s=%s is not word|wide — using the default (wide)\n",
PN_TT_ISSUE_ENV, value);
}
}
return (pn_tt_issue_shape)cached;
}
const char *pn_tt_issue_shape_string(pn_tt_issue_shape shape)
{
return shape == PN_TT_ISSUE_WIDE ? "wide" : "word";
}
pn_tt_issue_shape pn_tt_issue_shape_for(const ft_tt_runtime *runtime, const ft_tt_queue *queue)
{
if (pn_tt_issue_shape_selected() != PN_TT_ISSUE_WIDE)
return PN_TT_ISSUE_WORD;
/* A DEMOTION, NOT A FAILURE. A queue placed where a 16-byte write cannot reach it would
* otherwise refuse its first parameter block and take the whole family to the host — a
* loud failure, but a needless one when the shipped shape is right there. */
if (!runtime || !runtime->bulk_l1.write || !ft_tt_queue_issue_is_wide_addressable(queue))
return PN_TT_ISSUE_WORD;
return PN_TT_ISSUE_WIDE;
}
const ft_tt_noc_access *pn_tt_issue_access(const ft_tt_runtime *runtime,
pn_tt_issue_shape shape)
{
if (!runtime)
return NULL;
return shape == PN_TT_ISSUE_WIDE ? &runtime->bulk_l1 : &runtime->noc;
}
uint32_t pn_tt_issue_unit_bytes(pn_tt_issue_shape shape)
{
return shape == PN_TT_ISSUE_WIDE ? PN_TT_ISSUE_BLOCK_BYTES : PN_TT_ISSUE_WORD_BYTES;
}
uint64_t pn_tt_issue_accesses(pn_tt_issue_shape shape, uint32_t payload_bytes)
{
/* The head bump is ALWAYS one WORD32 access, on both shapes, because nothing else can
* carry a 4-byte write. That single word is why `wide` is 7 and not 6. */
return (uint64_t)payload_bytes / pn_tt_issue_unit_bytes(shape) + 1u;
}
bool pn_tt_issue_red_head_first(void)
{
static int cached = -1;
if (cached < 0) {
/* SELECTS, NOT CONTAINS. `pn_tt_name_list_contains()` answers TRUE for an UNSET list
* by design, so arming a red arm with it arms it on every run — measured on silicon
* 2026-08-25, and it cost a cross-agent bisect the wrong commit. */
cached = pn_tt_name_list_selects(getenv("PN_TT_RED"), "issue_head_first") ? 1 : 0;
}
return cached != 0;
}
#endif /* USE_TT */

Some files were not shown because too many files have changed in this diff Show More