/* SPDX-License-Identifier: AGPL-3.0-or-later */ /* Copyright (C) 2026 David (Corey) Herrera */ /** * @file ft_tt_runtime_udrv.h * @brief THE SEAM. The userspace-driver half of `ft_tt_runtime` — everything a * serving process needs from a card that has NO VENDOR DRIVER on the machine: * find it in sysfs, bring the ARC up, read the grid, claim two windows, and * hand back the SAME three `ft_tt_noc_access` accessors the tt-kmd path hands * back. * * --------------------------------------------------------------------------- * WHY THIS FILE IS THE WHOLE CHANGE, AND NOTHING ABOVE IT MOVES * --------------------------------------------------------------------------- * The backend above `ft_tt_runtime` — launch, queue, ops, residencies, * `ft_tt_core` — is written against `ft_tt_launch.h`'s `ft_tt_noc_access` * vtable and against `ft_tt_grid_info`. Neither mentions a device handle. So * the ONLY thing tt-kmd was supplying that could not be supplied otherwise was * the way those two get built, and that is exactly what this file replaces. * * There is deliberately no second copy of anything: * - the PIO copy shape is `ft_tt_noc_copy.h`, shared with `ft_tt_noc.c`; * - the memory-space bounds and alignment rules are * `ft_tt_noc_check_transfer()`, called here, so `bulk_dram` refuses a * register address on BOTH transports for the same reason and with the same * message; * - the telemetry table walk is `ft_tt_arc_telemetry_read_grid_via()`, shared; * - the coordinates are `ft_tt_coord`, shared. * If a file ever has to be duplicated to make this work, the cut is in the * wrong place. * * --------------------------------------------------------------------------- * IT REFUSES. IT NEVER FALLS BACK. * --------------------------------------------------------------------------- * Every failure here returns a negative code and leaves nothing open. There is * no "try the other transport" path in this file or in its caller, because this * project has shipped a silent fallback twice — the one-shot dispatch fallback * and the export 404 — and both times the symptom was a correct-looking answer * produced by the wrong thing. A serve asked for `FT_TT_DRIVER=udrv` on a * machine with no usable card must stop, not answer on the CPU. * * --------------------------------------------------------------------------- * WHAT IS LOST RELATIVE TO THE tt-kmd PATH — priced, not hidden * --------------------------------------------------------------------------- * - **No `ft_tt_ctx`.** `ft_tt_runtime.ctx` is NULL on this path, so anything * reached through it is unavailable: the DRAM allocation tally (accounting * only — `ft_tt_allocator_init()` takes a NULL ctx) and `SET_NOC_CLEANUP`, * the death-time stop write. That second one is a real loss and is stated at * `ft_tt_udrv.h` §5(d); the residency idle-spin bound is what remains. * - **No second reader of a window's registers.** Every aim is verified by * reading the registers back through the mapping that wrote them, which is a * real uncached PCIe read and catches a dropped word, a torn write or a * register that does not hold — but not a mapping that consistently points * at the wrong page. `ft_tt_udrv_noc.h` prices this exactly. * - **Window arbitration is a flock, and the flock cannot see tt-kmd's * bitmap.** While the module is loaded, a process using `ALLOCATE_TLB` and a * process using this one can pick the same window and neither is told. The * mitigation here is to claim from the TOP of the window range downward, * because the ioctl allocates from the bottom, and to reserve the window the * driver keeps for itself. It is a mitigation, not a lock. On the machine * this exists for there is no ioctl and the flock is the whole arbitrator, * which is the condition the rehearsal proved. * * @status **THE PRODUCT RUNS ON THE CARD OVER THIS PATH WITH `tenstorrent` * UNLOADED (2026-08-17).** A real `pantheon serve` — MiniCPM5-1B, * `PN_TT_SPARSE_LOGITS=1`, `PN_TT_MATMUL_CORES=4`, host attention, programs * from the embedded blob — answered a chat with `[admit] path=gpu` and * **6,760 matmuls on the card**, resident triad up on four cores, clean * shutdown. The same binary over tt-kmd returns BYTE-IDENTICAL text, which is * exactly why the text is not the gate. * * | arm | tt-kmd | path | RUN TOTAL | s/token | * |---|---|---|---|---| * | kmd (control) | loaded | gpu | 6760 | 0.106 | * | udrv | loaded | gpu | 6760 | 0.113 | * | udrv | UNLOADED | gpu | 6760 | 0.114 | * | kmd | UNLOADED | **cpu** | — | — | * * **THE 6.6% IS THE AIM READBACK AND THAT IS MEASURED.** All of the gap is in * `issue` (0.331 s / 49.0 µs per op against 0.104 / 15.3); `wait`, which is * the card's own work, is identical to 0.5%, and the aim counts agree to 10 * out of 68,071. Deleting the readback from `ft_tt_udrv_noc_window_aim()` in * a scratch build put `issue` back to 0.100 s and the token to **0.105 s — at * or below the kmd control**. So the transport costs nothing and the three * uncached PCIe register reads that verify each aim cost **3.3 µs per aim**, * 1,702 aims a token. They stay: with no vendor driver there is no second * reader, and that readback is the only thing that makes a wrong aim * detectable instead of silent. * * Everything under this file is on silicon with the module removed * (`ft_tt_udrv.h` @status): the ARC answers in A3, the bring-up sequence is * accepted, the telemetry walk is the only reader left, the flock is the only * arbitrator, and a 64 KiB L1 round trip lands. * @issues * - The bring-up sends `POWER_SETTING` asking for the maximum AI clock and * nothing lowers it at close, so a card this path owned stays at its top * clock until something asks otherwise. Visible, harmless, and tracked in * `ft_tt_udrv.h`'s @todo. * - MRRS is not set here. It lives in DEVCTL and is rewritten by the reset * path's config restore; a machine where tt-kmd has NEVER run has nobody * setting it. Same @todo. * @todo None */ #ifndef FT_TT_RUNTIME_UDRV_H #define FT_TT_RUNTIME_UDRV_H #include #include #include "ft_tt_arch.h" #include "ft_tt_coord.h" #include "ft_tt_launch.h" /** An open card over the userspace driver. Opaque: it owns mappings, claims and * paths whose lifetimes the accessors below depend on, and a caller has no * business reaching into any of them. Heap-allocated so the accessors' `user` * pointers stay valid however the caller moves its own runtime struct. */ typedef struct ft_tt_runtime_udrv ft_tt_runtime_udrv; /** * Open device `device_index` with NO vendor driver, bring it up, and build the * three accessors a serving process runs on. * * The sequence, and every step is one the rehearsal ran with `tenstorrent` * removed: * 1. find the card in `/sys/bus/pci/devices` by vendor and device id, and * select the architecture table from those ids alone; * 2. map BAR0's TLB register file, writable; * 3. claim two 2 MiB windows through `ft_tt_udrv_claim` — one UC for run * control and the ARC, one WC for bulk data; * 4. bind an ARC transport to the UC path and run * `ft_tt_udrv_arc_probe_hardware()`: `ASIC_STATE0`, `SET_WDT_TIMEOUT`, * `POWER_SETTING`. This is what `tt_cdev_open()` was doing for us; * 5. walk the ARC telemetry table over that same path for the grid, and * refuse a topology `ft_tt_coord` cannot map — a coordinate we cannot * derive is a coordinate we must not probe (spec §7.3); * 6. fill the vtables. * * @param device_index which Blackhole, in sysfs enumeration order * @param out_udrv receives the handle; release with * ft_tt_runtime_udrv_close() * @param out_grid the live topology * @param out_noc UC / WORD32 — registers, mailbox, images. NEVER bulk * @param out_bulk_dram BLOCK16 over the WC path, DRAM addresses only * @param out_bulk_l1 BLOCK16 over the WC path, L1 addresses only * @return FT_TT_OK, or a negative FT_TT_* code with the reason logged. On any * failure nothing is left open and no output is written. * * Thread-safe: no Reentrant: no Signal-safe: no */ int ft_tt_runtime_udrv_open(int device_index, ft_tt_runtime_udrv **out_udrv, ft_tt_grid_info *out_grid, ft_tt_noc_access *out_noc, ft_tt_noc_access *out_bulk_dram, ft_tt_noc_access *out_bulk_l1); /** Release every window claim, unmap every aperture and free the handle. NULL * is a no-op. The card is left in A0 at whatever clock the bring-up asked for * — see @issues. */ void ft_tt_runtime_udrv_close(ft_tt_runtime_udrv *udrv); /** MEASURE: read ARC telemetry tags (reads only, no ARC message) — see * ft_tt_arc_telemetry_read_tags_via(). */ int ft_tt_runtime_udrv_read_telemetry(ft_tt_runtime_udrv *udrv, const uint16_t *tags, unsigned count, uint32_t *out_values, bool *out_found); /** How many reader banks a runtime may hold (one per reader thread). */ #define FT_TT_UDRV_READER_BANKS_MAX 4u /** * PHASE W: DRAM accessors over READER bank `index` — extra WC window banks * (`FT_TT_UDRV_READER_BANKS`, default 2, of `FT_TT_UDRV_READER_WINDOWS` windows each, * default 2, 0 = none), each for exactly ONE thread other than the one driving the control * and bulk banks. The prefill's alpha/beta readback runs there so it stops serialising with * descriptor issue on the main thread. * * @return FT_TT_OK; FT_TT_EUNAVAIL when that bank was not opened; FT_TT_EINVAL * * Thread-safe: the returned accessor may be used concurrently with the runtime's other * accessors, by ONE thread at a time. */ int ft_tt_runtime_udrv_reader_access(ft_tt_runtime_udrv *udrv, uint32_t index, ft_tt_noc_access *out_dram); /** The same reader bank, L1 addresses (BLOCK16): the reader polls a queue's completion word * there before reading the strip that queue's descriptor writes (PHASE W wavefront). */ int ft_tt_runtime_udrv_reader_l1_access(ft_tt_runtime_udrv *udrv, uint32_t index, ft_tt_noc_access *out_l1); /** The card's PCI address, e.g. `0000:12:00.0`. Never NULL for an open handle; * the empty string for NULL. This is what `ft_tt_device_get_info()` answers on * the tt-kmd path, read from sysfs instead of from an ioctl. */ const char *ft_tt_runtime_udrv_pci_address(const ft_tt_runtime_udrv *udrv); /** * What each of a runtime's two windows carried. * * Declared HERE, in the udrv header, only because `ft_tt_runtime.h` includes * this file and the reverse would be a cycle. It describes both transports. * * **`aims` is the number that separates them.** Both paths write the same three * registers to move a window; the tt-kmd path then returns from an ioctl, while * this one READS ALL FOUR REGISTERS BACK and refuses the aperture on any * disagreement — a syscall can report an error and three stores through an mmap * cannot. Those readbacks are uncached PCIe reads on a Gen2 x1 link, so the aim * count is what turns a per-aim cost into a per-token one, and without it a * difference between the transports can only be guessed at. */ typedef struct ft_tt_runtime_window_stats { uint64_t register_read; /**< bytes card->host through the UC window */ uint64_t register_written; /**< bytes host->card through the UC window */ uint64_t register_aims; /**< times the UC window was re-aimed */ uint64_t bulk_read; uint64_t bulk_written; uint64_t bulk_aims; uint64_t aim_failures; /**< aims whose readback disagreed; udrv only, must be 0 */ /** WHAT THOSE AIMS COST, over BOTH windows, and the readback's share. * * The aim count above turns a per-aim cost into a per-token one only if the per-aim cost * is known, and it never was — it was quoted at 1.23 us from the tt-kmd ioctl path, which * is a different mechanism. These are the udrv path's own clocks. ZERO on tt-kmd, where * the aim is an ioctl and there is no readback to separate. */ double aim_seconds; double aim_verify_seconds; /** THE WINDOW BANK, AND WHY EVERY ONE OF THESE IS ON THE REPORT LINE. * * Each kind of window is a SET now, and the aim count above only falls if the set is * actually absorbing the endpoint rotation. `hits`/`misses` is that mechanism measured * directly rather than inferred from the aim count, and `evictions` separates "the set is * too small" from "the endpoint had never been seen". `misaims_injected` is the red arm's * own counter — an arm that never fired must read as a zero and not as a silence — and * `audit_failures` must be zero on every arm that is not the red one. ZERO on tt-kmd. */ uint32_t register_windows; /**< windows the UC bank actually claimed */ uint32_t bulk_windows; /**< windows the WC bank actually claimed */ /* PER BANK, NOT POOLED. The two streams are different mechanisms and pooling * them hides the one that matters: the WC bank's weight upload walks * thousands of consecutive slices inside one 2 MiB window, so it reads ~99% * hit however badly the dispatch path is thrashing. */ uint64_t register_hits; uint64_t register_misses; uint64_t register_evictions; uint64_t bulk_hits; uint64_t bulk_misses; uint64_t bulk_evictions; uint64_t misaims_injected; /**< FT_TT_UDRV_RED=misaim's own count */ uint64_t audit_failures; /**< FT_TT_UDRV_AIM_AUDIT refusals; must be 0 unless armed */ uint32_t register_endpoints; /**< distinct (x,y) seen; 0 without the histogram */ uint32_t bulk_endpoints; } ft_tt_runtime_window_stats; /** Fill `out` from an open udrv handle. Zeroed for NULL. */ int ft_tt_runtime_udrv_window_stats_read(const ft_tt_runtime_udrv *udrv, ft_tt_runtime_window_stats *out); #endif /* FT_TT_RUNTIME_UDRV_H */