David
/
Foundry
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Foundry/tt/ft_tt_udrv_noc_bank.h

351 lines
18 KiB
C
Raw Blame History

/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
* @file ft_tt_udrv_noc_bank.h
* @brief A SET of 2 MiB TLB windows driven as one accessor, so a dispatch path
* that talks to many cores stops re-pointing one window at each of them in
* turn.
*
* ---------------------------------------------------------------------------
* WHY THIS EXISTS: THE AIM IS A PER-DESCRIPTOR COST, AND IT IS THE LARGEST ONE
* ---------------------------------------------------------------------------
* `ft_tt_udrv_noc_path` owns ONE window and already skips the re-aim when the
* window is where the caller wants it (`aim_if_needed()`). That skip is a cache
* of ONE ENTRY, and the dispatch path's access stream is not one endpoint — it
* is a rotation over the resident cores, so the one entry misses on nearly
* every access.
*
* Measured on silicon 2026-08-26 (spec, "the issue side is not accesses, it is
* the TLB aim"): an aim is **8.92 us**, 98.0% of it four NON-POSTED register
* readbacks at 2.19 us each, and the decode token pays **4,333 of them =
* 0.0387 s = 25.1% of the wall**. All 25 PCIe accesses of a descriptor's issue
* together cost 0.79 us. The cost is the aim, not the traffic.
*
* A bank is the same cache with N entries, N being the number of windows it
* claimed. **202 windows exist on this part.** Two were being used.
*
* ---------------------------------------------------------------------------
* THE CACHE IS THE WINDOWS THEMSELVES — THERE IS NO SECOND COPY OF THE STATE
* ---------------------------------------------------------------------------
* A lookup scans the paths' OWN `aim`/`aimed` fields, which is what
* `ft_tt_udrv_noc_transfer()` maintains. There is deliberately no side table of
* "what I think window k holds": a side table can disagree with the register,
* and a cache that disagrees with the hardware aims at the wrong endpoint
* SILENTLY. Anything that aims a path in this bank — including the ARC
* transport, which is handed path 0 — updates the very field the lookup reads.
*
* ---------------------------------------------------------------------------
* AND A WRONG WINDOW IS A WRONG ADDRESS, SO THERE IS AN AUDIT AND A RED ARM
* ---------------------------------------------------------------------------
* The failure this module can introduce is routing a transfer to a window
* pointed at a DIFFERENT core. That does not fail; it reads and writes the
* wrong endpoint. So:
*
* - `FT_TT_UDRV_AIM_AUDIT=1` re-reads the selected window's register triple
* from the card and decodes it before a HIT is used. It is the same four
* non-posted reads the aim pays, which is why it is a check and not the
* shipped path — but it is an INDEPENDENT reading of where the window
* points, not the bank's own belief.
* - `FT_TT_UDRV_RED=misaim` falsifies a path's RECORDED aim so the re-aim is
* skipped and the read is dereferenced through a window pointed at another
* endpoint. It injects a STATE the whole transfer sees, not a race.
* **IT NEEDS A SECOND KEY AND IT IS NOT SAFE ON A LIVE MODEL** — it reads
* from whatever another window holds, which includes the ARC tile, and on
* 2026-08-26 one such read did not complete, wedged the NoC and cost the
* host a physical power cycle. Drive it through
* `tests/test_udrv_card_bank.c`, which uses two DRAM banks that always
* decode and compares the bytes.
*
* @status On silicon 2026-08-26. See the plan item "THE NEXT RUNG IS THE AIM".
* @issues
* - The audit reads back through the mapping that wrote the window, the same
* limit `ft_tt_udrv_noc.h` states for the aim's own verification: it cannot
* catch a mapping that consistently points at the wrong physical page.
* - A bank that could not claim every window it asked for runs with fewer
* cache entries. It says so loudly at open and reports the count; it never
* shares a window with another claimant.
* @todo None
*/
#ifndef FT_TT_UDRV_NOC_BANK_H
#define FT_TT_UDRV_NOC_BANK_H
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include "ft_tt_udrv_claim.h"
#include "ft_tt_udrv_noc.h"
/** Lowest window index a bank will claim, and it is NOT an arbitrary margin.
*
* WINDOWS BELOW `FT_TT_UDRV_TLB_STRIDED_COUNT` (32) CARRY A STRIDED-MULTICAST
* REGISTER, AND THAT MAKES THEM A DIFFERENT AND MORE EXPENSIVE KIND OF WINDOW.
* `ft_tt_udrv_noc_window_aim_timed()` guards both the strided CLEAR and the
* strided CHECK on `id < 32`, so on a window below the line every aim pays a
* fourth posted store and — at the shipped `full` verification — a FOURTH
* NON-POSTED READBACK, which is a whole PCIe round trip on the term this bank
* exists to remove. And a stale multicast pattern on one of those windows is
* this project's most reliable way to wedge the NoC, so it is also the one
* window kind whose aim has a fabric failure mode.
*
* There is no shortage that would justify reaching below it: indices 32..201
* are 170 windows, 169 once the vendor driver's is skipped, against two banks
* whose combined ceiling is 168. **A bank that wants more says so and runs
* SHORT rather than quietly buying windows that cost more per aim than the aim
* they save.** */
#define FT_TT_UDRV_NOC_BANK_MIN_WINDOW_INDEX FT_TT_UDRV_TLB_STRIDED_COUNT
/** Most windows one bank will ever hold.
*
* EIGHTY-FOUR, AND THE BOUND IS THE POOL ABOVE THE STRIDED LINE. Indices
* 32..201 are 170; the vendor driver keeps 201; two banks at 84 claim 168 of
* the remaining 169. So both banks can be satisfied in full, every window is
* one of the cheap kind, and one is left over.
*
* It was 32, and 32 was sized on a working set of 26 endpoints measured at
* EIGHT matmul members. The member count is a knob, the endpoint set grows
* with it, and at sixteen members the program touches more cores than a
* 32-window bank can hold — which turns the bank from a cache into an eviction
* machine: the bulk hit rate falls 99.74% / 99.50% / 98.20% at N = 8 / 12 / 16
* while the aims a token go 829.2 / 1,939.9 / 7,566.1.
*
* THIS IS A CEILING AND NOT A SIZE. What a bank actually claims is
* `FT_TT_UDRV_WINDOWS`, and its default is set from the reuse-distance
* measurement below, not from this number. Raising the ceiling costs nothing
* but `sizeof(ft_tt_udrv_noc_bank)`; claiming a window costs one file
* descriptor and one 2 MiB mapping of BAR0. */
#define FT_TT_UDRV_NOC_BANK_MAX_WINDOWS 84u
/** How deep the LRU reuse-distance instrument simulates.
*
* A BANK IS AN LRU CACHE, SO ITS HIT RATE AT EVERY SIZE IS ONE MEASUREMENT.
* For LRU, a reference hits a cache of `W` entries exactly when its reuse
* distance — how many DISTINCT other keys were touched since this key last
* was — is below `W`. So a histogram of reuse distances gives the hit rate at
* every bank size at once, and the bank size stops being a guess.
*
* Deeper than the window ceiling on purpose: a distance that lands past the
* ceiling has to be VISIBLE as "no bank this part can hold would catch it"
* rather than folded into the cold count. */
#define FT_TT_UDRV_NOC_BANK_REUSE_MAX 192u
/** Distinct (x, y) endpoints the histogram will name before it starts counting
* overflow. Sized well above the card's Tensix column count so a long tail is
* VISIBLE as overflow rather than silently truncated.
*
* IT IS KEYED ON (x, y) AND NOT ON (x, y, base), WHICH IS A MEASURED
* CORRECTION. A (x, y, base) table is FLOODED by the weight upload: 9.5 GB of
* DRAM through 2 MiB windows is ~4,750 distinct bases across four endpoints,
* and since the upload happens before the first token it fills any table and
* pushes every L1 endpoint of the dispatch path into overflow. Keyed on the
* core, DRAM is four rows however many bases it walks.
*
* IT WAS 64 AND 64 WAS NOT ENOUGH — measured 2026-09-03. "Sized well above the
* card's Tensix column count" is the wrong yardstick: the bulk bank's set is
* cores PLUS DRAM ports, and both grow with the matmul member count. At eight
* members it is 49 rows, at twelve 61, and at SIXTEEN it hit the 64-row ceiling
* and reported 7,219 selections past the table — so the one figure the round
* existed to produce read as ">= 64" instead of as a number. 160 is past the
* 140 Tensix the part has, which is the bound the table can never need to
* exceed for cores, plus the DRAM and ARC rows. */
#define FT_TT_UDRV_NOC_BANK_HISTOGRAM_MAX 160u
/** Distinct 2 MiB bases one row can count.
*
* PAST THIS THE ROW STOPS COUNTING AND SAYS SO. The first version kept
* incrementing once the list was full, which turns a DISTINCT-VALUE counter
* into a SELECTION counter the moment the set outgrows its list: a DRAM
* endpoint reported 21,790 "distinct bases" for 21,854 selections of a 32 GiB
* card. A cardinality that cannot be counted has to read as "at least N",
* never as a number. */
#define FT_TT_UDRV_NOC_BANK_BASES_PER_ROW 8u
/** One row of the endpoint histogram: which core, how often, and how much of
* its address space the window had to walk. */
typedef struct ft_tt_udrv_noc_bank_endpoint {
uint16_t x;
uint16_t y;
uint64_t selections; /**< transfers routed to this core */
uint64_t aims; /**< of those, how many had to re-point a window */
uint32_t base_count; /**< distinct 2 MiB bases seen, up to the cap */
bool base_uncounted; /**< true once the set outgrew the list: read base_count as ">=" */
uint64_t base[FT_TT_UDRV_NOC_BANK_BASES_PER_ROW];
} ft_tt_udrv_noc_bank_endpoint;
/** One routing key: the triple the window lookup actually compares.
*
* IT IS (x, y, base) AND NOT (x, y). `path_holds()` compares the 2 MiB base
* as well as the core, so two bases on one core are two windows' worth of
* working set. The endpoint histogram above is keyed on the CORE — which is
* the right key for "how many cores does the dispatch path talk to" and the
* WRONG key for "how many windows would hold it". */
typedef struct ft_tt_udrv_noc_bank_key {
uint16_t x;
uint16_t y;
uint64_t base;
} ft_tt_udrv_noc_bank_key;
/** Which deliberate defect is armed. Off unless FT_TT_UDRV_RED names it. */
typedef enum ft_tt_udrv_noc_bank_red {
FT_TT_UDRV_NOC_BANK_RED_NONE = 0,
FT_TT_UDRV_NOC_BANK_RED_MISAIM = 1
} ft_tt_udrv_noc_bank_red;
/**
* A set of claimed windows, their apertures, and the routing over them.
*
* Caller-owned storage. Treat the fields as private except through the
* accessors below; `path[0]` is public only because the ARC transport needs a
* single path and must share this bank's window rather than claim a third.
*/
typedef struct ft_tt_udrv_noc_bank {
ft_tt_udrv_noc_path path[FT_TT_UDRV_NOC_BANK_MAX_WINDOWS];
ft_tt_udrv_claim *claim[FT_TT_UDRV_NOC_BANK_MAX_WINDOWS];
uint64_t last_used[FT_TT_UDRV_NOC_BANK_MAX_WINDOWS]; /**< LRU stamp */
uint32_t window_count; /**< paths actually open */
uint32_t windows_requested; /**< what was asked for */
uint64_t clock; /**< LRU stamp source */
const char *name; /**< "register" / "bulk", for logs */
uint64_t selections; /**< transfers routed, one per window slice */
uint64_t hits; /**< routed to a window already pointed there */
uint64_t misses; /**< routed to a window that had to be re-aimed */
uint64_t evictions; /**< misses that displaced a live aim */
uint64_t misaims_injected; /**< the red arm's own count; 0 means it never fired */
uint64_t audit_failures; /**< audits that caught a wrong endpoint */
bool histogram_enabled;
bool audit_enabled;
bool reuse_enabled;
ft_tt_udrv_noc_bank_red red;
ft_tt_udrv_noc_bank_endpoint histogram[FT_TT_UDRV_NOC_BANK_HISTOGRAM_MAX];
uint32_t histogram_used;
uint64_t histogram_overflow; /**< selections past the table's capacity */
/** The reuse-distance instrument. `reuse_stack` is an LRU stack of routing
* keys, MRU first; `reuse_hist[d]` counts references found at depth `d`,
* so a bank of `W` windows would have hit exactly `sum(reuse_hist[0..W))`
* of them. `reuse_cold` is everything else — first touches and distances
* past the simulated depth, which are the references no bank catches. */
ft_tt_udrv_noc_bank_key reuse_stack[FT_TT_UDRV_NOC_BANK_REUSE_MAX];
uint32_t reuse_depth;
uint64_t reuse_hist[FT_TT_UDRV_NOC_BANK_REUSE_MAX];
uint64_t reuse_cold;
uint64_t reuse_references;
} ft_tt_udrv_noc_bank;
/**
* Claim up to `windows` windows, map each one's aperture, and build the bank.
*
* Claims walk DOWN from the top of the pool, skipping `reserved_window`, for
* the reason `ft_tt_runtime_udrv.c` documents: `ALLOCATE_TLB` allocates from
* the bottom, and while tt-kmd is loaded a bottom-up claim here collides with
* it on the first window every time. `reserved_window` is the index the vendor
* driver keeps for itself (`KERNEL_TLB_INDEX`, 201) and is NEVER handed out —
* claiming it out from under a loaded tt-kmd corrupts its telemetry reads.
*
* A short claim is NOT silently accepted as success: the achieved count is
* logged at error level with the shortfall named, and the bank runs with fewer
* cache entries. Zero windows is a hard failure.
*
* @param device an open device
* @param ns the "tlb2m" claim namespace for this card
* @param reserved_window index never to hand out
* @param write_combine true for the bulk path, false for registers
* @param windows how many to ask for, 1..FT_TT_UDRV_NOC_BANK_MAX_WINDOWS
* @param name "register" or "bulk"; used in log lines only
* @param out_bank zeroed and filled; close with ft_tt_udrv_noc_bank_close()
* @return FT_TT_UDRV_OK; FT_TT_UDRV_EINVAL; FT_TT_UDRV_EAGAIN when
* no window at all could be claimed; FT_TT_UDRV_EIO
*
* Thread-safe: no Reentrant: no Signal-safe: no
*/
int ft_tt_udrv_noc_bank_open(ft_tt_udrv_noc_device *device,
const ft_tt_udrv_claim_namespace *ns, uint32_t reserved_window,
bool write_combine, uint32_t windows, const char *name,
ft_tt_udrv_noc_bank *out_bank);
/** Unmap every aperture and release every claim. Safe on a zeroed bank. */
int ft_tt_udrv_noc_bank_close(ft_tt_udrv_noc_bank *bank);
/**
* Move `bytes` between host memory and (x, y, address) through whichever of the
* bank's windows is already pointed there, aiming the least recently used one
* when none is.
*
* Semantics are otherwise exactly `ft_tt_udrv_noc_transfer()`'s, which is what
* this calls once the window is chosen — the copy shape, the fence and the
* bounds are not re-derived here.
*
* @return FT_TT_UDRV_OK; FT_TT_UDRV_EINVAL; FT_TT_UDRV_ENOTSUP; FT_TT_UDRV_EIO
* when an aim did not verify or the audit caught a wrong endpoint (in
* both cases nothing was dereferenced)
*/
int ft_tt_udrv_noc_bank_transfer(ft_tt_udrv_noc_bank *bank, uint16_t x, uint16_t y,
uint64_t address, void *host, size_t bytes, bool is_read,
ft_tt_noc_transfer_width width);
/** `ft_tt_noc_access`-shaped adapters whose `user` is an
* `ft_tt_udrv_noc_bank *`. WORD32, matching the single-path pair. */
int ft_tt_udrv_noc_bank_access_write(void *user, uint16_t x, uint16_t y, uint64_t address,
const void *src, size_t bytes);
int ft_tt_udrv_noc_bank_access_read(void *user, uint16_t x, uint16_t y, uint64_t address,
void *dst, size_t bytes);
/** Totals over every window in the bank. */
uint64_t ft_tt_udrv_noc_bank_aim_count(const ft_tt_udrv_noc_bank *bank);
uint64_t ft_tt_udrv_noc_bank_aim_failures(const ft_tt_udrv_noc_bank *bank);
double ft_tt_udrv_noc_bank_aim_seconds(const ft_tt_udrv_noc_bank *bank);
double ft_tt_udrv_noc_bank_aim_verify_seconds(const ft_tt_udrv_noc_bank *bank);
uint64_t ft_tt_udrv_noc_bank_bytes_read(const ft_tt_udrv_noc_bank *bank);
uint64_t ft_tt_udrv_noc_bank_bytes_written(const ft_tt_udrv_noc_bank *bank);
/** How many distinct (x, y) endpoints the histogram saw. Zero when the
* histogram was never enabled. */
uint32_t ft_tt_udrv_noc_bank_endpoint_count(const ft_tt_udrv_noc_bank *bank);
/**
* Log the endpoint histogram at error level, rows sorted by selection count.
*
* A no-op unless the bank was opened with `FT_TT_UDRV_AIM_HISTOGRAM` set — the
* recording costs a linear scan per transfer and has no business on the shipped
* path. Prints the row count, the overflow count and the concentration, which
* is the number that decides whether a window per endpoint is a cache with a
* 100% hit rate or an eviction-policy problem.
*/
void ft_tt_udrv_noc_bank_histogram_log(const ft_tt_udrv_noc_bank *bank);
/**
* Log the reuse-distance ladder: what hit rate a bank of each size WOULD have
* had on the access stream this run actually produced.
*
* A no-op unless the bank was opened with `FT_TT_UDRV_AIM_REUSE` set. It costs
* a stack search and a move-to-front per routing decision, so a run carrying it
* is a run measuring the PATTERN and not the wall — its own timings are not
* comparable to an arm without it, and the line says so.
*
* IT PRINTS ITS OWN FALSIFIER. The last column reprints the ladder's entry at
* this bank's ACTUAL window count beside the hit rate the bank measured for
* itself. Those two are the same number computed two ways — an LRU cache of `W`
* entries hits exactly the references whose reuse distance is below `W` — so a
* disagreement means the simulation does not describe the cache and the ladder
* must not be used to size anything. An instrument that cannot be caught being
* wrong is this program's most expensive defect.
*/
void ft_tt_udrv_noc_bank_reuse_log(const ft_tt_udrv_noc_bank *bank);
/** References this bank's reuse instrument would have served from a bank of
* `windows` entries. Zero when the instrument was never enabled. */
uint64_t ft_tt_udrv_noc_bank_reuse_hits_at(const ft_tt_udrv_noc_bank *bank, uint32_t windows);
/** Parse `FT_TT_UDRV_WINDOWS` once. Clamped to
* 1..FT_TT_UDRV_NOC_BANK_MAX_WINDOWS; an unparseable value is refused loudly
* and the default is used. Exposed so the caller can echo what it parsed —
* a knob that reads as armed and is not is this project's most expensive
* instrument defect. */
uint32_t ft_tt_udrv_noc_bank_windows_selected(void);
#endif /* FT_TT_UDRV_NOC_BANK_H */
Reference in New Issue View Git Blame Copy Permalink