/* SPDX-License-Identifier: AGPL-3.0-or-later */ /* Copyright (C) 2026 David (Corey) Herrera */ /** * @file ft_tt_op_matmul_cooperative.h * @brief Host driver for a matmul that runs on N cores at once: who owns which output * columns, where the shared A operand lands, and the runtime-arg blocks that say so. * * The single-core driver is ft_tt_op_matmul.h and it stays exactly as it is. This is a * SEPARATE entry point rather than a mode of that one, because everything here is about a * SET of cores — a rectangle, a leader, a barrier, a per-core slice — and folding that into * a function whose whole signature is `(noc, x, y, ...)` would mean rewriting the op every * caller already uses in order to add one that nobody uses yet. * * --------------------------------------------------------------------------- * THE WORK SPLIT, AND WHY IT IS OVER N * --------------------------------------------------------------------------- * Core c of N owns `Nt / N` consecutive OUTPUT COLUMNS. It reads its own B panel from DRAM, * computes `Mt x Nt/N` output tiles, and writes them to its own output panel. Every core needs * the SAME A block at the same moment, and that is the only operand that is shared. * * over N every core needs the same A and a DISJOINT B. Exactly one operand is shared, and * it is the one whose re-reading the plan has named as the blocker since Phase 4 * ("every A tile re-read Nt times"). This is the split. * over K every core would hold a PARTIAL sum of the same output tile. The compute kernel * accumulates in one DST register across the whole K loop and has no way to emit or * take in a partial, and there is no cross-core add. It is not expressible without * changing the compute third. * over M every core would need a DIFFERENT A block, so the multicast would have nothing to * carry — the split would share nothing and cost the same DRAM traffic as today. * * **The B side wants nothing.** Each core's B panel is disjoint, so there is no broadcast to * make and every byte a core reads it is the only consumer of. Splitting over N leaves TOTAL B * traffic unchanged — the same bytes, read by N cores instead of one — which is also why B is * what limits the split: at large N a core's B panel becomes small enough that the fixed cost * of a launch dominates the work. * * **Nt MUST divide by N.** A ragged split would give members different block COUNTS, and the * per-block rendezvous is only correct because every member walks the same number of blocks in * the same order. It is refused here, not rounded. * * --------------------------------------------------------------------------- * THE THREE MODES ARE THE MEASUREMENT, NOT THREE FEATURES * --------------------------------------------------------------------------- * A cooperative run that is faster than one core proves nothing on its own — N cores are * faster than one core for reasons that have nothing to do with sharing an operand. So the * driver expresses all three arms and the kernel is one binary playing all of them: * * FT_TT_OP_MATMUL_COOPERATIVE N cores, disjoint slices, A multicast from the leader * FT_TT_OP_MATMUL_SPLIT N cores, disjoint slices, every core reads A from DRAM * itself. COOPERATIVE minus COOPERATIVE-minus-SPLIT is what * the MULTICAST is worth; SPLIT against one core is what the * extra cores are worth * FT_TT_OP_MATMUL_INDEPENDENT N cores, each doing the WHOLE matmul into its own output. * Not a split at all: it is N concurrent streams, and its * scaling is the DRAM/interconnect ceiling rather than this * op's * * --------------------------------------------------------------------------- * WHERE THE OUTPUT GOES * --------------------------------------------------------------------------- * Member c writes `Mt * Nt/N` tiles consecutively to `output.address + c * Mt * (Nt/N) * * tile_bytes` — a COLUMN PANEL, which is what the unchanged writer third can express (it * streams tiles from one base). **At Mt = 1 the panels concatenate into exactly the * single-core output**, tile for tile and byte for byte, which is what makes the correctness * gate a byte comparison against the solo arm rather than a PCC. At Mt > 1 they do not: the * whole result is there, in N panels, and a consumer must be told the layout. * * --------------------------------------------------------------------------- * THE MEASURED RECOMMENDATION * --------------------------------------------------------------------------- * MEASURED 2026-08-13, MiniCPM5-1B shapes at 32 tokens, one DRAM bank per operand: * * N 1 2 3 4 6 8 (ffn_gate/up, SPLIT, x vs one core) * speedup 1.00 1.80 1.87 1.88 1.95 2.07 * * **Use N = 2, and the rule is ONE CORE PER DRAM BANK THE OPERANDS OCCUPY.** A single core's * reader already runs at 39 GB/s against a measured ~42 GB/s per-bank read wall, so two cores * over two banks saturate everything the layout exposes and every core after that is idle * bandwidth-wise. Past N=2 the whole sweep buys 15% for four times the silicon. * * `attn_k/v` (Nt = 8) tops out at 1.24x at N=2 and goes BACKWARDS at N=8 (0.70x): at one * output tile per core the per-op fixed cost is the whole run. Leave it on one core, or two. * * --------------------------------------------------------------------------- * ...AND WHY THAT RECOMMENDATION HAS A PORT ROTATION UNDER IT NOW * --------------------------------------------------------------------------- * The wall above is not the BANK's, it is ONE DOOR of the bank. Every member read A through * port 0 of A's bank and B through port 0 of B's bank, because `ft_tt_coord_dram_bank()` IS * port 0 — so N members shared two endpoints at ~45 GB/s each, and 2 x 45 is exactly the 85-90 * GB/s the N=8 sweep flattened at. The bank behind each door serves ~64. * * `activation_ports` / `weight_ports` give the driver ALL of a bank's doors, and member `c` * reads through door `c % port_count`. **The bytes do not move**: three ports of one bank are * three doors into one memory (ft_tt_dram_stripe.h), so the same address answers through any * of them and the reader kernel is untouched — the endpoint arrives in the same two runtime * args it always did. Leave both sets empty and every member uses the operand's own (x, y), * i.e. exactly today's behaviour. * * @status Off-hardware in tests/test_cooperative_matmul_plan.c; MEASURED on the card by * tests/test_cooperative_matmul.c. * @issues None * @todo None */ #ifndef FT_TT_OP_MATMUL_COOPERATIVE_H #define FT_TT_OP_MATMUL_COOPERATIVE_H #include #include #include "ft_tt_dram_stripe.h" #include "ft_tt_group_shared_operand.h" #include "ft_tt_launch.h" #include "ft_tt_op_dispatch.h" #include "ft_tt_op_l1_layout.h" #include "kernels/cooperative_matmul_protocol.h" /** * The control region: the status block, the two semaphores, the leader's valid word and the * stop flag, at the SAME address on every member. * * It sits in the gap between the end of the kernel-config region (0x14000 + 69 KiB = 0x25400) * and the circular-buffer arena (0x26000) — 3 KiB that the Phase 4 op map leaves unowned. That * is deliberate rather than convenient: putting it inside the arena would make it a function * of how many CBs an op placed, and putting it in the kernel-config region would put it inside * a 16-bit offset field that the launch message also has to reach. * * Every address is the same on every member because the hardware requires it for the arrival * semaphore (a multicast writes ONE address on all destinations) and because one runtime-arg * block then serves every member, so a per-member address cannot drift. */ #define FT_TT_OP_COOP_CONTROL_BASE \ (FT_TT_OP_KERNEL_CONFIG_BASE + FT_TT_OP_KERNEL_CONFIG_BYTES) /** 16 bytes apart so no two control words share an L1 line. */ #define FT_TT_OP_COOP_STATUS_OFFSET 0x000u #define FT_TT_OP_COOP_READY_SEMAPHORE_OFFSET 0x040u #define FT_TT_OP_COOP_ARRIVAL_SEMAPHORE_OFFSET 0x050u #define FT_TT_OP_COOP_VALID_WORD_OFFSET 0x060u #define FT_TT_OP_COOP_STOP_FLAG_OFFSET 0x070u #define FT_TT_OP_COOP_CONTROL_BYTES 0x080u /** The widest group this driver will plan. The hardware bound is much lower and is a * RECTANGLE bound, not a count: Blackhole's Tensix columns are NoC x 1..7 and 10..16 because * x = 8 and 9 carry the ARC/L2CPU spine and the east DRAM column, so a multicast rectangle is * at most SEVEN columns wide and may not straddle the gap. 16 is what four rows of four fit, * which is the largest shape this project has evidence for. * * RAISED TO 32 ON 2026-08-19 for the SPLIT arm, which multicasts nothing and is therefore * not bound by any rectangle. The rectangle bound above still governs a COOPERATING group * and is enforced where a rectangle is actually built (pn_tt_group_place_rectangle), not * here — this number sizes the plan's arrays and nothing else. It was raised because it was * the ceiling on the experiment: with the hot activation strips mirrored across banks the * member count is no longer capped by one bank's ~59 GB/s, and "where does the split * actually scale to" cannot be asked at a limit of 16. */ #define FT_TT_OP_COOP_MAX_MEMBERS 64u /** * Where ONE member's own copy or slice of an operand lives, when that operand has been spread * over several DRAM banks instead of stored once in one. * * The two operands are spread differently and for different reasons, and the struct is the * same because the reader's view is the same — a bank, a door and a base: * * WEIGHT PARTITIONED. Member c owns a contiguous run of output columns and the weight * is stored N-major, so its panel is already a contiguous run of BYTES. Moving * that run into its own bank is a relocation, not a re-tiling, and the reader * then walks a grid `column_count` wide starting at column 0. * ACTIVATION REPLICATED. Every member needs ALL of A, so the copies are identical and only * the bank differs. It is affordable because A is Mt x Kt tiles against the * weight's Kt x Nt — 98 KiB against 14 MiB on ffn_gate/up. In a serve the cost * is not the capacity but the WRITES: N copies per token down the direction of * the card that does ~120 GB/s aggregate, not 500. */ typedef struct ft_tt_op_dram_shard { ft_tt_dram_endpoint endpoint; /**< the bank, and which of its doors this member uses */ uint32_t address; /**< base of THIS member's copy or panel inside that bank */ } ft_tt_op_dram_shard; /** How the work is spread. See the header block — these are the three arms of one * measurement, not three features. */ typedef enum ft_tt_op_matmul_cooperative_mode { /** N cores, disjoint column slices, A multicast once per block from the leader. */ FT_TT_OP_MATMUL_COOPERATIVE = 0, /** N cores, disjoint column slices, every core reads A from DRAM for itself. The control * that separates "the multicast helped" from "there were more cores". */ FT_TT_OP_MATMUL_SPLIT = 1, /** N cores, each computing the WHOLE matmul into its own output. N concurrent streams. */ FT_TT_OP_MATMUL_INDEPENDENT = 2 } ft_tt_op_matmul_cooperative_mode; /** One cooperative matmul: the same shape fields as ft_tt_op_matmul_request, plus the mode * and the bound every device-side wait takes. */ typedef struct ft_tt_op_matmul_cooperative_request { uint32_t tile_rows; /**< Mt */ uint32_t tile_inner; /**< Kt */ uint32_t tile_cols; /**< Nt of the WHOLE matmul, not of one member */ /** K-tiles per NoC barrier; shrunk to a divisor of tile_inner, exactly as the single-core * driver does. */ uint32_t prefetch_tiles; /** 0 = K-major weight tile grid (index kt*Nt + nt), 1 = N-major (nt*Kt + kt). A statement * about DRAM, not a request: both index formulas are in range, so a wrong value is a * wrong answer with no error anywhere. */ uint32_t weight_n_major; ft_tt_op_operand activation; /**< shared by every member */ ft_tt_op_operand weight; /**< the WHOLE K x N grid; the column offset does the slicing */ ft_tt_op_operand output; /**< base of member 0's panel; the rest follow it */ /** * The DOORS of the bank each read operand lives in — every NoC port of that one bank, * from ft_tt_dram_port_set_build(). Member `c` reads through port `c % port_count`. * * `port_count == 0` means "use the operand's own (x, y) on every member": one endpoint, * which is what every caller did before striping existed and what the recorded N=2..8 * baseline was measured with. Setting it changes no bytes and no kernel — a port is a * door into the same memory, so this is addressing only. * * The OUTPUT is deliberately not rotated. Writes do not behave like reads on this part * (aggregate ~120 GB/s, ports barely help, and the row-sharing limit does not apply to * them, plan Phase 6.7), so rotating it would add a variable to the measurement without a * measured reason to expect a gain. */ ft_tt_dram_port_set activation_ports; ft_tt_dram_port_set weight_ports; /** * Per-member placement, when the operand has been spread over BANKS and not just doors. * A count equal to `member_count` turns it on for that operand and OVERRIDES its port set; * 0 leaves it off. Anything else is refused rather than partly applied. * * This is the half that moves data, and it is what lifts the ceiling past ONE bank: the * port rotation above takes an operand from ~45 GB/s (one endpoint) to ~64 (one bank), and * only more banks go further. The card does 502. * * `weight_shards[c].address` is the base of member c's OWN panel, so the member reads a * grid `column_count` wide from column 0 — the full-grid column offset it would otherwise * carry is folded into the address by whoever laid the panels out. */ ft_tt_op_dram_shard activation_shards[FT_TT_OP_COOP_MAX_MEMBERS]; uint32_t activation_shard_count; ft_tt_op_dram_shard weight_shards[FT_TT_OP_COOP_MAX_MEMBERS]; uint32_t weight_shard_count; ft_tt_op_matmul_cooperative_mode mode; /** Spins before a device-side wait gives up and drains. Bounded AND askable is rule 4 of * the resident-kernel protocol; a wait that cannot expire is a wait only `assert_reset` * can end, and `assert_reset` on a blocked kernel is how cores were wedged. */ uint32_t spin_limit; } ft_tt_op_matmul_cooperative_request; /** What one member owns. */ typedef struct ft_tt_op_matmul_cooperative_member { /** FTCM_ROLE_*, from kernels/cooperative_matmul_protocol.h. */ uint32_t role; uint32_t column_first; /**< first output tile column, in the FULL grid */ uint32_t column_count; /**< how many; equal for every member by construction */ uint32_t output_address; /**< DRAM base of this member's output panel */ uint32_t output_tiles; /**< tile_rows * column_count — what the writer streams */ /** The DOOR this member reads each operand through, in the coordinate system the port * sets were built in. Equal to the request's own operand coordinate when that operand * was given no port set. These are the only two reader args striping changes. */ uint16_t activation_x; uint16_t activation_y; uint16_t weight_x; uint16_t weight_y; /** * Where this member's operands START, and the weight GRID it walks. * * Equal to the request's own address, `column_first` and the full Nt — unless the operand * was sharded, in which case the member reads its own panel from column 0 of a grid * exactly `column_count` wide. `column_first` above keeps naming the FULL grid either way, * because that is what says which part of the answer this member owns; these three say how * to reach the bytes. */ uint32_t activation_address; uint32_t weight_address; uint32_t weight_column_first; uint32_t weight_full_columns; } ft_tt_op_matmul_cooperative_member; /** A planned cooperative matmul: the slicing, the CB placement, and the shared-operand * contract. Pure arithmetic — nothing here has touched the device. */ typedef struct ft_tt_op_matmul_cooperative_plan { uint32_t member_count; /** Index into the caller's coordinate array of the core that reads A and multicasts it. * Meaningless outside FT_TT_OP_MATMUL_COOPERATIVE. */ uint32_t leader; uint32_t depth; /** The rectangle, the layout and the counts. Only built in COOPERATIVE mode. */ ft_tt_group_shared_operand shared; bool shared_valid; /** CB_IN0, CB_IN1, CB_OUT as placed — identical on every member, which is what makes one * multicast destination address correct for all of them. */ ft_tt_launch_cb_entry cb[3]; uint64_t cb_mask; unsigned char cb_buffer[FT_TT_NUM_CIRCULAR_BUFFERS * FT_TT_WORDS_PER_LOCAL_CB_CONFIG * 4]; ft_tt_op_matmul_cooperative_member member[FT_TT_OP_COOP_MAX_MEMBERS]; /** Distinct DRAM doors this plan's readers use, counted over the operands that were given * port sets. 0 when neither was. */ uint32_t distinct_endpoints; /** * Pairs of those doors that sit on ONE NoC row and therefore SHARE ~60 GB/s. * * REPORTED, NOT REFUSED. A colliding pair returns the right bytes, only slower, so * refusing would reject a legal plan for a performance reason; but it is also the single * mistake the striping rules name first, so it is logged when it happens and carried here * for a caller's own report to assert on. Zero is what a correct bank pairing gives. */ uint32_t row_collisions; } ft_tt_op_matmul_cooperative_plan; /** L1 address of one control word on any member. */ static inline uint32_t ft_tt_op_coop_status_address(void) { return FT_TT_OP_COOP_CONTROL_BASE + FT_TT_OP_COOP_STATUS_OFFSET; } static inline uint32_t ft_tt_op_coop_ready_semaphore_address(void) { return FT_TT_OP_COOP_CONTROL_BASE + FT_TT_OP_COOP_READY_SEMAPHORE_OFFSET; } static inline uint32_t ft_tt_op_coop_arrival_semaphore_address(void) { return FT_TT_OP_COOP_CONTROL_BASE + FT_TT_OP_COOP_ARRIVAL_SEMAPHORE_OFFSET; } static inline uint32_t ft_tt_op_coop_valid_word_address(void) { return FT_TT_OP_COOP_CONTROL_BASE + FT_TT_OP_COOP_VALID_WORD_OFFSET; } static inline uint32_t ft_tt_op_coop_stop_flag_address(void) { return FT_TT_OP_COOP_CONTROL_BASE + FT_TT_OP_COOP_STOP_FLAG_OFFSET; } /** * Plan a cooperative matmul over `member_count` cores, refusing every shape that cannot work. * * @param members TRANSLATED coordinates, from ft_tt_coord.h against live telemetry and * forming a RECTANGLE — use ft_tt_group_rectangle_coordinates(), which is * the partitioner that refuses a run of consecutive indices that wraps a * row or crosses the non-Tensix column gap. Only read in COOPERATIVE mode * @param member_count 1..FT_TT_OP_COOP_MAX_MEMBERS * @param leader index into `members` of the core that reads A and multicasts it * @param request the shape and the mode * @param out written only on success * @return FT_TT_OK; * FT_TT_EINVAL for a NULL argument, a zero dimension, a member count out of range, a * leader outside the group, or a `tile_cols` that `member_count` does not divide; * FT_TT_ENOMEM if the circular buffers do not fit the arena; * FT_TT_ERROR when the members are not a rectangle or the shared-operand layout is * refused. Every reason is logged * * Thread-safe: yes Reentrant: yes Signal-safe: no (logs on failure) */ int ft_tt_op_matmul_cooperative_plan_build(const ft_tt_noc_coord *members, uint32_t member_count, uint32_t leader, const ft_tt_op_matmul_cooperative_request *request, ft_tt_op_matmul_cooperative_plan *out); /** * The reader's FTCM_ARG_COUNT runtime args for one member. * * @param members the same array the plan was built from; only used for the leader's * coordinate, and may be NULL outside COOPERATIVE mode * @param out receives FTCM_ARG_COUNT words * @param capacity how many words `out` has * @return FT_TT_OK, FT_TT_EINVAL, or FT_TT_ENOMEM */ int ft_tt_op_matmul_cooperative_reader_arguments( const ft_tt_op_matmul_cooperative_plan *plan, const ft_tt_op_matmul_cooperative_request *request, const ft_tt_noc_coord *members, uint32_t member, uint32_t *out, uint32_t capacity); /** * The compute third's three args (Mt, Kt, this member's Nt) and the writer's four (tile count, * destination x, y, base). Neither kernel changes for a cooperative run; only these do. * * @return FT_TT_OK, FT_TT_EINVAL, or FT_TT_ENOMEM */ int ft_tt_op_matmul_cooperative_compute_arguments( const ft_tt_op_matmul_cooperative_plan *plan, const ft_tt_op_matmul_cooperative_request *request, uint32_t member, uint32_t *out, uint32_t capacity); int ft_tt_op_matmul_cooperative_writer_arguments( const ft_tt_op_matmul_cooperative_plan *plan, const ft_tt_op_matmul_cooperative_request *request, uint32_t member, uint32_t *out, uint32_t capacity); #endif /* FT_TT_OP_MATMUL_COOPERATIVE_H */