David
/
Foundry
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Foundry/tt/kernels/multicast_bench_reader.c

398 lines
19 KiB
C
Raw Blame History

/*
* SPDX-License-Identifier: AGPL-3.0-or-later
*
* multicast_bench_reader.c
* ONE BRISC kernel, in C99, that plays every role the multicast measurement needs:
* the multicast sender, a multicast receiver, or a core reading the same block out of
* DRAM for itself. The role is a RUNTIME argument.
*
* **One binary for all three roles is the measurement's method, not a convenience.**
* The whole question is "what does a multicast cost against N cores each reading DRAM",
* and if the two arms were two binaries then every difference between them -- inlining,
* text placement, which registers the loop got -- would be inside the number. Same
* binary, same L1 map, same issue layer (kernels/ft_tensix_noc_multicast.h), one runtime
* word apart. This is the same rule `weight_n_major` follows in matmul_reader.c for
* the same reason (docs/tt_kernel_model.md §14).
*
* It is C because the dataflow half of a kernel does not need C++: everything here is a
* store to a fixed register address or a do/while over one. It includes NOTHING from
* tt-metal -- only kernels/ft_tensix_dataflow.h (Phase 6.6's C99 dataflow library) and
* kernels/ft_tensix_noc_multicast.h on top of it. build_tensix_dataflow_kernel.sh's `.c`
* path is how a C translation unit becomes a Tensix ELF.
*
* ===========================================================================
* WHAT IT MEASURES, AND WHY IT IS TIMED THIS WAY
* ===========================================================================
* Both arms deliver the SAME thing: one block of `payload_bytes` into the SAME L1 address
* on N cores, `iterations` times. The measured quantity is the steady-state period of
* that, per core, in wall-clock cycles read off the core's own counter.
*
* MULTICAST arm 1 sender + N receivers. Per round the sender reads the block from
* DRAM once and multicasts it; each receiver announces readiness and
* waits for it. A receiver's per-round time is "how long until the
* block is in MY L1", which is what a cooperating matmul core actually
* waits on.
* DRAM arm N cores, no sender, no synchronisation: each reads the block from
* DRAM into the same L1 address itself. This is the status quo the
* plan calls out -- "every A tile re-read Nt times".
*
* **Timing is device-side on purpose.** The host is two 1.8 GHz cores that also drive
* eight K80s and drift enough over ten seconds to swamp a microsecond (DEV_NOTES.md §8),
* and a launch is ~14.5 us against a round that should be ~1 us. Each core reads
* RISCV_DEBUG_REG_WALL_CLOCK_0 itself, so launch skew across N cores affects how much the
* arms OVERLAP -- which is a real property of the arm -- and not the measurement.
*
* **Round 0 is not timed.** The clock is sampled AFTER round 0 and after round
* `iterations`, so the window is exactly `iterations` steady-state rounds and excludes
* the sender waiting for receivers to boot, the first DRAM row-buffer miss, and the
* instruction cache warming.
*
* ===========================================================================
* HOW THE RESULT IS PROVEN TO BE THE RESULT
* ===========================================================================
* A destination that matches the expected bytes could be stale L1 (DEV_NOTES.md §3), so
* the payload ALTERNATES between two different blocks staged at two different DRAM
* addresses, by round parity. After `iterations + 1` rounds every destination must hold
* block `iterations & 1` and not the other one, on every core -- which shows the data
* tracked the source and that the LAST round landed, not merely that something arrived.
*
* The alternation is by ADDRESS, never by mutating a buffer in place. A RISC-V store
* followed by a NoC read of the same L1 word is not ordered on this part, and it is a race
* you lose every time, not sometimes -- the group-completion post found that and an
* `invalidate_l1_cache()` between them did not fix it
* (kernels/resident_queue_protocol.h, ftq_group_post).
*
* ===========================================================================
* RUNTIME ARGUMENTS -- one layout for every role
* ===========================================================================
* One layout rather than three, because three arg maps is three things to keep in step
* with one host, and a runtime-arg block that did not say what its reader thought has
* already stalled a core here (docs/tt_launch_notes.md §3).
*
* 0 role FTMB_ROLE_*
* 1 iterations timed rounds; round 0 runs untimed on top
* 2 payload bytes multiple of 64 (DRAM read alignment is the strictest end)
* 3 staging A L1 address sender only: DRAM lands here on EVEN rounds
* 4 staging B L1 address sender only: odd rounds
* 5 shared destination address the SAME L1 address on every receiver / DRAM reader.
* The hardware requires it for multicast; the DRAM arm
* uses it too so the two arms differ in nothing else
* 6 rectangle x low TRANSLATED NoC coordinates, from the host's ft_tt_coord
* 7 rectangle y low
* 8 rectangle x high
* 9 rectangle y high
* 10 destination count receivers, EXCLUDING the sender
* 11 sender NoC x
* 12 sender NoC y
* 13 ready semaphore address on the SENDER; receivers atomically increment it
* 14 arrival semaphore address on EVERY receiver, same address
* 15 valid word address on the sender; the host stages FT_TENSIX_SEMAPHORE_VALID here
* 16 DRAM NoC x
* 17 DRAM NoC y
* 18 DRAM address A
* 19 DRAM address B
* 20 status block address
* 21 stop flag address
* 22 spin limit every wait is bounded as well as askable
*
* Compile-time args: none. Everything about the CALL is a runtime arg
* (docs/tt_kernel_model.md §14).
*
* Build (BRISC, NoC 0):
* CTARGS=0 NOC=0 bash foundry/tt/build_tensix_dataflow_kernel.sh \
* foundry/tt/kernels/multicast_bench_reader.c brisc /tmp/ttk_multicast
*
* Status: NEW 2026-08-13. Built and run on the p150a card host by tests/test_multicast_bench.c.
* Issues: None
* Todo: None
*/
#include "ft_tensix_dataflow.h"
#include "ft_tensix_noc_multicast.h"
/* ---------------------------------------------------------------------------
* The wire contract. tests/test_multicast_bench.c carries the same values; change
* both or neither.
* --------------------------------------------------------------------------- */
#define FTMB_ROLE_MULTICAST_SENDER 1u
#define FTMB_ROLE_MULTICAST_RECEIVER 2u
#define FTMB_ROLE_DRAM_READER 3u
/** "FTMB". A status block still reading the host's poison means the kernel never ran. */
#define FTMB_STATUS_MAGIC 0x46544D42u
#define FTMB_STATUS_MAGIC_WORD 0u
#define FTMB_STATUS_ROLE 1u
#define FTMB_STATUS_ROUNDS_COMPLETED 2u
#define FTMB_STATUS_ELAPSED_CYCLES 3u
#define FTMB_STATUS_ERROR 4u
#define FTMB_STATUS_START_CLOCK 5u
#define FTMB_STATUS_END_CLOCK 6u
#define FTMB_STATUS_LAST_ISSUE_CODE 7u
#define FTMB_STATUS_WORDS 8u
/** DRAM's read alignment, the stricter of the two ends of a DRAM->L1 transfer
* (ft_tt_noc.h, ft_tt_noc_alignment). A derived address that is only 16- or 32-byte
* aligned reads back byte-exact until its LAST 64 bytes, silently
* (tests/ft_tt_phase2_l1_layout.h). The NoC index this kernel issues on is NOC_INDEX,
* baked in by the build and read by the multicast header -- which is where the rectangle's
* corner order depends on it. */
#define FTMB_DRAM_READ_ALIGNMENT 64u
/** L1's alignment, which is what a multicast destination takes. */
#define FTMB_L1_ALIGNMENT 16u
/* --------------------------------------------------------------------------- */
/** Everything the three roles share, read once at entry so the timed loops index
* registers instead of re-reading L1. */
typedef struct ftmb_context {
uint32_t role;
uint32_t iterations;
uint32_t payload_bytes;
uint32_t staging_address[2];
uint32_t shared_destination_address;
uint32_t rectangle_x_low;
uint32_t rectangle_y_low;
uint32_t rectangle_x_high;
uint32_t rectangle_y_high;
uint32_t destination_count;
uint32_t sender_x;
uint32_t sender_y;
uint32_t ready_semaphore_address;
uint32_t arrival_semaphore_address;
uint32_t valid_word_address;
uint32_t dram_x;
uint32_t dram_y;
uint32_t dram_address[2];
volatile uint32_t *status;
volatile const uint32_t *stop_flag;
uint32_t spin_limit;
} ftmb_context;
static void ftmb_read_arguments(ftmb_context *context)
{
context->role = ft_tensix_runtime_arg((int)0u);
context->iterations = ft_tensix_runtime_arg((int)1u);
context->payload_bytes = ft_tensix_runtime_arg((int)2u);
context->staging_address[0] = ft_tensix_runtime_arg((int)3u);
context->staging_address[1] = ft_tensix_runtime_arg((int)4u);
context->shared_destination_address = ft_tensix_runtime_arg((int)5u);
context->rectangle_x_low = ft_tensix_runtime_arg((int)6u);
context->rectangle_y_low = ft_tensix_runtime_arg((int)7u);
context->rectangle_x_high = ft_tensix_runtime_arg((int)8u);
context->rectangle_y_high = ft_tensix_runtime_arg((int)9u);
context->destination_count = ft_tensix_runtime_arg((int)10u);
context->sender_x = ft_tensix_runtime_arg((int)11u);
context->sender_y = ft_tensix_runtime_arg((int)12u);
context->ready_semaphore_address = ft_tensix_runtime_arg((int)13u);
context->arrival_semaphore_address = ft_tensix_runtime_arg((int)14u);
context->valid_word_address = ft_tensix_runtime_arg((int)15u);
context->dram_x = ft_tensix_runtime_arg((int)16u);
context->dram_y = ft_tensix_runtime_arg((int)17u);
context->dram_address[0] = ft_tensix_runtime_arg((int)18u);
context->dram_address[1] = ft_tensix_runtime_arg((int)19u);
context->status = (volatile uint32_t *)(unsigned long)ft_tensix_runtime_arg((int)20u);
context->stop_flag = (volatile const uint32_t *)(unsigned long)ft_tensix_runtime_arg((int)21u);
context->spin_limit = ft_tensix_runtime_arg((int)22u);
}
static void ftmb_publish(const ftmb_context *context, uint32_t rounds_completed,
uint32_t start_clock, uint32_t end_clock, int error,
int last_issue_code)
{
/* Magic LAST, so a host reading a torn block sees "never ran" rather than a
* half-filled result -- the same ordering rule the resident queue's
* token-before-count uses (kernels/resident_queue_protocol.h rule 1). */
context->status[FTMB_STATUS_ROLE] = context->role;
context->status[FTMB_STATUS_ROUNDS_COMPLETED] = rounds_completed;
context->status[FTMB_STATUS_START_CLOCK] = start_clock;
context->status[FTMB_STATUS_END_CLOCK] = end_clock;
context->status[FTMB_STATUS_ELAPSED_CYCLES] = end_clock - start_clock;
context->status[FTMB_STATUS_ERROR] = (uint32_t)error;
context->status[FTMB_STATUS_LAST_ISSUE_CODE] = (uint32_t)last_issue_code;
context->status[FTMB_STATUS_MAGIC_WORD] = FTMB_STATUS_MAGIC;
}
/**
* One block out of DRAM into local L1, with the guards the shared read path leaves to its
* caller.
*
* `ft_tensix_noc_read()` deliberately does not check alignment, because the rule depends on
* the SOURCE space -- 64 bytes reading DRAM, 16 reading L1 -- and a 64-bit NoC address does
* not say which endpoint it names. This function knows the source is DRAM, so it applies
* the rule and REFUSES rather than issuing. The zero-length guard is the one that matters:
* a zero-length NoC transfer is what took this card from working to `Mem- BusMaster-` on
* 2026-08-12 with no error anywhere until the next read returned 0xFFFFFFFF
* (DEV_NOTES.md §1), and a host-side check cannot see a kernel-issued transfer.
*/
static int ftmb_read_from_dram(const ftmb_context *context, uint32_t dram_address,
uint32_t destination_l1_address)
{
if (context->payload_bytes == 0u)
return FT_TENSIX_MULTICAST_ERROR_ZERO_LENGTH;
if ((context->payload_bytes % FTMB_DRAM_READ_ALIGNMENT) != 0u ||
(dram_address % FTMB_DRAM_READ_ALIGNMENT) != 0u ||
(destination_l1_address % FTMB_DRAM_READ_ALIGNMENT) != 0u)
return FT_TENSIX_MULTICAST_ERROR_ALIGNMENT;
ft_tensix_noc_read(ft_tensix_noc_address(context->dram_x, context->dram_y, dram_address),
destination_l1_address, context->payload_bytes);
return FT_TENSIX_MULTICAST_OK;
}
/* ---------------------------------------------------------------------------
* The three roles
* --------------------------------------------------------------------------- */
/**
* One sender round: collect every receiver, fetch the block from DRAM, multicast it, then
* announce it.
*
* The order of the last three is the protocol and none of it is rearrangeable --
* ft_tt_multicast_kernel.h's header block has the full account, including why the flush
* between the two multicasts is mandatory on Blackhole and nowhere else.
*/
static int ftmb_sender_round(const ftmb_context *context, uint32_t round, int *out_issue_code)
{
volatile uint32_t *ready_semaphore =
(volatile uint32_t *)(unsigned long)context->ready_semaphore_address;
const uint32_t parity = round & 1u;
uint64_t payload_destination;
uint64_t arrival_destination;
int code;
code = ft_tensix_wait_semaphore_at_least(ready_semaphore, context->destination_count,
context->spin_limit, context->stop_flag);
if (code != FT_TENSIX_MULTICAST_OK)
return code;
ft_tensix_set_local_semaphore(ready_semaphore, 0u);
code = ftmb_read_from_dram(context, context->dram_address[parity],
context->staging_address[parity]);
*out_issue_code = code;
if (code != FT_TENSIX_MULTICAST_OK)
return code;
ft_tensix_noc_read_barrier();
code = ft_tensix_multicast_address(context->rectangle_x_low, context->rectangle_y_low,
context->rectangle_x_high, context->rectangle_y_high,
context->shared_destination_address,
&payload_destination);
if (code != FT_TENSIX_MULTICAST_OK) {
*out_issue_code = code;
return code;
}
code = ft_tensix_noc_write_multicast(context->staging_address[parity], payload_destination,
context->payload_bytes, context->destination_count);
*out_issue_code = code;
if (code != FT_TENSIX_MULTICAST_OK)
return code;
/* BLACKHOLE ONLY, and load-bearing: the data multicast and the semaphore multicast go
* into different command-buffer FIFOs and are not guaranteed to leave in issue order,
* so without this a receiver can be told the block arrived before it did and read the
* PREVIOUS round's bytes with nothing reporting an error. */
ft_tensix_noc_writes_flushed_barrier();
code = ft_tensix_multicast_address(context->rectangle_x_low, context->rectangle_y_low,
context->rectangle_x_high, context->rectangle_y_high,
context->arrival_semaphore_address,
&arrival_destination);
if (code != FT_TENSIX_MULTICAST_OK) {
*out_issue_code = code;
return code;
}
code = ft_tensix_noc_semaphore_set_multicast(context->valid_word_address,
arrival_destination,
context->destination_count);
*out_issue_code = code;
return code;
}
/**
* One receiver round: clear the arrival flag, announce readiness, wait for the block.
*
* The local clear comes FIRST. The sender cannot set the flag until it has seen this
* core's atomic, and this core issues that atomic after the clear -- so a VALID can never
* be erased by the clear that was meant to precede it.
*/
static int ftmb_receiver_round(const ftmb_context *context, int *out_issue_code)
{
volatile uint32_t *arrival_semaphore =
(volatile uint32_t *)(unsigned long)context->arrival_semaphore_address;
int code;
ft_tensix_set_local_semaphore(arrival_semaphore, FT_TENSIX_SEMAPHORE_INVALID);
ft_tensix_noc_semaphore_increment(
ft_tensix_noc_address(context->sender_x, context->sender_y,
context->ready_semaphore_address),
1u);
*out_issue_code = FT_TENSIX_MULTICAST_OK;
return ft_tensix_wait_semaphore_equals(arrival_semaphore, FT_TENSIX_SEMAPHORE_VALID,
context->spin_limit, context->stop_flag);
}
/** One DRAM-reader round: exactly the sender's fetch, with no sender and no rendezvous. */
static int ftmb_dram_reader_round(const ftmb_context *context, uint32_t round,
int *out_issue_code)
{
const uint32_t parity = round & 1u;
int code = ftmb_read_from_dram(context, context->dram_address[parity],
context->shared_destination_address);
*out_issue_code = code;
if (code != FT_TENSIX_MULTICAST_OK)
return code;
ft_tensix_noc_read_barrier();
return FT_TENSIX_MULTICAST_OK;
}
static int ftmb_run_round(const ftmb_context *context, uint32_t round, int *out_issue_code)
{
if (context->role == FTMB_ROLE_MULTICAST_SENDER)
return ftmb_sender_round(context, round, out_issue_code);
if (context->role == FTMB_ROLE_MULTICAST_RECEIVER)
return ftmb_receiver_round(context, out_issue_code);
if (context->role == FTMB_ROLE_DRAM_READER)
return ftmb_dram_reader_round(context, round, out_issue_code);
return FT_TENSIX_MULTICAST_ERROR_RECTANGLE; /* an unknown role is a refusal, never a no-op */
}
/* --------------------------------------------------------------------------- */
void kernel_main(void)
{
ftmb_context context;
uint32_t round;
uint32_t rounds_completed = 0u;
uint32_t start_clock = 0u;
uint32_t end_clock = 0u;
int issue_code = FT_TENSIX_MULTICAST_OK;
int error = FT_TENSIX_MULTICAST_OK;
ftmb_read_arguments(&context);
/* Round 0 is untimed: it absorbs the sender waiting for receivers to boot, the first
* DRAM row-buffer miss and the instruction cache. Then `iterations` timed rounds. */
for (round = 0u; round <= context.iterations; round++) {
error = ftmb_run_round(&context, round, &issue_code);
if (error != FT_TENSIX_MULTICAST_OK)
break;
rounds_completed++;
if (round == 0u)
start_clock = ft_bh_wall_clock_low();
}
end_clock = ft_bh_wall_clock_low();
/* Outside the timed window, and unconditional: an abandoned round may have left a
* write or an atomic in flight, and brisck.cc asserts both counters are settled after
* kernel_main() returns. Leaving a transaction outstanding is also how L1 the next
* kernel is about to use gets written underneath it. */
ft_tensix_noc_write_barrier();
ft_tensix_noc_atomic_barrier();
ftmb_publish(&context, rounds_completed, start_clock, end_clock, error, issue_code);
}
Reference in New Issue View Git Blame Copy Permalink