Foundry/tt/convert/ft_tt_weight_convert.h

/* SPDX-License-Identifier: AGPL-3.0-or-later */
/* Copyright (C) 2026 David (Corey) Herrera */
/**
 * @file ft_tt_weight_convert.h
 * @brief The Phase 3 weight conversion path: a GGUF checkpoint in, a tiled
 *   Blackhole weight artifact out, streaming, never full-model resident.
 *
 * ---------------------------------------------------------------------------
 * RUN IT ON THE CARD HOST, NOT THE MAC — MEASURED, and it reverses spec §8.4
 * ---------------------------------------------------------------------------
 * §8.4 put this on the Mac because it is bulk scalar work and the card host is 2
 * no-AVX cores at 1.8 GHz. That reasoning is about CPU and the binding
 * constraint turned out to be MEMORY: peak RSS is **76-215 MB on the card host
 * against 641 MB on macOS**, because `MADV_DONTNEED` is advisory-weak on Darwin
 * and the Mac holds the whole GGUF mapping instead of dropping it behind the
 * read cursor. The card host is also where the checkpoints already are, so nothing
 * crosses the LAN. **MiniCPM5-1B is ~45 s there; the full 50.9 GiB Qwen3.8 is
 * ~23 minutes.** Nothing here uses an SIMD intrinsic, so the same source builds
 * on both and the artifact is byte-identical either way — this is an operating
 * instruction, not a portability limit.
 *
 * ---------------------------------------------------------------------------
 * WHY IT NEVER HOLDS A TENSOR
 * ---------------------------------------------------------------------------
 * The unit of work is a **strip**: 32 logical rows of one tensor, the full
 * width. A strip of the widest tensor in MiniCPM5-1B is a few hundred KB, so
 * peak host memory is set by the strip buffers and not by the model. The GGUF
 * side is an mmap that is read once, sequentially, and advised away behind us.
 * The acceptance number the plan asks for — peak RSS under 8 GB for a full
 * model — is not a close call under this shape; it is measured anyway, because
 * "it should be small" is not a measurement.
 *
 * ---------------------------------------------------------------------------
 * THE ARTIFACT
 * ---------------------------------------------------------------------------
 * Little-endian throughout, which both the Mac and the card host are, and which the
 * card is.
 *
 *   magic 'FTTW' | version | tensor_count | flags
 *   directory_offset | data_offset                       (uint64 each)
 *   ... tensor data, each tensor 64-byte aligned ...
 *   ... directory: tensor_count fixed-size entries ...
 *
 * The directory is written LAST because a tensor's size is not known until it
 * has been converted — which is what lets the whole thing stream to a pipe or
 * a sequentially written file with no seeking back over the data.
 *
 * ---------------------------------------------------------------------------
 * TWO 2-D LAYOUTS, AND WHY A MATMUL WEIGHT GETS THE SECOND ONE
 * ---------------------------------------------------------------------------
 * `layout` in each directory entry says which, and it is a FORMAT field, not a
 * hint: read the wrong one and every index is in range and every answer is
 * wrong.
 *
 *   FT_TT_LAYOUT_SOURCE (0)
 *     GGUF's own order. ceil(rows/32) x ceil(cols/32) tiles in ROW-major tile
 *     order, `rows` = dims[1], `cols` = dims[0] (the fastest axis). This is
 *     what an EMBEDDING table wants, because it is indexed BY ROW: token id
 *     picks a row, and transposing it would turn a lookup into a gather.
 *
 *   FT_TT_LAYOUT_MATMUL_NMAJOR (1)
 *     A matmul weight. TRANSPOSED to K x N = [in_features x out_features] —
 *     the orientation Phase 4 decided (foundry/tt/ft_tt_op_matmul.h) — and its
 *     tile grid stored COLUMN-major, tile index nt*Kt + kt.
 *
 * The transpose has to happen here and nowhere else: a Bfp8_b/Bfp4_b tile
 * shares one exponent across 16 elements ALONG A ROW, so transposing a PACKED
 * tile means unpacking and repacking it. Upstream of the packer it is free.
 *
 * The column-major tile grid is the second half of the same change and it is a
 * measured one. The matmul reader fetches a block of `depth` K-tiles per NoC
 * barrier; in row-major tile order those are strided by Nt, so each is its own
 * transaction, and transaction COUNT is what the reader is bound by — the same
 * observation was worth 1.28x on the A side alone (docs/tt_bringup_log.md
 * §16.2). Column-major makes a block's B tiles consecutive, so the whole block
 * is one read.
 *
 * **It costs nothing to produce.** A conversion strip is 32 SOURCE rows, i.e.
 * one out-feature tile column, and the Kt target tiles it yields are exactly
 * the ones that are consecutive in N-major order. The streaming order the
 * converter already had IS the new order; the only added work is a 32x32
 * transpose per tile.
 *
 * Edge tiles are zero-padded, in whichever axis the padding lands after the
 * transpose.
 *
 * **1-D tensors are stored row-major and untiled, and that is a decision, not
 * an omission.** Blackhole broadcasts a norm weight in hardware, so it never
 * needs 32x replication in DRAM — but ROW broadcast wants a tile with a filled
 * 0-row and COL wants one with a filled 0-column, and which of the two RMSNorm
 * needs follows from the matmul orientation, which is Phase 4's to decide. The
 * untiled vector is the only form BOTH are derivable from, so the artifact
 * stores it and the loader or the reader kernel shapes it. 16 fp32 = 64 B =
 * one face row = the artifact's alignment, so that shaping is an aligned copy.
 * `docs/tt_bringup_log.md` §13.2.
 *
 * @status Converts MiniCPM5-1B end to end at every ladder rung.
 * @issues None
 * @todo None
 */

#ifndef FT_TT_WEIGHT_CONVERT_H
#define FT_TT_WEIGHT_CONVERT_H

#include <stdint.h>
#include <stdio.h>

#include "ft_tt_precision_ledger.h"

/** 'FTTW' little-endian. */
#define FT_TT_ARTIFACT_MAGIC 0x57545446u

/**
 * Bumped whenever the on-disk meaning changes.
 *
 *   1  every 2-D tensor in GGUF order, row-major tile grid, `reserved` zero
 *   2  `reserved` became `layout`, and a matmul weight is transposed to
 *      [in x out] with a column-major tile grid
 *
 * The bump is the point: a v2 artifact fed to a v1 consumer would be read with
 * the wrong orientation and the wrong tile stride, both in range and both
 * silent. A hard version refusal is the only thing that turns that into an
 * error. Readers here accept both and branch on `layout`.
 */
#define FT_TT_ARTIFACT_VERSION 2u

/** The oldest artifact version a reader in this tree still understands. */
#define FT_TT_ARTIFACT_VERSION_MIN 1u

/** How a 2-D tensor's tiles are arranged. See the header note. */
typedef enum {
    FT_TT_LAYOUT_SOURCE = 0,       /**< GGUF [out x in], row-major tile grid */
    FT_TT_LAYOUT_MATMUL_NMAJOR = 1 /**< [in x out], column-major tile grid */
} ft_tt_tensor_layout;

/**
 * Header `flags` bits. A reader that meets a bit it does not know must refuse
 * the file rather than ignore it: every flag here changes what the BYTES MEAN.
 */
/** `attn_q`/`attn_k` output rows are permuted from INTERLEAVED to SPLIT-HALF. */
#define FT_TT_ARTIFACT_FLAG_ROPE_SPLIT_HALF 0x00000001u

/**
 * The rotary PAIR INDEX is reflected: the slot a consumer calls pair `p` holds
 * the model's pair `half - 1 - p`.
 *
 * **ORTHOGONAL to the bit above, and deliberately so.** The two bits answer two
 * independent questions and the four combinations are all meaningful:
 *
 * | flags | WHERE a pair's partner is | WHICH pair a slot holds |
 * |---|---|---|
 * | `0x0` | adjacent (GGUF interleaved) | `p` |
 * | `0x1` | `+half` (split-half) | `p` |
 * | `0x2` | adjacent (GGUF interleaved) | `half - 1 - p` |
 * | `0x3` | `+half` (split-half) | `half - 1 - p` |
 *
 * The consumer owes exactly one thing for this bit: apply frequency
 * `half - 1 - p` at slot `p`. Do that and the attention score is EXACTLY the
 * unreflected one — the sum over pairs is the same set of terms in a different
 * order — and fail to and every rotary feature gets the wrong frequency, which
 * is why this is a header bit and not a naming convention.
 */
#define FT_TT_ARTIFACT_FLAG_ROPE_PAIRS_REVERSED 0x00000002u

/** Every flag bit this build understands. Anything else is from the future. */
#define FT_TT_ARTIFACT_FLAGS_KNOWN                                                                            \
    (FT_TT_ARTIFACT_FLAG_ROPE_SPLIT_HALF | FT_TT_ARTIFACT_FLAG_ROPE_PAIRS_REVERSED)

/**
 * Which rotary pairing `attn_q`/`attn_k` are stored for.
 *
 * ---------------------------------------------------------------------------
 * WHY A WEIGHT PERMUTATION IS A ROPE DECISION
 * ---------------------------------------------------------------------------
 * A head's rotary pairing is a property of the CHECKPOINT, not of the kernel.
 * HF publishes Llama-family q/k projections in SPLIT-HALF order — feature `j`
 * pairs with `j + rot/2` — and llama.cpp's HF->GGUF conversion PERMUTES the
 * output rows into INTERLEAVED order, pairing `(2i, 2i+1)`, because that is
 * what its own `rope_norm` wants. So a GGUF `llama` checkpoint is a permuted
 * checkpoint, and that permutation is the only reason our on-card RoPE kernel
 * cannot serve it: `ft_tt_op_rope.h` refuses interleaved pairing because an
 * interleaved partner is an adjacent column INSIDE a face, where a split-half
 * partner (with `rot/2` a multiple of 32) is the SAME position in a different
 * TILE — a permutation of tiles and the identity inside one.
 *
 * Running llama.cpp's permutation backwards in the converter therefore turns a
 * kernel capability gap into a build-time row shuffle.
 *
 * **IT CHANGES NO MATH, AND THAT IS PROVABLE RATHER THAN ASSERTED.** Attention
 * consumes q and k only through `q . k`, which is invariant under a permutation
 * applied to BOTH — and the same one is, because it is a function of the head
 * geometry, not of the tensor. `v` is not permuted and `attn_output` is not
 * permuted, so nothing downstream of the score ever sees a reordered feature
 * axis.
 *
 * What DOES move, and it is worth stating rather than discovering: a
 * block-float format shares one exponent across 16 elements of a stored tile
 * ROW, which after the N-major transpose is 16 consecutive OUTPUT features.
 * Permuting the output rows regroups those 16, so a Bfp8_b artifact is not
 * bit-identical to its unpermuted twin — it is a different, equally valid
 * quantization of the same weights. At the bf16 and fp32 rungs there are no
 * blocks and the permutation is EXACTLY inert, which is the rung the unit test
 * pins it at.
 *
 * ---------------------------------------------------------------------------
 * THE CONSUMER HAS TO BE TOLD, AND THE ARTIFACT IS WHERE
 * ---------------------------------------------------------------------------
 * A permuted `attn_q` fed to an interleaved RoPE is silently wrong in exactly
 * the way this converter's `layout` field exists to prevent, so the choice is
 * recorded in the header as FT_TT_ARTIFACT_FLAG_ROPE_SPLIT_HALF and the host
 * forward selects its rotary form from the FLAG, not from the architecture.
 */
/**
 * ---------------------------------------------------------------------------
 * AND WHY THERE ARE TWO MORE THAT NO CHECKPOINT ASKS FOR
 * ---------------------------------------------------------------------------
 * The last two values are INSTRUMENTS, not checkpoint formats, and they exist to
 * answer one question with a control rather than an argument.
 *
 * `--rope-pairing split-half` took the chat-parity gate's worst document from
 * **1.227205 to 1.032823** with nothing else moving — the artifacts are the same
 * size, the same tensors, the same ledger, and the only difference is which 16
 * output features share a block-float exponent. Two artifacts is a sample of
 * two, so `SPLIT_HALF_REVERSED` is a THIRD draw over the SAME two tensors: the
 * pair index is reflected, so the exponent blocks regroup again while the
 * attention score stays exact (`q . k` sums one term per pair, and reflecting
 * the labelling permutes the summands without changing the set).
 *
 * `INTERLEAVED_REVERSED` is the crossed control that separates the two things
 * split-half changed at once. Split-half did two things: it moved a pair's
 * partner from ADJACENT to `+half`, and it therefore stopped a pair's two
 * components sharing an exponent block. This value reflects the pair index while
 * KEEPING the partner adjacent — a different draw of the same shape as source.
 * So the four values isolate the variable:
 *
 * | value | partner | exponent block holds |
 * |---|---|---|
 * | `SOURCE` | adjacent | both components of 8 pairs |
 * | `SPLIT_HALF` | `+half` | one component of 16 pairs |
 * | `SPLIT_HALF_REVERSED` | `+half` | one component of 16 other pairs |
 * | `INTERLEAVED_REVERSED` | adjacent | both components of 8 other pairs |
 *
 * Neither instrument is loadable by the on-card RoPE kernel: its table is built
 * with the frequency in slot order, and it refuses an adjacent partner outright.
 * The host forward refuses both by name rather than serving them wrong.
 */
typedef enum {
    FT_TT_ROPE_PAIRING_SOURCE = 0,     /**< store `attn_q`/`attn_k` exactly as GGUF has them */
    FT_TT_ROPE_PAIRING_SPLIT_HALF = 1, /**< undo llama.cpp's permutation; NeoX/split-half order */
    FT_TT_ROPE_PAIRING_SPLIT_HALF_REVERSED = 2, /**< split-half, pair index reflected. Instrument. */
    FT_TT_ROPE_PAIRING_INTERLEAVED_REVERSED = 3 /**< GGUF order, pair index reflected. Instrument. */
} ft_tt_rope_pairing;

/** Tensor payloads start on this boundary; the NoC's DRAM alignment is 64 B. */
#define FT_TT_ARTIFACT_ALIGNMENT 64u

/** Name field width in a directory entry. Matches ft_gguf_map's limit. */
#define FT_TT_ARTIFACT_NAME_BYTES 128

/** On-disk artifact header. 40 bytes, no padding by construction. */
typedef struct {
    uint32_t magic;
    uint32_t version;
    uint32_t tensor_count;
    uint32_t flags;
    uint64_t directory_offset;
    uint64_t data_offset;
    uint64_t total_bytes;
} ft_tt_artifact_header;

/** On-disk directory entry. Fixed size, so the directory is indexable. */
typedef struct {
    char name[FT_TT_ARTIFACT_NAME_BYTES];
    uint32_t format;    /**< ft_tt_tensor_format */
    uint32_t role;      /**< ft_tt_tensor_role, for diagnostics */
    uint32_t n_dims;
    /** The STORED tile grid, not the logical one. For FT_TT_LAYOUT_MATMUL_NMAJOR that
     *  means tile_rows = Kt = ceil(in/32) and tile_cols = Nt = ceil(out/32), i.e. the
     *  transpose of what `dims` says. `dims` stays logical and in GGML order. */
    uint32_t tile_rows; /**< 0 when the tensor is stored untiled */
    uint32_t tile_cols;
    uint32_t layout;    /**< ft_tt_tensor_layout; always 0 in a v1 artifact */
    uint64_t dims[4];   /**< logical, GGML order */
    uint64_t data_offset;
    uint64_t byte_count;
} ft_tt_artifact_entry;

/** The two on-disk structures are a format, so their sizes are pinned. */
#define FT_TT_ARTIFACT_HEADER_BYTES 40u
#define FT_TT_ARTIFACT_ENTRY_BYTES 200u

/** Per-tensor result, reported as the conversion runs. */
typedef struct {
    char name[FT_TT_ARTIFACT_NAME_BYTES];
    ft_tt_tensor_role role;
    ft_tt_tensor_format format;
    uint64_t rows;
    uint64_t cols;
    uint64_t source_bytes;
    uint64_t converted_bytes;
    double pcc;      /**< against the fp32 source; 1.0 when the format is fp32 */
    double max_abs_error;
    double rms_error;
    /**
     * rms_error divided by the RMS of the SOURCE, i.e. the error as a fraction
     * of the signal. This, not `rms_error`, is the number that compares across
     * tensor types: a norm weight and an ffn_down live at different scales, so
     * their absolute RMS errors are not on the same axis and ranking them by it
     * ranks them by magnitude. 0.0 for an all-zero tensor.
     */
    double relative_rms_error;
    /**
     * Relative RMS of the SOURCE against the reference, when
     * ft_tt_convert_measure_against() supplied one. Zero otherwise.
     *
     * This is the half of a double transcode that the on-card format is not
     * responsible for: converting a Q8_0 checkpoint measures
     * `relative_rms_error` against the bf16 the model was released as, and this
     * field says how much of that was already spent before the packer ran.
     */
    double source_relative_rms_error;
} ft_tt_convert_tensor_report;

/** Called once per tensor, in directory order. Return non-zero to abort. */
typedef int (*ft_tt_convert_progress_fn)(const ft_tt_convert_tensor_report *report, void *user);

/** Options for one conversion run. */
typedef struct {
    ft_tt_ledger_mode mode;
    int measure_pcc; /**< round-trip every tensor and report PCC */
    ft_tt_convert_progress_fn progress;
    void *progress_user;
    ft_tt_rope_pairing rope_pairing; /**< default 0 = FT_TT_ROPE_PAIRING_SOURCE */
    /**
     * A MEASUREMENT INSTRUMENT, and deliberately not a ledger rung.
     *
     * Every 2-D tensor whose name CONTAINS this substring is encoded at
     * `format_override` instead of the format the ledger chose. NULL disables
     * it, and 1-D tensors (norms, biases) are never touched — a shared exponent
     * over a normalization scale is not what this measures.
     *
     * It exists to answer a question the ledger cannot express: what share of
     * the artifact's total squared error does ONE tensor family carry? Degrade
     * that family alone, read the residual-divergence plateau, and the share
     * inverts directly, because divergence is linear in injected error. Naming a
     * SUBSTRING rather than a role is the whole point — `attn_output` and
     * `attn_q` are one role, and the question is about one of them.
     *
     * It is not a rung because an artifact built this way must never be mistaken
     * for a shipping one: it has no name on the ladder, the converter announces
     * every tensor it moves, and `ft_tt_fttw_list` reads the real formats back
     * out of the written file.
     */
    const char *format_override_name_contains;
    ft_tt_tensor_format format_override; /**< read only when the substring is set */
} ft_tt_convert_options;

/** Status codes. Negative is failure. */
enum {
    FT_TT_CONVERT_OK = 0,
    FT_TT_CONVERT_ERROR = -1,
    FT_TT_CONVERT_ENOMEM = -2,
    FT_TT_CONVERT_EINVAL = -3,
    FT_TT_CONVERT_EIO = -4,
    FT_TT_CONVERT_EUNSUPPORTED = -8,   /**< a tensor type this path cannot read */
    FT_TT_CONVERT_EUNCLASSIFIED = -9,  /**< a 2-D tensor whose role no rule recognises */
    FT_TT_CONVERT_EROPEGEOMETRY = -10  /**< rope pairing asked for, head geometry does not allow it */
};

/**
 * Which layout a tensor is stored in, from its role and rank — or a REFUSAL.
 *
 * Only a MATMUL WEIGHT is transposed. The distinction that matters is
 * `token_embd`, which is 2-D and looks exactly like a weight: it is indexed BY
 * ROW by a token id, so transposing it would turn every embedding lookup into
 * a strided gather across the whole tensor. `output`/`lm_head` IS a matmul
 * weight and is transposed, even though it shares the embedding's shape.
 *
 * **AN UNRECOGNISED 2-D TENSOR HAS NO ANSWER AND THIS FUNCTION SAYS SO.** It
 * used to return FT_TT_LAYOUT_SOURCE, on the reasoning that a transpose we have
 * not reasoned about is the change that would be silently wrong. That reasoning
 * was half right and the half it missed is the dangerous one: NOT transposing a
 * tensor the kernels read N-major is equally silent and equally wrong, and it
 * is what happened to `ssm_out` — a real 6144->5120 matmul weight — the first
 * time this classifier met a GDN model. There is no safe default because BOTH
 * defaults are a guess about a kernel's expectations. So the guess is refused
 * and the caller has to add a rule.
 *
 * @param role    from ft_tt_ledger_role()
 * @param n_dims  tensor rank, as GGUF declares it
 * @param out     receives FT_TT_LAYOUT_SOURCE or FT_TT_LAYOUT_MATMUL_NMAJOR;
 *                untouched when the role has no layout
 * @return        FT_TT_CONVERT_OK, or FT_TT_CONVERT_EUNCLASSIFIED for a 2-D
 *                tensor of unknown role
 *
 * Thread-safe: yes    Reentrant: yes    Signal-safe: yes
 */
int ft_tt_convert_layout_for_checked(ft_tt_tensor_role role, uint32_t n_dims, ft_tt_tensor_layout *out);

/**
 * Convert a GGUF checkpoint to a Blackhole weight artifact.
 *
 * @param gguf_path    input checkpoint
 * @param output_path  artifact to create, truncating any existing file
 * @param options      may be NULL for defaults (production ledger, no PCC)
 * @return             FT_TT_CONVERT_OK, or a negative status
 *
 * Thread-safe: yes (no shared state)
 * Reentrant: yes
 * Signal-safe: no
 */
int ft_tt_convert_gguf(const char *gguf_path, const char *output_path, const ft_tt_convert_options *options);

/**
 * Measure only: walk every tensor and report the PCC each candidate format
 * would give, writing nothing. This is the Phase 3 quality gate.
 *
 * @param gguf_path  input checkpoint
 * @param format     the candidate format to score every 2-D tensor at
 * @param progress   called once per tensor
 * @param user       passed through to `progress`
 * @return           FT_TT_CONVERT_OK, or a negative status
 */
int ft_tt_convert_measure(const char *gguf_path, ft_tt_tensor_format format, ft_tt_convert_progress_fn progress,
                          void *user);

/**
 * Measure, but score the round trip against a SECOND checkpoint rather than
 * against the input's own values.
 *
 * The question this answers is the one a lossy source raises and a self-scoring
 * gate cannot: converting a Q8_0 checkpoint and comparing the result to the
 * Q8_0 values reports the block-float error only, and quietly treats the Q8_0
 * quantization as if it were the truth. Point `reference_gguf_path` at the
 * BF16 release the model was published as and the report is against the actual
 * weights, with `source_relative_rms_error` splitting out how much of it the
 * source had already spent.
 *
 * The two checkpoints must agree on every tensor's name and shape; a tensor
 * missing from the reference, or a shape mismatch, is an error rather than a
 * skip — silently scoring 90% of a model is how a regression hides.
 *
 * @param gguf_path            the checkpoint being converted
 * @param reference_gguf_path  the truth to score against; NULL means self-score
 * @param format               the candidate format for every 2-D tensor
 * @return                     FT_TT_CONVERT_OK, or a negative status
 */
int ft_tt_convert_measure_against(const char *gguf_path, const char *reference_gguf_path,
                                  ft_tt_tensor_format format, ft_tt_convert_progress_fn progress, void *user);

#endif /* FT_TT_WEIGHT_CONVERT_H */