# SR4 Operating Guide — how it behaves, and how to train with it well

> This is the **operational** knowledge about the SR4 optimizer — distilled from the
> SR4 MNIST research campaign (reached the FP32 ceiling, memory-neutral) and the first RWKV-x
> LM runs. `HOW_IT_WORKS.md` covers the *mechanism*; this covers *how to actually get good
> results with it and how to read its behavior.* If you are training with SR4, read this first.

## The one root fact everything follows from

**SR4 is stochastic on every update.** The int4 accumulator `q` is integrated by *stochastic
rounding* — so the per-update gradient **noise is amplified** into the weights, every step,
every weight. A deterministic optimizer (SGD/Adam) only touches randomness at init + data
order; SR4 threads it through all ~10⁵ updates. Almost every operating rule below is a
consequence of this.

## Rule 1 — Batch size is the DOMINANT lever. Never run tiny batches.

Bigger batch → lower per-update gradient variance → the stochastic rounding integrates a
**cleaner** signal. This is the single most important knob.
- On MNIST, going batch **128 → 2048** was the path from "SR4 works" to "SR4 *matches
  full-precision-dense*." Every variance-reducer we found was subsumed by big batch at the top.
- **`batch=1` is pathological** — maximum per-update noise, and SR4 will **plateau** while
  Adam (batch-insensitive due to per-coordinate adaptivity) descends right past it. If you see
  that pattern, the fix is **batch, not learning rate.**
- Diagnostic signature of "SR4 starved of batch": **noisy/spiky train loss + an early
  plateau**, while a reference optimizer keeps improving.

## Rule 2 — SR4 has its own learning rate, and it is NOT Adam's

The step size is `TARGET_STEP` (the mean proposed step in LSBs per update). It must be tuned
**per optimizer and per scale** — do a small sweep at each new rung. Observed optima:
MNIST-MLP `0.001`, LM memorization `0.003`, LM real-data `~0.001`. **Never share an LR with
Adam** in an A/B — that rigs the comparison. Compare each optimizer *at its own best LR*, at
matched **tokens**, and report wall-clock separately.

## Rule 3 — Report seed BANDS, never single runs

Because SR4 is stochastic every step, final quality varies by seed. Run **≥3 seeds** and
compare **band means with a noise margin** (MNIST band ≈ 0.25pp; a gain must *clear* the band
to count). A single lucky/unlucky seed is not a result.

## Rule 4 — Variance-reducers compound (super-additively)

Anything that lowers the noise floor lets small mean-gains become visible and stackable:
- **Low-discrepancy (Weyl) dither** — built into the core; variance-reduced *unbiased*
  stochastic rounding (free, always on).
- **Batch** — the big one (Rule 1).
- **Dropout** — a *tunable reliability dial*: more dropout → tighter seed band (its cost is
  mean, so anneal/gentle-decay it; regime-dependent — see Rule 6).
Small sub-band mechanisms that look like noise individually start *stacking* once the floor
drops.

## Rule 5 — The recipe (core vs knobs)

The **core** is small: int4 accumulator + stochastic-rounded integer update + ternary forward.
Everything else is **recipe**, and it lives in different places:
- **Batch size** → training loop (Rule 1, dominant).
- **Co-trained per-column FP32 scales** → per layer; restore the dynamic range ternary weights
  throw away (`Z = scale ⊙ (ternaryW·x)`). ~0.04 b/w. Basically required for quality.
- **Label smoothing** → loss side (biggest single regularizer on MNIST).
- **Dropout / annealed dropout** → the reliability dial (Rule 4).
The core got MNIST to ~97; the recipe got it to the FP32 ceiling (~98.3). *Don't judge SR4
without its recipe.*

## Rule 6 — Regime matters

A lever's value depends on its base. Examples that actually flipped: early-release dropout was
the **worst** schedule at batch 1024 but the **best** at batch 2048; a higher step helped at
moderate batch and was null at large batch. **Tune levers in the target regime**; do not assume
a lever that helped at one scale/batch transfers.

## Rule 7 — Which params are ternary (the split)

SR4 ternarizes the **large weight matrices** (the projections — the bulk of params, the memory
win). Keep **small/precision-sensitive params continuous** (float, plain optimizer): embeddings,
the LM head, LayerNorm γ/β, LoRA/low-rank gates, biases, and the per-column scales themselves.
This mirrors the BitNet/ternary-LLM convention. Each ternary matrix gets its own `idx_base` so
the SR dither lanes stay independent.

## Rule 8 — Think in updates/tokens, not epochs

The meaningful invariant is the number of weight **updates** (and, at LM scale, **tokens**) —
not passes over the data. When you change batch size, hold updates constant (scale steps
accordingly), or you are changing two things at once.

## Quick reference — reading an SR4 run

| You see | It usually means | Do |
|---|---|---|
| Spiky train loss + early plateau; Adam descends past it | per-update noise too high | **raise batch** (Rule 1) |
| Whole seed band wide | noise floor high | add a variance-reducer (batch / dither / dropout) |
| Loss won't drop at all | LR wrong or scales missing | check `TARGET_STEP` sweep + that per-column scales are on |
| Matches Adam only after the recipe | expected | that's the point — SR4 core + recipe ≈ full precision, at ~24× less optimizer memory |

---
*Provenance: the SR4 MNIST research campaign and the RWKV-x SR4-vs-AdamW scale test (both
unpublished). Update this as behavior at LM scale is learned.*