# Barswap — Canonical Quick Reference

**This is the authoritative current-state doc.** The other files in this folder are
deep-dives and history; where they show interim die counts ("1 of 2 dies", "6/6"),
those are *historical* — the current state is below. Read this first.

> One sentence: the BTC-S37 board can map **only one 16 GB GPU die into CPU
> address space at a time**, so all 16 K80 dies are **time-multiplexed** through a
> single MMIO slot via `bar_swap.ko` — acquire a die, use it, release, repeat.

## Current hardware reality (verified 2026-03-25)

| Fact | Value |
|---|---|
| Board / CPU | BTC-S37, Intel Pentium 2117U (Ivy Bridge, **36-bit phys = 64 GB**), 8 GB DDR3 |
| GPUs | 8× K80 = **16 dies, 192 GB VRAM** — **16/16 tested and passing** |
| Working MMIO slot | **exactly one**, at `0xC00000000` (16 GB) |
| Simultaneous dies | **1** (`bs_num_slots()` == 1). The other two 16 GB-aligned addresses (`0x400000000`, `0x800000000`) hang/timeout — see `MULTI_SLOT.md` |
| Multi-slot | **hardware-impossible** on this board without a risky BIOS reflash (closed; `MULTI_SLOT.md`) |
| Swap cost | ~0.2–1.2 ms BAR remap (nvidia stays loaded, context persists); ~2.5 s only if you reload nvidia |
| VRAM + context | **persist across release/acquire** — don't destroy the context on swap (Architecture A, confirmed) |
| **In-flight compute** | **continues executing on-die while the die is released** — the BAR is only a CPU access window, not needed during kernel execution (**`compute_persist` PASS 2026-06-26**) |
| Driver / toolkit | nvidia **470** (Kepler→Turing); CUDA **11.4**; K80 = **sm_37** |

**Architectural consequence — parallel compute behind a serialized control plane.**
The single MMIO slot multiplexes only the **control/IO** path (launch kernels, copy
inputs/results in and out). It is **not** a compute bottleneck: once work is launched
on a die, that die keeps computing on its local VRAM after it's released from the slot
(BAR = doorway, not the room). So all 16 dies can compute **in parallel**;
the slot is time-multiplexed (~0.2–1.2 ms/visit) just to dispatch work and collect
finished results. Effective aggregate throughput ≈ N × single-die, minus control-plane
serialization — bounded by *(slot-time per die-visit) ÷ (on-die compute-time per step)*,
which is small when compute ≫ launch/copy (true on the slow K80). This is the
`round-robin-training.md` scheduler pattern applied to inference.

> **Load-bearing assumption — CONFIRMED (2026-06-26, `compute_persist` die 1).**
> Calibrated a 16.2 s kernel, launched it async, **released the die from the slot at
> 128 ms** (GPU unmapped from MMIO), waited **48.7 s with the die released**, then
> re-acquired: `cuCtxSynchronize` returned in 0.02 ms ("work already done") with
> **1,048,576 / 1,048,576 elements correct, 0 zeros**. In-flight GPU compute continues
> after BAR unmap — the parallel-compute model holds. Switch overhead ~0.2 ms (launch
> only); "top-2 cost = top-1 cost (parallel, not sequential)."

## Build recipe (CUDA on this box)

The default toolchain does **not** work out of the box; these are required:

- Use the **separately-installed CUDA 11.4** at `/usr/local/cuda-11.4`.
  **Do NOT** use the apt `nvidia-cuda-toolkit` package — it installs CUDA 12, which
  **dropped sm_37** (K80). 11.4 is the last toolkit supporting Kepler.
- nvcc host compiler must be **gcc-11** (`-ccbin gcc-11`). Ubuntu 24.04's default is
  gcc-13; nvcc 11.4 refuses any gcc > 11 (`unsupported GNU version`).
- glibc 2.39 needs the **`-isystem nvcc_shim`** include shim (a patched `sys/cdefs.h`),
  or nvcc dies on `error: attribute "__malloc__" does not take arguments` from
  `/usr/include/stdio.h`.

All three are already wired into `cuda_kernels/Makefile`. To (re)compile the kernels
to PTX and regenerate the embedded `cuda/fc_cuda_ptx.h`:

```bash
cd foundry/cuda_kernels && make      # nvcc -ptx -arch=sm_37 -ccbin gcc-11 -isystem nvcc_shim
```

This needs only the host CPU (no GPU, no driver). The generated `fc_cuda_ptx.h` is
committed so foundry/Pantheon build without nvcc on dev machines.

## Manual bring-up (today's procedure)

nvidia is **blacklisted on purpose** (`/etc/modprobe.d/blacklist-nvidia.conf` with
`install … /bin/false`) so it never auto-loads and corrupts GPU state. Bring it up by
hand, **in this order**:

```bash
# 1. Load the barswap module — registers /dev/bar_swap, slot starts empty
sudo insmod ~/foundry/barswap/bar_swap_ko/bar_swap.ko
sudo dmesg | grep bar_swap            # expect 16 dies listed, slot empty

# 2. Acquire ONE die into the 0xC00000000 slot (userspace lib: bs_acquire(die_id))
#    e.g. via a small program linking barswap.c, or a barswap CLI

# 3. Load the nvidia driver (it's blacklisted, so --ignore-install)
sudo modprobe --ignore-install nvidia nvidia_modeset nvidia_drm nvidia_uvm

# 4. Verify — read CACHED driver data, never probe hardware
cat /proc/driver/nvidia/gpus/*/information
```

### Hazards (this box is remote/headless — a wrong move means a physical trip)

- **Never `nvidia-smi` on a swapped die.** It probes hardware (temp/power/ECC) and can
  hang or crash the die. Use `/proc/driver/nvidia/gpus/<BDF>/information` instead.
- **Never automate PCI manipulation at boot.** A systemd service doing setpci/rescan/
  modprobe froze the box and required a physical power cycle + recovery mode. Barswap
  bring-up must be **on-demand from userspace**, never a boot unit. (The intended home
  for this is *inside Pantheon*, acquiring dies as needed — see the Nex-N2 spec.)

### Why nvidia is blacklisted — and the cleaner replacement

**Why (root cause):** at boot the 16 dies are in a broken-BAR state (single-slot
constraint — most have no valid CPU-routable BAR). If nvidia auto-loads it **binds all
16 dies** and probes them (`RmInitAdapter`); accessing the broken BARs / config space
soft-locks the kernel (`pci_conf1_read` never returns), and even when it survives,
nvidia stays bound and **corrupts state when bar_swap later disables a die's decode**.
The blacklist (`blacklist` + `install … /bin/false` + `update-initramfs -u`) is the
blunt fix: stop nvidia loading at all, load it by hand after a die is acquired.

**Why it's hacky:** it's system-wide — it kills nvidia for the *whole box*, so you can't
run a normal nvidia GPU alongside the K80s; and it needs `/etc/modprobe.d` edits +
initramfs regen that don't survive cleanly as a shipped product.

**Ship-grade replacement (planned): make `bar_swap` *bind* the dies (vfio-pci / pci-stub
pattern).** Today `bar_swap` claims die *resources* (`pci_claim_resource`) but is **not a
registered `pci_driver`** — so nothing stops nvidia binding them; hence the blacklist.
Register `bar_swap` as a `pci_driver` matching the GK210 device ID so the kernel **binds
all 16 dies to bar_swap at probe** → they're invisible to nvidia → no probe, no panic, no
corruption — *without* a global blacklist. On `acquire`, hand the one slot die to nvidia
(unbind → nvidia binds it); the other 15 stay parked with bar_swap. A box could then run
a normal nvidia GPU *and* the K80s at once. **Zero-code de-risk first:** put the GK210
ID in `pci-stub.ids=`/`vfio-pci.ids=` on the kernel cmdline and check nvidia boots clean,
before writing the bind path into `bar_swap`.
- **One die *mapped* at a time** (slot occupancy, not compute — released dies keep
  computing). Acquiring a new die implicitly releases the active one.
- **Don't destroy the CUDA context on swap** — it survives release/acquire; tearing it
  down forces a full ~2.5 s reload.
- **CUDA atexit teardown hangs on this rig (confirmed 2026-06-26).** When a process with
  a CUDA context exits normally, the atexit context-destroy wedges a thread against the
  bar_swap'd die — the process becomes a `Zl` zombie and nvidia ends up half-unloaded
  (`NVRM: kernel module is unloaded` while `lsmod` still lists it). These zombies can't
  be killed and accumulate until the box needs a power cycle (it also makes clean
  shutdown crawl). **Workaround for short-lived processes (tests, one-shots): call
  `_Exit()` instead of `return`/`exit()`** — skip the userspace teardown and let the
  kernel reclaim the context on process death (clean exit, no zombie; verified with the
  kernel-parity harness). **Proper fix (TODO, product-critical — every model process
  exit hits this):** an ordered context-destroy + `bs_release` before exit, and/or the
  device-binding driver work above. A long-lived Pantheon server avoids it by not
  exiting; it still needs a clean teardown path for shutdown.

## barswap userspace API (`barswap.h`)

```c
bs_context* bs_init(void);                       // open /dev/bar_swap
uint32_t    bs_num_dies(ctx);                    // 16 on this rig
uint32_t    bs_num_slots(ctx);                   // 1 (v2 — hardware limit)
bs_gpu*     bs_acquire(ctx, int die_id);         // program die into the slot, mmap VRAM
void        bs_release(ctx, bs_gpu*);            // vacate the slot
int         bs_active_die(ctx);                  // die in the slot, or -1
int         bs_is_mapped(ctx, int die_id);
int         bs_refresh(ctx);
void        bs_free(ctx);
```

Kernel module ioctls on `/dev/bar_swap`: `BAR_SWAP_ACQUIRE` / `…RELEASE` / `…QUERY`.

## Field service — identifying a physical card (future feature, not built)

This rig has **no BMC/IPMI** and the K80 bracket LED isn't software-programmable, so
there's no turnkey "locator LED." Two practical ways to find a specific/failing card,
worth building into the barswap-in-Pantheon tooling later:

1. **BDF → physical map (deterministic).** Every die has a fixed PCI address (die 0 =
   `03:00.0`, die 1 = `04:00.0`, Card B = `07/08:00.0`, … — see `CARDS.md`).
   A failing die's BDF → its card → its riser. **One-time win: physically label each
   riser with its BDF at install** so a bad die maps to a card by eye.
2. **`identify <die>` by workload (lights up the card).** Acquire *only* the suspect die
   and run a continuous busy kernel for ~30 s — its **activity LED flickers** while the
   others stay dark, so you can physically spot it. barswap's single-die acquire makes
   this clean. This is the small operability feature to add (a `pantheon --barswap
   identify <die>` mode).

Also useful and already possible: a *failed* K80 often shows **amber** on the bracket LED
(passive), and barswap can **disable/skip a die by BDF** in software to route around a
flaky card until it's pulled.

## Where to read more

- **`MULTI_SLOT.md`** — *why* only one slot works (Ivy Bridge 36-bit addressing; closed).
- **`CARDS.md`** — GPU compatibility (which cards need barswap; T4/P4 don't) + current rig.
- **`memoryTricks.md`** — barswap-as-scheduler notes for multi-model hosting.