179 lines
11 KiB
Markdown
179 lines
11 KiB
Markdown
# Barswap — Canonical Quick Reference
|
||
|
||
**This is the authoritative current-state doc.** The other files in this folder are
|
||
deep-dives and history; where they show interim die counts ("1 of 2 dies", "6/6"),
|
||
those are *historical* — the current state is below. Read this first.
|
||
|
||
> One sentence: the BTC-S37 board can map **only one 16 GB GPU die into CPU
|
||
> address space at a time**, so all 16 K80 dies are **time-multiplexed** through a
|
||
> single MMIO slot via `bar_swap.ko` — acquire a die, use it, release, repeat.
|
||
|
||
## Current hardware reality (verified 2026-03-25)
|
||
|
||
| Fact | Value |
|
||
|---|---|
|
||
| Board / CPU | BTC-S37, Intel Pentium 2117U (Ivy Bridge, **36-bit phys = 64 GB**), 8 GB DDR3 |
|
||
| GPUs | 8× K80 = **16 dies, 192 GB VRAM** — **16/16 tested and passing** |
|
||
| Working MMIO slot | **exactly one**, at `0xC00000000` (16 GB) |
|
||
| Simultaneous dies | **1** (`bs_num_slots()` == 1). The other two 16 GB-aligned addresses (`0x400000000`, `0x800000000`) hang/timeout — see `MULTI_SLOT.md` |
|
||
| Multi-slot | **hardware-impossible** on this board without a risky BIOS reflash (closed; `MULTI_SLOT.md`) |
|
||
| Swap cost | ~0.2–1.2 ms BAR remap (nvidia stays loaded, context persists); ~2.5 s only if you reload nvidia |
|
||
| VRAM + context | **persist across release/acquire** — don't destroy the context on swap (Architecture A, confirmed) |
|
||
| **In-flight compute** | **continues executing on-die while the die is released** — the BAR is only a CPU access window, not needed during kernel execution (**`compute_persist` PASS 2026-06-26**) |
|
||
| Driver / toolkit | nvidia **470** (Kepler→Turing); CUDA **11.4**; K80 = **sm_37** |
|
||
|
||
**Architectural consequence — parallel compute behind a serialized control plane.**
|
||
The single MMIO slot multiplexes only the **control/IO** path (launch kernels, copy
|
||
inputs/results in and out). It is **not** a compute bottleneck: once work is launched
|
||
on a die, that die keeps computing on its local VRAM after it's released from the slot
|
||
(BAR = doorway, not the room). So all 16 dies can compute **in parallel**;
|
||
the slot is time-multiplexed (~0.2–1.2 ms/visit) just to dispatch work and collect
|
||
finished results. Effective aggregate throughput ≈ N × single-die, minus control-plane
|
||
serialization — bounded by *(slot-time per die-visit) ÷ (on-die compute-time per step)*,
|
||
which is small when compute ≫ launch/copy (true on the slow K80). This is the
|
||
`round-robin-training.md` scheduler pattern applied to inference.
|
||
|
||
> **Load-bearing assumption — CONFIRMED (2026-06-26, `compute_persist` die 1).**
|
||
> Calibrated a 16.2 s kernel, launched it async, **released the die from the slot at
|
||
> 128 ms** (GPU unmapped from MMIO), waited **48.7 s with the die released**, then
|
||
> re-acquired: `cuCtxSynchronize` returned in 0.02 ms ("work already done") with
|
||
> **1,048,576 / 1,048,576 elements correct, 0 zeros**. In-flight GPU compute continues
|
||
> after BAR unmap — the parallel-compute model holds. Switch overhead ~0.2 ms (launch
|
||
> only); "top-2 cost = top-1 cost (parallel, not sequential)."
|
||
|
||
## Build recipe (CUDA on this box)
|
||
|
||
The default toolchain does **not** work out of the box; these are required:
|
||
|
||
- Use the **separately-installed CUDA 11.4** at `/usr/local/cuda-11.4`.
|
||
**Do NOT** use the apt `nvidia-cuda-toolkit` package — it installs CUDA 12, which
|
||
**dropped sm_37** (K80). 11.4 is the last toolkit supporting Kepler.
|
||
- nvcc host compiler must be **gcc-11** (`-ccbin gcc-11`). Ubuntu 24.04's default is
|
||
gcc-13; nvcc 11.4 refuses any gcc > 11 (`unsupported GNU version`).
|
||
- glibc 2.39 needs the **`-isystem nvcc_shim`** include shim (a patched `sys/cdefs.h`),
|
||
or nvcc dies on `error: attribute "__malloc__" does not take arguments` from
|
||
`/usr/include/stdio.h`.
|
||
|
||
All three are already wired into `cuda_kernels/Makefile`. To (re)compile the kernels
|
||
to PTX and regenerate the embedded `cuda/fc_cuda_ptx.h`:
|
||
|
||
```bash
|
||
cd foundry/cuda_kernels && make # nvcc -ptx -arch=sm_37 -ccbin gcc-11 -isystem nvcc_shim
|
||
```
|
||
|
||
This needs only the host CPU (no GPU, no driver). The generated `fc_cuda_ptx.h` is
|
||
committed so foundry/Pantheon build without nvcc on dev machines.
|
||
|
||
## Manual bring-up (today's procedure)
|
||
|
||
nvidia is **blacklisted on purpose** (`/etc/modprobe.d/blacklist-nvidia.conf` with
|
||
`install … /bin/false`) so it never auto-loads and corrupts GPU state. Bring it up by
|
||
hand, **in this order**:
|
||
|
||
```bash
|
||
# 1. Load the barswap module — registers /dev/bar_swap, slot starts empty
|
||
sudo insmod ~/foundry/barswap/bar_swap_ko/bar_swap.ko
|
||
sudo dmesg | grep bar_swap # expect 16 dies listed, slot empty
|
||
|
||
# 2. Acquire ONE die into the 0xC00000000 slot (userspace lib: bs_acquire(die_id))
|
||
# e.g. via a small program linking barswap.c, or a barswap CLI
|
||
|
||
# 3. Load the nvidia driver (it's blacklisted, so --ignore-install)
|
||
sudo modprobe --ignore-install nvidia nvidia_modeset nvidia_drm nvidia_uvm
|
||
|
||
# 4. Verify — read CACHED driver data, never probe hardware
|
||
cat /proc/driver/nvidia/gpus/*/information
|
||
```
|
||
|
||
### Hazards (this box is remote/headless — a wrong move means a physical trip)
|
||
|
||
- **Never `nvidia-smi` on a swapped die.** It probes hardware (temp/power/ECC) and can
|
||
hang or crash the die. Use `/proc/driver/nvidia/gpus/<BDF>/information` instead.
|
||
- **Never automate PCI manipulation at boot.** A systemd service doing setpci/rescan/
|
||
modprobe froze the box and required a physical power cycle + recovery mode. Barswap
|
||
bring-up must be **on-demand from userspace**, never a boot unit. (The intended home
|
||
for this is *inside Pantheon*, acquiring dies as needed — see the Nex-N2 spec.)
|
||
|
||
### Why nvidia is blacklisted — and the cleaner replacement
|
||
|
||
**Why (root cause):** at boot the 16 dies are in a broken-BAR state (single-slot
|
||
constraint — most have no valid CPU-routable BAR). If nvidia auto-loads it **binds all
|
||
16 dies** and probes them (`RmInitAdapter`); accessing the broken BARs / config space
|
||
soft-locks the kernel (`pci_conf1_read` never returns), and even when it survives,
|
||
nvidia stays bound and **corrupts state when bar_swap later disables a die's decode**.
|
||
The blacklist (`blacklist` + `install … /bin/false` + `update-initramfs -u`) is the
|
||
blunt fix: stop nvidia loading at all, load it by hand after a die is acquired.
|
||
|
||
**Why it's hacky:** it's system-wide — it kills nvidia for the *whole box*, so you can't
|
||
run a normal nvidia GPU alongside the K80s; and it needs `/etc/modprobe.d` edits +
|
||
initramfs regen that don't survive cleanly as a shipped product.
|
||
|
||
**Ship-grade replacement (planned): make `bar_swap` *bind* the dies (vfio-pci / pci-stub
|
||
pattern).** Today `bar_swap` claims die *resources* (`pci_claim_resource`) but is **not a
|
||
registered `pci_driver`** — so nothing stops nvidia binding them; hence the blacklist.
|
||
Register `bar_swap` as a `pci_driver` matching the GK210 device ID so the kernel **binds
|
||
all 16 dies to bar_swap at probe** → they're invisible to nvidia → no probe, no panic, no
|
||
corruption — *without* a global blacklist. On `acquire`, hand the one slot die to nvidia
|
||
(unbind → nvidia binds it); the other 15 stay parked with bar_swap. A box could then run
|
||
a normal nvidia GPU *and* the K80s at once. **Zero-code de-risk first:** put the GK210
|
||
ID in `pci-stub.ids=`/`vfio-pci.ids=` on the kernel cmdline and check nvidia boots clean,
|
||
before writing the bind path into `bar_swap`.
|
||
- **One die *mapped* at a time** (slot occupancy, not compute — released dies keep
|
||
computing). Acquiring a new die implicitly releases the active one.
|
||
- **Don't destroy the CUDA context on swap** — it survives release/acquire; tearing it
|
||
down forces a full ~2.5 s reload.
|
||
- **CUDA atexit teardown hangs on this rig (confirmed 2026-06-26).** When a process with
|
||
a CUDA context exits normally, the atexit context-destroy wedges a thread against the
|
||
bar_swap'd die — the process becomes a `Zl` zombie and nvidia ends up half-unloaded
|
||
(`NVRM: kernel module is unloaded` while `lsmod` still lists it). These zombies can't
|
||
be killed and accumulate until the box needs a power cycle (it also makes clean
|
||
shutdown crawl). **Workaround for short-lived processes (tests, one-shots): call
|
||
`_Exit()` instead of `return`/`exit()`** — skip the userspace teardown and let the
|
||
kernel reclaim the context on process death (clean exit, no zombie; verified with the
|
||
kernel-parity harness). **Proper fix (TODO, product-critical — every model process
|
||
exit hits this):** an ordered context-destroy + `bs_release` before exit, and/or the
|
||
device-binding driver work above. A long-lived Pantheon server avoids it by not
|
||
exiting; it still needs a clean teardown path for shutdown.
|
||
|
||
## barswap userspace API (`barswap.h`)
|
||
|
||
```c
|
||
bs_context* bs_init(void); // open /dev/bar_swap
|
||
uint32_t bs_num_dies(ctx); // 16 on this rig
|
||
uint32_t bs_num_slots(ctx); // 1 (v2 — hardware limit)
|
||
bs_gpu* bs_acquire(ctx, int die_id); // program die into the slot, mmap VRAM
|
||
void bs_release(ctx, bs_gpu*); // vacate the slot
|
||
int bs_active_die(ctx); // die in the slot, or -1
|
||
int bs_is_mapped(ctx, int die_id);
|
||
int bs_refresh(ctx);
|
||
void bs_free(ctx);
|
||
```
|
||
|
||
Kernel module ioctls on `/dev/bar_swap`: `BAR_SWAP_ACQUIRE` / `…RELEASE` / `…QUERY`.
|
||
|
||
## Field service — identifying a physical card (future feature, not built)
|
||
|
||
This rig has **no BMC/IPMI** and the K80 bracket LED isn't software-programmable, so
|
||
there's no turnkey "locator LED." Two practical ways to find a specific/failing card,
|
||
worth building into the barswap-in-Pantheon tooling later:
|
||
|
||
1. **BDF → physical map (deterministic).** Every die has a fixed PCI address (die 0 =
|
||
`03:00.0`, die 1 = `04:00.0`, Card B = `07/08:00.0`, … — see `CARDS.md`).
|
||
A failing die's BDF → its card → its riser. **One-time win: physically label each
|
||
riser with its BDF at install** so a bad die maps to a card by eye.
|
||
2. **`identify <die>` by workload (lights up the card).** Acquire *only* the suspect die
|
||
and run a continuous busy kernel for ~30 s — its **activity LED flickers** while the
|
||
others stay dark, so you can physically spot it. barswap's single-die acquire makes
|
||
this clean. This is the small operability feature to add (a `pantheon --barswap
|
||
identify <die>` mode).
|
||
|
||
Also useful and already possible: a *failed* K80 often shows **amber** on the bracket LED
|
||
(passive), and barswap can **disable/skip a die by BDF** in software to route around a
|
||
flaky card until it's pulled.
|
||
|
||
## Where to read more
|
||
|
||
- **`MULTI_SLOT.md`** — *why* only one slot works (Ivy Bridge 36-bit addressing; closed).
|
||
- **`CARDS.md`** — GPU compatibility (which cards need barswap; T4/P4 don't) + current rig.
|
||
- **`memoryTricks.md`** — barswap-as-scheduler notes for multi-model hosting.
|