6.7 KiB

Raw Blame History

Compatible GPU Cards — BTC-S37 / 8× K80 rig

Rig Constraints

Board: BTC-S37 (8 PCIe slots, x1/x4 electrical)
CPU: Intel Pentium 2117U (Ivy Bridge, 36-bit physical address = 64 GB max)
RAM: 8 GB DDR3
GPUs installed: 8x K80 (16 dies, 192 GB VRAM) — all 16 dies tested and passing
PSU: 3x 750W (2,250W total)
MMIO budget: ~56 GB (64 GB address space minus 8 GB RAM)
Working BAR1 slot: 0xC00000000 (16 GB, the only slot that routes correctly)
Driver: nvidia 470 (supports Kepler through Turing)
CUDA: 11.4 (sm_35 through sm_86)

Card Compatibility Table

Card	Arch	sm	VRAM	Mem BW	FP16 TFLOPS	INT8 TOPS	BAR1	TDP	Power Conn	Slots	Bar Swap?	Max on Board	Status
K80	Kepler	37	2x12 GB	240 GB/s/die	0.37/die	—	16 GB	300W	1x8-pin	2 (dual)	Required	8 (16 dies)	16/16 passing
K40	Kepler	35	12 GB	288 GB/s	0.43	—	16 GB	235W	1x8-pin	2 (full-length)	Required	3-4	Compatible
K20X	Kepler	35	6 GB	250 GB/s	0.39	—	8 GB	235W	1x8-pin	2 (full-length)	Required	3-4	Compatible
M40 24GB	Maxwell	52	24 GB	288 GB/s	0.21	—	16 GB	250W	1x8-pin	2 (full-length)	Required	3-4	Likely works
M60	Maxwell	52	2x8 GB	160 GB/s/die	0.15/die	—	16 GB	300W	1x8-pin	2 (dual-die)	Required	3 (6 dies)	Likely works
P100 PCIe 16GB	Pascal	60	16 GB HBM2	732 GB/s	21.2	—	16 GB	250W	1x8-pin	2 (full-length)	Required	3-4	Likely works
P4	Pascal	61	8 GB	192 GB/s	5.5	22	256 MB	75W	None	1 (low-profile)	Not needed	8	Likely works
V100 PCIe 16GB	Volta	70	16 GB HBM2	900 GB/s	28.3	—	16 GB	250W	1x8+6-pin	2 (full-length)	Required	3-4	Likely works
T4	Turing	75	16 GB GDDR6	320 GB/s	65	130	256 MB	75W	None	1 (low-profile)	Not needed	8	Likely works

INT8 TOPS note: Cards marked — have no hardware INT8 dot product. Kepler and Maxwell lack INT8 math entirely. Pascal sm_60 (P100) has FP16 Tensor-like throughput but no INT8 path. Volta Tensor Cores only accelerate FP16→FP32 mixed precision, not INT8. The first INT8 dot product arrived with Pascal sm_61 (P4) via the dp4a instruction, and Turing (T4) added INT8 Tensor Core mode for a massive throughput jump. On cards without INT8 hardware, INT8 data must be upcast to FP32 for computation — no speed benefit over FP32 math.

Cards That Will NOT Fit

Card	Reason
V100 32GB	BAR1 = 32 GB, exceeds single working 16 GB slot
P40 24GB	BAR1 = 32 GB
A30 24GB	BAR1 = 32 GB, also needs driver 495+
A100 40/80GB	BAR1 = 64/128 GB, needs driver 495+
H100	BAR1 = 128 GB, needs driver 525+
Any card with BAR1 > 16 GB	Only one 16 GB MMIO slot available at 0xC00000000

Notes

Bar Swap vs No Bar Swap

Cards with BAR1 <= 16 GB need bar_swap because only one 16 GB MMIO slot works on this board. All such cards share the slot via time-multiplexing. Only one die is mapped at a time.

Cards with BAR1 = 256 MB (P4, T4) have a small enough MMIO footprint that all 8 fit simultaneously in the address space (8 x 256 MB = 2 GB, well within the 56 GB MMIO budget). The nvidia driver manages VRAM internally through page tables — the 256 MB window is enough for the driver to function. Bar_swap is not needed because all cards are visible to nvidia at once. Standard multi-GPU CUDA works out of the box.

Current Rig: 8x K80 (16 Dies, 192 GB VRAM)

The board currently has 8 K80 cards installed. 3 cards (6 dies) are tested and working with bar_swap. The remaining 5 cards (10 dies) are installed but untested.

All 16 dies tested and passing (2026-03-25):

192 GB total VRAM
All 16 dies time-multiplexed through the single 0xC00000000 BAR1 slot
Enough VRAM to hold a 70B+ model in FP16 or a 140B+ model in INT8
~1.2ms acquire time per die, ~2.5s full cycle (with nvidia load/unload)
Power: 8x 300W = 2,400W (needs 30A breaker for sustained load)

T4 — Best Upgrade Option

8x T4 is the most interesting configuration for this board:

128 GB total VRAM (vs 192 GB with 8x K80)
520 TFLOPS FP16 / 1,040 TOPS INT8 (vs ~17 TFLOPS FP16 with K80s)
600W total power, no power cables (vs 2,400W + cables)
Single-slot passive cooling — designed for dense server deployment
All 8 visible to nvidia simultaneously, standard multi-GPU CUDA
Used price: ~$150-200 each (~$1,200-1,600 for 8)

P4 — Budget Small-BAR Option

8x P4 is the budget version of the same idea:

64 GB total VRAM (8x 8 GB)
Pascal sm_61 (same as GTX 1080)
75W, no power cables, single-slot, passive
Cheaper than T4 (~$50-80 used)
Less compute than T4 but still faster than K80

P100 / V100 16GB — Maximum Single-Card Performance

These are the fastest cards that fit the 16 GB BAR1 slot:

P100: ~21 TFLOPS FP16, 16 GB HBM2 (732 GB/s bandwidth)
V100 16GB: ~125 TFLOPS FP16, 16 GB HBM2 (900 GB/s bandwidth)
Both require bar_swap (16 GB BAR1)
A single V100-16GB through bar_swap outperforms all 16 K80 dies in compute

PCIe Bandwidth

All BTC-S37 slots are x1 or x4 electrical:

PCIe 2.0 x1: 500 MB/s
PCIe 3.0 x1: 1 GB/s
PCIe 3.0 x4: 4 GB/s

This limits host-to-device transfers (model weight uploads) but does not affect on-device compute. With VRAM persistence (bar_swap) or standard multi-GPU (T4/P4), weights are uploaded once and stay resident.

Driver Compatibility

Driver	Architectures Supported
470 (installed)	Kepler, Maxwell, Pascal, Volta, Turing
495+	Adds Ampere (sm_80/86)
525+	Adds Hopper (sm_90)

Driver 470 covers all cards in the compatibility table. No driver change needed.

Dual-Die Cards

K80 and M60 are dual-die cards (2 GPUs on one PCB) connected via an on-card PLX PEX 8747 switch. The BTC-S37 already handles this topology — the PLX downstream ports appear as separate PCI devices. Bar_swap treats each die independently.

What Has NOT Been Tested

All 8 K80 cards (16 dies) tested and passing with bar_swap as of 2026-03-25
No card other than K80 has been tested on this board
BAR1 sizes for non-K80 cards are based on published specs and typical configurations
T4/P4 small-BAR behavior assumes the card does not request resizable BAR expansion on a board that doesn't support it (expected behavior, but unverified)
BIOS compatibility with non-Kepler cards is assumed but not confirmed
8x K80 power draw (2,400W) may exceed PSU capacity — verify PSU wiring before testing all 8 cards under load

6.7 KiB Raw Blame History Unescape Escape