History

David Herrera 775df8ab31 Initial public release		2026-10-19 13:24:09 -05:00
..
docker	Initial public release	2026-10-19 13:24:09 -05:00
tests	Initial public release	2026-10-19 13:24:09 -05:00
README.md	Initial public release	2026-10-19 13:24:09 -05:00
config.c	Initial public release	2026-10-19 13:24:09 -05:00
config.h	Initial public release	2026-10-19 13:24:09 -05:00
coordinator.c	Initial public release	2026-10-19 13:24:09 -05:00
coordinator.h	Initial public release	2026-10-19 13:24:09 -05:00
coordinator_main.c	Initial public release	2026-10-19 13:24:09 -05:00
network.c	Initial public release	2026-10-19 13:24:09 -05:00
network.h	Initial public release	2026-10-19 13:24:09 -05:00
protocol.c	Initial public release	2026-10-19 13:24:09 -05:00
protocol.h	Initial public release	2026-10-19 13:24:09 -05:00
worker.c	Initial public release	2026-10-19 13:24:09 -05:00
worker.h	Initial public release	2026-10-19 13:24:09 -05:00
worker_main.c	Initial public release	2026-10-19 13:24:09 -05:00

README.md

swarm/

Custom Extension - Distributed CPU training across Docker Swarm.

This is a project-specific addition to Foundry, not part of the core toolset. It enables training models across heterogeneous hardware (Raspberry Pis, laptops, desktops, servers) connected via WiFi.

Purpose

Train a custom 1B parameter RWKV-x model using:

~20 Raspberry Pi 4s
4-10 laptops
~10 desktops (3rd-5th gen Intel i-series)
A few old servers

All connected via WiFi/Starlink with Docker Swarm orchestration.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         DOCKER SWARM                            │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   ┌─────────────────────┐                                       │
│   │    COORDINATOR      │  (1 instance, on server)              │
│   │  - Holds weights    │                                       │
│   │  - Aggregates grads │                                       │
│   │  - Checkpoints      │                                       │
│   └──────────┬──────────┘                                       │
│              │                                                  │
│              │ TCP (port 7100)                                  │
│              │                                                  │
│   ┌──────────┴──────────────────────────────────────────┐       │
│   │                                                     │       │
│   ▼                    ▼                    ▼           ▼       │
│ ┌──────────┐     ┌──────────┐     ┌──────────┐   ┌──────────┐   │
│ │ WORKER   │     │ WORKER   │     │ WORKER   │   │ WORKER   │   │
│ │ server   │     │ desktop  │     │ laptop   │   │ rpi4     │   │
│ │ batch=128│     │ batch=64 │     │ batch=32 │   │ batch=8  │   │
│ └──────────┘     └──────────┘     └──────────┘   └──────────┘   │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Key Features

Async gradient updates - Don't wait for slow nodes (critical for WiFi)
Gradient compression - TopK/FP16/QSGD to reduce bandwidth
Auto hardware detection - Adjusts batch size per device capability
Fault tolerance - Workers can drop and rejoin without killing training
LoRA training mode - Train specialized adapters with minimal bandwidth

Files

File	Purpose
`protocol.h/.c`	Message types, serialization, compression
`network.h/.c`	TCP socket communication layer
`coordinator.h/.c`	Central training server
`worker.h/.c`	Training worker client
`docker/`	Swarm deployment configs

Dependencies

This module builds on:

tensor/ - Tensor operations and autograd
optim/ - Adam optimizer, LR scheduling
distributed/ - Gradient accumulation primitives

Usage

Local Development

# Build images
cd swarm/docker
./deploy.sh build

# Start with docker-compose (2 workers by default)
docker-compose up -d

# Scale to more workers
docker-compose up -d --scale worker=5

Production Swarm Deployment

# On manager node - initialize swarm
./deploy.sh init

# On worker nodes - join swarm
./deploy.sh join <token> <manager-ip>:2377

# Label nodes by hardware type
./deploy.sh label <node-id> server
./deploy.sh label <node-id> desktop
./deploy.sh label <node-id> laptop
./deploy.sh label <node-id> rpi

# Build and push images
./deploy.sh build
REGISTRY=myregistry:5000 ./deploy.sh push

# Deploy the stack
./deploy.sh deploy

# Monitor
./deploy.sh status
./deploy.sh logs coordinator

# Scale workers
./deploy.sh scale worker-rpi 20

Manual Binary Usage

# On coordinator node (server)
./bin/swarm_coordinator --config config/coordinator.toml

# On worker nodes
./bin/swarm_worker --config config/worker.toml

Integration with RWKV-x

This module is designed to be pulled into the RWKV-x repo as a git submodule:

rwkv-x/
├── model/           # RWKV-x architecture
├── data/            # Training data
└── foundry/         # ← Git submodule
    ├── tensor/
    ├── optim/
    ├── swarm/       # ← This module
    └── ...