# Foundry Documentation

Pure C implementation of essential ML tools for training large language models on CPU clusters.

---

## Quick Links

- [Getting Started](getting-started.md)
- [API Reference](#api-reference)
- [Examples](examples.md)
- [Architecture](architecture.md)

---

## API Reference

### Core Modules

| Module | Description | Docs |
|--------|-------------|------|
| **tensor** | Tensor operations with autograd | [tensor.md](tensor.md) |
| **optim** | Optimizers (SGD, Adam, AdamW) | [optim.md](optim.md) |
| **tokenizer** | BPE tokenization | [tokenizer.md](tokenizer.md) |
| **safetensors** | Model serialization | [safetensors.md](safetensors.md) |
| **data** | Dataset loading and batching | [data.md](data.md) |

### Training Modules

| Module | Description | Docs |
|--------|-------------|------|
| **attention** | Efficient attention with KV cache | [attention.md](attention.md) |
| **lora** | Low-Rank Adaptation fine-tuning | [lora.md](lora.md) |
| **quantization** | INT8/INT4/ternary quantization, training + export | [quantization.md](quantization.md) |
| **distributed** | Distributed training primitives | [distributed.md](distributed.md) |
| **swarm** | Docker Swarm distributed training | [swarm.md](swarm.md) |

### GPU Acceleration

| Module | Description | Docs |
|--------|-------------|------|
| **metal** | Apple Metal GPU compute backend | [metal.md](metal.md) |
| **tt** | Tenstorrent Blackhole compute backend (p150a; own userspace driver, tt-kmd optional) | [tenstorrent.md](tenstorrent.md) |
| **barswap** | K80 BAR reassignment (kernel module + userspace) | [bar-swap.md](bar-swap.md) |

### Utility Modules

| Module | Description | Docs |
|--------|-------------|------|
| **progress** | Progress bars and metrics | [progress.md](progress.md) |
| **eval** | Model evaluation framework | [eval.md](eval.md) |

### CLI Tools

| Tool | Description | Docs |
|------|-------------|------|
| **rwkvx_generate** | RWKV-X text generation (INT8, HybridMem) | [rwkvx-generate.md](rwkvx-generate.md) |
| **rwkvx_train** | RWKV-X LoRA training (moba attention blocks) | [rwkvx-train.md](rwkvx-train.md) |
| **rfa_generate** | RFA hybrid text generation (RWKV-7 + Qwen) | [rfa-generate.md](rfa-generate.md) |
| **rfa_train** | RFA hybrid LoRA training (parallel, INT8) | [rfa-train.md](rfa-train.md) |
| **rfa_lora_merge** | LoRA adapter averaging (cross-machine merge) | [rfa-lora-merge.md](rfa-lora-merge.md) |
| **quantize_model** | Offline model quantization (INT8/INT4/ternary) | [quantize-model.md](quantize-model.md) |

---

## Building Foundry

### Prerequisites

- GCC or Clang with C11 support
- Make
- POSIX-compatible system (Linux, macOS)

### Build Commands

```bash
# Build all modules
make all

# Build specific module
make tensor
make swarm

# Run all tests
make test

# Clean build artifacts
make clean
```

### Build Options

| Option | Description |
|--------|-------------|
| `DEBUG=1` | Build with debug symbols |
| `SANITIZE=1` | Build with address sanitizer |
| `OPTIMIZE=1` | Build with -O3 optimization |

---

## Project Structure

```
foundry/
├── tensor/          # Core tensor operations
│   ├── tensor.h
│   ├── tensor.c
│   └── tests/
├── optim/           # Optimizers
├── tokenizer/       # Tokenization
├── safetensors/     # Model I/O
├── data/            # Data loading
├── attention/       # Attention mechanisms
├── lora/            # LoRA adapters
├── quantization/    # Quantization
├── metal/           # Apple Metal GPU backend
├── tt/              # Tenstorrent Blackhole backend (host C + Tensix kernels + converter)
├── barswap/         # K80 BAR reassignment (kernel module + userspace lib)
├── distributed/     # Distributed primitives
├── swarm/           # Docker Swarm training
├── progress/        # Progress bars
├── eval/            # Evaluation
├── examples/        # Example programs
├── docs/            # Documentation (you are here)
├── Makefile
└── roadmap.md
```

---

## Python Counterparts

Each module covers the core function of one Python library, in C:

| Python Tool | Foundry Module | Notes |
|-------------|----------------|-------|
| PyTorch | tensor, optim | tensor ops + autograd; PyTorch parity 80/83 comparison tests |
| HuggingFace Tokenizers | tokenizer | BPE (WordPiece planned) |
| HuggingFace Safetensors | safetensors | read/write, mmap |
| HuggingFace Datasets | data | JSONL, batching, sqlite chat loading |
| tqdm | progress | |
| bitsandbytes | quantization | INT8/INT4, export |
| PyTorch attention | attention | standard attention, KV cache, RoPE; no fused flash kernel yet |
| PEFT | lora | |
| Accelerate/DeepSpeed | distributed | single-process primitives; real multi-node planned |
| lm-eval-harness | eval | perplexity, accuracy, F1 |

---

## Design Principles

1. **Zero Python Dependencies** - Pure C, no Python runtime required
2. **Minimal External Dependencies** - Only POSIX APIs
3. **Memory Efficient** - No unnecessary copies, mmap support
4. **Embeddable** - Designed to be embedded in other applications
5. **Portable** - Works on x86_64 and ARM (Raspberry Pi)

---

## Target Use Case

Training 1B+ parameter RWKV-x models across heterogeneous CPU clusters:
- Raspberry Pi 4/5
- Laptops and desktops
- Small servers
- Connected via WiFi/Starlink

The swarm module provides distributed training with:
- Async gradient updates
- TopK gradient compression (90% bandwidth reduction)
- Hardware-adaptive batch sizes
- Fault tolerance

---

## License

See [`../LICENSE`](../LICENSE) and the
License section of the top-level [`README.md`](../README.md).

---

## Contributing

See [`../CONTRIBUTING.md`](../CONTRIBUTING.md).