345 lines
10 KiB
Markdown
345 lines
10 KiB
Markdown
# Foundry Roadmap
|
||
|
||
Pure C implementations of 10+ Python ML tools with **full feature parity**.
|
||
|
||
---
|
||
|
||
## Module Status Overview
|
||
|
||
| Module | Replaces | Status | Notes |
|
||
|--------|----------|--------|-------|
|
||
| **tensor** | PyTorch/NumPy | ✅ API Complete | Autograd, broadcasting, all core ops |
|
||
| **optim** | torch.optim | ✅ API Complete | SGD, Adam, AdamW, schedulers, clipping |
|
||
| **tokenizer** | Tokenizers | ✅ API Complete | BPE, HuggingFace JSON loading |
|
||
| **safetensors** | Safetensors | ✅ API Complete | Read/write, mmap, metadata |
|
||
| **data** | Datasets | ✅ API Complete | JSONL loading, batching, shuffling |
|
||
| **progress** | Tqdm | ✅ API Complete | Progress bars, training metrics |
|
||
| **quantization** | Bitsandbytes | ✅ API Complete | INT8/INT4, per-channel, per-group |
|
||
| **attention** | FlashAttention | ✅ API Complete | Multi-head, KV cache, RoPE |
|
||
| **lora** | PEFT | ✅ API Complete | LoRA layers, adapters, merging |
|
||
| **distributed** | DeepSpeed/FSDP | ✅ API Complete | Collectives, gradient accum |
|
||
| **eval** | lm-eval-harness | ✅ API Complete | Perplexity, accuracy, F1 |
|
||
| **swarm** | Custom | ✅ Complete | Docker Swarm distributed training |
|
||
| **tracker** | Wandb | ❌ Missing | Experiment tracking |
|
||
| **transformer** | Transformers | ❌ Missing | Model architectures (GPT, Llama, RWKV) |
|
||
|
||
---
|
||
|
||
## Module Details
|
||
|
||
### 1. tensor (PyTorch/NumPy) ✅
|
||
|
||
**File:** `tensor/tensor.h`
|
||
|
||
**Implemented:**
|
||
- [x] Tensor creation (zeros, ones, full, randn, rand, empty)
|
||
- [x] Data types (F32, F64, F16, BF16, I32, I64)
|
||
- [x] Shape operations (reshape, transpose, squeeze, unsqueeze)
|
||
- [x] Element-wise ops (add, sub, mul, div, neg, exp, log, sqrt, tanh)
|
||
- [x] Activations (relu, sigmoid, gelu)
|
||
- [x] Matrix ops (matmul, bmm)
|
||
- [x] Reductions (sum, mean, max, min)
|
||
- [x] NN ops (softmax, log_softmax, layer_norm, embedding, dropout)
|
||
- [x] Loss functions (cross_entropy, mse_loss, l1_loss, smooth_l1_loss, nll_loss, kl_div, binary_cross_entropy)
|
||
- [x] Autograd (requires_grad, backward, zero_grad, detach, no_grad)
|
||
- [x] In-place ops (zeros_like, ones_like, fill_, add_, scale_)
|
||
- [x] Broadcasting support
|
||
|
||
**Needs verification against PyTorch:**
|
||
- [ ] Edge cases in broadcasting
|
||
- [ ] Gradient correctness for all ops
|
||
- [ ] Numerical stability
|
||
|
||
---
|
||
|
||
### 2. optim (torch.optim) ✅
|
||
|
||
**File:** `optim/optim.h`
|
||
|
||
**Implemented:**
|
||
- [x] SGD with momentum, weight decay, Nesterov
|
||
- [x] Adam with AMSGrad
|
||
- [x] AdamW (decoupled weight decay)
|
||
- [x] Schedulers: step, cosine, linear, warmup_cosine
|
||
- [x] Gradient clipping (norm and value)
|
||
|
||
**Needs verification:**
|
||
- [ ] Optimizer state save/load
|
||
- [ ] Comparison with PyTorch implementations
|
||
|
||
---
|
||
|
||
### 3. tokenizer (HuggingFace Tokenizers) ✅
|
||
|
||
**File:** `tokenizer/tokenizer.h`
|
||
|
||
**Implemented:**
|
||
- [x] Load from HuggingFace JSON format
|
||
- [x] Load BPE from vocab/merges files
|
||
- [x] Encode/decode text
|
||
- [x] Special tokens (BOS, EOS, PAD, UNK)
|
||
- [x] Batch encoding
|
||
|
||
**Needs verification:**
|
||
- [ ] Unicode handling edge cases
|
||
- [ ] Large vocabulary performance
|
||
- [ ] Comparison with `tokenizers` library
|
||
|
||
---
|
||
|
||
### 4. safetensors (HuggingFace Safetensors) ✅
|
||
|
||
**File:** `safetensors/safetensors.h`
|
||
|
||
**Implemented:**
|
||
- [x] Read safetensors files (mmap)
|
||
- [x] Read from memory buffer
|
||
- [x] Write safetensors files
|
||
- [x] All data types (F32, F64, F16, BF16, I8-I64, U8-U64, BOOL)
|
||
- [x] Metadata read/write
|
||
- [x] Zero-copy data access
|
||
|
||
**Needs verification:**
|
||
- [ ] Sharded checkpoint support
|
||
- [ ] Large file handling
|
||
- [ ] Comparison with Python safetensors
|
||
|
||
---
|
||
|
||
### 5. data (HuggingFace Datasets) ✅
|
||
|
||
**File:** `data/data.h`
|
||
|
||
**Implemented:**
|
||
- [x] Load JSONL files
|
||
- [x] Sample creation/manipulation
|
||
- [x] Dataset shuffling
|
||
- [x] Train/test splitting
|
||
- [x] DataLoader with batching
|
||
|
||
**Needs verification:**
|
||
- [ ] Streaming large datasets
|
||
- [ ] Memory-mapped loading
|
||
- [ ] Comparison with `datasets` library
|
||
|
||
---
|
||
|
||
### 6. progress (Tqdm) ✅
|
||
|
||
**File:** `progress/progress.h`
|
||
|
||
**Implemented:**
|
||
- [x] Progress bars with ETA
|
||
- [x] Training state logging
|
||
- [x] Nested progress bars
|
||
- [x] Postfix formatting
|
||
- [x] Rate calculation
|
||
|
||
**Needs verification:**
|
||
- [ ] Terminal width handling
|
||
- [ ] Nested bar rendering
|
||
|
||
---
|
||
|
||
### 7. quantization (Bitsandbytes) ✅
|
||
|
||
**File:** `quantization/quantization.h`
|
||
|
||
**Implemented:**
|
||
- [x] INT8 quantization (symmetric)
|
||
- [x] INT8 per-channel quantization
|
||
- [x] INT4 per-group quantization
|
||
- [x] Dequantization
|
||
- [x] Quantized matmul
|
||
|
||
**Needs verification:**
|
||
- [ ] NF4 quantization (QLoRA)
|
||
- [ ] Dynamic quantization
|
||
- [ ] Comparison with bitsandbytes
|
||
|
||
---
|
||
|
||
### 8. attention (FlashAttention) ✅
|
||
|
||
**File:** `attention/attention.h`
|
||
|
||
**Implemented:**
|
||
- [x] Scaled dot-product attention
|
||
- [x] Multi-head attention
|
||
- [x] Self-attention
|
||
- [x] Attention with mask
|
||
- [x] KV cache for inference
|
||
- [x] RoPE (Rotary Position Embedding)
|
||
- [x] Causal masking
|
||
|
||
**Needs verification:**
|
||
- [ ] Flash Attention algorithm (memory-efficient)
|
||
- [ ] Numerical stability at scale
|
||
- [ ] Comparison with FlashAttention
|
||
|
||
---
|
||
|
||
### 9. lora (PEFT) ✅
|
||
|
||
**File:** `lora/lora.h`
|
||
|
||
**Implemented:**
|
||
- [x] LoRA layer creation
|
||
- [x] LoRA forward pass
|
||
- [x] Weight merging
|
||
- [x] Adapter save/load (safetensors)
|
||
- [x] Configurable rank/alpha/dropout
|
||
|
||
**Needs verification:**
|
||
- [ ] QLoRA support
|
||
- [ ] Comparison with PEFT library
|
||
|
||
---
|
||
|
||
### 10. distributed (DeepSpeed/FSDP basics) ✅
|
||
|
||
**File:** `distributed/distributed.h`
|
||
|
||
**Implemented:**
|
||
- [x] Process group initialization
|
||
- [x] All-reduce, broadcast, reduce, all-gather
|
||
- [x] Barrier synchronization
|
||
- [x] Gradient accumulation
|
||
- [x] Data sharding
|
||
|
||
**Needs verification:**
|
||
- [ ] MPI backend integration
|
||
- [ ] Multi-node support
|
||
|
||
---
|
||
|
||
### 11. eval (lm-eval-harness) ✅
|
||
|
||
**File:** `eval/eval.h`
|
||
|
||
**Implemented:**
|
||
- [x] Perplexity evaluation
|
||
- [x] Accuracy/Top-K accuracy
|
||
- [x] F1 score
|
||
- [x] Multiple choice tasks
|
||
- [x] Few-shot support
|
||
- [x] HellaSwag-style tasks
|
||
|
||
**Needs verification:**
|
||
- [ ] Standard benchmark compatibility
|
||
- [ ] Comparison with lm-eval-harness
|
||
|
||
---
|
||
|
||
### 12. swarm (Distributed Training) ✅ COMPLETE
|
||
|
||
**Files:** `swarm/` directory
|
||
|
||
**Implemented:**
|
||
- [x] Parameter server architecture
|
||
- [x] Coordinator (weight distribution, gradient aggregation)
|
||
- [x] Worker (gradient computation, compression)
|
||
- [x] TopK gradient compression
|
||
- [x] FP16 compression
|
||
- [x] Hardware profile auto-detection
|
||
- [x] Docker Swarm deployment
|
||
- [x] Fault tolerance (reconnection)
|
||
- [x] TOML config parsing
|
||
- [x] 14 passing tests
|
||
|
||
---
|
||
|
||
## Missing Modules
|
||
|
||
### 13. tracker (Wandb) ❌ NOT IMPLEMENTED
|
||
|
||
**Should implement:**
|
||
- [ ] Experiment initialization
|
||
- [ ] Metric logging (scalars, histograms)
|
||
- [ ] Config tracking
|
||
- [ ] Artifact storage
|
||
- [ ] Local logging (SQLite/JSON fallback)
|
||
- [ ] Optional cloud sync
|
||
|
||
---
|
||
|
||
### 14. transformer (Transformers) ❌ NOT IMPLEMENTED
|
||
|
||
**Should implement:**
|
||
- [ ] Embedding layer (token + position)
|
||
- [ ] TransformerBlock (attention + MLP)
|
||
- [ ] GPT-2 architecture
|
||
- [ ] Llama architecture (RMSNorm, SwiGLU, RoPE)
|
||
- [ ] RWKV architecture
|
||
- [ ] Weight loading from safetensors
|
||
- [ ] HuggingFace weight name mapping
|
||
|
||
---
|
||
|
||
### 15. corpus-archaeology (research idea — back-pocket, not committed) 💡
|
||
|
||
A model-forensics tool: take **open weights** and infer *what a model was trained on* and *which data actually moved which capability* — across many models, comparatively. Not data reconstruction (training is lossy compression; the full corpus is information-theoretically unrecoverable past the memorized slice) — it's **distributional/provenance analysis via forward passes only** (no gradients). Core signal: **comparative perplexity/loss fingerprinting** — score each model over a battery of candidate datasets; the loss profile is a fingerprint of what it knows comfortably. From that: cluster models by **data lineage**, run **membership/dataset inference** ("was source X in training?"), regress **dataset presence → eval score** to find what's *effective* (and what's redundant). Optional tier: a **memorization-extraction sweep** (sample heavily, flag abnormally-low-loss verbatim spans) to recover the small memorized fraction. Natural Foundry fit (it's the tensor/compute/research base, not serving) and a **capacity-bound batch workload** — hold many models resident, fire probes — i.e. *exactly* what a slow-but-huge-VRAM multi-K80 host is good at. Turns such a host into a model-archaeology rig, not just an inference server.
|
||
|
||
- [ ] Perplexity/loss fingerprinting harness (model × dataset matrix)
|
||
- [ ] Membership / dataset inference scorers
|
||
- [ ] dataset-presence → capability correlation (regress vs eval scores)
|
||
- [ ] (tier 2) memorization-extraction sweep (low-loss verbatim detection + dedup)
|
||
|
||
---
|
||
|
||
## Verification Checklist
|
||
|
||
Before each module is production-ready, verify against the original Python implementation:
|
||
|
||
1. **API Compatibility** - Does it support the same core operations?
|
||
2. **Numerical Accuracy** - Do outputs match within tolerance?
|
||
3. **Edge Cases** - Does it handle empty inputs, large inputs, special values?
|
||
4. **Performance** - Is it at least as fast for common operations?
|
||
5. **Memory** - Does it avoid unnecessary copies/allocations?
|
||
|
||
---
|
||
|
||
## Next Steps
|
||
|
||
1. **transformer** - Model architectures (GPT, Llama, RWKV)
|
||
2. **tracker** - Experiment tracking (Wandb replacement)
|
||
3. **Full feature parity passes** - Implement ALL missing PyTorch/library features
|
||
4. **GPU support** - CUDA/Metal backends for all ops
|
||
5. Verification passes on all modules
|
||
|
||
---
|
||
|
||
## Test Coverage
|
||
|
||
| Module | Tests | Status |
|
||
|--------|-------|--------|
|
||
| tensor | Yes | Passing |
|
||
| optim | Yes | Passing |
|
||
| tokenizer | Yes | Passing |
|
||
| safetensors | Yes | Passing |
|
||
| data | Yes | Passing |
|
||
| progress | Yes | Passing |
|
||
| quantization | Yes | Passing |
|
||
| attention | Yes | Passing |
|
||
| lora | Yes | Passing |
|
||
| distributed | Yes | Passing |
|
||
| eval | Yes | Passing |
|
||
| swarm | 14 tests | Passing |
|
||
|
||
---
|
||
|
||
## Hardware Targets
|
||
|
||
Primary target: Train 1B RWKV-x model across ~40 heterogeneous nodes:
|
||
- Raspberry Pi 4/5
|
||
- Laptops (Intel/AMD)
|
||
- Desktop PCs
|
||
- Small servers
|
||
- Connected via WiFi/Starlink
|
||
|
||
The swarm module is designed specifically for this use case with:
|
||
- Async gradient updates (workers don't block each other)
|
||
- TopK compression (90% bandwidth reduction)
|
||
- Hardware-adaptive batch sizes
|
||
- Fault tolerance (nodes can drop/rejoin)
|