256 lines
8.1 KiB
Markdown
256 lines
8.1 KiB
Markdown
# Architecture
|
|
|
|
Overview of Foundry's design and architecture.
|
|
|
|
---
|
|
|
|
## Design Goals
|
|
|
|
1. **Zero Python Dependencies** - Pure C, no Python runtime
|
|
2. **Minimal External Dependencies** - Only POSIX APIs
|
|
3. **Memory Efficient** - No unnecessary copies, mmap support
|
|
4. **Embeddable** - Static library, simple integration
|
|
5. **Portable** - x86_64 and ARM (Raspberry Pi)
|
|
|
|
---
|
|
|
|
## Module Dependency Graph
|
|
|
|
```
|
|
┌─────────┐
|
|
│ tensor │ ◄── Foundation
|
|
└────┬────┘
|
|
│
|
|
┌───────────────────┼───────────────────┐
|
|
│ │ │
|
|
┌────▼────┐ ┌────▼────┐ ┌────▼────┐
|
|
│ optim │ │attention│ │quantize │
|
|
└────┬────┘ └────┬────┘ └─────────┘
|
|
│ │
|
|
│ ┌────▼────┐
|
|
│ │ lora │
|
|
│ └─────────┘
|
|
│
|
|
┌────▼────┐
|
|
│ swarm │ ◄── Distributed training
|
|
└─────────┘
|
|
|
|
┌─────────┐ ┌─────────┐ ┌─────────┐
|
|
│tokenizer│ │safetens │ │ data │ ◄── I/O modules
|
|
└─────────┘ └─────────┘ └─────────┘
|
|
|
|
┌─────────┐ ┌─────────┐
|
|
│progress │ │ eval │ ◄── Utilities
|
|
└─────────┘ └─────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Backends
|
|
|
|
CPU is the baseline; every op works there first.
|
|
|
|
| Backend | Directory | Reaches Foundry ops via | Notes |
|
|
|---------|-----------|-------------------------|-------|
|
|
| CPU | (core modules) | — | the reference path |
|
|
| CUDA | `cuda/`, `cuda_kernels/` | `tensor.c` / `rwkvx.c` dispatch | `dlopen`, embedded PTX, `sm_37`+ |
|
|
| Metal | `metal/` | `tensor.c` / `rwkvx.c` dispatch | Apple Silicon, embedded MSL |
|
|
| Tenstorrent Blackhole | `tt/` | **driven directly, no tensor dispatch** | own userspace driver (tt-kmd optional), SFPI to rebuild kernels; kernels are reader/compute/writer triads on Tensix cores, embedded as ELF |
|
|
| Vulkan | — | — | planned |
|
|
|
|
`tt/` is the odd one out on purpose: there is no vendor runtime to hand a buffer to, so the
|
|
backend owns the whole stack from device open to op dispatch. It is not yet wired into
|
|
`tensor/`. See [tenstorrent.md](tenstorrent.md).
|
|
|
|
---
|
|
|
|
## Memory Model
|
|
|
|
### Tensor Memory
|
|
|
|
- Tensors own their data by default
|
|
- Reference counting for safe memory management
|
|
- Views share underlying data (no copy)
|
|
- Explicit `tm_contiguous()` forces copy when needed
|
|
|
|
```c
|
|
tm_tensor* a = tm_randn(shape, 2, TM_F32); // Owns data
|
|
tm_tensor* b = tm_transpose(a, 0, 1); // View, shares data
|
|
tm_tensor* c = tm_contiguous(b); // Copy, owns new data
|
|
tm_free(c);
|
|
tm_free(b); // Only frees view struct, not data
|
|
tm_free(a); // Frees data
|
|
```
|
|
|
|
### Autograd Graph
|
|
|
|
- Operations create graph nodes
|
|
- Nodes hold references to inputs
|
|
- `tm_backward()` traverses graph in reverse
|
|
- Graph freed after backward (or via `tm_detach()`)
|
|
|
|
---
|
|
|
|
## Autograd Implementation
|
|
|
|
```
|
|
Forward:
|
|
a ──┐
|
|
├── add_backward ── c
|
|
b ──┘
|
|
|
|
c.grad_fn = {
|
|
.backward = add_backward,
|
|
.inputs = {a, b},
|
|
.num_inputs = 2
|
|
}
|
|
|
|
Backward:
|
|
tm_backward(c)
|
|
├── c.grad = 1.0
|
|
└── c.grad_fn.backward(c)
|
|
├── a.grad += c.grad
|
|
└── b.grad += c.grad
|
|
```
|
|
|
|
---
|
|
|
|
## File Format: Safetensors
|
|
|
|
```
|
|
┌──────────────────────────────────────────┐
|
|
│ Header Length (8 bytes, little-endian) │
|
|
├──────────────────────────────────────────┤
|
|
│ JSON Header (variable length) │
|
|
│ { │
|
|
│ "__metadata__": {...}, │
|
|
│ "tensor_name": { │
|
|
│ "dtype": "F32", │
|
|
│ "shape": [1024, 1024], │
|
|
│ "data_offsets": [0, 4194304] │
|
|
│ }, │
|
|
│ ... │
|
|
│ } │
|
|
├──────────────────────────────────────────┤
|
|
│ Tensor Data (contiguous, aligned) │
|
|
│ [tensor 1 data][tensor 2 data][...] │
|
|
└──────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Swarm Protocol
|
|
|
|
### Wire Format
|
|
|
|
```
|
|
┌──────────────────────────────────────────┐
|
|
│ Magic (4 bytes): "SWRM" │
|
|
├──────────────────────────────────────────┤
|
|
│ Version (2 bytes) │
|
|
├──────────────────────────────────────────┤
|
|
│ Message Type (1 byte) │
|
|
├──────────────────────────────────────────┤
|
|
│ Flags (1 byte) │
|
|
├──────────────────────────────────────────┤
|
|
│ Payload Length (4 bytes) │
|
|
├──────────────────────────────────────────┤
|
|
│ Checksum (4 bytes, CRC32) │
|
|
├──────────────────────────────────────────┤
|
|
│ Payload (variable length) │
|
|
└──────────────────────────────────────────┘
|
|
```
|
|
|
|
### Compression: TopK
|
|
|
|
Only send the K largest gradients by magnitude:
|
|
|
|
```
|
|
Original: [0.01, 5.0, 0.02, -3.0, 0.03, 2.0, 0.0, -1.0]
|
|
TopK (k=3):
|
|
Indices: [1, 3, 5]
|
|
Values: [5.0, -3.0, 2.0]
|
|
|
|
Compression ratio: 3/8 = 37.5%
|
|
With k=10%: 90% bandwidth reduction
|
|
```
|
|
|
|
---
|
|
|
|
## Threading Model
|
|
|
|
- Main thread: Coordinator event loop
|
|
- Worker threads: Forward/backward computation
|
|
- I/O threads: Network send/receive
|
|
|
|
Thread safety:
|
|
- Tensors are NOT thread-safe
|
|
- Coordinator uses mutex for worker list
|
|
- Worker uses mutex for state transitions
|
|
|
|
---
|
|
|
|
## Build System
|
|
|
|
```makefile
|
|
# Top-level Makefile structure
|
|
all: tensor optim tokenizer safetensors data progress \
|
|
quantization attention lora distributed eval swarm
|
|
|
|
tensor:
|
|
$(CC) -c tensor/tensor.c -o obj/tensor.o
|
|
|
|
# ... other modules ...
|
|
|
|
lib/libfoundry.a: $(ALL_OBJS)
|
|
ar rcs $@ $^
|
|
```
|
|
|
|
---
|
|
|
|
## Error Handling
|
|
|
|
- Functions return NULL or false on error
|
|
- No exceptions (C doesn't have them)
|
|
- Errors logged to stderr
|
|
- Assertions for programmer errors
|
|
|
|
```c
|
|
tm_tensor* t = tm_zeros(shape, 2, TM_F32);
|
|
if (t == NULL) {
|
|
fprintf(stderr, "Failed to allocate tensor\n");
|
|
return 1;
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Future Considerations
|
|
|
|
### GPU Support
|
|
|
|
Planned abstraction:
|
|
|
|
```c
|
|
typedef enum {
|
|
TM_DEVICE_CPU,
|
|
TM_DEVICE_CUDA,
|
|
TM_DEVICE_METAL,
|
|
} tm_device;
|
|
|
|
tm_tensor* tm_zeros_on(shape, ndim, dtype, device);
|
|
tm_tensor* tm_to(tensor, device);
|
|
```
|
|
|
|
### Python Bindings
|
|
|
|
If needed, via ctypes or cffi:
|
|
|
|
```python
|
|
import ctypes
|
|
|
|
lib = ctypes.CDLL("libfoundry.so")
|
|
lib.tm_zeros.restype = ctypes.c_void_p
|
|
lib.tm_zeros.argtypes = [ctypes.POINTER(ctypes.c_int64), ctypes.c_int, ctypes.c_int]
|
|
```
|