79 lines
2.1 KiB
Markdown
79 lines
2.1 KiB
Markdown
# tensor/
|
|
|
|
C implementation of tensor operations with automatic differentiation.
|
|
|
|
## Python counterpart
|
|
|
|
**PyTorch** - https://github.com/pytorch/pytorch
|
|
|
|
## Why PyTorch?
|
|
|
|
PyTorch is the dominant framework for LLM training. TensorFlow and JAX are alternatives but PyTorch has won the LLM space. Core features:
|
|
|
|
- N-dimensional tensor operations
|
|
- Automatic differentiation (autograd)
|
|
- GPU acceleration
|
|
- Neural network primitives
|
|
|
|
## Scope
|
|
|
|
This module implements:
|
|
|
|
1. **Tensor data structure** - N-dimensional arrays with strides, shapes, dtypes
|
|
2. **Basic operations** - add, sub, mul, div, matmul, transpose, reshape
|
|
3. **Reductions** - sum, mean, max, min
|
|
4. **Autograd** - Computation graph, backward pass, gradient accumulation
|
|
5. **NN operations** - softmax, layer norm, embedding lookup
|
|
|
|
## Getting Started
|
|
|
|
### Dependencies
|
|
|
|
- C99 compiler
|
|
- BLAS library (OpenBLAS, MKL, or Accelerate)
|
|
- Optional: CUDA toolkit, Metal SDK
|
|
|
|
### Key Files to Study
|
|
|
|
From PyTorch:
|
|
- `c10/core/TensorImpl.h` - Tensor implementation
|
|
- `torch/csrc/autograd/` - Autograd engine
|
|
- `aten/src/ATen/native/` - Native operations
|
|
|
|
From ggml:
|
|
- `ggml.h` / `ggml.c` - Simpler tensor implementation
|
|
- Already C, good reference
|
|
|
|
### Core API
|
|
|
|
```c
|
|
// Tensor creation
|
|
tm_tensor* tm_zeros(int64_t* shape, int ndim, tm_dtype dtype);
|
|
tm_tensor* tm_randn(int64_t* shape, int ndim, tm_dtype dtype);
|
|
|
|
// Operations (all support autograd)
|
|
tm_tensor* tm_add(tm_tensor* a, tm_tensor* b);
|
|
tm_tensor* tm_matmul(tm_tensor* a, tm_tensor* b);
|
|
tm_tensor* tm_softmax(tm_tensor* x, int dim);
|
|
|
|
// Autograd
|
|
void tm_backward(tm_tensor* loss);
|
|
void tm_zero_grad(tm_tensor** params, int n);
|
|
```
|
|
|
|
## Implementation Order
|
|
|
|
1. Tensor struct and memory management
|
|
2. Basic ops (add, mul, matmul) - CPU only
|
|
3. Autograd graph building
|
|
4. Backward pass for basic ops
|
|
5. NN ops (softmax, layernorm)
|
|
6. GPU backends
|
|
|
|
## References
|
|
|
|
- [PyTorch Internals](http://blog.ezyang.com/2019/05/pytorch-internals/)
|
|
- [ggml](https://github.com/ggerganov/ggml)
|
|
- [tinygrad](https://github.com/tinygrad/tinygrad) - Minimal autograd
|
|
- [micrograd](https://github.com/karpathy/micrograd) - Educational autograd
|