Foundry/tensor/README.md

tensor/

C implementation of tensor operations with automatic differentiation.

Python counterpart

PyTorch - https://github.com/pytorch/pytorch

Why PyTorch?

PyTorch is the dominant framework for LLM training. TensorFlow and JAX are alternatives but PyTorch has won the LLM space. Core features:

• N-dimensional tensor operations
• Automatic differentiation (autograd)
• GPU acceleration
• Neural network primitives

Scope

This module implements:

1. Tensor data structure - N-dimensional arrays with strides, shapes, dtypes
2. Basic operations - add, sub, mul, div, matmul, transpose, reshape
3. Reductions - sum, mean, max, min
4. Autograd - Computation graph, backward pass, gradient accumulation
5. NN operations - softmax, layer norm, embedding lookup

Getting Started

Dependencies

• C99 compiler
• BLAS library (OpenBLAS, MKL, or Accelerate)
• Optional: CUDA toolkit, Metal SDK

Key Files to Study

From PyTorch:

• c10/core/TensorImpl.h - Tensor implementation
• torch/csrc/autograd/ - Autograd engine
• aten/src/ATen/native/ - Native operations

From ggml:

• ggml.h / ggml.c - Simpler tensor implementation
• Already C, good reference

Core API

// Tensor creation
tm_tensor* tm_zeros(int64_t* shape, int ndim, tm_dtype dtype);
tm_tensor* tm_randn(int64_t* shape, int ndim, tm_dtype dtype);

// Operations (all support autograd)
tm_tensor* tm_add(tm_tensor* a, tm_tensor* b);
tm_tensor* tm_matmul(tm_tensor* a, tm_tensor* b);
tm_tensor* tm_softmax(tm_tensor* x, int dim);

// Autograd
void tm_backward(tm_tensor* loss);
void tm_zero_grad(tm_tensor** params, int n);

Implementation Order

1. Tensor struct and memory management
2. Basic ops (add, mul, matmul) - CPU only
3. Autograd graph building
4. Backward pass for basic ops
5. NN ops (softmax, layernorm)
6. GPU backends

References

• PyTorch Internals
• ggml
• tinygrad - Minimal autograd
• micrograd - Educational autograd