86 lines
2.1 KiB
Markdown
86 lines
2.1 KiB
Markdown
# data/
|
|
|
|
C implementation of dataset loading and processing.
|
|
|
|
## Python counterpart
|
|
|
|
**datasets** - https://github.com/huggingface/datasets
|
|
|
|
## Why datasets?
|
|
|
|
HuggingFace datasets provides:
|
|
|
|
- Unified dataset loading (local files, HuggingFace Hub)
|
|
- Memory-mapped files for large datasets
|
|
- Streaming for datasets that don't fit in memory
|
|
- Data transforms and filtering
|
|
|
|
Built on Apache Arrow for efficient columnar storage.
|
|
|
|
## Scope
|
|
|
|
This module implements:
|
|
|
|
1. **JSONL loading** - Line-delimited JSON (most common format)
|
|
2. **Memory mapping** - Handle datasets larger than RAM
|
|
3. **Batching** - Create training batches
|
|
4. **Shuffling** - Randomize data order
|
|
5. **binidx format** - Pre-tokenized binary format (fast loading)
|
|
|
|
## Getting Started
|
|
|
|
### Dependencies
|
|
|
|
- tokenizer/ module (for on-the-fly tokenization)
|
|
- Optional: mmap support (POSIX)
|
|
|
|
### Key Files to Study
|
|
|
|
From datasets:
|
|
- `src/datasets/arrow_dataset.py` - Core dataset class
|
|
- `src/datasets/iterable_dataset.py` - Streaming datasets
|
|
|
|
From PyTorch:
|
|
- `torch/utils/data/dataloader.py` - DataLoader pattern
|
|
|
|
### Core API
|
|
|
|
```c
|
|
// Load dataset
|
|
fd_dataset* fd_load_jsonl(const char* path);
|
|
fd_dataset* fd_load_binidx(const char* path); // Pre-tokenized
|
|
|
|
// Dataset info
|
|
int64_t fd_len(fd_dataset* ds);
|
|
tm_tensor* fd_get(fd_dataset* ds, int64_t index);
|
|
|
|
// DataLoader
|
|
fd_dataloader* fd_dataloader_new(fd_dataset* ds, int batch_size, bool shuffle);
|
|
tm_tensor* fd_next_batch(fd_dataloader* dl); // Returns NULL at end
|
|
void fd_reset(fd_dataloader* dl);
|
|
|
|
// Transforms
|
|
fd_dataset* fd_map(fd_dataset* ds, tm_tensor* (*fn)(tm_tensor*));
|
|
fd_dataset* fd_filter(fd_dataset* ds, bool (*fn)(tm_tensor*));
|
|
fd_dataset* fd_shuffle(fd_dataset* ds, int seed);
|
|
|
|
// Memory-mapped loading
|
|
fd_dataset* fd_load_mmap(const char* path);
|
|
```
|
|
|
|
## Implementation Order
|
|
|
|
1. JSONL parser
|
|
2. In-memory dataset
|
|
3. Batching and iteration
|
|
4. Shuffling
|
|
5. Memory-mapped loading
|
|
6. binidx format (pre-tokenized)
|
|
7. Streaming for very large datasets
|
|
|
|
## References
|
|
|
|
- [HuggingFace datasets](https://huggingface.co/docs/datasets)
|
|
- [Apache Arrow](https://arrow.apache.org/)
|
|
- [PyTorch DataLoader](https://pytorch.org/docs/stable/data.html)
|