81 lines
2.1 KiB
Markdown
81 lines
2.1 KiB
Markdown
# lora/
|
|
|
|
C implementation of Low-Rank Adaptation (LoRA) for efficient fine-tuning.
|
|
|
|
## Python counterpart
|
|
|
|
**PEFT** - https://github.com/huggingface/peft
|
|
|
|
## Why PEFT?
|
|
|
|
PEFT (Parameter-Efficient Fine-Tuning) is HuggingFace's library for adapter-based training. Core technique is LoRA:
|
|
|
|
- Freeze base model weights
|
|
- Add small trainable low-rank matrices
|
|
- Train only ~0.1% of parameters
|
|
- Merge adapters back into base model
|
|
|
|
Alternatives (Unsloth, Axolotl, LLaMA-Factory) are wrappers around PEFT with extra features.
|
|
|
|
## Scope
|
|
|
|
This module implements:
|
|
|
|
1. **LoRA layers** - Low-rank A and B matrices
|
|
2. **Target selection** - Apply LoRA to specific layers
|
|
3. **Merge/unmerge** - Combine adapters with base weights
|
|
4. **Adapter save/load** - safetensors format
|
|
5. **Multi-adapter** - Load and switch between adapters
|
|
|
|
## Getting Started
|
|
|
|
### Dependencies
|
|
|
|
- tensor/ module (for autograd)
|
|
- safetensors/ module (for save/load)
|
|
|
|
### Key Files to Study
|
|
|
|
From PEFT:
|
|
- `src/peft/tuners/lora/layer.py` - LoRA layer implementation
|
|
- `src/peft/tuners/lora/model.py` - Model wrapping
|
|
|
|
From the paper:
|
|
- [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)
|
|
|
|
### Core API
|
|
|
|
```c
|
|
// Create LoRA layer wrapping a linear layer
|
|
fl_lora* fl_lora_new(tm_linear* base, int rank, float alpha, float dropout);
|
|
|
|
// Forward pass (base + lora contribution)
|
|
tm_tensor* fl_lora_forward(fl_lora* layer, tm_tensor* x);
|
|
|
|
// Merge LoRA into base weights (for inference)
|
|
void fl_lora_merge(fl_lora* layer);
|
|
void fl_lora_unmerge(fl_lora* layer);
|
|
|
|
// Save/load adapters
|
|
void fl_lora_save(fl_lora** layers, int n, const char* path);
|
|
fl_lora** fl_lora_load(const char* path, int* n);
|
|
|
|
// Get trainable parameters only
|
|
tm_tensor** fl_lora_params(fl_lora** layers, int n, int* num_params);
|
|
```
|
|
|
|
## Implementation Order
|
|
|
|
1. LoRA linear layer (forward only)
|
|
2. LoRA backward pass
|
|
3. Initialization (Kaiming for A, zero for B)
|
|
4. Merge/unmerge
|
|
5. Save/load (safetensors format)
|
|
6. Multi-adapter support
|
|
|
|
## References
|
|
|
|
- [LoRA Paper](https://arxiv.org/abs/2106.09685)
|
|
- [QLoRA Paper](https://arxiv.org/abs/2305.14314)
|
|
- [PEFT Documentation](https://huggingface.co/docs/peft)
|