2.7 KiB

Raw Blame History

lora

Low-Rank Adaptation (LoRA) for efficient fine-tuning. Python counterpart: HuggingFace PEFT.

Header: lora/lora.h Source: lora/lora.c Status: API Complete - Pending Verification

Overview

The LoRA module provides:

LoRA layer creation and forward pass
Weight merging for inference
Adapter save/load (safetensors format)
Configurable rank, alpha, and dropout

For weight matrix W (d_out x d_in), LoRA adds:

output = Wx + (alpha/r) * B @ A @ x

where A is (r x d_in) and B is (d_out x r)

Quick Start

#include "lora/lora.h"

// Configure LoRA
fl_config cfg = fl_config_default();  // rank=8, alpha=16, dropout=0
cfg.rank = 16;

// Create LoRA layer
fl_layer* layer = fl_layer_new("model.attn.q_proj", 4096, 4096, &cfg);

// Forward pass (adds LoRA output to base output)
tm_tensor* lora_out = fl_layer_forward(layer, x);

// Merge into base weights for inference
tm_tensor* merged = fl_layer_merge(layer, base_weight);

// Save adapter
fl_adapter* adapter = fl_adapter_new();
fl_adapter_add(adapter, layer);
fl_adapter_save(adapter, "adapter.safetensors");

fl_adapter_free(adapter);

API Reference

Configuration

typedef struct {
    int rank;            // LoRA rank (r)
    float alpha;         // Scaling factor
    float dropout;       // Dropout rate (0.0 = no dropout)
    bool fan_in_fan_out; // For Conv1D compatibility
} fl_config;

fl_config fl_config_default(void);  // rank=8, alpha=16, dropout=0

LoRA Layer

fl_layer* fl_layer_new(const char* name, int d_in, int d_out, fl_config* config);
fl_layer* fl_layer_from_tensors(const char* name, tm_tensor* A, tm_tensor* B, float alpha);
void fl_layer_free(fl_layer* layer);

// Forward: returns BAx scaled by alpha/r
tm_tensor* fl_layer_forward(fl_layer* layer, tm_tensor* x);

// Merge: W_new = W + scale * B @ A
tm_tensor* fl_layer_merge(fl_layer* layer, tm_tensor* W);

// Get merged without modifying original
tm_tensor* fl_layer_get_merged(fl_layer* layer, tm_tensor* W);

Adapter (Collection of Layers)

fl_adapter* fl_adapter_new(void);
void fl_adapter_free(fl_adapter* adapter);

void fl_adapter_add(fl_adapter* adapter, fl_layer* layer);
fl_layer* fl_adapter_get(fl_adapter* adapter, const char* name);
int fl_adapter_size(fl_adapter* adapter);
fl_layer* fl_adapter_get_idx(fl_adapter* adapter, int idx);

fl_adapter* fl_adapter_load(const char* path, fl_config* config);
bool fl_adapter_save(fl_adapter* adapter, const char* path);

Utility

int64_t fl_adapter_num_params(fl_adapter* adapter);
void fl_adapter_print(fl_adapter* adapter);

2.7 KiB Raw Blame History