2.2 KiB

Raw Permalink Blame History

rfa_lora_merge

CLI tool for averaging multiple LoRA adapters. Used for cross-machine parallel training.

Source: examples/rfa_lora_merge.c Build: make USE_BLAS=1 rfa-lora-merge

Overview

Loads N LoRA adapter safetensors files, averages A and B weight matrices element-wise across all adapters (FedAvg), and saves the merged result. All input adapters must have the same structure (same number of layers, same dimensions).

Supports up to 64 input adapters.

Usage

./examples/rfa_lora_merge \
  --output merged_adapter.safetensors \
  adapter_0.safetensors adapter_1.safetensors [... adapter_N.safetensors]

Cross-machine merge workflow

# After training on M4 and M3 separately:
scp m3:loras_m3/adapter_final.safetensors ./loras_m3/

./examples/rfa_lora_merge \
  --output adapter_merged.safetensors \
  loras_m4/adapter_final.safetensors \
  loras_m3/adapter_final.safetensors

Merging many thread adapters

./examples/rfa_lora_merge \
  --output adapter_merged.safetensors \
  thread_0.safetensors thread_1.safetensors \
  thread_2.safetensors thread_3.safetensors \
  thread_4.safetensors thread_5.safetensors

CLI Options

Flag	Default	Description
`--output PATH`	(required)	Output merged adapter path
`ADAPTER1 ADAPTER2 ...`	(required)	At least 2 input adapter files

Averaging Method

Standard FedAvg — element-wise average of all A and B matrices:

For each layer i:
    merged.A[i] = (adapter[0].A[i] + adapter[1].A[i] + ... ) / N
    merged.B[i] = (adapter[0].B[i] + adapter[1].B[i] + ... ) / N

The tool verifies that all input adapters have the same number of layers before merging.

Output

Loading 3 adapters...
  [0] adapter_0.safetensors: 49 layers, 9200000 params
  [1] adapter_1.safetensors: 49 layers, 9200000 params
  [2] adapter_2.safetensors: 49 layers, 9200000 params
Averaging 3 adapters (49 layers each)...
Saved: merged.safetensors (49 layers, 9200000 params)

2.2 KiB Raw Permalink Blame History