LoRA

Overview

From-scratch PyTorch implementation of LoRA (Low-Rank Adaptation). Rather than fine-tuning all parameters, LoRA freezes the pre-trained weights and injects trainable low-rank decomposition matrices (A and B) into each attention projection. The update ΔW = BA where B ∈ Rᵈˣʳ and A ∈ Rʳˣᵏ, with rank r ≪ d. Based on LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2022).

Architecture

• LoRA adapters injected into Q, V projections
• Rank r configurable; original weights frozen
• Zero-initialised B, random Gaussian A — ensures ΔW=0 at start

Training

Results

Paper

LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., 2022