θ′ = θ + Δθ:
why small updates matter.
A plain-language look at parameter-efficient fine-tuning, low-rank adaptation, and the questions DeltaTune Lab wants to test.
The equation in our logo describes fine-tuning in its simplest form. A model has parameters θ. Training on new data produces a change Δθ, and the adapted model uses θ′ = θ + Δθ.
Full fine-tuning updates every parameter. For a large model, that means storing gradients and optimizer state for billions of values, and saving a full-size copy for every specialised version. That cost puts adaptation out of reach for many small teams.
Changing less
Parameter-efficient fine-tuning (PEFT) starts from a different assumption: the useful change may live in a much smaller space than the model itself. Instead of learning every entry of Δθ, these methods learn a compact update and keep the original weights frozen.
One widely used example is low-rank adaptation (LoRA), introduced by Hu et al. in 2021. For a weight matrix W, LoRA learns the update as a product of two thin matrices:
When the rank r is small, A and B contain far fewer values than W. The base model can be shared, and each task only needs its own small adapter.
The questions we want to test
Methods like this raise practical questions that we think deserve careful measurement:
- How does adapter size trade off against what a model can actually learn?
- Which layers benefit most from adaptation, and does that change by task?
- What are the real memory and time savings on accessible GPUs, not just on paper?
- When does a small update fall short, and how do we detect it reliably?
None of these have a single answer. That is exactly why they are worth studying.
This post is an explainer of existing ideas, not a report of DeltaTune results. Experimental findings will be published with their configurations, baselines, and limitations.