DEV Community

Cover image for LoRA Explained in Plain Words: I Fine-Tuned a Small LLM by Training 2% of Its Weights, and Here Is What It Cost
AI Explore
AI Explore

Posted on

LoRA Explained in Plain Words: I Fine-Tuned a Small LLM by Training 2% of Its Weights, and Here Is What It Cost

Part 6 of ten. Each post is a short, plain-words tour of one lesson from e=mc², a free build-an-LLM-from-scratch course; the full lesson, notebook and numbers are on the site.

Almost every fine-tuned model you hear about today was not really retrained. It was given a sticky note. The technique is called LoRA, and it is how people teach a huge language model a new skill without touching, or even saving, most of it. Yesterday I fine-tuned my tiny storyteller the expensive way, turning every one of its 5.85 million dials. Today I did the same job touching 2% of them, and I want to show you what that bought and what it cost, with the real numbers.

Ella freezes the dials

You know Momo by now: the monkey who is really a model, with millions of dials in his head. Ella, the elephant who never forgets, has been his teacher since Lesson 01.

In Lesson 05 Ella taught Momo to follow requests ("tell a story about a fox in a lighthouse") by nudging all of his dials. It worked, more or less: 44% of requests he had never seen came back with the right character in the right place. But now she had two complete Momos to keep, and every new skill would mean another full copy.

So tonight she tries something different. She freezes every dial. Nobody may turn them. Then on each of Momo's big trays of dials (every large weight matrix in the model) she sticks a small note: "when this tray gives an answer, add this little correction."

Peel the notes off and the old Momo is back, untouched. Swap in a different set of notes and he has a different skill. That is LoRA, low-rank adaptation. One set of notes is an adapter.

Why the note is small

Here is the one idea I want you to walk away with.

Each note is not a second tray of the same size. It is two thin strips, A and B. A squeezes whatever comes in down to just a few numbers; B spreads those few numbers back out to the full width. The number of numbers in the narrow middle is the rank, written r.

The tray that does attention in my model is 256 by 768, which is 196,608 dials. With r = 5, its note is 5,120 numbers. The note can only change the tray along five directions, and LoRA's bet is that a new skill rarely needs more than a few. A knob called alpha sets how loudly the note speaks; here alpha is 10, so the correction is scaled by alpha / r = 2.

B starts at zero, so at step one the wrapped model is exactly the original (I checked: the largest difference in output was zero). Training writes the note.

The whole thing is one small class in PyTorch. This is the heart of the lesson's notebook:

class LoRALinear(nn.Module):
 def __init__(self, base: nn.Linear, r: int, alpha: float):
 super().__init__()
 self.base = base
 for p in self.base.parameters():
 p.requires_grad_(False) # the original weights are frozen
 self.scale = alpha / r
 self.A = nn.Parameter(torch.empty(r, base.in_features))
 self.B = nn.Parameter(torch.zeros(base.out_features, r)) # starts at zero: no change at step 0
 nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
 def forward(self, x):
 return self.base(x) + (x @ self.A.t() @ self.B.t()) * self.scale
 def merged_weight(self):
 return self.base.weight + self.scale * (self.B @ self.A) # same shape as the frozen weight
Enter fullscreen mode Exit fullscreen mode

Wrap the four big layers in each of Momo's six blocks with that, add a tiny correction for the two new request tokens, and you have LoRA by hand. No library.

The numbers, honestly

  • 2.11% of the weights trained: 123,392 of 5,853,696. Rank 4 would have been 1.69%; I used rank 5.
  • Same lesson plan as the full fine-tune: the same 4,000 request-and-story pairs, 500 steps of 32, same seed, same optimizer.
  • The file to ship: 0.51 MB for the adapter, against 23.44 MB for the full set of weights. About 2.2% of the size.
  • Optimizer memory: 0.99 MB against 46.8 MB. That one is arithmetic rather than a measurement, but it is the number that matters most at scale, because the optimizer's bookkeeping for a billion-parameter model is what stops fine-tuning from fitting on one GPU.

Now the part people skip.

It was not faster. Per step, LoRA ran at 95.6 milliseconds against 104.7 for the full fine-tune in the final run, and across four back-to-back timings it was 1% to 9% quicker. The frozen weights still run forward and backward; LoRA only skips computing their gradients and updating them. The time saving is real but modest, even on big models. What LoRA saves is memory and storage.

It did not match the full fine-tune. At Lesson 05's learning rate the adapter barely learned the job: 6% of held-out requests followed, no better than the untuned model's 12%. A note that starts at zero has to grow, so I widened the learning-rate grid (the first notebook tried only two rates; I added two more after seeing the results). Higher rates helped: 14% at 3e-3 and 26% at 1e-2, against 44% for the full fine-tune.

One more honest detail: before the run I fixed the rule that I would keep the adapter with the lowest loss on held-out text, not the best score on the check. The loss barely separated two of them, 2.211 against 2.216, so the kept adapter is the 14% one, not the 26% one. That way the check stays a test.

The merge, which did work

The other thing LoRA promises is that the notes can be copied onto the dials and thrown away. You add each scaled B times A into its frozen weight and you get a plain model, same shape as before, no adapter machinery left.

I did that. The merged model's outputs differed from the adapter model's by at most 3.29e-05, which is rounding, and all 50 test stories came out character for character identical. A separate little script rebuilt the same merged weights too. That part of the story is clean.

What I actually learned

On this tiny model, for this task, a rank-5 correction learned "read the request and choose from it" less well than 5.85 million free dials did in the same 500 steps. LoRA is usually reported close to full fine-tuning on much larger models. I did not test that, and I am not going to pretend I did.

What I can say is what LoRA buys: a half-megabyte file instead of a 23-megabyte one, a fiftieth of the optimizer memory, an untouched original you can always return to, and a merge that gives every output back. On a model a thousand times this size, that is the difference between fine-tuning on one GPU and not fine-tuning at all. It is why the original LoRA paper made exactly those savings its headline, and why so many of the "fine-tuned" models you download are really a base model plus a note.

The full lesson has the glossary, the results and cost tables, the notebook that trains all four adapters on a free GPU, the four adapters themselves, and the merge script. On my Mac, with the GPU shared, it took 11 minutes end to end.

Want to actually build it?

This post is the postcard. The lesson is the trip. Lesson 06 of e=mc² on theaivibe.org walks you through every piece with widgets you can poke, a notebook that runs on a free GPU in minutes, and the honest numbers at every step. No account, no paywall, nothing to install.

Tomorrow a brand-new tiny Momo gets a real job: Lesson 07 trains a small transformer to pick the right operation instead of writing a story.

Top comments (0)