Fine-tuning

"Let's just train the model on our own data." The sentence comes up in almost every project, usually after the third answer that misses. It is rarely wrong. It is almost always premature.

What you should have tried first

You have three levers, and they cost wildly different amounts:

1. The system prompt. A few sentences of instruction, ten minutes of work, reversible on the spot. How far that carries is in the System Prompt component.
2. RAG. You put the right documents into the context instead of hoping the model knows them. Effort: an afternoon for the first version.
3. Fine-tuning. You change the weights. For that you need a dataset, an hour of GPU time or more, and then a second thing nobody has on their list beforehand: a test bench that shows you whether anything got better.

Why that order? Because the first two steps are reversible and the third one is not: a system prompt that achieved nothing gets deleted and is done with, while a fine-tune that achieved nothing has cost you two days and stays in production anyway, because afterwards nobody wants to touch it. And because the first two often suffice. A surprisingly large share of the cases where somebody wants to retrain are context problems in disguise.

Form and behaviour, not knowledge

This is where the most common misunderstanding sits. Fine-tuning feels like teaching, and that is mostly how it gets sold. We pour the company documents in, afterwards the model knows our company. That is not how it works.

What does work well:

• Format. Always JSON, always these fields, never a sentence of preamble.
• Tone. Short, factual, no apologies.
• Selection behaviour. Picking the right one out of forty tools. Or none at all.
• One fixed task. Pulling the four fields your ticket system needs out of an email.

What works badly is feeding in facts. Gekhman et al. studied this in 2024 (arXiv:2405.05904) and brought back an uncomfortable finding. Examples carrying knowledge the model never saw in pre-training are learned far more slowly than familiar ones. And the more of them sit in the dataset, the more often the model invents things elsewhere too. So whoever wants to supply facts uses RAG. Whoever wants to nail down behaviour uses fine-tuning. Whoever needs both does both.

The rule of thumb is short. If the answer sits in a document you could send along with the request, fine-tuning is the wrong tool.

Train everything, or add one layer?

With full fine-tuning every weight gets adjusted. For an 8B model that means weights, gradients and optimiser states have to sit in memory at the same time, which puts us well past a hundred gigabytes of VRAM. That is data centre work. And you end up with a complete second model that you then have to manage, version and ship completely, with everything that hangs off a model.

The other shape is called LoRA, and it is the reason fine-tuning is within reach of small teams at all. The idea comes from Hu et al. (2021, arXiv:2106.09685). The base weights stay frozen, and training touches only a small, low-rank extra piece that gets added onto each layer's output at runtime. In the paper this cut the number of trainable parameters by several orders of magnitude, at comparable quality on the tasks they measured.

QLoRA (Dettmers et al., 2023, arXiv:2305.14314) pushes that further by quantising the frozen base model to 4 bits as well. That fits the training of a very large model onto a single fat GPU. For the model sizes local agents usually deal with, considerably less will do.

So what is an adapter, then?

The result of a LoRA run is a file next to the model. Not inside it. A few dozen megabytes, depending on the chosen rank sometimes far less, and beside it sit several gigabytes of base weights that never changed and that you therefore never have to download, store or distribute again.

What does that size ratio change? Almost everything. You can keep several adapters for the same base model and switch one on at startup, a different one tomorrow. You can version it like code. You can throw it away. What that looks like in server operation, which engine offers which switch for it and what several adapters cost at once is in the deep dive.

Fine-tuning is the third lever, not the first. Prompt, then context, then weights - and in practice the first two solve the larger share of cases.

It teaches a model form and behaviour, not knowledge. Facts belong in the context, behaviour in the weights.

And these days it almost never happens as full training. The normal case is a LoRA adapter, a small extra piece that sits beside the base model and gets switched on.