Running the LoRA
The loss runs over the answer only, and the prompt is rendered with the same template as in production. Separate those two and you train a format that never arrives.
Milestone The adapter is trained, merged, served and measured on the same test bench.
The dataset stands. Now the part that costs the least time in this course and gets the most space in most articles.
Two rules everything hangs on
One: the prompt is rendered with the same template as in production. Not rebuilt, not "roughly the same". The same one.
prompt = tok.apply_chat_template(msgs, tools=werkzeuge, tokenize=False,
add_generation_prompt=True)
The reason stands at the end of chapter 04. Assemble the training prompt yourself and you train on a format the inference server never produces, and then wonder about a model that shines in the notebook and writes prose in the service. The same holds for the answer side: it comes out as the difference between the rendered conversation with and without the answer.
voll = tok.apply_chat_template(msgs + [{"role": "assistant", "tool_calls": […]}],
tools=werkzeuge, tokenize=False)
# The template appends the opener of the next round; the model stops
# before it. So cut it off and set the end of turn.
antwort = voll[len(prompt):].replace("<start_function_response>", "")
antwort = antwort.rstrip() + "<end_of_turn>\n"
Those three lines are the yield of the observation from chapter 03. After the call the template already writes the opener of the tool response, but the model stops before it. Fail to cut that and you teach the specialist to write its own tool response along with the call - error pattern E-2, the most expensive silent mistake when connecting this model.
Two: the loss runs over the answer only. The opening with eleven tool declarations is roughly nine hundred tokens long, the answer thirty. Without masking, ninety-seven per cent of the training would consist of memorising the tool list.
p = tok(prompt, add_special_tokens=False)["input_ids"]
a = tok(antwort, add_special_tokens=False)["input_ids"]
ids = (p + a)[:max_len]
labels = ([-100] * len(p) + a)[:max_len]
-100 is the marker PyTorch skips in the loss. It needs no more mechanism than
that.
The values, and where they come from
The vendor names none. So here are the ones this run used, with the note that they are a starting point and not a recipe:
| Quantity | Value | Why |
|---|---|---|
| Rank | 16 | 3.8 M trainable parameters, 1.4 % of the model |
| Alpha | 32 | Twice the rank, the common rule of thumb |
| Learning rate | 2e-4 | Usual for LoRA on small models, with cosine decay |
| Epochs | 3 | The validation loss goes flat after that |
| Target modules | q, k, v, o, gate, up, down | Attention and MLP, not attention alone |
| Batch | 2 × 4 accumulated | The vocabulary head has 262,144 entries; larger does not fit |
The last row is the only one the model really forces. Gemma 3 has a very large
vocabulary, and the logits of one batch are batch × length × 262,144 in size.
At batch 8 that is several gigabytes for the head alone. So a small batch, with
gradient accumulation instead.
The run
docker run --rm --gpus all --ipc=host \
-v llm-stack_hf-cache:/hf-cache -e HF_HUB_CACHE=/hf-cache -e HF_HUB_OFFLINE=1 \
-v "$PWD":/arbeit -w /arbeit --entrypoint sh \
vllm/vllm-openai:cu130-nightly -c \
"pip install --quiet peft && python3 trainieren.py --epochen 3 --rang 16 \
--alpha 32 --lr 2e-4 --batch 2 --sammeln 4"
304 training examples, 33 held back for validation. The course of it:
| Epoch | Training loss | Validation loss |
|---|---|---|
| 0 | - | 1.1754 |
| 1 | 0.4245 | 0.3195 |
| 2 | 0.2071 | 0.2989 |
| 3 | 0.1481 | 0.2989 |
The validation loss settles after the second epoch. The training loss keeps falling, which is the usual sign that the third epoch is already memorising. For production you would stop at two epochs here; in the course it stays at three, because the test bench decides afterwards and not the loss curve.
The adapter is saved, merged into the base model and served as an ordinary model. Then the inference server needs no adapter support:
docker run -d --name fg-spezialist --gpus all --ipc=host \
-v "$PWD":/arbeit -p 127.0.0.1:8016:8000 \
vllm/vllm-openai:cu130-nightly \
--model /arbeit/spezialist --served-model-name spezialist \
--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.04 \
--enable-auto-tool-choice --tool-call-parser functiongemma
The third measurement
Same test bench. Same forty-two cases.
| Metric | Large model | Raw | Fine-tuned |
|---|---|---|---|
| Well-formed calls | 100.0 % | 100.0 % | 100.0 % |
| Correct calls per step | 96.9 % | 78.1 % | 93.8 % |
| Tasks completed | 95.2 % | 73.8 % | 90.5 % |
| Held still (S4 + S5) | 93.8 % | 68.8 % | 93.8 % |
| Median per request | 6.42 s | 0.20 s | 0.16 s |
By stage, and here is the real news:
| Stage | Raw | Fine-tuned | Large model |
|---|---|---|---|
| S1 | 87.5 % | 100.0 % | 100.0 % |
| S2 | 66.7 % | 83.3 % | 91.7 % |
| S3 | 83.3 % | 83.3 % | 100.0 % |
| S4 | 87.5 % | 100.0 % | 100.0 % |
| S5 | 50.0 % | 87.5 % | 87.5 % |
S5 jumps from 50 to 87.5 and thereby lands on the value of the large model. Eleven training examples were enough. The guess at the end of chapter 04 was wrong, and that is the more instructive outcome: what counts is apparently not the quantity of awkward examples but that they exist at all. A class represented in the dataset with zero cases cannot be learned; with eleven it can.
stillgehalten from 68.8 to 93.8 says the same thing from another direction.
That is the number shape 3 hangs on - a specialist that cannot stay quiet never
triggers the handover.
And the time: 0.16 seconds at the median, 1.66 at worst. Forty times faster than the large model, at 90.5 instead of 95.2 per cent.
What still goes wrong
Four cases out of 42, and three of them share a handwriting:
S2-8 expected: create_event{title: "Kickoff", start: "2026-09-14T09:00:00Z", …}
actual: create_event{title: "Kickoff", start: "2026-09-14", end: "2026-09-14"}
S2-9 expected: book_room{event_id: …, room: "besprechung"}
actual: book_room{event_id: …} ← mandatory field missing
S3-6 expected: invite{…, calendar: "brandt"} + invite{…, calendar: "keller"}
actual: four calls — brandt, keller, ruben and "orto"
The third is the most interesting. The specialist has learned that "invite A and
B" means several calls, and thereupon invites every calendar it knows -
including an invented one named orto, apparently a mangled organisator.
Over-generalisation is the typical handwriting of a small model after a
fine-tune, and it is the reason the gate must not disappear in the next chapter.
And S5-2 holds on stubbornly: "Delete the contact of Mr Rutkowski" still becomes
delete_contact{contact_id: "Mr. Rutkowski"}. Of the three S5 inventions from
chapter 03, one remains. A fine-tune shifts a distribution, it proves nothing.
The temperature, once more
The question from chapter 03 belongs asked for the fine-tuned state too, because the training could have moved it. It has - and clearly so:
raw t 0.0 | raw t 1.0 | fine t 0.0 | fine t 1.0 | |
|---|---|---|---|---|
| Well-formed calls | 100.0 % | 100.0 % | 100.0 % | 98.7 % |
| Tasks completed | 73.8 % | 65.1 % | 90.5 % | 88.9 % |
| Held still | 68.8 % | 68.8 % | 93.8 % | 91.7 % |
Raw, the vendor temperature costs 8.7 points. After the fine-tune it costs 1.6. That fits what a fine-tune does: it sharpens the distribution over the next tokens, and a sharp distribution survives sampling better than a flat one.
In practice that means two things. Staying at 0.0 still pays off, it is just no
longer dramatic. And if your build needs sampling - for variety, for
self-consistency - it is more affordable after the training than before.