Fine-tune FunctionGemma on your own MCP
A 550 MB model learns the eleven tools of the sim worlds. Three measurements on one test bench, a distilled dataset and a LoRA run that fits on a small card.
Why does a small model choose so badly among many tools? And what actually helps against it?
The first question is familiar to anyone running local models. To the second you usually hear "better prompt", sometimes "bigger model". This course answers with a fine-tune and shows across three measurements what it buys and what it does not.
The subject
The training runs against the sim worlds of this repo. Two finished MCP servers
live in the mcp container: the calendar of Simhaven
(mcp/tools/sim_calendar.ts, five tools) and the CRM of the same town
(mcp/tools/sim_crm.ts, six tools). Eleven tools together, with real schemas,
real mandatory fields and real enum values.
That is not a side detail. The chapter Working together names three conditions under which a fine-tune pays off, and all three have to hold:
| Condition | How it holds here |
|---|---|
| Stable action set | Eleven tools, unchanged for months, with a zod schema |
| Logged real cases | The parcours runs produce traces against exactly these worlds |
| Test bench | S1 to S5 are spelled out in the chapter and arrive here as code |
Practise against an invented weather API and you cannot say afterwards whether a call was right. Here you can.
The arc
Six steps, and the order is the point. Why not train right away? Because the chapter Working together calls the specialist "stage four, not stage one": whoever starts with the training ends up with a second model in production and still does not know whether anything got better.
- The starting state. The agent runs against the sim MCP with the large model, and every round is logged.
- The bar. The test bench S1 to S5 as runnable code, four metrics, first measurement.
- The raw specialist. FunctionGemma as its own service in front, second measurement. It comes out worse, and that is the reason for step four.
- The dataset. Pairs of request, tool list and call, distilled from the runs of the large model.
- The LoRA run. An adapter on 270 million parameters, third measurement against the same test bench.
- Back into the harness. Specialist as the fast path, large model as the fallback.
What ends up on the table
Three numbers on the same test bench, measured on a DGX Spark:
| State | Tasks completed | Median per request |
|---|---|---|
Large model (qwen3.8-27b) | 95.2 % | 6.42 s |
Raw specialist (functiongemma-270m-it) | 73.8 % | 0.20 s |
| Fine-tuned specialist | 90.5 % | 0.16 s |
The middle row is the interesting one. Raw, the small model answers thirty-two times faster and trails by twenty-one points. After the training those twenty-one points have become barely five, at forty times the speed - and on the test stage where it failed most clearly before, it now sits level with the 27B.
What you need
A machine with a card that fits a 270-million-parameter model. Which is not
much. In bfloat16 that is around 550 MB, and the LoRA training gets by on
about four gigabytes. On top of that you need a large model to do the labelling;
whether it runs locally or comes over an API is all the same to the course, but
over an API it costs money.
Before you start you should have read the two chapters of the building block Tool calling in production that the course builds on. The building block Fine-tuning classifies what becomes manual work here.
The roadmap
6 steps, one insight and one working state each. Every step stands on the one before it.
- 01The starting state25 min
Before anything is improved, every round has to be readable. Raw output, parsed call, result, finish_reason.
Milestone Eleven tools from the sim worlds stand there as a call list, and one run of the large model is fully logged.
- 02The bar35 min
A test bench is not a benchmark rig. It is forty-two fixed expectations and a loop.
Milestone S1 to S5 run through with structural grading, and the large model stands at 95.2 % of tasks completed.
- 03The raw specialist30 min
Thirty-two times faster and twenty points worse. And when a mandatory value is missing, it invents one.
Milestone The specialist runs as its own service, the second measurement stands, and the temperature question from chapter 06 is settled.
- 04Distilling the dataset35 min
The awkward cases are half the work: no tool needed, mandatory value missing. Without them the fine-tuned specialist calls something on everything.
Milestone A dataset from real runs of the large model exists, filtered through the gate, with S4 and S5 cases in it.
- 05Running the LoRA35 min
The loss runs over the answer only, and the prompt is rendered with the same template as in production. Separate those two and you train a format that never arrives.
Milestone The adapter is trained, merged, served and measured on the same test bench.
- 06Back into the harness30 min
The specialist is the fast path, the large model the fallback. What decides the path is the gate.
Milestone The switch is in place, and the calculation of when the effort pays off is done with your own numbers.