The raw specialist

Now the small model comes in between. The chapter FunctionGemma describes the connection in full; this chapter makes it and measures what comes out.

Up front, so you are not surprised. The measurement comes out worse. It is meant to come out worse - a model that keeps up with a 27B on a foreign tool set right away would need no training, and then this course would have no subject.

The service

Where does the specialist belong? Next to the application, not inside it. Its own model volume, its own health check, and it may fail without taking the application with it. The building block Sidecars describes the shape.

docker run -d --name functiongemma --gpus all \
  -v llm-stack_hf-cache:/hf-cache -e HF_HUB_CACHE=/hf-cache \
  -p 127.0.0.1:8008:8000 \
  vllm/vllm-openai:cu130-nightly \
  --model google/functiongemma-270m-it --served-model-name functiongemma-270m \
  --host 0.0.0.0 --port 8000 --dtype bfloat16 --max-model-len 32768 \
  --gpu-memory-utilization 0.035 --max-num-seqs 8 \
  --enable-auto-tool-choice --tool-call-parser functiongemma

The ENTRYPOINT of this image is already vllm serve, which is why no serve appears in the line - a second one aborts the call with unrecognized arguments. And three lines carry the rest. --gpu-memory-utilization 0.035 reserves three and a half per cent of the card, because the model and its context need no more and the rest belongs to the large models. --enable-auto-tool-choice opens the tool interface. And --tool-call-parser functiongemma connects the notation from chapter 06 - the server reads <start_function_call>call:name{…}<end_function_call> and hands it out as ordinary tool_calls, so your harness never notices that a model with its own language sits here.

If you work with llama.cpp, you need --jinja there for the shipped chat template and <start_function_response> in the stop sequences. vLLM does both by itself.

A look at the rendered prompt

What does the model actually get to read? Take a look once. The chat template from the weights turns an ordinary tool schema into this:

<bos><start_of_turn>developer
You are the tool layer of Simhaven. …<start_function_declaration>declaration:get_events{description:<escape>Read events.<escape>,parameters:{properties:{calendar:{description:<escape>Calendar key.<escape>,type:<escape>STRING<escape>}},type:<escape>OBJECT<escape>}}<end_function_declaration><end_of_turn>
<start_of_turn>user
Show brandt.<end_of_turn>
<start_of_turn>model

All of it as chapter 06 describes: role developer, <escape> around every string value, type names in capitals. And the answer to it, pulled raw from the completion endpoint:

<start_function_call>call:get_events{}<end_function_call>

The model stops after <end_function_call> by itself. That observation matters twice. It confirms the connection stands, and it fixes where your training answer has to end in chapter 5.

The second measurement

Same test bench, same forty-two cases, same opening:

python3 messen.py --url http://localhost:8008/v1/chat/completions \
  --modell functiongemma-270m --temp 0.0 --n 1 --tag roh-t0

By stage: S1 87.5 %, S2 66.7 %, S3 83.3 %, S4 87.5 %, S5 50.0 %.

The first row is the good news, and it is no small thing. A hundred per cent well-formed means template, parser and notation are right. Whatever still goes wrong now is a matter of understanding and not of wiring. Had that number collapsed, you would have to look there first.

And the last row? 0.20 instead of 6.42 seconds. Thirty-two times faster, on the same machine.

Where it fails

The interesting errors sit in S5, and they all look alike. The request does not name the mandatory value. The model puts something in anyway:

The third case is the prettiest. The model writes the placeholder from the schema down as a value and mangles the enum value on the way - error pattern B-5 in its purest form. A delete_contact with an invented id would be no cosmetic flaw in production.

Next to it a pattern from S2 with the same root:

S2-7  expected: update_contact{contact_id: …, status: "kunde"}
      actual:   update_contact{contact_id: …, status: "kanada"}
S2-10 expected: list_contacts{status: "neu"}
      actual:   list_contacts{}
S3-6  expected: invite{event_id: …, calendar: "brandt"} + invite{…, calendar: "keller"}
      actual:   invite{event_id: …, calendar: <the event id>}

kunde turns into kanada, a mandatory filter drops out, and the event id ends up in the calendar field. The model has understood which tool is meant. It just does not know what belongs in the fields, because it has never seen these fields.

Here a fine-tune can do something. The tool is usually chosen correctly already, the arguments are not.

The temperature question, settled

Chapter 06 leaves one question open. The vendor names 1.0, the rule of thumb of the chapter says low. Who is right? The contradiction belongs on the test bench, so onto the test bench it goes: three times per case at the vendor values, against a deterministic run.

For this tool set the answer is unambiguous. The vendor temperature costs 8.7 points. S3 suffers most, falling from 83.3 to 50.0, and that fits: writing two related calls cleanly one after the other is the kind of output sampling takes apart first.

That confirms the rule of thumb and does not refute the vendor figure. It presumably holds for the tool set the model was trained on. For yours, what your test bench says holds. The rest of the course runs at 0.0.