The starting state

Before you improve anything, you have to be able to see what happens. That is stage one from the chapter Working together, and it costs a day. Skip it and you later measure against a feeling.

Where the tools come from

Why not just invent a few tools? Because afterwards you cannot say whether a call was right. So the course takes the ones that already exist. Two sim worlds run in the mcp container of this repo, built for the parcours tasks:

Eleven tools. To the model they are a list of schemas, so you pull them out once in exactly the shape a chat API expects. An excerpt from werkzeuge.json:

{
  "type": "function",
  "function": {
    "name": "update_contact",
    "description": "Change fields of one contact. Only the fields you pass are touched - everything else stays as it is. `notes` REPLACES the note; if you want to keep what is there, read it first and pass the combined text. `stand` is set to now.",
    "parameters": {
      "type": "object",
      "properties": {
        "contact_id": { "type": "string", "description": "Id of the contact, a uuid." },
        "status": { "type": "string", "description": "One of: neu, kontaktiert, kunde, verloren." }
      },
      "required": ["contact_id"]
    }
  }
}

The descriptions are literally the ones from the server. Smoothing them would be tempting and wrong. The specialist should learn what really stands in its prompt in production, and in production what stands there is a sentence about a field called notes that replaces the existing entry rather than adding to it - with the warning behind it that you have to read first if you want to keep anything.

The opening all three states share

A large model with tools is a planner. On "show me the appointments in the calendar brandt" it likes to call list_calendars first to look up the key - sensible agent behaviour, and still the wrong move for this test bench, because what should be measured is how an intention becomes a call, not the planning around it.

So all three states get the same opening, word for word:

AUFTAKT = (
    "You are the tool layer of Simhaven. The calendar keys are brandt, keller, "
    "ruben, raum and organisator; contact ids appear literally in the request. "
    "Answer a request with exactly the tool calls it asks for, no exploratory "
    "lookups. If no tool fits, or a required argument is missing, answer in "
    "words instead."
)

Three sentences, and the last one matters most. It permits doing nothing. Without it every refusal would be a rule violation, and the test stages S4 and S5 would measure something nobody asked for.

What gets logged

What belongs in such a log? Four things per round, and you may leave out none of them:

• The raw output, unchanged. Not the parsed version, not the cleaned one. If the parser fails, that is the only place you see why.
• The parsed call, meaning name and arguments after parsing.
• The result in the shape the model would get to see it.
• finish_reason and duration. A call that the token limit cut off looks like a format error in the log. finish_reason is the only difference.

In the measuring script of the course it looks like this:

protokoll.append({
    "tag": tag, "fall": fall["id"], "stufe": fall["stufe"], "runde": k,
    "roh_inhalt": (msg.get("content") or "")[:400],
    "ist": ist, "soll": soll, "form_ok": form_ok,
    "treffer": treffer, "schritte_soll": len(soll),
    "schritte_ist": len(ist), "gelungen": gelungen,
    "finish_reason": finish, "dauer_s": round(dauer, 3), "fehler": fehler,
})

One line per request, as JSON on disk. That is enough. You need no benchmark infrastructure for it, and whoever builds one before the first number exists has lost their way.

What the log gives away immediately

The first run of the large model against the eleven tools takes 6.42 seconds per request at the median, 2.28 at best and 25.42 at worst. That spread is the real finding of this chapter. A model that thinks before it calls is not four times slower than a specialist on a simple translation - at worst it is a hundred times slower.

Whether that bothers you depends on where the call sits. In an overnight batch nobody notices. In front of a waiting person they do, and in practice that is where it shows up first.

That settles the starting state. What is missing is the number to hold it against.