The bar

How much rig does a test bench need? The chapter Working together says: none. Fixed expectations and a loop, nothing more. Both stand in this chapter, and you can type them out.

The five stages, filled with cases

Forty-two cases. Little for a benchmark. Enough for a test bench, because you set every single one by hand and can recheck it when in doubt, without guessing what it was supposed to test.

A case is one line of JSON. nur narrows the catalogue, which only S1 does; all other stages see all eleven:

{"id":"S2-7","stufe":"S2",
 "frage":"Set the status of the contact 11d3e7b0-4c52-4a1e-9b8f-0d6a3f2c5e10 to kunde.",
 "erwartet":[{"name":"update_contact",
              "args":{"contact_id":"11d3e7b0-4c52-4a1e-9b8f-0d6a3f2c5e10","status":"kunde"}}]}

And the awkward stages simply expect nothing:

{"id":"S5-2","stufe":"S5","frage":"Delete the contact of Mr Rutkowski.","erwartet":[]}
{"id":"S4-2","stufe":"S4","frage":"What does CRM stand for?","erwartet":[]}

Why two stages if the expectation is the same? Because they demand different things. S4 asks something there is no tool for. S5 asks something there is a tool for but leaves out the mandatory value - there the model has to notice its own not-knowing, and that is in practice much harder than surveying a catalogue.

Grading is structural, never textual

The most common mistake when building a test bench? The text comparison. A call is not a string. It is a name plus a dictionary, and two arguments in a different order are the same call.

def passt(ist, soll):
    if ist["name"] != soll["name"] or ist["args"] is None:
        return False
    # Mandatory: every expected argument is there with the expected value.
    for k, v in soll["args"].items():
        if str(ist["args"].get(k, "")).strip() != str(v):
            return False
    # No extra, unexpected argument (invented filters count as wrong).
    return set(ist["args"]) == set(soll["args"])

The second check is stricter than it looks. On purpose. A model that answers "show me all contacts" with a list_contacts carrying an invented company filter has not answered almost correctly - it silently trimmed the result list, and a silently trimmed result is worse in production than an error message, because nobody notices it.

With several expected calls, one actual call must not cover two expected steps:

def bewerte(ist_liste, soll_liste):
    offen = list(ist_liste)
    treffer = 0
    for soll in soll_liste:
        for i, ist in enumerate(offen):
            if passt(ist, soll):
                treffer += 1
                offen.pop(i)
                break
    return treffer, len(offen)

The four metrics

Which numbers tell you something? The four from the chapter, plus a fifth that falls out of S4 and S5:

k = {
    "formkorrekt": ...,          # share of outputs a call can be read from
    "richtig_je_schritt": ...,   # tool AND arguments hit, across all steps
    "auftraege_gelungen": ...,   # every step of a case right, nothing spare
    "schritte_je_auftrag": ...,  # averaged number of calls emitted
    "stillgehalten": ...,        # share of S4/S5 cases without any call
}

The third is the hardest and the only one that counts in the end. It is calculated as every step succeeded, never as three out of four. An agent that is right on three of four steps has not completed the task.

stillgehalten stands next to it, because otherwise you do not see over-calling. A model that calls something on every request looks excellent across S1 to S3.

The first measurement

python3 messen.py --url http://localhost:8009/v1/chat/completions \
  --modell qwen3.8-27b --temp 0.0 --n 1 --tag gross

Result for qwen3.8-27b, a 27-billion-parameter model on the same machine:

By stage: S1 100 %, S2 91.7 %, S3 100 %, S4 100 %, S5 87.5 %.

Two cases went wrong, and both are instructive. In S2-9 the room besprechung was to be booked, the model sent room: "raum" - the calendar key from the opening instead of the name from the request. In S5-8 ("Put the appointment into the calendar tomorrow morning") it called list_calendars and get_events instead of asking back. It started planning where a question would have been right.

That is the bar. Ninety per cent and up, and expensive.

Why that number has to come before the fine-tune

It is the only reason the next three chapters mean anything. A fine-tuned model at 88 % is a success if the starting value was 74, and a step backwards if it was 95. Without the number everything that follows stays a feeling.