Distilling the dataset

Where do the training examples come from? Chapter 03 showed that the specialist usually finds the right tool and fills the fields wrongly. So it needs examples of what those fields look like - many of them, in the notation it is later supposed to answer in.

Inventing them is the wrong road. An invented dataset contains the cases somebody thought of, and afterwards the model can handle only those.

The three sources

A training example has three parts, and each comes from somewhere else:

The middle row is a concession, and it belongs said out loud. Real user requests would be better, because they are crooked - incomplete, half phrased, sometimes in the wrong language. The course does not have them, so it produces them from templates and varies phrasing, ids, companies and times. For the labelling that changes nothing, because the labelling comes from the large model either way.

("get_events", ["What is in the calendar {kal}?", "Show the appointments of {kal}.",
 "Read the events of {kal}.", "I want to see {kal}'s schedule.",
 "Pull the entries of calendar {kal}.", "List everything booked in {kal}.",
 "Show the events of {kal} between {t1} and {t2}.", …]),

A quarter of the tool cases gets a shrunken catalogue - four tools instead of eleven. Otherwise the specialist learns on the side that the same eleven declarations always stand there, and collapses as soon as a catalogue looks different. In this run that is 42 of 337 examples.

The gate filters, not the eye

The large model labels, but it does not have the last word. Every call runs through the same checks that stand before execution in production:

def durchs_gatter(aufruf):
    """Whitelist, schema, mandatory fields, enums. Returns a reason or None."""
    w = NACH_NAME.get(aufruf["name"])
    if not w:
        return f"unbekanntes Werkzeug: {aufruf['name']}"
    schema = w["function"]["parameters"]
    args = aufruf["args"]
    fremd = set(args) - set(schema["properties"])
    if fremd:
        return f"unbekannte Argumente: {sorted(fremd)}"
    fehlend = [f for f in schema.get("required", []) if f not in args]
    if fehlend:
        return f"Pflichtangabe fehlt: {fehlend}"
    for k, v in args.items():
        if k in ENUMS and v not in ENUMS[k]:
            return f"{k}={v} ist kein zulässiger Wert"
        if k.endswith("_id") and len(str(v)) != 36:
            return f"{k} sieht nicht wie eine uuid aus"
    return None

Ten examples got stuck there. Ten out of 371 - the large model works cleanly, and still those ten would have landed in the dataset without the gate and taught the specialist what the large model gets wrong. Distillation inherits mistakes when nobody looks.

The awkward cases

Now the part naive datasets leave out, because nothing happens in it.

For a request that needs no tool there is no call to learn. For a request with a missing mandatory value there is none either. Both are training examples all the same, only with a text as the target instead of a call:

Q: Read that contact for me.
T: I'd be happy to, but I need the contact's ID (the UUID) to read it.
   If you only know the name, tell me the name and I'll find the ID first.

Q: Create an event called Jour fixe.
T: I can create the event "Jour fixe", but I'm missing the required start
   and end times.

The targets are cut to two sentences. A specialist should decline, not write essays, and a 270-million-parameter model trained on paragraphs starts writing paragraphs.

And the admission condition is strict: only cases in which the large model too held still get in. If it calls, the case is discarded. Otherwise the dataset would carry a call under a request that permits none, and that is exactly the behaviour the training is supposed to drive out.

What ends up there

behalten: 337 von 371
nach Gattung: {'werkzeug': 260, 'doppelt': 36, 'kein_werkzeug': 30, 'pflicht_fehlt': 11}
verworfen: {'gatter': 10, 'pflicht_fehlt:hat_aufgerufen': 8,
            'pflicht_fehlt:ohne_antwort': 10, 'werkzeug:kein_aufruf': 3,
            'doppelt:nur_einer': 3}

Look at the fourth line. Of the 29 requests with a missing mandatory value, eleven survive. Eight dropped out because the large model called something itself - on "Delete the contact of Mr Oltmanns" it fires off a list_contacts to look up the id. Sensibly planned, and useless for this class. Ten more had no text at all, because the model spent its token budget on thinking.

So the class that matters most turns up with 3.3 % of the dataset. That is little, and it is not sloppiness but the finding: the awkward cases are not only the ones you forget. They are also the ones hardest to harvest, because in exactly those cases the teacher model is undecided itself.

Whether eleven examples are enough is decided by the test bench in the next chapter. The guess up front: probably not.

What the trained example looks like

No JSON, no chat message. What the model gets to see is exactly what the inference server will later put in front of it - rendered with the same template:

<bos><start_of_turn>developer
You are the tool layer of Simhaven. …<start_function_declaration>…<end_function_declaration><end_of_turn>
<start_of_turn>user
Read contact efca05dd-9fc4-43a7-a5fe-186bc1eda055.<end_of_turn>
<start_of_turn>model
<start_function_call>call:get_contact{contact_id:<escape>efca05dd-9fc4-43a7-a5fe-186bc1eda055<escape>}<end_function_call><end_of_turn>

Why that matters stands in the next chapter. Briefly: build the training prompt differently from the production one and you train a format that never arrives.