Back into the harness

A trained model on disk is not yet production. What is missing is the switch: who answers which request, and who decides that?

The chapter Working together describes three shapes. The course builds the third, because it brings the biggest gain and carries the sharpest condition.

The switch

The specialist stands right at the front. If it recognises a clean call, the request never reaches the large model. If it recognises none, it is passed through.

user ──► specialist ──► gate ──ok──► tool ──► answer
                          │
                       rejected
                          ↓
                     large model

And who decides whether a call is clean? Not the specialist. The gate. That is the one place where many builds turn off the road: they ask the small model for its confidence, or they draw a threshold over the probabilities. Both measure how sure a model feels, not whether the call is executable.

def weiche(frage, werkzeuge):
    aufrufe = spezialist(frage, werkzeuge)      # 0.16 s
    if not aufrufe:
        return grosses_modell(frage, werkzeuge)  # nothing recognised -> pass on
    for a in aufrufe:
        grund = durchs_gatter(a)                 # whitelist, schema, enums, rights
        if grund:
            protokoll.warnen(a, grund)
            return grosses_modell(frage, werkzeuge)
    return [ausfuehren(a) for a in aufrufe]

Six checks, the same ones as in the chapter Repairing, and all six are answerable mechanically. Does the whitelist know the tool? Are the mandatory fields there? Is the enum value one of the allowed ones? Does the caller have the right? Is it the same request as a moment ago, byte for byte? Is the budget still open?

A status: "kanada" falls through here without anyone having to think about confidence. And so does the invented calendar orto from chapter 05 - it is not in the whitelist of calendar keys. A call that falls through is not a failure. It is the trigger for the fallback path.

What the fallback path really costs

In shape 3 the slow path is the more expensive one, because it runs on top of the specialist. The calculation goes like this:

share q  = requests the specialist answers cleanly
duration = q · 0.16 s  +  (1 − q) · (0.16 s + 6.42 s)

At q = 0 you are 0.16 seconds slower than before. At q = 0.9 - the value the fine-tuned specialist reached on the test bench - you land at 0.80 seconds instead of 6.42. Eight times faster, with the same result, because everything doubtful ends up at the large model anyway.

That is the real number of the course. Not "the small model is just as good", but: how large is the share you can divert safely?

What this does not change

The chain exponent stays what it was. A task with five steps and a 90 % hit rate per step succeeds 59 % of the time, no matter which model translates the steps. The specialist raises the base and lowers the waiting time. It does not shorten the chain.

And the four rounds from the chapter Working together that demand judgement keep demanding judgement. Breaking tasks into steps, clearing up ambiguities, interpreting results, writing text for people - the large model stays responsible for those, and no dataset changes that.

When the effort pays off

The honest answer is an effort calculation, and it often comes out against the training:

The training is the cheapest item. The test bench is the most expensive, and it is at the same time the only one you would have needed without any training. That is exactly why it comes before the dataset in this course.

Three conditions should hold before you start, and the building block Fine-tuning spells them out: the action set stands still, there are logged real cases, and there is a test bench. If one of them wobbles, the system prompt is the cheaper adjustment - you can change it in five minutes, an adapter in half a day.

Production afterwards

Two more things belong to it, otherwise you only notice at the outage that something has shifted.

The service may fail. If the specialist is unreachable, every request goes to the large model. That is slower and more expensive and it works, and it is the reason it stands as its own container next to the application.

The gate's rejections belong in the log, with a reason. They are the dataset of the next round. A field that is often filled wrongly shows up there as a pattern long before anyone notices it in production.