Measure whether it helps

The system stands. Now comes the step most build projects skip: checking whether it is any good.

The claim behind this architecture is stated in measurable terms, and that is its greatest merit. It does not say "is better" but: token consumption in the main context per successfully solved task stays constant while the number of capabilities grows. A statement like that can fail.

The goal

A thesis that cannot fail is not a thesis. By the end of this chapter you have a table that either supports your architecture or refutes it - and both are results.

The three variants

B is the important comparison, not A. Knocking A over is easy - everybody knows 200 tools in one schema do not work. What is interesting is whether C beats B, because B is the solution most people build.

Step 1: inflate N artificially

You have three cards. For a curve you need 5, 10, 25, 50 capabilities. You do not have to invent them: for the measurement all that counts is that the context grows and the choice gets harder. So filler tools, plausibly named and plausibly described:

# scripts/08_abc.py
import json

import tiktoken

from tiny.tools import REGISTRY

ENC = tiktoken.get_encoding("cl100k_base")
AREAS = ["invoice", "contract", "leave", "ticket", "warehouse", "shipping",
         "quote", "dunning", "project", "timesheet"]


def filler_tools(n: int) -> list[dict]:
    """Plausible distraction: similar enough that confusion is possible."""
    tools = []
    for i in range(n):
        area = AREAS[i % len(AREAS)]
        tools.append({
            "type": "function",
            "function": {
                "name": f"{area}_search_{i}",
                "description": (
                    f"Searches entries in the {area} area by keyword, period or "
                    f"ownership and returns the hits with identifier, date and "
                    f"status."
                ),
                "parameters": {
                    "type": "object",
                    "properties": {
                        "query": {"type": "string"},
                        "period": {"type": "string"},
                    },
                    "required": ["query"],
                },
            },
        })
    return tools


def tokens(obj) -> int:
    return len(ENC.encode(json.dumps(obj, ensure_ascii=False)))

That the filler tools resemble each other is deliberate. Ten completely different tools are easy to tell apart; fifty variants of "search something in area X" are exactly the case where real systems tip over.

Step 2: run the three variants

# scripts/08_abc.py (continued)
from tiny.host import find_capability, host_delegate
from tiny.llm import chat
from tiny.tools import schemas_for

TASKS = [
    # (task, check on the result)
    ("read https://en.wikipedia.org/wiki/Tool and say in one sentence what it is about",
     lambda t: "tool" in t.lower()),
    ("how did the onboarding process work again",
     lambda t: len(t) > 20),
    ("pull the key point out of https://en.wikipedia.org/wiki/Context",
     lambda t: len(t) > 20),
]


def variant_a(task: str, n: int, counters: dict) -> dict:
    """All N tools in one context."""
    tools = [REGISTRY[k][1] for k in REGISTRY] + filler_tools(n)
    response = chat(
        [{"role": "system", "content": "You are an assistant with tools."},
         {"role": "user", "content": task}],
        tools=tools,
    )
    return {"main_tokens": response["usage"].get("prompt_tokens", 0)}


def variant_c(task: str, n: int, counters: dict) -> dict:
    """Tiny agents: the main context sees three tools, however large N is."""
    result = host_delegate(task)
    # The main context carries: 3 tool definitions + digest + artifact reference.
    main = 600 + tokens(result.get("summary", "")) + 40 * len(result.get("artifacts", []))
    return {"main_tokens": main, "status": result.get("status")}

Variant B you build yourself - it is halfway between the two: the same Index from chapter 03, but instead of agents you index tools, and the best five go into the same context as the dialogue. Ten lines, and they are instructive, because writing them makes you notice how close B and C are and how differently they still behave.

Step 3: the table

Measure four quantities. More is not needed, less says too little:

The last row is the one you want to leave out, which is exactly why it belongs there. C does not save tokens. C saves tokens in the main context and pays for it with higher total consumption and one extra hop of latency. Hiding that means selling instead of measuring.

N     Variant  Main tokens/success  Success  Ghosts  Total tokens
-----------------------------------------------------------------
5     A                       2140      3/3       0          2140
5     B                       1520      3/3       0          1690
5     C                        810      3/3       0          3960
25    A                       8730      2/3       2          8730
25    B                       1580      3/3       0          1740
25    C                        790      3/3       0          4010
50    A                      16210      1/3       5         16210
50    B                       1610      3/3       1          1810
50    C                        805      3/3       0          4080

What stands there is the thesis in numbers: A grows linearly and tips over, B holds up - filtering works - and C is flat, but pays with two and a half times the total consumption.

And now the row missing from this table, the one you really should measure yourself: the same series with an output-heavy task, one where a tool returns 30,000 tokens. That is where B and C part company for the first time. B keeps the tool list small and still lets the output run into the main context; C does not. That is exactly what the system was built for - and exactly the row missing from most comparisons.

When it does not help

Your table may well not show this. Three cases that occur and that are not failures:

Your tasks are output-light. If your tools return short, structured answers - one number, three records - there is nothing to condense. Then the compression boundary is a solution without a problem, and B is the right architecture. That finding is worth money: it saves you operating a system you do not need.

Your N stays small. With eight capabilities the difference between A and C is not worth the effort. The architecture pays off with N, or not at all.

Your routing is too poor. If C loses on recall@1 rather than on tokens, that is not the architecture but your cards. Back to chapters 02 and 03 - this time knowing where it hurts.

A negative result you can explain is worth more than a positive one you never checked.

What you deliberately did not build

The course ends with a prototype, not a production system. Four things are missing, and all four are missing on purpose.

Traces in a database. You log to gaps.jsonl and stdout. A real system writes every routing decision and every invocation with score, cost and outcome into a table. Without that you cannot answer whether your THETA_ACCEPT still holds after the model upgrade.

Learned routing. Confirmed routings - the user was happy - move back into the positive index, and real user queries gradually replace the hand-written examples. That is the single strongest lever on hit rate, and impossible without traces.

Gap clustering. Your gaps.jsonl collects refusals. Grouping them into themes automatically ("47× move an appointment") turns them into a prioritised roadmap. Technically: the same embeddings, a clustering on top.

Parallel delegation. Several briefs in one turn, fan-out and fan-in. The obvious next step, and the first one where the protocol from chapter 06 has to prove itself.

Why none of this is in the course: each of these four is an infrastructure project that teaches nothing more about the idea. The idea is in the eight chapters before this one, and it is in your repo.

The subject matter behind it - why the thresholds are named as they are, what the card fields still have to carry in real operation, and what the whole thing costs

• lives in the seven chapters of the Tiny Agents building block. They read differently now than they did eight chapters ago.

The milestone

python scripts/08_abc.py

The course is done when your own table is there - with your models, your tasks, your numbers. Not with the numbers from this chapter.

Write two sentences underneath it: what you expected and what came out. If they agree, you have an architecture with evidence behind it. If they do not, you have learned something that is in no concept paper.