Tiny Agents and the Compression Boundary
Two hundred tools do not fit into any context - and the tool list is the cheaper of the two problems. Why dynamic filtering falls short, and what a compression boundary does instead.
Basics
Free for everyone: the concept, the analogy, the why.
Tool calling in production covers what breaks about a single tool call: six layers of translation, a catalogue of failures, forced syntax, repair. This page starts one floor above that.
Because once an agent no longer operates five tools but two hundred, a question appears that the single call never had to answer: which of them do you show the model at all? And right behind it the second, far more expensive one: what actually happens to everything those tools return?
Two problems that keep getting confused
The industry trend is towards ever larger agents: long system prompts, dozens of skills, hundreds of tools. What goes wrong there are two independent sources of failure - and they are orders of magnitude apart:
| Problem | Cause | Order of magnitude |
|---|---|---|
| Confusability | too many similar tool definitions in the schema | ~200 tokens per definition |
| Ballast | the tools' outputs flood the context | 100 to 1000 times that |
The first row produces ghost calls, hallucinated parameter names, the wrong pick between two tools that sound almost the same. The second row produces 40,000 tokens of scraped HTML for a single number - and a context that is full after three calls.
That is the one idea everything on this page hangs on: these are two problems, not one.
Filtering solves the cheaper problem
The obvious answer to too many tools is to show fewer of them: a retrieval step picks the five that fit the current turn, the rest stays out. That works - for the top row of the table. Two hundred definitions become five, 40,000 tokens of tool description become 1,000.
It changes nothing about the bottom row. The five selected tools then run inside the very context that also holds the conversation, and they dump their raw output right there. Filtering alone solves the problem that costs 200 tokens per tool and leaves untouched the one that costs a hundred times that per call.
No filter helps with the bottom row. Only a boundary does.
A boundary instead of a filter
The shape behind it has two levels and exactly one seam between them:
Host agent
Big model, long conversation - and permanently exactly three tools in its context, no matter how many capabilities are registered.
- find_capabilityIs there anyone at all for this goal?
- delegateHand off a sub-task - with a goal and the fields that have to come back.
- expandAsk the raw material about what the summary left out.
Only compressed material passes upwards: a summary, the requested fields, a handle on the original.
- web-readerReads a page and answers one question about it.
- crm-lookupLooks up a contact in the CRM.
- calendar-readerReads appointments and finds free slots.
Every tiny agent: its own fresh context, 1-5 tools, a fixed output schema, a hard token budget. Five of them or five hundred - nothing changes up top.
A tiny agent is specialised on exactly one capability: its own fresh context, one to five tools, a fixed output schema, a hard token budget. Its job is explicitly not "call the tool" - its job is to compress. The 40,000 tokens still come into existence, but they do so below the boundary, and that is where they stay.
The difference to a filter is where the saving happens. A filter decides what gets written into the context. The compression boundary decides what is allowed to reach it at all. That is why the main context stays constantly small - whether five capabilities are registered or five hundred. It is also the target metric of this architecture, and an unusual one: not accuracy, but main-context token footprint per solved task.
Three tools, and that is the whole list
What the host agent sees does not grow along. It is always the same three:
find_capabilityanswers "is there anyone for this?" before the agent promises it to the user. And it does so without a single model call - the answer comes from a retrieval step over the registered capabilities.delegatehands off a bounded sub-task. What gets passed down is not the task verbatim but what is being looked for: a goal in one sentence and the fields that have to come back. That explicitness is half the trick - whatever the host cannot name, it cannot delegate either.expandasks the raw material. Because the specialist compresses without knowing the overall task; it may well discard exactly the paragraph the host would have needed.
The third tool is what makes the compression bearable in the first place. Without it, condensing is a one-way street: whatever was left out is gone. Because the raw output stays down there instead of being deleted, it remains retrievable - the host gets the short answer and keeps access to the original. That is the difference between a summary and a loss.
A fourth case belongs here too, even though it is not a tool: when nobody is responsible for a goal, the system has to be allowed to say so. A retrieval step always returns a best match, even for nonsense; without a threshold, "make me a sandwich" ends up at the next best specialist. An honest "nobody here for that" is not a failure but a feature - and, as a side effect, the most honest roadmap you can get: it keeps count of which capability was missing most often.
Leaves, not trees
One thing about this architecture is deliberately unspectacular: there are exactly two levels. Tiny agents may not delegate themselves - they simply do not have the tool.
That sounds like a restriction and is a relief. A strict two-level hierarchy makes recursion structurally impossible and cost estimation trivial: at most one hop, always. Multi-level delegation sounds more powerful but immediately drags in cycle detection, inherited budgets and errors that travel across three levels. Anyone who has ever debugged an agent calling an agent calling an agent gives that up gladly.
This tier is currently being written. Here is the outline it will follow:
- The capability card - the manifest per capability that is registration, documentation, routing index and test fixture all at once
- The router - example queries instead of descriptions, a separate veto index, and the threshold below which nothing gets routed
- The delegation protocol - brief down, digest up, and why the budget has to be hard instead of polite
- The artifact store - how a raw output becomes a handle, and what
expandfinds inside it - The runner - one process, many configurations: isolation comes from the fresh context, not from separate containers
Two problems
Two problems, two orders of magnitude
Tool definitions cost ~200 tokens, tool outputs a hundred times that. Anyone filtering tools dynamically has solved the cheaper problem and left the expensive one untouched.
There is one sentence that comes up in almost every discussion about oversized agents: "we'll just load the tools dynamically." The sentence is correct and it solves the wrong problem - more precisely: it solves one of two, and it is the smaller one.
This chapter separates the two cleanly, because every following chapter rests on that separation. Once you have made it, you can spot instantly which architectural proposals address only the upper half.
The separation
| Confusability | Ballast | |
|---|---|---|
| What there is too much of | definitions in the schema | outputs in the transcript |
| What it depends on | number of registered tools | number and kind of calls |
| Cost per unit | ~200 tokens per definition | 10,000 to 40,000 tokens per call |
| Symptom | ghost calls, invented parameters, wrong pick between two similar names | context full after three calls, model loses the start of the dialogue |
| Grows with | the catalogue | the work |
The last row is the uncomfortable one. Confusability grows with what an agent could do - a number you hold in your hand. Ballast grows with what it does. An agent with five tools that reads three web pages has a full context window and not a single selection problem.
Why filtering only hits the upper row
The usual intervention is called tool filtering or tool injection: a retrieval step picks the five tools that match the request, the other 195 stay out. 40,000 tokens of tool description turn into 1,000. That is a real improvement, and it is cheap to get.
Except that the five selected tools then run in the same context the dialogue lives in. Their raw outputs land right there. The filter decided which tools you see, not what comes out of them.
The panel below shows both side by side. The slider moves the number of registered capabilities, the switch makes the task output-heavy - three calls that return actual text instead of a single number.
Three designs, the same task
The slider moves the number of registered capabilities, the switch makes the task output-heavy. Only the main context is shown - the place where tokens crowd out the dialogue.
- AMonolith3,700 tokens
All definitions in the schema, all raw outputs in the same transcript.
- BTool filtering3,700 tokens
Retrieval puts five definitions in. It changes nothing about the outputs.
- CTiny agents4,500 tokens
Three tools, constant - and digests come up instead of raw outputs.
5 capabilities, small outputs: the monolith sits at 3,700 tokens, tiny agents at 4,500. Variant C is the **more expensive** option here - the effort only starts paying off once one of the two quantities grows.
Assumptions behind this calculation: 1,500 tokens of base load (system prompt and dialogue), 200 tokens per tool definition, 3 tool calls per task, 400 tokens of raw output per call for a light task and 12,000 for an output-heavy one, 800 tokens per digest, and retrieval puts 5 definitions into the context. Calculated, not measured. And across all contexts combined, variant C spends more than A - that downside is deliberately not drawn here, it is in chapter 07.
Guided in four steps
Each step sets the slider and switch for you.
What is usually misunderstood here
- Dynamic tool filtering solves the context problem.
- →It solves the definitions half. The outputs of the selected tools still land in the same context as the dialogue - step 3 shows this with only five tools.
- Tiny agents save tokens.
- →Overall they spend more. They move the spend to where it does not hurt: out of the main context and into short-lived contexts that expire with the answer.
- A larger context window handles both.
- →It moves the boundary, it does not remove it - and it makes the output row more expensive, not cheaper: every turn re-reads the grown transcript.
The bars are calculated, not measured; the formula is in the source of this panel and the assumptions are underneath it. The point does not hang on the exact numbers but on the order-of-magnitude gap between a definition and a raw output.
The interesting state is step 3: five capabilities, output-heavy task. The catalogue is tiny, the selection problem does not exist - and the context is full anyway. A filter has nothing to filter there.
What helps instead
For the lower row no filter helps, only a boundary: a place in the architecture where things are compressed rather than passed through. Below that boundary the 40,000 tokens are allowed to exist. Above it, only what has passed through an output schema and a hard budget arrives.
That is exactly what a tiny agent is: not a "small model" but its own fresh context with one to five tools, a fixed output schema and a hard token budget. Its job is explicitly not "call the tool" but compress.
From that follows the target metric of this architecture, and it is an unusual one:
Tokens in the main context per successfully solved task.
Not accuracy. Accuracy alone tempts you to optimise against a test set; an agent with 95 % success that burns 200,000 tokens doing it is unusable on your own hardware. Coupling quality and cost in a single number is precisely the quantity that decides feasibility in practice.
What this means for the remaining chapters
Every following chapter can be checked against this separation:
- The capability card (chapter 02) and the concierge (chapter 03) work on the upper row - they decide which capability is even a candidate, and they do it without spending a single model call.
- Brief and digest (chapter 05) and the artifact store (chapter 06) work on the lower one. They are the reason this architecture exists at all.
- Abstain (chapter 04) belongs to neither row and still cannot be left out: a router without a threshold routes even the things nobody can serve.
- Chapter 07 adds up what it costs - because it is not free.
If you take away one thing while reading: there are two problems, not one, and they lie two orders of magnitude apart.
Continue
The solutions start here - 6 chapters for Pro members.
- The capability card - One manifest per capability that is registration, documentation, routing index and test fixture at once. And the reason it holds example requests instead of a description.
- The concierge - Routing by retrieval, without a single model call. Why the counter examples must live in a second index, and why three thresholds are the only real knobs in the system.
- Abstain is a feature - A top-k router always hits something, even for nonsense - the same calibration error as a model that never says no to the question “can you do this?”. And why the abstain log is the most honest product roadmap you can get.
- Brief and digest - What goes down is not the task but what is being looked for. What comes up is a schema-validated digest with five possible states - and a budget enforced in the runner instead of politely requested in the prompt.
- The artifact store - Throw nothing away, just do not send it along. A handle plus expand() make compression retrievable - and that is the difference between a summary and a loss.
- What it costs - One extra hop of latency, roughly one and a half to two times the total tokens - and an expectation that can be falsified. If variant C does not show the advantage, the concept is wrong.