The positive index

Now the concierge. Its job sounds like a case for a language model: "which of my agents can do this?" It is not. An LLM call at this point costs latency, costs tokens, is not deterministic and - this is the real objection - not calibrated. It says yes to everything.

The router in this course makes no LLM call at all. It embeds text and computes cosine similarity. That is the whole of it.

The goal

You route query against query, not query against description. This is the one trick the hit rate hangs on, and it costs no model training, only the decision to embed the right thing.

Why it matters so much: users say "i need meier's invoice again". The description says "agent for document retrieval from the DMS with full-text search". There is a lot of air between those two sentences, even though they belong together exactly. Between the query and the example "get me the invoice from again" there is almost none. This is called asymmetric retrieval, and the countermeasure is as simple as it is effective: put in the index what looks like the query.

Step 1: the client

One file, two functions, no vendor SDKs. Anything speaking /v1/chat/completions and /v1/embeddings works:

# tiny/llm.py
from __future__ import annotations

import os

import httpx

BASE_URL = os.environ.get("TINY_BASE_URL", "http://localhost:11434/v1")
API_KEY = os.environ.get("TINY_API_KEY", "not-needed")
CHAT_MODEL = os.environ.get("TINY_CHAT_MODEL", "qwen3:8b")
EMBED_MODEL = os.environ.get("TINY_EMBED_MODEL", "bge-m3")

_client = httpx.Client(
    base_url=BASE_URL,
    headers={"Authorization": f"Bearer {API_KEY}"},
    timeout=120,
)


def embed(texts: list[str]) -> list[list[float]]:
    """Embeddings for a list of texts, order preserved."""
    response = _client.post("/embeddings", json={"model": EMBED_MODEL, "input": texts})
    response.raise_for_status()
    data = response.json()["data"]
    return [entry["embedding"] for entry in sorted(data, key=lambda d: d["index"])]


def chat(messages: list[dict], tools: list[dict] | None = None, **kwargs) -> dict:
    """One chat completion call. Returns the first choice message plus usage."""
    payload = {"model": CHAT_MODEL, "messages": messages, **kwargs}
    if tools:
        payload["tools"] = tools
    response = _client.post("/chat/completions", json=payload)
    response.raise_for_status()
    data = response.json()
    return {"message": data["choices"][0]["message"], "usage": data.get("usage", {})}

The sorted(...) call is not decoration. Not every server returns embeddings in input order, and once they have slipped, every example belongs to the wrong agent. The failure is silent: the router works, just badly.

Step 2: the index

Here is the decision that separates this course from the reference architecture: no vector database. Three cards with ten examples each are thirty vectors. Thirty. A NumPy matrix and a dot product handle that in microseconds.

# tiny/index.py
from __future__ import annotations

from dataclasses import dataclass

import numpy as np

from tiny.llm import embed


@dataclass
class Hit:
    agent_id: str
    text: str
    score: float


class Index:
    """A positive or veto index: example texts plus where they came from."""

    def __init__(self) -> None:
        self._agents: list[str] = []
        self._texts: list[str] = []
        self._matrix: np.ndarray | None = None

    def add(self, agent_id: str, texts: list[str]) -> None:
        self._agents.extend([agent_id] * len(texts))
        self._texts.extend(texts)
        self._matrix = None  # stale, recomputed on the next build()

    def build(self) -> None:
        vectors = np.array(embed(self._texts), dtype=np.float32)
        # Normalise, so the dot product is the cosine similarity directly.
        lengths = np.linalg.norm(vectors, axis=1, keepdims=True)
        self._matrix = vectors / np.maximum(lengths, 1e-12)

    def search(self, query: str, limit: int = 10) -> list[Hit]:
        if self._matrix is None:
            self.build()
        q = np.array(embed([query])[0], dtype=np.float32)
        q /= max(float(np.linalg.norm(q)), 1e-12)
        scores = self._matrix @ q
        best = np.argsort(-scores)[:limit]
        return [Hit(self._agents[i], self._texts[i], float(scores[i])) for i in best]

That is the entire vector store. Eleven lines of arithmetic.

Step 3: the router

# tiny/concierge.py
from __future__ import annotations

from dataclasses import dataclass

from tiny.cards import Card, load_registry
from tiny.index import Index


@dataclass
class Decision:
    kind: str                      # "match" | "abstain" | "ambiguous", complete in chapter 04
    agent_id: str | None
    score: float
    alternatives: list[tuple[str, float]]


class Concierge:
    def __init__(self, cards: dict[str, Card]) -> None:
        self.cards = cards
        self.positive = Index()
        for card in cards.values():
            self.positive.add(card.id, card.examples)
        self.positive.build()

    def route(self, goal: str) -> Decision:
        hits = self.positive.search(goal, limit=20)

        # Best hit per agent: an agent with many similar examples should not win
        # because it has many, but because one of them fits.
        best: dict[str, float] = {}
        for h in hits:
            best[h.agent_id] = max(best.get(h.agent_id, 0.0), h.score)

        ranking = sorted(best.items(), key=lambda p: -p[1])
        if not ranking:
            return Decision("abstain", None, 0.0, [])
        agent_id, score = ranking[0]
        return Decision("match", agent_id, score, ranking[1:3])


def load_concierge(folder: str = "registry") -> Concierge:
    return Concierge(load_registry(folder))

The line with max(...) is the only place with a real decision in it. An agent with fifteen examples would otherwise have more hits in the top 20 than one with eight - and would win on sheer quantity. What counts is the best hit, not the number of them.

Step 4: measure instead of believe

A router you tried on three queries is a router you know nothing about. Write yourself a test set: twelve queries that do not repeat any card example verbatim, each with the answer you expect.

# scripts/03_router_eval.py
from tiny.concierge import load_concierge

TESTSET = [
    ("pull me the price list off their website", "web-reader"),
    ("does the article say anything about delivery times", "web-reader"),
    ("what do they write about themselves on their home page", "web-reader"),
    ("is there a phone number on the page", "web-reader"),
    ("please create the new customer for me", "crm-writer"),
    ("note on the company that we called them", "crm-writer"),
    ("enter the new address for the customer", "crm-writer"),
    ("set the status of that deal to won", "crm-writer"),
    ("how does the travel expense process work again", "docs-finder"),
    ("where do i find the vpn guide", "docs-finder"),
    ("what do we have internally on parental leave", "docs-finder"),
    ("is there a document about onboarding", "docs-finder"),
]


def main() -> None:
    concierge = load_concierge()
    correct = 0
    for query, expected in TESTSET:
        d = concierge.route(query)
        ok = d.agent_id == expected
        correct += ok
        mark = "  " if ok else "XX"
        print(f"{mark} {d.score:.3f}  {d.agent_id or '-':<14} (expected {expected:<14}) {query}")
    print(f"\nRecall@1: {correct}/{len(TESTSET)} = {correct / len(TESTSET):.2f}")


if __name__ == "__main__":
    main()

The milestone

python scripts/03_router_eval.py

   0.734  web-reader     (expected web-reader    ) pull me the price list off their website
   0.681  web-reader     (expected web-reader    ) does the article say anything about delivery times
   ...
XX 0.612  web-reader     (expected docs-finder   ) is there a document about onboarding

Recall@1: 11/12 = 0.92

Above 0.90 is a good state for this step. If you are clearly below it, the cause is almost always one of three.

The embedding model is English-leaning. Check it in ten seconds: embed "leg den kunden an" and "create the customer" and compute the similarity. If it is below 0.7, your model handles the bridge between languages badly - and then it handles colloquial phrasing badly too. Use bge-m3.

The examples read like documentation. If your card says "determines price information from web pages" instead of "what does it cost with them according to the website", you made exactly the mistake this chapter is built against.

Two cards genuinely overlap. If docs-finder and web-reader both contain "read that to me", that is not a router fault but a scoping fault. The router cannot separate what you did not separate.

When a vector database pays off

The arithmetic is simple: your matrix holds N × D floats. For 500 capabilities with 15 examples each and 1024 dimensions that is 7,500 × 1,024 × 4 bytes, so around 30 MB in memory, and one search costs a dot product over 7,500 vectors - under a millisecond. Up to that point Qdrant is a dependency without a return.

When it does pay off depends not on the number but on three properties your NumPy index lacks: persistence (your index is re-embedded on every start - with 7,500 examples that is seconds and API cost), concurrency (several processes, one index) and pre-filtering (only the agents of one tenant). As soon as you need one of those, take a database. Before that, the one-liner is more honest.