Veto and abstain

The router from chapter 03 has a fault you cannot see as long as you feed it sensible queries: it always answers. Retrieval returns top-k, and top-k is populated even when nothing fits. Ask it for a sandwich and it sends you to docs-finder, because of the three that one is the least unsuitable.

This is the same calibration failure as a language model that says yes to "can you do this?" on principle. Only at a place where you can fix it - here there is a number, not a model.

The goal

A router without a threshold routes nonsense too. Refusal is not a shortcoming but the feature that makes the router usable in the first place: only when it can say no does its yes mean anything.

Step 1: the veto index

The counter_examples in your cards have been sitting unused so far. They go into an index of their own - the reason is in chapter 02, and it is important enough to repeat: "I do not do web search" contains the words web search. In the positive index, the very sentence meant to exclude a request would sit right next to that request.

Indexed separately, the effect inverts. A hit in the veto index does not mean "fits well" but "not that one".

# tiny/concierge.py (extended; also needed: import json, from pathlib import Path)
THETA_VETO = 0.72     # from here a counter-example counts as exclusion
THETA_ACCEPT = 0.62   # below this: abstain
DELTA_AMBIG = 0.04    # gap below which two agents count as equivalent


class Concierge:
    def __init__(self, cards: dict[str, Card], gap_log: Path | None = None) -> None:
        self.cards = cards
        self.gap_log = gap_log or Path("gaps.jsonl")
        self.positive = Index()
        self.veto = Index()
        for card in cards.values():
            self.positive.add(card.id, card.examples)
            self.veto.add(card.id, card.counter_examples)
        self.positive.build()
        self.veto.build()

    def route(self, goal: str) -> Decision:
        hits = self.positive.search(goal, limit=20)
        veto_hits = self.veto.search(goal, limit=10)

        # Binary, not a score penalty: whoever is above the threshold here is
        # out - no matter how good their best positive hit was.
        blocked = {h.agent_id for h in veto_hits if h.score >= THETA_VETO}

        best: dict[str, float] = {}
        for h in hits:
            if h.agent_id in blocked:
                continue
            best[h.agent_id] = max(best.get(h.agent_id, 0.0), h.score)

        ranking = sorted(best.items(), key=lambda p: -p[1])

        if not ranking or ranking[0][1] < THETA_ACCEPT:
            self._remember_gap(goal, ranking)
            return Decision("abstain", None, ranking[0][1] if ranking else 0.0, [])

        if len(ranking) > 1 and ranking[0][1] - ranking[1][1] < DELTA_AMBIG:
            return Decision("ambiguous", ranking[0][0], ranking[0][1], ranking[1:3])

        return Decision("match", ranking[0][0], ranking[0][1], ranking[1:3])

    def _remember_gap(self, goal: str, ranking: list[tuple[str, float]]) -> None:
        entry = {
            "goal": goal,
            "best_candidate": ranking[0][0] if ranking else None,
            "best_score": round(ranking[0][1], 4) if ranking else 0.0,
        }
        with self.gap_log.open("a", encoding="utf-8") as file:
            file.write(json.dumps(entry, ensure_ascii=False) + "\n")

The blocked filter sits before the scoring, not after it. A veto you can compute away later is not a veto.

Step 2: three answers instead of one

The router now returns three different things, and the difference matters to the calling agent:

ambiguous is the answer you most want to optimise away. Do not. It says something true about your registry: two cards are scoped too similarly, and no amount of threshold fiddling changes that. DELTA_AMBIG is therefore less a router parameter than a measuring device for your scoping.

Step 3: calibrate the thresholds

The three numbers above are starting values, not truths. They depend on your embedding model: some models squeeze similarities into a narrow band between 0.6 and 0.9, others spread from 0.1 to 0.95. A THETA_ACCEPT of 0.62 can let everything through on one model and reject everything on another.

So sweep it. Extend your test set from chapter 03 by six queries that should match nothing:

UNCOVERED = [
    "make me a sandwich",
    "what's the weather like in hamburg tomorrow",
    "translate that into italian",
    "call the customer and ask",
    "how much is 17 percent of 4200",
    "delete the whole project",
]

# scripts/04_threshold_sweep.py
import numpy as np

from tiny import concierge as c
from scripts.router_eval import TESTSET, UNCOVERED   # test sets factored out


def evaluate(concierge, theta_accept: float) -> tuple[float, float]:
    c.THETA_ACCEPT = theta_accept
    hits = sum(concierge.route(q).agent_id == e for q, e in TESTSET)
    refusals = sum(concierge.route(q).kind == "abstain" for q in UNCOVERED)
    return hits / len(TESTSET), refusals / len(UNCOVERED)


def main() -> None:
    concierge = c.load_concierge()
    print(f"{'theta':>7}{'Recall@1':>11}{'Abstain':>10}{'F1':>8}")
    for theta in np.arange(0.40, 0.86, 0.02):
        recall, abstain = evaluate(concierge, float(theta))
        f1 = 0.0 if recall + abstain == 0 else 2 * recall * abstain / (recall + abstain)
        print(f"{theta:>7.2f}{recall:>11.2f}{abstain:>10.2f}{f1:>8.2f}")


if __name__ == "__main__":
    main()

What you see is a curve with a hump: at a low threshold you route everything correctly and all the nonsense with it. At a high one you refuse every piece of nonsense and half the real queries along with it. The best F1 marks the spot in between. Put THETA_ACCEPT there and write the number, the date and the model name into your README - it is only valid for this embedding model.

Step 4: the refusals are the build list

The gaps.jsonl writer above looks like bookkeeping. It is the most underestimated part of the whole architecture.

Every line in it is a request a real user made and for which your system had nobody. After two weeks of operation the file looks like this:

 47x  "move the appointment with ..."   -> candidate: calendar-writer
 31x  "what was revenue in ..."         -> candidate: reporting
 12x  "translate that into italian"     -> candidate: translator

That is not an error list, that is a product roadmap prioritised by actual demand. A system that routes every request somewhere would never have produced this information - the 47 appointment requests would have vanished as bad answers from docs-finder.

Grouping those lines into themes is a small project of its own and does not belong in this course (see the outlook in chapter 08). You still have to write the file now - later on you no longer have the data.

The milestone

python -c "
from tiny.concierge import load_concierge
c = load_concierge()
for query in ['make me a sandwich',
              'pull me the price list off their website',
              'google who the managing director is there']:
    d = c.route(query)
    print(f'{d.kind:<10} {d.agent_id or \"-\":<14} {d.score:.3f}  {query}')
"

abstain    -              0.412  make me a sandwich
match      web-reader     0.751  pull me the price list off their website
abstain    -              0.583  google who the managing director is there

Two things have to be true. The sandwich is refused, not routed. And the third line is the more interesting one: "google" is a counter-example of the web-reader. Without a veto index it would have won here - it does something with web pages, after all. With one it is out, and because nobody replaces it, the request honestly ends up in gaps.jsonl.

Then look at the file:

cat gaps.jsonl

Two lines. Your first two candidates for agents number four and five.