BenchmarksImplementation 01 / 01

AgentLens - our own benchmark

The benchmark the basics tier demands, actually built: real agent tasks in two real harnesses (Pi and Hermes), repeated per model, with a clickable execution trace. Not whether a model fails, but where, when and why.

Implementations

One data sheet per system that implements this part - always the same grid.

Model report 02

Qwen3.6-35B-A3B

70 pages · 24 chapters · 10,250 words · PDF

Eight failure patterns drawn from 726 runs, each with the move from the transcript where the run tips over: the mistyped write path the tool silently creates; the list that loses an element while being copied out; the rm -rf that stands in for merging; the file deleted because it could not be read. Ahead of them the statistics across eighteen tasks, two harnesses and the same model with and without its thinking block; after them every task in turn - and for each failure pattern a countermeasure with an estimated effect. 22 charts, the measurement setup as a reference section at the back.

The volume is written in German - the station itself is available in both languages.

Sign inThe full report with every run, trace excerpt and the hardware calculation is open to Pro members.

Model report 03

Qwen3.8-27B

68 pages · 26 chapters · 9,539 words · PDF

The highest score this test bench has measured so far: 343 of 364 runs across twenty tasks and three harnesses - and 1,344 tool calls under Hermes without a single error. The eight remaining failure patterns are errors of judgement, each with the move from the transcript where the run tips over: the 50,000-character thinking block instead of the one-second check script; the self-test that verifies with its own blind spot; the mv that overwrites a case number while merging; the training idiom that overrules the specification. Plus the comparison of vLLM against local Q8 weights, a countermeasure with estimated effect for every pattern, 22 charts, and the measurement setup as a reference section at the back.

The volume is written in German - the station itself is available in both languages.

Sign inThe full report with every run, trace excerpt and the hardware calculation is open to Pro members.

Model report 04

Qwen vs. Qwen

67 pages · 21 chapters · 11,846 words · PDF

Second edition with the right opponent: Qwen3.6-27B against Qwen3.8-27B - both dense, both 27 billion parameters, both on the same endpoint, 744 scored runs across all 20 tasks. The first version accidentally compared the mixture-of-experts Qwen3.6-35B-A3B and made the generational jump look twice as large at twelve points; measured it is 5.8 (88.4 against 94.2 percent). The volume reads 19 runs against each other in the protocols: four chapters where the new generation wins (tidy-up, work rules, course corrector, building an application), three where the old one wins (error reader, second half, CRM curation) - because the progress sits in the middle: T2 and T3 gain, T4 goes to the old generation, T5 ends 88 to 88. Plus the token bill (+32 percent median, +80 mean, at practically the same number of moves), all 65 failed runs on both sides individually, and 26 charts. The experimental setup comes first - including the chapter on how easily one puts the wrong opponent in the ring.

The volume is written in German - the station itself is available in both languages.

Sign inThe full comparison volume with every trace face-off and the token accounting is open to Pro members.

The basics tier ends on an uncomfortable sentence: your own benchmark beats any leaderboard. We take it seriously - AgentLens is that benchmark, built on this very platform and fed with the very tasks people here give their agents.

Why another benchmark?

Public benchmarks deliver a number. That number doesn't answer what actually matters in practice: did the model never even attempt the tool call and merely describe it (a ghost call)? Did the parser reject a syntactically broken invocation? Did the chain tip over at step 4 - and what happened at step 3? Anyone picking a mid-sized model for a productive, locally hosted agent stack needs this diagnosis, not a grade.

AgentLens therefore stores the complete execution log of every run and evaluates at step level. The price: more storage and a normalisation layer. The gain: diagnosis instead of a leaderboard.

The method in four sentences

Real tasks. No needle-in-haystack riddles but work orders: create and restructure files, analyse a server log, clean up a customer list, place a meeting diplomatically against three calendars, tidy a CRM without losing a single note - two of these are literally this platform's parcours tasks. Plus memory tasks across several conversation turns, with distractions in between.
Real harnesses. Every task runs in the complete system of model plus agent harness - once under PiExternal - Opens in a new tab, once under the Hermes agent. Failures arise across the whole chain (chat template, decoding, parser, retry logic), so we measure the whole chain. "Qwen under Pi" and "Qwen under Hermes" are two different results - and that difference is one of the most interesting readings.
Repeated, not once. Every cell (task × model × harness) runs several times. We separately show what a model achieves sometimes and what it achieves always - for production decisions, the gap between the two is the real answer.
Deterministic verdicts. Passing means the checking code can measure it in the end state: file present, calendar correct, no note lost. No LLM grades an LLM.

When a run fails, the trace analyzer assigns it a failure class - ghost call, parser rejection, endless loop, error spiral after a tool failure, forgotten instructions as context grows, premature "done!". Outages of our own infrastructure never count against the model.

What you can see here

As a member (free) you see selected models in the results overview and can click through individual session traces step by step - every tool call, every result, every failed attempt.

As a Pro member you see everything: the full comparison matrix across all models and both harnesses, the failure-class fingerprints, harness deltas, endurance curves and every single trace - plus recommendations on which building blocks from Block II (parser, retry, context compaction) specifically compensate the weaknesses found in a combination.

We plug in new models as soon as they exist - endpoints are pure configuration. The moment Qwen 3.8 ships, it runs here against the same tasks as everything before it.

→ To the results overview