Agent benchmark
AgentLens is this platform's benchmark for agent readiness. Models get real multi-step tasks - creating files, tidying a calendar, maintaining a CRM - and have to solve them with tools, not with a nicely worded answer. Every run is recorded in full.
It measures what single scores hide: whether a model solves a task every time or only sometimes (pass^k versus pass@k), at which step a chain tips over, and which of ten failure classes is behind it - from ghost calls to error spirals. Infrastructure failures never count against the model.
And because the same task runs under two harnesses, it becomes visible how much of the performance is the model and how much is the scaffolding around it. Every result stays verifiable: behind each cell of the matrix sits the run itself, step by step.
Model × Harness × Task
A meaningful agent benchmark measures every combination of model, harness and task - and analyses the failures instead of counting tokens per second. Every number here has a full trace behind it.
The inventory of the test series: these tasks, harnesses and models make up the benchmark. Every task links to its detail page with prompt, goal and reference solution.
20 tasks
- T1Create a status fileWorkspace taskThe smallest real task: create one file with exact content. Failing here means failing the tool call itself.
- T2Extract errors from a server logWorkspace taskRead a log, count ERROR lines, report the request ids — read, filter, write as one chain.
- T3Split a configuration fileWorkspace taskA refactor with a side condition: extract one part without damaging the rest — the classic chain-breaker.
- T4Clean up a customer listWorkspace taskResolve duplicates by rule, drop the unusable, lose nothing valid — data care as in real operations.
- T4Calendar diplomacy (Simhaven calendar)Sim world: calendarParcours task P5 as a benchmark: read three calendars, find a valid slot, create the event with invites and room — real multi-step tool calling against an MCP server.
- T5The CRM caretaker (Simhaven CRM)Sim world: CRMParcours task P8 as a benchmark: fifty contacts, merge duplicates without losing notes, import leads, raise statuses — the hardest chain in the suite.
- T3Retain project facts across distractionsDialogue taskTurn 1 states the facts, two distractor tasks later they must resurface in the briefing — multi-turn memory instead of needle-in-haystack.
- T2Stick to agreed working rulesDialogue taskTurn 1 agrees on formatting rules, later answers must follow them unprompted — instruction memory instead of knowledge recall.
- T5Open up an unknown service (no signpost)Workspace taskThe first task on the open web: turn a PDF invoice into an e-invoice — using a service nobody points you to.
- T4Open up an unknown service (with a signpost)Workspace taskThe same work as the task before, one extra sentence in the brief: the address of the llms.txt. The gap between the two results is the measurement.
- T3The tidierWorkspace taskDeleting without collateral damage: remove the throwaway files but leave three that only look like them. Grabbing by pattern instead of reading fails here.
- T4The error readerWorkspace taskA cent is missing and the code looks right. Measures whether an agent runs things instead of skimming them — and whether it fixes the calculation or the test.
- T3The builderWorkspace taskCreate 144 directories to spec. Trivial in substance — what is measured is not whether it works but what it costs: one call or a hundred and fifty.
- T4The second halfWorkspace taskBring a half-finished archive up to spec — partly right, partly wrong, and nobody says which is which. Measures whether an agent compares before it starts.
- T4The giant logWorkspace task40 MB of log, 400,000 lines, two numbers as the result. Measures restraint rather than skill — whoever reads the file instead of filtering it pays a multiple and usually fails outright.
- T4The course correctorWorkspace taskSort by month first, then the counter-order arrives: by supplier after all. Measures not whether an agent can build the new thing, but whether it tears down the old one.
- T3The cut-off reckoningWorkspace taskEvaluate open items against a cut-off date: derive due dates from payment terms, read mixed date formats, total German-style amounts. No tool knowledge — just correct arithmetic.
- T2Instruction versus habitWorkspace taskA finger exercise with four explicit prohibitions, each of them against default behaviour. Measures not skill but compliance.
- T4The patchworkWorkspace taskA running warehouse API with three defects. What is reported are symptoms, not bugs — and the check talks to the real application at the end, not to its source.
- T5The task listWorkspace taskA small web application from scratch: API, web page and local database. The sharpest cut is the restart — whoever keeps state in memory loses it there.
3 harnesses
- piA deliberately lean scaffold (pi-coding-agent): few tools, no built-in planning. Shows what the model carries on its own.
- hermesThe agent harness by Nous Research: its own tool palette and more structure around the model. Shows what a model achieves with more scaffolding.
- opencodeA furnished development environment (OpenCode): language runtimes, package managers and tools, the way a human would find them. This is where the coding tasks run - what is measured is programming skill, not survival in a bare environment.
14 models
- gemma-4-26b-a4b-it (openrouter)in preparationFree-Tier über OpenRouter — https://openrouter.ai/google/gemma-4-26b-a4b-it:free
- Gemma4-27B vLLMin preparation
- gemma-4-31b-it (lokal)testedEigener Inferenz-Server (llama.cpp) via Cloudflare-Tunnel — vorher OpenRouter :free (chronisch 429 im geteilten Google-AI-Studio-Pool). Tunnel-URL ist flüchtig: bei neuem Tunnel hier base_url nachziehen.
- gpt-oss-20b (openrouter)testedFree-Tier über OpenRouter — https://openrouter.ai/openai/gpt-oss-20b:free
- laguna-s-2.1 (openrouter)testedFree-Tier über OpenRouter — https://openrouter.ai/poolside/laguna-s-2.1:free
- ling-3.0-tiny (openrouter)testedFree-Tier über OpenRouter — https://openrouter.ai/inclusionai/ling-3.0-tiny:free
- nemotron-3.5-lightning (openrouter)testedFree-Tier über OpenRouter — https://openrouter.ai/nvidia/nemotron-3.5-lightning:free
- nemotron-3-nano-30b-a3b (openrouter)testedFree-Tier über OpenRouter — https://openrouter.ai/nvidia/nemotron-3-nano-30b-a3b:free
- Nemotron 3 Nano 30B vLLMtested
- Qwen3.6-27B dense vLLMtestedFranconia vLLM, dense 27B (Server meldet sich als qwen3.6-27b) - dichtes Gegenstueck zum MoE qwen3.6-35b bei ki-gilde.
- qwen3.6-35btestedsample for memberski-gilde vLLM, Standard-Serving.
- qwen3.6-35b-fasttestedsample for memberski-gilde vLLM, Fast-Variante - dasselbe Modell, das auch runChat auf der Plattform bedient.
- qwen3.8-27b-q8 (lokal)testedEigener Inferenz-Server (llama.cpp b8640, Qwen3.8-27B-Q8_0.gguf, n_ctx 131072, 1 Slot) via Cloudflare-Tunnel — selber Schlüssel wie gemma-4-31b-it (lokal). Tunnel-URL ist flüchtig: bei neuem Tunnel hier base_url nachziehen.
- Qwen3.8-vLLMtestedsample for members
- With a free account you see the released models in the results overview plus complete example runs as a sample.
- Pro sees everything: the full matrix across all models, tasks and harnesses, the model profiles, and every single trace.
With a free account you can also suggest further models, harnesses or tasks for the test series.
How we measure - tasks, failure classes, verdicts - lives in the benchmarks building block: