The Shelf · Signal Library
No. 206

Benchmarks as software: a third of Terminal-Bench 2.0 needed repair

StrongEvidence grade
Nov 6, 2026Shelved
Agentic AISection
sha256:8dae57d49eae…Fingerprint

benchmarks validated at release go stale as models find valid solutions verifiers reject; the authors call the gap benchmark maintenance debt and ran a continuous-validation workflow (community reports, traces, oracle checks, human-gated repairs) over Terminal-Bench 2.0. The audit found issues in roughly 30% of the benchmark; the repairs, released as Terminal-Bench 2.1, moved accuracy by over 10 points and leaderboard positions by up to three ranks, and fixed wall-clock limits alone closed a 20-point harness gap on one model.

Read the source — arXivBrowse the Shelf

Cite as — Sawdust Shelf v2026-11-06.1 · No. 206 · fingerprint sha256:8dae57d49eae…
Share image rendered from the living shelf — the number, the grade and the fingerprint travel with the card.