The Shelf · Signal Library
No. 193

Cached traces cut agent-evaluation error from a single executed task

StrongEvidence grade
Nov 5, 2026Shelved
Agentic AISection
sha256:a363a9754967…Fingerprint

instead of judging a new agent system only by binary correctness, this method embeds LLM-summarized execution traces and predicts the system's full benchmark score from its position among reference systems. On SWE-bench Verified, adding the trace geometry to an item-response model cut mean absolute error from 0.13 to 0.08 given just one random executed task; on Terminal-Bench 2.0 it went from 0.14 to 0.12.

Read the source — arXivBrowse the Shelf

Cite as — Sawdust Shelf v2026-11-05.3 · No. 193 · fingerprint sha256:a363a9754967…
Share image rendered from the living shelf — the number, the grade and the fingerprint travel with the card.