The Shelf · Signal Library
No. 167

Latent Sparse Attention: subquadratic attention that holds accuracy at 10M tokens

StrongEvidence grade
Nov 2, 2026Shelved
Tools & modelsSection
sha256:b4e8456ea780…Fingerprint

full attention is prohibitive past 1M tokens, and existing sparse methods either stay quadratic in indexing or lose accuracy at ultra-long context. LSA clusters keys hierarchically in a learned latent space so queries score clusters, not every key; the distilled variant speeds one attention layer up 22.7× in prefill and 4.6× in decode at 10M tokens on GLM-5.3-FLASH while keeping similar accuracy, and the native variant matches full attention's accuracy with a 1.6× per-layer training speedup at 128k. If it replicates, long-horizon agent context gets much cheaper.

Read the source — arXivBrowse the Shelf

Cite as — Sawdust Shelf v2026-11-05.3 · No. 167 · fingerprint sha256:b4e8456ea780…
Share image rendered from the living shelf — the number, the grade and the fingerprint travel with the card.