AuthorsJiangnan Yu, Kisson Songqi Lin, Jilong Wu
TL;DR
WhenLoss uses a four-condition diagnostic protocol plus Expected Predictive Compression to cut the write-side gap on LongMemEval to 0.04 and reach 0.49 CSM vs 0.44 for Summary LLM.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Long-context memory systems often fail under fixed budgets, and end-to-end scores cannot show whether evidence was discarded or just never retrieved.
On LongMemEval, most tested systems in WhenLoss have larger write-side gaps than retrieval-side gaps, so practitioners may optimize retrieval while the main losses happen during compression.
HOW IT WORKS
WhenLoss introduces four controlled conditions—Truncated Full Context, Oracle Evidence, Complete Stored Memory, and Retrieved Memory—to evaluate a fixed reader and localize performance drops.
You can think of this like profiling a computer: write behaves like disk compression, retrieval like RAM lookup, and Oracle Evidence like a perfect cache of the needed bytes.
By adding Expected Predictive Compression, WhenLoss moves the “what to keep” decision to write time using self-generated future questions, preserving minimal supporting evidence in memory that a plain context window or generic summarizer would omit.
DIAGRAM
This diagram shows how WhenLoss feeds different contexts into a fixed reader to compute write-side and retrieval-side gaps.
DIAGRAM
This diagram shows how WhenLoss runs seven memory systems across LongMemEval and LoCoMo under fixed budgets and readers.
PROCESS
Write stage
WhenLoss runs each memory system’s write strategy under a fixed token budget, producing a Complete Stored Memory representation of the conversation history.
Oracle Evidence condition
WhenLoss feeds the reader only the gold evidence turns for the question, creating the Oracle Evidence reference score for answer correctness.
Complete Stored Memory condition
WhenLoss gives the reader the entire stored memory without retrieval filtering, measuring how much performance is lost during the write stage.
Retrieved Memory condition
WhenLoss lets each system run its retrieval over stored memory, then compares this Retrieved Memory score to CSM to quantify retrieval-side degradation.
KEY CONTRIBUTIONS
Four-condition diagnostic protocol
WhenLoss defines Truncated Full Context, Oracle Evidence, Complete Stored Memory, and Retrieved Memory, and uses Contains Match and F1 to compute write-side and retrieval-side gaps on 500 LongMemEval questions.
Operational bottleneck indicators
WhenLoss introduces ∆write = ϕ(OE) − ϕ(CSM) and ∆retr = ϕ(CSM) − ϕ(RM) with a ϵ = 0.02 margin, showing four of six baselines are robustly write-dominant.
Expected Predictive Compression
WhenLoss proposes Expected Predictive Compression, which uses LLM self-questioning and a coverage–specificity–redundancy utility to reach 0.49 CSM vs 0.44 for Summary LLM and reduce ∆write to 0.04.
RESULTS
CSM Contains Match
0.49
+0.05 over Summary (LLM)
RM Contains Match
0.42
+0.04 over Summary (LLM)
Write-side gap Δwrite
0.04
0.05 smaller than Verbatim Chunk’s 0.31
Retrieval-side gap Δretr
0.07
0.01 larger than Summary (LLM)’s 0.06
On LongMemEval, which has 500 multi-session questions with gold evidence, WhenLoss shows Expected Predictive Compression achieves the highest Complete Stored Memory score (0.49 CM vs 0.44 for Summary LLM) and the lowest write-side gap (0.04). This proves that improving write-time evidence preservation under fixed budgets can yield larger gains than changing retrieval for the tested systems.
BENCHMARK
On LongMemEval, which has 500 multi-session questions with gold evidence, WhenLoss shows Expected Predictive Compression achieves the highest Complete Stored Memory score (0.49 CM vs 0.44 for Summary LLM) and the lowest write-side gap (0.04). This proves that improving write-time evidence preservation under fixed budgets can yield larger gains than changing retrieval for the tested systems.
BENCHMARK
Contains Match scores for Complete Stored Memory across memory systems on LongMemEval.
KEY INSIGHT
WhenLoss finds that write-side gaps exceed retrieval-side gaps for most baselines, with Verbatim Chunk losing 0.31 CM at write vs only 0.04 at retrieval.
This is counterintuitive because many practitioners focus on improving retrievers, yet the main performance loss in these long-context systems comes from compression decisions made before questions arrive.
WHY IT MATTERS
WhenLoss gives builders a concrete way to see whether their long-context system is bottlenecked by write or retrieval, using four simple evaluation conditions.
With Expected Predictive Compression, developers can design memory modules that anticipate future questions and preserve the exact entities, dates, and preferences that downstream answers depend on, instead of only optimizing generic summaries or larger context windows.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…