AuthorsEnze Ma, Yufan Zhou, Wei-Chieh Huang et al.
TL;DR
MEMPROBE uses hidden user-state banks plus dual dump_all vs retrieve audits to show memory agents reach only ~0.62 recovery despite ~99.9% task success.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
MEMPROBE shows agents can reach 99.935% task completion while a memoryless baseline achieves a 0.000 category-balanced reconstruction score.
This means assistance agents appear highly capable, yet their long-term user state and memory artifacts remain largely unaudited and potentially empty.
HOW IT WORKS
MEMPROBE’s core mechanism chains Simulated Users and Leak-Controlled Tasks, Simulation Rollout, and Recovery Scoring with Failure Attribution to audit long-term memory.
Think of MEMPROBE like a black-box hard drive test: it writes structured user evidence, then later reads only the stored artifact to see what user “files” survive.
This design lets MEMPROBE measure recoverable user state directly, something a plain context window or task-success metric cannot reveal.
DIAGRAM
This diagram shows how MEMPROBE exposes hidden user state, collects it into memory, and then recovers it under dump_all and retrieve modes.
DIAGRAM
This diagram shows how MEMPROBE runs 50 users × 31 dimensions, scores reconstruction, and attributes failures across causes.
PROCESS
Simulated Users and Leak-Controlled Tasks
MEMPROBE builds each synthetic user with a hidden taxonomy-anchored memory bank and designs leak-controlled tasks that indirectly expose specific dimensions.
Simulation Rollout
During Simulation Rollout, MEMPROBE lets the user simulator interact with a memory-equipped agent while the agent reads and writes its memory across 31 tasks.
Recovery Scoring
In Recovery Scoring, MEMPROBE queries the final store with dump_all and retrieve, uses slot-filling readers, and judges reconstruction on a five-level scale.
Failure Attribution
Through Failure Attribution, MEMPROBE labels low-recovery cases as task-design failure, memory failure, agent elicitation failure, or simulator too strict.
KEY CONTRIBUTIONS
MEMPROBE recovery benchmark
MEMPROBE introduces a recovery benchmark with 50 simulated users and 31 hidden dimensions each, yielding 1,550 user-state targets grounded in established taxonomies.
Dual dump_all and retrieve audit
MEMPROBE separates written versus reachable memory by comparing full-store dump_all access against top-k retrieve access using the systems’ own search interfaces.
Failure attribution for memory agents
MEMPROBE adds Failure Attribution that partitions 1,550 targets into recovered, memory failures, task-design failures, agent-elicitation failures, and simulator-too-strict cases.
RESULTS
Task (A, %)
99.935%
+0.064 over mem0
Reconstruction (B) Full store
0.624
+0.013 over amem
Reconstruction (B) Top-k
0.540
+0.037 over mem0
Recovered targets
982
+21 over mem0 full-store
On MEMPROBE’s 1,550-target benchmark, longctx_full reaches 99.935% task completion yet only 0.624 category-balanced recovery under full-store and 0.503 under top-k retrieval. These results show MEMPROBE exposes a large gap between apparent task competence and actual recoverable user-state memory.
BENCHMARK
On MEMPROBE’s 1,550-target benchmark, longctx_full reaches 99.935% task completion yet only 0.624 category-balanced recovery under full-store and 0.503 under top-k retrieval. These results show MEMPROBE exposes a large gap between apparent task competence and actual recoverable user-state memory.
BENCHMARK
Category-balanced reconstruction score B under dump_all access on 1,550 user-state targets.
KEY INSIGHT
MEMPROBE shows a memoryless baseline reaches 99.935% task completion while achieving exactly 0.000 recovery of hidden user state.
This breaks the assumption that high task success implies good long-term memory, revealing that agents can be helpful yet effectively remember nothing durable about users.
WHY IT MATTERS
MEMPROBE gives the field a direct, quantitative handle on recoverable user state, not just downstream task behavior.
Builders can now design and tune memory systems explicitly for higher recovery scores, targeting episodic and relational user state that was previously invisible to evaluation.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…