AuthorsZeyu Liu, Jian Zhong, Rongduo Han et al.
TL;DR
EvalMem uses three parallel Examiners plus recall-first agentic RAG to diagnose memory failures, revealing retrieval defects up to 22.1% on LOCOMO and enabling +2.5 pp accuracy via MemWiki.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing long-term memory evaluations only report end-to-end QA correctness, so a wrong answer gives just 1-bit feedback and hides the failure layer.
On LOCOMO, EvalMem shows retrieval defects reach 22.1%, while encoding and generation defects are only 7.7% and 6.5%, meaning systems misattribute many errors and struggle to fix them.
HOW IT WORKS
EvalMem’s core mechanism is three parallel components: Encoding Examiner, Retrieval Examiner, Generation Examiner, coordinated by an Attribution Agent that outputs 11-code multi-label diagnoses.
You can think of EvalMem like a computer debugger: one probe checks what’s written to disk, another inspects cache hits, and a third tests whether the CPU can compute correctly from perfect inputs.
This operation-level view lets EvalMem pinpoint whether a long-term assistant failed to store, retrieve, or use a fact, something a plain context window or single QA score cannot reveal.
DIAGRAM
This diagram shows how EvalMem sequences Encoding, Retrieval, and Generation Examiners plus the Attribution Agent to diagnose each query.
DIAGRAM
This diagram shows how EvalMem runs seven memory systems on LOCOMO, LONGMEMEVAL-S, and DynaMem-Bench and then aggregates defect statistics.
PROCESS
Encoding Examiner
EvalMem uses the Encoding Examiner to check whether key facts Fkey are stored in the memory store M, assigning states like EXIST, MISS, CORRUPTAmbig, CORRUPTWrong, or DIRTY.
Retrieval Examiner
EvalMem’s Retrieval Examiner inspects the native Coriginal, tests whether Fkey is supported, and classifies retrieval as HIT, MISS, or NOISE with extra RF, LATE, NOI, and NIR codes.
Generation Examiner
EvalMem’s Generation Examiner feeds Q plus oracle evidence Coracle to the answer model, judging PASS or FAIL and subtyping failures into GF, GRF, or GH depending on reasoning and hallucination.
Attribution Agent
EvalMem’s Attribution Agent merges Denc, Dret, and Dgen into an 11-code multi-label defect set, masking downstream retrieval defects when encoding MISS or dirty memory explains the error.
KEY CONTRIBUTIONS
Fine-grained attribution for memory-system evaluation
EvalMem introduces three Examiners plus an Attribution Agent that produce an 11-code multi-label diagnosis, separating EM, RF, GF, GRF, GH, and other defects for each query.
Reliable and dynamic diagnostic evaluation
EvalMem adapts recall-first agentic RAG inside the Encoding Examiner, reducing observation MISS from 29.8% to 4.4% on LOCOMO, and builds DynaMem-Bench with 962 questions over 10 personas.
Retrieval as the most frequently attributed failure layer
EvalMem shows retrieval defects dominate encoding and generation across datasets, reaching 22.1% vs 7.7% and 6.5% on LOCOMO, and guides MemWiki to improve accuracy by +2.5 pp.
RESULTS
Mean Accuracy LOCOMO
68.1%
+? over static end-to-end scores (EvalMem adds defect attribution, not a new baseline)
Retrieval Defect Rate LOCOMO
22.1%
+14.4 pp over Encoding defect rate 7.7%
MemWiki Accuracy Gain LOCOMO
+2.5 pp
guided by retrieval diagnosis using EvalMem
Agentic RAG MISS Reduction
29.8% → 4.4%
−25.4 pp observation MISS on LOCOMO POS with present evidence
EvalMem evaluates seven memory systems on LOCOMO, LONGMEMEVAL-S, and DynaMem-Bench, exposing retrieval as the dominant failure layer. The MemWiki intervention and recall-first agentic RAG show that EvalMem’s diagnoses are reliable and actionable for improving long-term memory systems.
BENCHMARK
EvalMem evaluates seven memory systems on LOCOMO, LONGMEMEVAL-S, and DynaMem-Bench, exposing retrieval as the dominant failure layer. The MemWiki intervention and recall-first agentic RAG show that EvalMem’s diagnoses are reliable and actionable for improving long-term memory systems.
BENCHMARK
Encoding vs Retrieval vs Generation defect incidence across seven memory systems on LOCOMO.
KEY INSIGHT
EvalMem shows that retrieval defects consistently exceed encoding and generation defects, reaching 22.1% vs 7.7% and 6.5% on LOCOMO and 25.9% vs 9.7% and 5.0% on DynaMem-Bench.
This is counterintuitive because many memory-system designs focus on better storage or summarization, yet EvalMem reveals that the main bottleneck is actually surfacing already stored facts reliably.
WHY IT MATTERS
EvalMem unlocks operation-level visibility into long-term memory systems, letting builders see exactly whether encoding, retrieval, or generation failed for each query.
With EvalMem, practitioners can now design targeted interventions like MemWiki or retriever changes, validate them quantitatively, and iterate beyond opaque end-to-end QA scores that previously hid these failure modes.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Okan Bursa
· 2026
Adaptive RAG Memory (ARM) augments a standard retriever–generator stack with a Dynamic Embedding Layer and Remembrance Engine that track usage statistics and apply selective remembrance and decay to embeddings. On a lightweight retrieval benchmark, ARM achieves NDCG@5 ≈ 0.9401 and Recall@5 = 1.000 with 22M parameters, matching larger baselines like gte-small while providing the best efficiency among ultra-efficient models.
Wei Zhou, Xuanhe Zhou et al.
arXiv 2026 · 2026
Are We Ready For An Agent-Native Memory System? analyzes Memory Representation and Storage, Memory Extraction, Memory Retrieval and Routing, and Memory Maintenance across 12 real systems like Mem0, Zep, MemTree, LightMem, MemOS, MemoryOS, and A-MEM. The study’s main result is that structured systems such as Zep reach 48.0 LLM Judge Accuracy on LongMemEval while Long Context reaches 19.0, and that localized maintenance strategies like LightMem achieve 48.3% normalized utility at only 3.67 s per query.
Answers use this explainer on Memory Papers.
Checking…