AuthorsWen-Yu Chang, Yun-Nung Chen
TL;DR
LOCOMO-CONV uses four conversational query styles plus unified retrieval/response evaluation to reveal large implicit/composed memory gaps, with AnchorMem reaching 0.754 dialog recall@10 (+0.095 over its base).
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing long-term memory benchmarks mostly use explicit QA probing, masking failures when users speak naturally without asking direct questions. LOCOMO-CONV shows Naive RAG and AnchorMem implicit retrieval recall@10 at only 0.312 and 0.368.
When conversational agents rely on these systems, they miss implicit and composed cues, leading to weak grounding and incorrect responses even when relevant memories exist.
HOW IT WORKS
LOCOMO-CONV introduces four query styles—dialog, implicit, counterfactual, and composed—and evaluates memory systems with retrieval recall and response quality on the same LoCoMo evidence. LOCOMO-CONV tests AnchorMem, A-MEM, mem0, Memora, and Naive RAG under realistic conversational framing.
You can think of LOCOMO-CONV as swapping a multiple-choice exam for a live conversation, then watching how each memory system behaves when users hint instead of ask. Like comparing a neatly indexed library catalog to a librarian who must recall books from casual chat.
This design lets LOCOMO-CONV expose failures that a plain context window or third-person QA cannot, especially silent grounding and the retrieval-to-response gap on implicit and composed queries.
DIAGRAM
This diagram shows how LOCOMO-CONV rewrites LoCoMo QA items into four conversational query styles before evaluation.
DIAGRAM
This diagram shows how LOCOMO-CONV runs retrieval and response evaluation across five memory systems and four query styles.
PROCESS
Conversational Query Styles
LOCOMO-CONV first rewrites each LoCoMo question into dialog, implicit, counterfactual, or composed style while preserving gold answers and evidence dia_ids.
Composed Multi Memory Clusters
LOCOMO-CONV builds composed clusters by pairing QAs with overlapping evidence E(qi) ∩ E(qj) ≠ ∅ and non identical evidence, yielding 1,069 composed clusters.
Retrieval Recall
LOCOMO-CONV computes retrieval recall@10 by matching retrieved memories against gold evidence dia_ids, using verbatim text or metadata for abstractive systems.
Response Quality Judging
LOCOMO-CONV feeds retrieved memories to an answer model and scores responses with style specific LLM judges using fact_used, unaware hedge corrected, or atomic fact coverage.
KEY CONTRIBUTIONS
Introduction of LOCOMO-CONV
LOCOMO-CONV adds four first person conversational query styles over fixed LoCoMo memories, exposing retrieval failures on implicit and composed queries where recall@10 falls to 0.312–0.368.
Unified Retrieval and Response Evaluation
LOCOMO-CONV provides a unified framework scoring retrieval recall and free form response quality, enabling systematic comparison of AnchorMem, A-MEM, mem0, Memora, and Naive RAG.
Supportive Memory Annotation Layer
LOCOMO-CONV releases supportive_memory annotations for implicit queries, capturing conversationally supportive turns beyond original gold evidence to study silent grounding.
RESULTS
Retrieval Recall@10 Dialog
0.754
+0.095 over AnchorMem base
Retrieval Recall@10 Implicit
0.524
+0.156 over AnchorMem base
Response Quality Dialog
0.669
+0.071 over AnchorMem base
Response Quality Implicit
0.470
+0.106 over AnchorMem base
These metrics come from LOCOMO-CONV’s main table, evaluating AnchorMem with multi facet query rewriting on LoCoMo derived conversations. The results show that conversational query rewriting in LOCOMO-CONV substantially boosts both retrieval recall and response quality compared to vanilla AnchorMem on dialog and implicit styles.
BENCHMARK
These metrics come from LOCOMO-CONV’s main table, evaluating AnchorMem with multi facet query rewriting on LoCoMo derived conversations. The results show that conversational query rewriting in LOCOMO-CONV substantially boosts both retrieval recall and response quality compared to vanilla AnchorMem on dialog and implicit styles.
BENCHMARK
Retrieval Recall@10 on LOCOMO-CONV dialog queries for five memory systems.
KEY INSIGHT
LOCOMO-CONV shows that mem0 reaches strong implicit retrieval recall@10 of 0.456 yet delivers weaker dialog response quality at 0.366. Memora, with similar implicit recall of 0.445, achieves higher dialog fact_used of 0.501.
This breaks the assumption that better retrieval automatically yields better responses, revealing how lossy abstractive compression can discard details needed for grounded conversational answers.
WHY IT MATTERS
LOCOMO-CONV gives researchers a way to stress test memory systems under realistic conversational framing, including implicit, counterfactual, and composed user turns. Builders can now design and compare memory architectures that focus on reasoning based elaboration and silent grounding, rather than only optimizing QA style fact recall.
Jingbiao Mei, Jinghong Chen et al.
arXiv 2026 · 2026
ATM-Bench structurally evaluates long-term multimodal personal memory using Memory Ingestion, Retrieval, and Answer Generation with Schema-Guided Memory and Descriptive Memory variants. On ATM-Bench-Hard, Oracle with SGM reaches 47.3% QS while the best full system stays under 20% accuracy, revealing a large gap.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Answers use this explainer on Memory Papers.
Checking…