AuthorsBoqin Yuan, Yue Su, Kun Yao
TL;DR
Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory shows that retrieval quality, not fancy writing, drives up to 23-point accuracy gains on LoCoMo.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Memory-augmented agents show average accuracy spanning 20 points across retrieval methods, from 57.1% to 77.2% on LoCoMo.
Without diagnostics, systems cannot tell whether failures come from write strategy, retrieval, or memory utilization, obscuring where to invest engineering effort.
HOW IT WORKS
Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory combines Write Strategies, Retrieval Methods, and a Probing Framework to isolate memory bottlenecks.
Think of the memory pipeline like RAM and disk: write strategies decide what gets stored, retrieval methods decide what gets loaded, and probes inspect how it is used.
This KEY_MECHANISM lets Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory separate storage, retrieval, and reasoning failures, which a plain context window cannot disentangle.
DIAGRAM
This diagram shows how Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory processes a question through retrieval methods and diagnostic probes.
DIAGRAM
This diagram shows how Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory crosses three write strategies with three retrieval methods on LoCoMo.
PROCESS
Write Strategies
Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory applies Basic RAG, Extracted Facts, or Summarized Episodes to store multi session conversations.
Retrieval Methods
For a question, Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory runs Cosine similarity, BM25, or Hybrid reranking to fetch top k memories.
Probing Framework
Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory uses the Probing Framework to compute retrieval precision and classify memory utilization outcomes.
Failure Classification
Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory assigns each wrong answer to Retrieval failure, Utilization failure, or Hallucination for targeted diagnosis.
KEY CONTRIBUTIONS
Diagnostic probing framework
Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory introduces a Probing Framework that separates retrieval relevance, memory utilization, and failure modes with Retrieval Precision@k and utilization labels.
3 by 3 factorial study
Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory runs a 3×3 grid over Basic RAG, Extracted Facts, Summarized Episodes and Cosine, BM25, Hybrid reranking on 1,540 LoCoMo questions.
Retrieval bottleneck evidence
Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory shows retrieval precision and accuracy correlate at r=0.98, with hybrid reranking averaging 77.2% versus 57.1% for BM25.
RESULTS
Accuracy (%)
81.1%
+21.7 over Extracted Facts BM25
Token F1
0.240
vs 0.184 for Basic RAG BM25
Accuracy (%)
77.2%
average hybrid vs 57.1% average BM25
Precision
29.4%
vs 14.8% for Basic RAG BM25 retrieval
On LoCoMo, Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory evaluates 1,540 non adversarial questions to compare memory pipelines. The MAIN_RESULT shows retrieval method choice can move accuracy by 14–23 points, dwarfing write strategy effects.
BENCHMARK
On LoCoMo, Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory evaluates 1,540 non adversarial questions to compare memory pipelines. The MAIN_RESULT shows retrieval method choice can move accuracy by 14–23 points, dwarfing write strategy effects.
BENCHMARK
Accuracy (%) for write strategies and retrieval methods from Table 1.
KEY INSIGHT
Raw chunked Basic RAG with zero LLM calls reaches 81.1% accuracy under hybrid, beating more expensive fact extraction and summarization.
This breaks the assumption that sophisticated write pipelines always help; lossy compression and extraction can discard useful context that better retrieval cannot recover.
WHY IT MATTERS
Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory shows builders should prioritize retrieval precision, reranking, and query understanding over complex write time memory engineering.
With this, practitioners can confidently deploy cheap raw chunk storage and invest effort in hybrid retrieval and diagnostic probes to debug agent memory behavior.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…