AuthorsAhmed Cherif
TL;DR
AgentMemBench uses a unified MemoryStrategy pipeline with provenance-aware retrieval to show External Key-Value (EKV) memory reaches macro Recall@5 0.792 while ICW stays at 0.468.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
LLMs operate within finite context windows of 4,096–128,000 tokens, far smaller than days-long multi-session histories, creating a persistent memory gap.
In long-term dialogue tasks like LoCoMo, agents forget earlier sessions entirely, leading to repetition, contradiction, and failure on questions whose gold evidence lies thousands of tokens back.
HOW IT WORKS
AgentMemBench defines a common MemoryStrategy interface with Store, Retrieve, and Generate operations, implemented by ICW, EKV, GEM, CBS, and WAM over the same conversation logs.
Think of AgentMemBench as a lab bench where different memory “RAM vs disk vs card catalog vs notebook vs web browser” backends plug into the same agent and are stress-tested identically.
This unified harness plus provenance-aware retrieval lets AgentMemBench expose where plain recency windows fail and where dense retrieval, summaries, or graphs actually preserve long-term recall.
DIAGRAM
This diagram shows how AgentMemBench replays conversations, updates memory, retrieves context, and generates answers at each annotated question turn.
DIAGRAM
This diagram shows how AgentMemBench feeds three datasets through five strategies and computes seven metrics, including Recall@5 and Memory Footprint.
PROCESS
Store
AgentMemBench calls Store to update the memory state Mt using the latest user utterance ut and agent response at under the chosen MemoryStrategy.
Retrieve
AgentMemBench invokes Retrieve to produce context Ct, using ICW’s window, EKV’s FAISS embeddings, GEM’s graph neighbourhood, CBS’s summaries, or WAM’s web-augmented window.
Generate
AgentMemBench passes ut+1 and Ct into Qwen2.5-7B-Instruct with greedy decoding, ensuring deterministic answer generation for all 491 annotated question turns.
MADS heuristic
AgentMemBench optionally applies the MADS Memory-Adaptive Dynamic Selector, choosing a strategy from corpus statistics like Csessions, Tavg, Edensity, and Qrel via a fixed priority cascade.
KEY CONTRIBUTIONS
Unified reproducible benchmark
AgentMemBench evaluates 5 memory strategies × 3 task types = 15 configurations over 491 annotated question turns, reporting seven metrics including Recall@5, F1, Faithfulness, Memory Footprint, and Latency.
Provenance-aware retrieval evaluation
AgentMemBench scores Recall@k, MRR, and nDCG against true session and turn provenance, correcting ad-hoc evaluations that match only retrieval indices instead of gold evidence locations.
MADS heuristic and external baselines
AgentMemBench introduces the zero-shot MADS Memory-Adaptive Dynamic Selector and implements MemGPT/Letta and HippoRAG under the same harness to situate ICW, EKV, GEM, CBS, and WAM in the wider literature.
RESULTS
Recall@5
0.792
+0.324 over ICW
MRR
0.677
+0.227 over ICW
Answer F1
0.156
+0.019 over ICW
Memory Footprint
5126.4 tokens
≈25× ICW’s 297.5 tokens
AgentMemBench reports these macro-averaged metrics across LoCoMo, MultiDoc2Dial, and MSC, showing EKV’s dense retrieval as the only strategy that maintains long-range Recall@5 while trading off a much larger memory footprint.
BENCHMARK
AgentMemBench reports these macro-averaged metrics across LoCoMo, MultiDoc2Dial, and MSC, showing EKV’s dense retrieval as the only strategy that maintains long-range Recall@5 while trading off a much larger memory footprint.
BENCHMARK
Macro Recall@5 over LoCoMo, MultiDoc2Dial, and MSC for the five AgentMemBench strategies.
KEY INSIGHT
On LoCoMo, ICW, WAM, GEM, and CBS all score Recall@5 ≤0.005, while EKV alone reaches 0.573 on long-range questions.
This breaks the intuition that larger windows, summaries, or graphs suffice for long-term memory; AgentMemBench shows only dense retrieval scales when gold evidence lies many sessions back.
WHY IT MATTERS
AgentMemBench gives builders a quantitative map of when to use ICW, EKV, GEM, CBS, or WAM, including explicit accuracy–efficiency trade-offs in tokens and latency.
With AgentMemBench’s numbers and MADS heuristic, developers can design conversational agents that choose dense retrieval only when long-range recall is needed, keeping API costs manageable while avoiding catastrophic forgetting.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…