AuthorsYuyao Wang, Zhongjian Zhang, Mo Chi et al.
TL;DR
EvoMemBench organizes agent memory along scope and content axes, showing no single memory form wins universally and long-context LLMs often remain competitive.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing benchmarks mainly test reasoning, planning, and execution, leaving agent memory mechanisms under-evaluated and fragmented across tasks and protocols.
Without a unified view over in-episode and cross-episode, knowledge and execution evolution, developers cannot tell when memory helps, when it hurts, or which mechanisms fit specific agent demands.
HOW IT WORKS
EvoMemBench defines four settings using In-Episode Knowledge Evolution, In-Episode Execution Evolution, Cross-Episode Knowledge Evolution, and Cross-Episode Execution Evolution over six curated datasets.
You can think of EvoMemBench as a test rig that separates short-term “RAM-like” episode state from long-term “disk-like” experience, and splits factual knowledge from procedural execution.
This design lets EvoMemBench expose where memory abstractions fail, revealing that simply extending a context window cannot replace structured, reusable, and revisable memory across episodes.
DIAGRAM
This diagram shows how EvoMemBench organizes tasks into four memory evolution settings across six datasets.
DIAGRAM
This diagram shows how EvoMemBench runs memory-free and memory-augmented agents across six datasets with standardized metrics.
PROCESS
Agents With Memory Mechanisms
EvoMemBench instantiates agents with memory state me_t and policy π(· | h_t−1, o_t, me_t) to capture interaction history and observations compactly.
Scope of Memory Evolution
EvoMemBench assigns each episode to in-episode memory or cross-episode memory, deciding whether me_t resets after the task or persists across episodes.
Content of Memory Evolution
EvoMemBench labels tasks as knowledge evolution or execution evolution, focusing memory on facts and rules versus action state and procedures.
Benchmark Construction
EvoMemBench maps the episode to one of six datasets like INEP-KNOW or CROSSEP-EMB and evaluates accuracy or success rate plus token usage.
KEY CONTRIBUTIONS
Benchmarking Agent Memory From a Self-Evolving Perspective
EvoMemBench organizes evaluation along in-episode vs cross-episode and knowledge vs execution axes, covering four forms of self-evolving memory in one unified framework.
Unified and Open Benchmark EvoMemBench
EvoMemBench reconstructs six datasets into INEP-KNOW, INEP-EXEC, CROSSEP-KNOW, CROSSEP-TOOL, CROSSEP-WEB, and CROSSEP-EMB, with at least five episodes per shared context in CROSSEP-KNOW.
Standardized Evaluation Protocol and Comparative Analysis
EvoMemBench compares 15 memory methods plus Gemini-3-Flash, GPT-5-mini, and DeepSeek-V3.2, revealing that no single memory form works consistently across all settings.
RESULTS
Memory Retention Accuracy
98.20%
+17.6 over DeepSeek-V3.2 on Memory Retention Event QA
Memory Revision Accuracy
96.00%
Gemini-3-Flash vs 65.00% for DeepSeek-V3.2 on Memory Revision FC SH
INEP EXEC Success Rate
66.9%
Gemini-3-Flash at 64K vs 45.3% for DeepSeek-V3.2 overall
CROSSEP KNOW Hard Split
13.0%
ACE vs 0.0% for DeepSeek-V3.2 on overall hard tasks
On INEP-KNOW, EvoMemBench shows Gemini-3-Flash reaching 98.20% memory retention accuracy and 96.00% revision accuracy, far above DeepSeek-V3.2. On CROSSEP-KNOW hard tasks, EvoMemBench reveals ACE at 13.0% overall accuracy while DeepSeek-V3.2 collapses to 0.0%, highlighting when explicit memory becomes essential.
BENCHMARK
On INEP-KNOW, EvoMemBench shows Gemini-3-Flash reaching 98.20% memory retention accuracy and 96.00% revision accuracy, far above DeepSeek-V3.2. On CROSSEP-KNOW hard tasks, EvoMemBench reveals ACE at 13.0% overall accuracy while DeepSeek-V3.2 collapses to 0.0%, highlighting when explicit memory becomes essential.
BENCHMARK
Answer accuracy on Memory Retention Event QA in INEP-KNOW.
KEY INSIGHT
On CROSSEP-KNOW easy tasks, EvoMemBench shows memory-free DeepSeek-V3.2 at 52.1% overall accuracy, while the best memory method Qwen3-Emb-4B only reaches 50.0%.
This is surprising because we expect more stored knowledge to help, yet EvoMemBench reveals that added memory can introduce irrelevant evidence and actually reduce performance on easy contexts.
WHY IT MATTERS
EvoMemBench gives developers a clear map of when retrieval-based, long-term, or procedural memory helps across in-episode and cross-episode, knowledge and execution settings.
With EvoMemBench, builders can now stress-test agent memory designs systematically, choosing mechanisms that match task difficulty, context limits, and decision structure instead of relying on ad hoc long-context prompts.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…