Best for
- • Multi-dimensional agent-memory evaluation
Memory benchmark
MemBench
A benchmark for LLM-agent memory across factual and reflective memory, participation and observation scenarios, and effectiveness, efficiency, and capacity dimensions.
Evaluation style
- • Agent interaction scenarios
Memory horizon
- • Factual and reflective memory
Scale
- • Multiple memory levels, interaction scenarios, and evaluation dimensions in one agent-memory suite.
Tasks
- • Factual memory
- • Reflective memory
- • Participation scenarios
- • Observation scenarios
What it measures
- • Effectiveness
- • Efficiency
- • Capacity
Original research
MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents
This is the paper that introduced MemBench.
arXiv:2506.21605 ↗Used in the field
Papers evaluated on MemBench
Benchmark
AgentMemBench: A Systematic Benchmark for Evaluating Long-Term Memory Management Strategies in Conversational AI Agents
Ahmed Cherif
arXiv 2026 · 2026
AgentMemBench wires five strategies—In-Context Windowing (ICW), External Key-Value Store (EKV), Graph-Based Episodic Memory (GEM), Compression-Based Summarisation (CBS), and Web-Augmented Memory (WAM)—into a shared Store Read Generate interface over multi-session dialogues. AgentMemBench’s main result is that EKV achieves macro Recall@5 0.792 on LoCoMo, MultiDoc2Dial, and MSC, a +0.324 gain over ICW’s 0.468, while also leading MRR, Answer F1, and Faithfulness.
RAG
Can Agent Memory Systems Track Evolving State?
Xinyi Fan, Miri Liu et al.
arXiv 2026 · 2026
StateMem represents conversations as structured StateStore entries built by a TurnEncoder, updated via deterministic Rechecker passes, and queried through a guided test-time recomputation prompt. On StateMemBench, StateMem reaches 0.363 gold rate on DeepSeek-V4-Flash, a +0.158 gain over the best retrieval baseline (Dense at 0.205) and +0.164 over long-context (0.149).
Long-Term Memory
EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents
Xinze Li, Ziyue Zhu et al.
· 2026
EMemBench builds on Interaction episode, Benchmark generator, Structured logging, State reconstruction, and Programmatic QA generation to turn each agent’s own game trajectory into verifiable episodic-memory questions. On text-only games, EMemBench shows A-MEM with Qwen3-32B achieves 51.9% Overall ACC compared to 44.9% for the Qwen3-32B in-context baseline (+7.0 points).
Agent Memory
EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
Yuyao Wang, Zhongjian Zhang et al.
arXiv 2026 · 2026
EvoMemBench evaluates self-evolving agent memory using four settings and six datasets spanning in-episode and cross-episode, knowledge and execution evolution. EvoMemBench’s main result is that explicit memory only helps reliably when context is insufficient, tasks are hard, or stored experience matches the decision process, while Gemini-3-Flash and other long-context baselines often match or beat 15 memory methods.
Benchmark
GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations
Jingbo Yang, Kwei-Herng Lai et al.
arXiv 2026 · 2026
GroupMemBench combines Graph-grounded Message Synthesis, Path-Based Context Sampling, Message Generation, Adversarial Question Synthesis, and a Solve–Judge–Refine Loop to stress-test group memory in LLM agents. On GroupMemBench, Hindsight reaches 46.01% average accuracy while a simple BM25 baseline attains 43.22%, revealing that current memory ingestion pipelines fail to preserve crucial multi-user structure.
Agent Memory
VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents
Yuhao Chen, Yi Xu et al.
· 2026
VehicleMemBench builds multi-user interaction histories via Persona Group Generation, Event Chain Construction, Temporal Interleaving, and Conversation Generation, then evaluates agents in an executable in-vehicle environment. On VehicleMemBench, Gemini-3-Pro-Preview reaches 90.60 Exact State Match with Gold Memory but drops to 64.80 under Recursive Summarization, a 25.8-point loss that exposes memory as the main bottleneck.