AuthorsXinze Li, Ziyue Zhu, Siyuan Liu et al.
TL;DR
EMemBench uses an experience-conditioned benchmark generator with programmatic QA templates to show A-MEM reaches 51.9% ACC on text games vs 44.9% for Qwen3-32B in-context (+7.0 points).
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
EMemBench reveals that induction peaks at only 34.2 ACC on text games and 20.2 ACC on visual games, with visual spatial best at 24.3 ACC.
These low scores mean VLM agents fail to form reliable episodic memory for induction and spatial reasoning, limiting long-term decision making in interactive environments.
HOW IT WORKS
EMemBench centers on Interaction episode, Benchmark generator, Structured logging, State reconstruction, and Programmatic QA generation to turn each agent’s trajectory into QA instances.
Think of EMemBench like a game engine plus a black-box recorder and an auto-grader: it logs everything the agent sees and does, then computes ground truth from underlying game signals.
This KEY_MECHANISM lets EMemBench test individualized episodic memory over dynamic environments, rather than static long-context reasoning within a single prompt window.
DIAGRAM
This diagram shows how EMemBench converts an agent’s interaction episode into a QA set with deterministic ground truth and metadata.
DIAGRAM
This diagram shows how EMemBench runs text-only and visual games, applies query horizon control, and aggregates accuracy across seeds.
PROCESS
Interaction episode
EMemBench records an Interaction episode as a sequence of observations, actions, and rewards while the agent plays Jericho or Crafter.
Game State Collection
EMemBench performs Structured logging of each timestep, capturing scores, coordinates, inventory, player view, and achievements as JSON.
State reconstruction
EMemBench builds a Timeline index and Event index to reconstruct higher level events like inventory changes and location transitions efficiently.
Programmatic QA Generation
EMemBench applies the Benchmark generator and Programmatic QA generation templates to create balanced, answerable questions with deterministic ground truth.
KEY CONTRIBUTIONS
Experience-conditioned memory evaluation framework
EMemBench defines Interaction episode and Benchmark generator to turn each agent’s own trajectory into QA, supporting individualized episodic memory evaluation across 16 environments and infinite trajectories.
Programmatic QA generation with controlled answerability
EMemBench uses Structured logging, State reconstruction, and Programmatic QA generation templates to cover seven abilities, including single hop, multi hop, induction, spatial, temporal, logical, and adversarial reasoning.
Query horizon control and stability analysis
EMemBench introduces query horizon control and reports low seed standard deviations (e.g., 0.0364 ACC SD for GPT-5.1 on text games), ensuring fair and reproducible comparisons between memory agents.
RESULTS
Overall ACC
51.9%
+7.0 over Qwen3-32B in-context
Single-Hop ACC
66.9%
+17.9 over Qwen3-32B in-context
Multi-Hop ACC
28.1%
-4.6 vs Qwen3-32B in-context
Visual Overall ACC
43.8%
best in-context score for GPT-5.1 on visual games
These metrics come from EMemBench’s main results table on text-only and visual games, which test seven memory abilities across Jericho and Crafter. The 51.9% Overall ACC for A-MEM with Qwen3-32B on text games shows EMemBench can quantify gains from persistent memory modules over pure in-context prompting baselines.
BENCHMARK
These metrics come from EMemBench’s main results table on text-only and visual games, which test seven memory abilities across Jericho and Crafter. The 51.9% Overall ACC for A-MEM with Qwen3-32B on text games shows EMemBench can quantify gains from persistent memory modules over pure in-context prompting baselines.
BENCHMARK
Overall ACC on EMemBench text games for Qwen3-32B with different memory settings.
KEY INSIGHT
EMemBench shows that memory agents can reduce Single-Hop accuracy for GPT-5.1, where in-context Single-Hop ACC is already 76.7% but some memory variants drop below this.
This is surprising because persistent memory is expected to help all question types, yet EMemBench reveals that adding retrieval and organization can hurt simple recall when the backbone’s prompt-based memory is already strong.
WHY IT MATTERS
EMemBench unlocks a way to stress-test episodic memory as an interactive, individualized competence, not just long-context reading over static logs.
With EMemBench, builders can now design and compare memory agents that must survive dynamic text and visual games, revealing nuanced trade-offs between in-context reasoning and persistent memory modules.
Guilin Zhang, Wei Jiang et al.
· 2026
A-MAC scores candidate memories using Utility, Confidence, Novelty, Recency, and Type Prior combined by a learned linear admission policy with Algorithm 1 A-MAC Memory Admission. On the LoCoMo benchmark, A-MAC achieves F1 0.583 and 2644 ms latency, improving F1 by 0.042 and reducing latency by 1187 ms compared to A-mem.
Robbyant Team, Zelin Gao et al.
arXiv 2026 · 2026
LingBot-World combines a Data Engine, Fundamental World Model, Action-Conditioned World Model, and Post-Training causal adaptation to turn a 28B-parameter video generator into a real-time interactive world simulator. On the VBench benchmark, LingBot-World achieves a dynamic degree of 0.8857 versus 0.7612 for Yume-1.5, while also improving imaging quality to 0.6683.
Manoj Madushanka Perera, Adnan Mahmood et al.
· 2026
AgenticAI-DialogGen chains ChatPreprocessor, KnowledgeExtractor, TopicAnalyzer, KnowledgeGraphBuilder, PersonaGenerator, DuelingChat Agent, ConversationValidator, ConversationRefiner, QAGeneration, and PostProcessing to turn raw multi-session chats into topic-guided, persona-grounded conversations with explicit short- and long-term memories. On the TGC / KG memory QA benchmark, Mistral-7B fine-tuned within AgenticAI-DialogGen achieves 87.36 F1, compared to GPT-4’s 83.77 F1 in a zero-shot setting on the same task.
Answers use this explainer on Memory Papers.
Checking…