AuthorsYijun Chen, Yaqi Zheng, Yanya Li et al.
TL;DR
EM2Mem binds multimodal evidence to event anchors, yielding event-centric memory cells that boost Video-MME (L) accuracy to 76.8% vs 73.1% for WorldMM† (+3.7pp).
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing multimodal memory systems store captions, frames, and transcripts as isolated fragments, forcing LLMs to reconstruct cross-modal alignments at inference time.
For long-video QA, this fragment-centric design causes sparse evidence across hours of video to be hard to localize, making attribution difficult and increasing latency and token cost.
HOW IT WORKS
EM2Mem’s core mechanism is Event-Centric Multimodal Memory Cells µi, combining Multimodal Event Records, Temporal Context Views, and event-linked Episodic Graph GE and Semantic Graph GS.
You can think of EM2Mem like a card catalog where each card is an event anchor, and all related visual, textual, and graph evidence is filed under that card instead of scattered shelves.
This event-anchored align-then-retrieve design lets EM2Mem fetch compact, generation-ready evidence units that a plain context window or fragment-level RAG cannot provide.
DIAGRAM
This diagram shows how EM2Mem retrieves, expands, and compiles event-centric evidence when answering a long-video question.
DIAGRAM
This diagram summarizes how EM2Mem is evaluated across benchmarks and ablated on EgoLifeQA.
PROCESS
Event-Centric Multimodal Memory Schema
EM2Mem first defines event anchors ei and wraps each into Event-Centric Multimodal Memory Cells µi, preparing a schema that binds local records Ri and context views Ci.
Multimodal Memory Construction
EM2Mem constructs Multimodal Event Records with captions ci, transcripts ri, keyframes ki, and structured metadata zi, then builds Temporal Context Views at multiple scales.
Event-Linked Memory Graph Construction
EM2Mem builds an Episodic Graph GE over entities and temporal transitions and a Semantic Graph GS over habits, preferences, and routines, all grounded to event anchors.
Memory Retrieval and Ranking
Given a question, EM2Mem selects relevant event cells using Ri, Ci, GE, and GS, expands them, compiles Eq, and lets LLMans generate the final grounded answer.
KEY CONTRIBUTIONS
Event-Centric Multimodal Memory Schema
EM2Mem introduces Event-Centric Multimodal Memory Cells µi that unify Multimodal Event Records and Temporal Context Views, improving EgoLifeQA accuracy to 66.0% vs 64.0% for WorldMM†.
Event-Linked Episodic and Semantic Graphs
EM2Mem builds Episodic Graph GE and Semantic Graph GS over event anchors, capturing cross-event relations and long-term habits while keeping all semantic facts evidence-grounded.
Align-then-Retrieve Long-Video QA Framework
EM2Mem’s align-then-retrieve design shifts fusion to construction time, yielding a 4.67× per-query latency speedup and 63.66% fewer total inference tokens compared with WorldMM on EgoLifeQA.
RESULTS
Avg. accuracy Video-MME (L)
76.8%
+3.7 points over WorldMM† (73.1%)
Avg. accuracy EgoLifeQA
66.0%
+2.0 points over WorldMM† (64.0%)
Avg. latency per query
98.21 s
4.67× faster than WorldMM at 459.00 s
Total inference tokens
15.27M
63.66% fewer tokens than WorldMM at 42.03M
On EgoLifeQA, Ego-R1 Bench, and Video-MME (L), EM2Mem is evaluated for multiple-choice long-video QA. The 76.8% Video-MME (L) result and efficiency gains show EM2Mem’s event-centric memory makes long-video reasoning both more accurate and more scalable than fragment-centric baselines like WorldMM†.
BENCHMARK
On EgoLifeQA, Ego-R1 Bench, and Video-MME (L), EM2Mem is evaluated for multiple-choice long-video QA. The 76.8% Video-MME (L) result and efficiency gains show EM2Mem’s event-centric memory makes long-video reasoning both more accurate and more scalable than fragment-centric baselines like WorldMM†.
BENCHMARK
Average accuracy (%) across categories on Video-MME (L).
BENCHMARK
Overall accuracy (%) on EgoLifeQA for EM2Mem and ablation variants.
KEY INSIGHT
Structured event fields plus construction-time unification give EM2Mem 71.2% accuracy on the first 250 EgoLifeQA questions, beating raw frames at 68.0%.
This is surprising because many assume more pixel-level visual detail is always better, yet EM2Mem shows typed, event-anchored structure can be more answerable than raw imagery.
WHY IT MATTERS
EM2Mem unlocks event-level, multimodal, and graph-grounded memory that can be read in single passes, with strict Top-5 event recall reaching 30.8% vs 23.8% for WorldMM.
Builders can now design long-video agents that reuse pre-aligned event memories across many queries, achieving both higher accuracy and dramatically lower latency and token budgets.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…