AuthorsHuacan Chai, Yukai Wang, Yingxuan Yang et al.
TL;DR
SMMBench uses source-distributed multimodal environments to stress-test agent memory, revealing overall scores up to 0.4933 but still far below 0.7473 with golden evidence.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
SMMBench shows that each QA instance needs evidence from 2.82 sources on average, and no single source alone can determine the gold answer.
Under this source-distributed setting, multimodal agent memory systems struggle to retrieve and compose fragmented evidence, leading to unreliable answers and actions even with strong backbones.
HOW IT WORKS
SMMBench builds environments via QA Preparation, Conversational Source Synthesis, and Source-Aware Evidence Insertion, then evaluates with Memory Construction and an Evaluation Framework.
Think of SMMBench as a long-lived workspace where facts are scattered across many folders, chats, and files, instead of one neatly organized document.
This design forces agents to bridge source boundaries and resolve conflicts, enabling tests of capabilities that a plain context window cannot expose.
DIAGRAM
This diagram shows how SMMBench drives an agent from observing distributed sources to answering benchmark questions.
DIAGRAM
This diagram shows how SMMBench constructs source-distributed multimodal environments from raw benchmarks.
PROCESS
QA Preparation
SMMBench converts raw multimodal samples into unified question answer pairs with verified evidence units, forming the basis for source-distributed evaluation.
Conversational Source Synthesis
SMMBench instantiates agents with profiles and generates group and private chats, creating parallel conversational sources with long interaction histories.
Source-Aware Evidence Insertion
SMMBench routes complementary, updated, and conflicting evidence units into different sources and positions, preserving temporal dependencies and local coherence.
Memory Construction And Evaluation Framework
SMMBench merges turns by timestamp, builds agent memory, triggers retrieval with the benchmark question, and evaluates responses across five task types.
KEY CONTRIBUTIONS
Problem Identification
SMMBench identifies source-distributed memory composition as a bottleneck where each QA needs evidence from 2.82 sources and 4.54 evidence items on average.
Challenge Characterization
SMMBench characterizes source-level incompleteness, cross-source context bridging, and conflict resolution under different authority levels and time states.
Benchmark Construction
SMMBench builds 1877 samples over 5 task types and 264 sources, covering conversations, profiles, screenshots, tables, images, and documents for multimodal agent memory.
RESULTS
Overall
0.7473 score
+0.2473 over HMRAG
SINGLE-HOP QA
0.8753 score
+0.0624 over HMRAG
MULTI-HOP QA
0.7768 score
+0.0615 over HMRAG
FUNCTION CALL
0.2778 score
+0.1667 over HMRAG
On SMMBench’s 1877 samples, the GOLDEN EVIDENCE BASELINE reaches 0.7473 overall, while the best baseline HMRAG only achieves 0.4933. This gap shows that SMMBench exposes substantial room for improvement in source-distributed multimodal agent memory.
BENCHMARK
On SMMBench’s 1877 samples, the GOLDEN EVIDENCE BASELINE reaches 0.7473 overall, while the best baseline HMRAG only achieves 0.4933. This gap shows that SMMBench exposes substantial room for improvement in source-distributed multimodal agent memory.
BENCHMARK
Overall unweighted average score across all five SMMBench task types.
KEY INSIGHT
SMMBench shows that even native multimodal memory systems like MEMGPT drop from 0.3708 to 0.2713 overall when switching to native multimodal access.
This is surprising because one might expect richer multimodal input to help, but SMMBench reveals that cross-source composition, not modality access alone, is the dominant difficulty.
WHY IT MATTERS
SMMBench unlocks a way to systematically test whether agents can retrieve, align, and compose evidence scattered across independent multimodal sources.
With SMMBench, builders can now benchmark memory systems on realistic cross-source tasks like conflict resolution, preference reasoning, and memory-grounded function calling that were previously under-evaluated.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…