AuthorsYuanchen Bei, Tianxin Wei, Xuying Ning et al.
TL;DR
Mem-Gallery uses a three-dimensional evaluation framework over 1,711 QA pairs to show that explicit multimodal memory like MuRAG boosts overall F1 to 0.6966 on long-term conversations.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing benchmarks either drop visual modality or restrict to single-session context, so they cannot test evolving multimodal memory over long conversations.
This breaks evaluation of MLLM agents in realistic multi-session settings, where missing multimodal memory leads to poor reasoning, weak knowledge management, and unsafe behavior.
HOW IT WORKS
Mem-Gallery combines Benchmark Construction, a unified Conversational Environment for Memory, and an Evaluation Framework with Memory Extraction and Adaptation, Memory Reasoning, and Memory Knowledge Management.
You can think of Mem-Gallery like a long-term lab notebook for agents, where every multimodal session is logged, indexed, and then probed with targeted tests.
This design lets Mem-Gallery stress-test capabilities such as conflict-aware knowledge resolution and multimodal visual-centric reasoning that a plain context window cannot reliably expose.
DIAGRAM
This diagram shows how Mem-Gallery feeds multi-session multimodal conversations into memory agents and evaluates their responses across the three task families.
DIAGRAM
This diagram shows how Mem-Gallery constructs conversations and QA tasks, then runs benchmarking and analysis.
PROCESS
Data Sourcing
Mem-Gallery performs Data Sourcing by curating images and text, then generating structured backgrounds like role profiles and topics for Benchmark Construction.
Conversation Generation
Mem-Gallery runs Conversation Generation with newly created stories and topic-based clustering, producing coherent multi-session multimodal dialogues for the Conversational Environment for Memory.
Evaluation Data Generation
Mem-Gallery executes Evaluation Data Generation by prompting LLMs and humans to create QA pairs aligned with the Evaluation Framework and annotated evidence clues.
Benchmark Evaluation
Mem-Gallery conducts Benchmark Evaluation where memory agents store conversations, retrieve with unified protocols, and are scored on F1, BLEU-1, EM, and LLM-Judge.
KEY CONTRIBUTIONS
New Scenario and Dataset
Mem-Gallery formulates multimodal long-term conversational memory as an evolving system and builds multi-session conversations with 240 sessions, 3,962 rounds, and 1,003 images.
Evaluation Framework
Mem-Gallery introduces an Evaluation Framework that structures 1,711 QA pairs into Memory Extraction and Adaptation, Memory Reasoning, and Memory Knowledge Management with explicit evidence clues.
Benchmark Takeaways
Mem-Gallery benchmarks thirteen memory systems and reveals that MuRAG reaches 0.6966 overall F1 while naive Full Memory multimodal storage performs 8.08 percent worse F1.
RESULTS
Overall F1
0.6966
+0.0992 over NaiveRAG
VS LLM-Judge
0.8856
vs NaiveRAG 0.7402
FR F1
0.6724
FR F1 +0.0852 over NaiveRAG 0.5852
TTL F1
0.8177
TTL F1 +0.1651 over NaiveRAG 0.6526
Mem-Gallery evaluates memory systems on its own multimodal long-term conversational benchmark, testing extraction, reasoning, and knowledge management. The 0.6966 overall F1 and 0.8856 LLM-Judge on visual-centric search show that explicit multimodal memory in MuRAG is crucial for Mem-Gallery tasks.
BENCHMARK
Mem-Gallery evaluates memory systems on its own multimodal long-term conversational benchmark, testing extraction, reasoning, and knowledge management. The 0.6966 overall F1 and 0.8856 LLM-Judge on visual-centric search show that explicit multimodal memory in MuRAG is crucial for Mem-Gallery tasks.
BENCHMARK
Overall F1 on Mem-Gallery across representative memory systems.
KEY INSIGHT
Mem-Gallery shows that Full Memory multimodal storage achieves only 0.3354 overall F1, which is 8.08 percent worse than Full Memory text at 0.3625.
This is surprising because adding images should help, but Mem-Gallery reveals that naive multimodal accumulation can crowd out useful text and inject irrelevant visual noise.
WHY IT MATTERS
Mem-Gallery unlocks a controlled way to stress-test multimodal long-term memory, including conflict detection, knowledge resolution, and answer refusal across 1,711 QA pairs.
Builders can now compare memory systems like MuRAG, A-Mem, and MemoryOS under identical multimodal conditions, making it practical to design agents with robust, efficient long-horizon memory.
Guilin Zhang, Wei Jiang et al.
· 2026
A-MAC scores candidate memories using Utility, Confidence, Novelty, Recency, and Type Prior combined by a learned linear admission policy with Algorithm 1 A-MAC Memory Admission. On the LoCoMo benchmark, A-MAC achieves F1 0.583 and 2644 ms latency, improving F1 by 0.042 and reducing latency by 1187 ms compared to A-mem.
Robbyant Team, Zelin Gao et al.
arXiv 2026 · 2026
LingBot-World combines a Data Engine, Fundamental World Model, Action-Conditioned World Model, and Post-Training causal adaptation to turn a 28B-parameter video generator into a real-time interactive world simulator. On the VBench benchmark, LingBot-World achieves a dynamic degree of 0.8857 versus 0.7612 for Yume-1.5, while also improving imaging quality to 0.6683.
Manoj Madushanka Perera, Adnan Mahmood et al.
· 2026
AgenticAI-DialogGen chains ChatPreprocessor, KnowledgeExtractor, TopicAnalyzer, KnowledgeGraphBuilder, PersonaGenerator, DuelingChat Agent, ConversationValidator, ConversationRefiner, QAGeneration, and PostProcessing to turn raw multi-session chats into topic-guided, persona-grounded conversations with explicit short- and long-term memories. On the TGC / KG memory QA benchmark, Mistral-7B fine-tuned within AgenticAI-DialogGen achieves 87.36 F1, compared to GPT-4’s 83.77 F1 in a zero-shot setting on the same task.
Answers use this explainer on Memory Papers.
Checking…