AuthorsShiping Zhu, Yibo Yang, Zhengyang Wang et al.
TL;DR
H2HMem uses a human-in-the-loop multimodal dialogue pipeline plus a 9-task memory taxonomy to show best methods reach only 0.5757 overall LLM-as-Judge score.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing memory benchmarks are single-user, text-only, and human–assistant focused, missing multimodal, multi-party human–human interactions.
In these settings, agents must resolve anaphora, deixis, and conflicting participant information across sessions, but current evaluations cannot expose these failures.
HOW IT WORKS
H2HMem builds on Participant Profile Generation, Scenario Construction, Image Collection and Human Refinement, Image Captioning and Dialogue Generation, and Question-Answer Pairs Construction to create realistic multimodal dialogues.
Think of H2HMem like a scripted TV series: profiles define recurring characters, scenarios outline episodes, images set scenes, and QA tasks quiz long-term plot understanding.
This design lets H2HMem probe memory behaviors—recall, reasoning, and application—far beyond what a plain context window or simple RAG setup can test.
DIAGRAM
This diagram shows how H2HMem feeds multi-session multimodal dialogues into agents that store memories, retrieve them, and answer benchmark questions.
DIAGRAM
This diagram visualizes the five-stage human-in-the-loop pipeline H2HMem uses to build multimodal dyadic and multi-party conversations.
PROCESS
Participant Profile Generation
H2HMem first runs Participant Profile Generation to create structured personas that drive consistent behavior across sessions and anchor memory cues.
Scenario Construction
Next, Scenario Construction uses these profiles to sample topics and generate multi-session outlines plus image keywords that distribute facts over time.
Image Collection and Human Refinement
Image Collection and Human Refinement aligns retrieved or generated images with outlines, ensuring visual evidence supports later memory questions.
Question-Answer Pairs Construction
Finally, Question-Answer Pairs Construction targets recall, reasoning, and application, turning dialogues into graded memory tasks for H2HMem.
KEY CONTRIBUTIONS
Human-to-Human Multimodal Memory Benchmark
H2HMem defines a unified benchmark over dyadic and multi-party conversations, totaling 25 dialogues, 309 sessions, 7,078 rounds, and 1,300 images.
Human-in-the-loop Dataset Construction Pipeline
H2HMem introduces a five-stage pipeline with Participant Profile Generation and Scenario Construction to safely synthesize realistic, privacy-preserving multimodal interactions.
Comprehensive Memory Evaluation Taxonomy
H2HMem designs nine tasks across Memory Recall, Memory Reasoning, and Memory Application, yielding 2,236 QA pairs with 618 image-grounded questions.
RESULTS
Overall LLM-as-Judge score
0.5757
+0.1188 over Full (Text) on H2HMem weighted D&M
Memory Recall LLM-as-Judge
0.6652
vs NaiveRAG 0.5181 on H2HMem recall tasks
Memory Reasoning LLM-as-Judge
0.4572
tests MCR, RET, TR difficulty on H2HMem
Memory Application LLM-as-Judge
0.6045
shows TTL, CD, AR remain challenging on H2HMem
On the H2HMem benchmark, GPT-4.1-Nano with A-Mem achieves an overall weighted LLM-as-Judge score of 0.5757 across recall, reasoning, and application tasks. This demonstrates that even advanced memory systems struggle to integrate multimodal, multi-participant information over long horizons.
BENCHMARK
On the H2HMem benchmark, GPT-4.1-Nano with A-Mem achieves an overall weighted LLM-as-Judge score of 0.5757 across recall, reasoning, and application tasks. This demonstrates that even advanced memory systems struggle to integrate multimodal, multi-participant information over long horizons.
BENCHMARK
LLM-as-Judge overall scores on H2HMem for different memory methods.
KEY INSIGHT
On H2HMem, the best overall LLM-as-Judge score is only 0.5757, even though A-Mem reaches 0.6652 on Memory Recall alone.
This is surprising because many assume better retrieval solves memory, but H2HMem shows reasoning and conflict handling remain weak despite strong recall.
WHY IT MATTERS
H2HMem gives researchers a controlled way to stress-test multimodal, multi-party memory across recall, reasoning, and application with realistic human–human dialogues.
With H2HMem, builders can now benchmark observer-style agents for meetings, clinics, and chats under conditions that were previously unmeasured and easy to overestimate.
Jingbiao Mei, Jinghong Chen et al.
arXiv 2026 · 2026
ATM-Bench structurally evaluates long-term multimodal personal memory using Memory Ingestion, Retrieval, and Answer Generation with Schema-Guided Memory and Descriptive Memory variants. On ATM-Bench-Hard, Oracle with SGM reaches 47.3% QS while the best full system stays under 20% accuracy, revealing a large gap.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Answers use this explainer on Memory Papers.
Checking…