AuthorsTiantian Chen, Jiaqi Lu, Ying Shen, Lin Zhang
TL;DR
ES-MemEval integrates QA, summarization, and dialogue generation over EvoEmo’s 18-user, 401-session corpus to expose long-term memory limits in emotional support agents, with RAG boosting Mistral-24B QA F1 from 15.5% to 18.8%.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing long-term dialogue benchmarks “primarily focus on static and explicit fact retrieval,” failing to evaluate agents when user information is implicit and evolving.
In online emotional support scenarios, this causes agents to miss fragmented disclosures and evolving user states, leading to broken personalization and trust-damaging hallucinations.
HOW IT WORKS
ES-MemEval combines Question Answering, Summarization, and Dialogue Generation over EvoEmo’s User Profile Construction, Event Timeline Expansion, and Chat Data Generation pipeline to probe five memory abilities.
You can think of EvoEmo as a long-term “user disk” and ES-MemEval as RAM tests that repeatedly read, summarize, and converse over that disk.
This design lets ES-MemEval stress-test information extraction, temporal reasoning, conflict detection, abstention, and user modeling in ways a plain context window or short QA benchmark cannot.
DIAGRAM
This diagram shows how EvoEmo constructs long-term emotional support conversations via user profiles, event timelines, and chat generation.
DIAGRAM
This diagram shows how ES-MemEval evaluates different LLM paradigms with and without RAG across QA, summarization, and dialogue generation.
PROCESS
User Profile Construction
ES-MemEval relies on EvoEmo’s User Profile Construction to define 18 virtual users with demographics, social relationships, and core beliefs for realistic personalization.
Event Timeline Expansion
Event Timeline Expansion uses GPT-4o plus human review to build temporally and causally structured event sequences, averaging 24.8 events per user.
Chat Data Generation
Chat Data Generation conditions GPT-4o on profiles, timelines, and prior summaries to create multi-session emotional support dialogues with 401 sessions and 13.3K tokens per conversation.
Evaluation Task Formulation
Evaluation Task Formulation turns these dialogues into QA, summarization, and dialogue generation sets that probe information extraction, temporal reasoning, conflict detection, abstention, and user modeling.
KEY CONTRIBUTIONS
EvoEmo Multi Session Dataset
ES-MemEval introduces EvoEmo with 18 conversations, 401 sessions, and an average 13,291.6 tokens per conversation to capture evolving user states in emotional support.
ES-MemEval Benchmark Design
ES-MemEval defines QA, summarization, and dialogue generation tasks that jointly evaluate information extraction, temporal reasoning, conflict detection, abstention, and user modeling.
Systematic LLM Paradigm Study
ES-MemEval benchmarks open-source long-context, commercial, and RAG-augmented LLMs, revealing RAG boosts Mistral-24B QA F1 from 15.5% to 18.8% and summarization F1 from 26.8% to 48.1%.
RESULTS
QA F1 Score
18.8%
+3.3 points over Mistral-24B base on ES-MemEval QA
QA LLM-as-Judge
1.27
+0.26 over Mistral-24B base on 0–2 semantic consistency
Summarization Event F1
48.1%
+21.3 points over Mistral-24B base on event alignment
Summarization LLM Score
2.79
+1.34 over Mistral-24B base on 0–5 summary quality
These metrics come from ES-MemEval’s QA and summarization tables on EvoEmo, which test long-term memory over multi-session emotional support. The gains show ES-MemEval exposes how RAG and explicit histories change information extraction, temporal reasoning, and user modeling performance.
BENCHMARK
These metrics come from ES-MemEval’s QA and summarization tables on EvoEmo, which test long-term memory over multi-session emotional support. The gains show ES-MemEval exposes how RAG and explicit histories change information extraction, temporal reasoning, and user modeling performance.
BENCHMARK
Overall F1 Score (%) on ES-MemEval QA task.
BENCHMARK
Event-based F1 (%) on ES-MemEval summarization benchmark.
KEY INSIGHT
ES-MemEval shows GPT-3.5-turbo scores 2.95 on long-term memory without any history, yet only 0.73 on QA LLM-as-Judge with full context.
This is surprising because it reveals that high perceived memory in dialogue can coexist with weak factual consistency, breaking the assumption that fluent personalization implies accurate recall.
WHY IT MATTERS
ES-MemEval unlocks a way to stress-test long-term emotional support agents across QA, summarization, and live dialogue using the same EvoEmo trajectories.
Builders can now design and compare memory–retrieval integrations that explicitly target information extraction, temporal reasoning, conflict detection, abstention, and user modeling in multi-session support systems.
Jingbiao Mei, Jinghong Chen et al.
arXiv 2026 · 2026
ATM-Bench structurally evaluates long-term multimodal personal memory using Memory Ingestion, Retrieval, and Answer Generation with Schema-Guided Memory and Descriptive Memory variants. On ATM-Bench-Hard, Oracle with SGM reaches 47.3% QS while the best full system stays under 20% accuracy, revealing a large gap.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Answers use this explainer on Memory Papers.
Checking…