AuthorsJingbo Yang, Kwei-Herng Lai, Xiaowen Wang et al.
TL;DR
GroupMemBench uses graph-grounded multi-party conversation synthesis plus adversarial queries to show leading agent memories peak at only 46.01% accuracy, often matched by BM25.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
GroupMemBench shows that even the strongest agent memory system reaches only 46.01% average accuracy, with knowledge update at 27.10% and term ambiguity at 37.74%.
These failures mean LLM agents in group conversations mis-track speaker-grounded beliefs and audience-adapted language, producing wrong or generic answers for different users in the same shared thread.
HOW IT WORKS
GroupMemBench builds multi-party conversations using Graph-grounded Message Synthesis, Path-Based Context Sampling, Message Generation, Adversarial Question Synthesis, and a Solve–Judge–Refine Loop to expose group memory failures.
You can think of GroupMemBench as a structured card catalog for conversations, where each message is filed by domain, topic, phase, user, and thread before being attacked by tailored questions.
This design lets GroupMemBench probe dynamics, speaker grounding, and lexical shifts that a plain context window or dyadic memory benchmark cannot reveal.
DIAGRAM
This diagram shows how GroupMemBench synthesizes realistic multi-party workplace threads from a typed graph before any questions are generated.
DIAGRAM
This diagram shows how GroupMemBench builds hard questions and only keeps those that defeat a competent retrieval baseline.
PROCESS
Graph-grounded Message Synthesis
GroupMemBench first uses Graph-grounded Message Synthesis to create multi-party conversations over domains, topics, phases, and users, preserving reply structure and personas.
Path-Based Context Sampling
Next, Path-Based Context Sampling walks the graph to pick authors, recipients, and phase context, inducing channel posts, threaded replies, and cross-project bridges.
Adversarial Question Synthesis
Adversarial Question Synthesis then selects anchor messages and entities to draft questions across six categories, binding each query to a specific asking user.
Solve–Judge–Refine Loop
Finally, the Solve–Judge–Refine Loop iteratively tests each question against a retrieval baseline, refining it until the baseline fails, and then adds it to GroupMemBench.
KEY CONTRIBUTIONS
Defining group-memory evaluation axes
GroupMemBench formalizes group-memory evaluation along dynamics, speaker grounding, and audience adaptation, and constructs 120K turns over 123 topics with 2.7M tokens.
Graph-grounded multi-party synthesis
GroupMemBench introduces Graph-grounded Message Synthesis and Path-Based Context Sampling to generate realistic threaded multi-user conversations with per-user personas and Theory-of-Mind.
Adversarial multi-category memory benchmark
GroupMemBench builds adversarial queries across six categories and shows leading systems, including Hindsight and HippoRAG, reach only 46.01% and 39.72% average accuracy respectively.
RESULTS
Average accuracy
46.01%
+2.79 over BM25
Knowledge Update
27.10%
+1.87 over BM25
Term Ambiguity
37.74%
+23.59 over BM25
Abstention
77.06%
-0.92 vs BM25
On GroupMemBench, which tests multi-hop, updates, ambiguity, user-implicit, temporal, and abstention, Hindsight achieves 46.01% average accuracy while BM25 attains 43.22%. This MAIN_RESULT proves that GroupMemBench exposes gaps where sophisticated agent memories barely surpass a simple lexical retriever.
BENCHMARK
On GroupMemBench, which tests multi-hop, updates, ambiguity, user-implicit, temporal, and abstention, Hindsight achieves 46.01% average accuracy while BM25 attains 43.22%. This MAIN_RESULT proves that GroupMemBench exposes gaps where sophisticated agent memories barely surpass a simple lexical retriever.
BENCHMARK
Average accuracy (%) over all categories from Table 2.
KEY INSIGHT
GroupMemBench reveals that BM25, with zero LLM ingestion cost, reaches 43.22% average accuracy, matching or exceeding most agent memory systems.
This is surprising because we expect structured memory ingestion to help, yet GroupMemBench shows these pipelines often erase useful group structure and vocabulary instead of enhancing retrieval.
WHY IT MATTERS
GroupMemBench unlocks a way to stress-test LLM agents on speaker-conditioned retrieval, belief tracking, and role-specific vocabulary in realistic multi-party settings.
With GroupMemBench, builders can now design and validate memory systems that truly handle channels, threads, and teams, rather than assuming a single user and a flat conversation history.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…