AuthorsYiting Shen, Kun Li, Wei Zhou, Songlin Hu
TL;DR
Mem2ActBench builds long conversational Fact Evolution Chains and reverse-generated tool calls to show that oracle retrieval boosts F1 from 30.7 to 53.8 on memory-driven tasks.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing long term memory benchmarks mostly test explicit fact retrieval instead of memory driven task execution with tools.
Mem2ActBench shows that 91.3% of its 400 tool use tasks are strongly memory dependent, yet agents still misground parameters and misapply historical constraints.
HOW IT WORKS
Mem2ActBench builds on Heterogeneous Data Integration, Fact Extraction and Grouping, Memory Evolution Chain Construction, and Memory-anchored Q&A Construction to synthesize realistic long term sessions.
Think of Mem2ActBench as constructing a card catalog of evolving facts, then asking agents to fill tool forms using only the catalog rather than a short clipboard.
By enforcing reverse implicit query generation and leakage detection, Mem2ActBench enables testing memory application that a plain context window benchmark cannot capture.
DIAGRAM
This diagram shows how Mem2ActBench turns a memory evolution chain into underspecified user queries and grounded tool calls for evaluation.
DIAGRAM
This diagram shows how Mem2ActBench integrates ToolACE, BFCL, and Oasst1 into globally consistent memory evolution chains and memory dependent tasks.
PROCESS
Heterogeneous Data Integration
Mem2ActBench merges ToolACE and BFCL task oriented dialogues with Oasst1 conversational noise, creating long interrupted sessions for downstream Fact Extraction and Grouping.
Fact Extraction and Grouping
Mem2ActBench prompts an LLM to extract attribute fact source triples and clusters attributes with BERTopic, preparing consistent groups for Memory Evolution Chain Construction.
Memory Evolution Chain Construction
Mem2ActBench performs local conflict resolution and global evolution sequence construction using a dependency graph and modified Kahns algorithm to build a coherent Fact Evolution Chain.
Memory anchored Q and A Construction
Mem2ActBench selects target tools, anchors parameters to the memory chain, and applies reverse implicit query generation with leakage detection to produce 400 memory dependent tasks.
KEY CONTRIBUTIONS
Inference driven memory utilization benchmark design
Mem2ActBench formalizes inference driven long term memory utilization by combining Heterogeneous Data Integration and Memory Evolution Chain Construction into 2,029 dialogue sessions with evolving constraints.
Mem2ActBench dataset release
Mem2ActBench releases 400 memory dependent tool use tasks derived from 2,029 sessions, with human verification showing 91.3 percent of tasks require long term memory.
Comprehensive evaluation of memory frameworks
Mem2ActBench evaluates seven memory frameworks including LTMemory, A mem, Generative Agents, SCM, Langmem, MemTree, and Mem0, revealing a 23.1 F1 gap between passive and oracle retrieval.
RESULTS
F1
53.8
+23.1 over Hybrid passive retrieval at k=5
BLEU
53.7
vs Hybrid passive retrieval BLEU 29.7
Tool Accuracy
97.00
Mem0 with Qwen2.5 32B on Mem2ActBench
Memory Dependency Validity
91.3%
expert verified tasks requiring long term memory
Mem2ActBench evaluates memory grounded tool call generation with F1, BLEU, and Tool Accuracy on 400 tasks, showing oracle retrieval F1 53.8 versus best passive hybrid retrieval F1 30.7 and confirming strong memory dependence in 91.3 percent of tasks.
BENCHMARK
Mem2ActBench evaluates memory grounded tool call generation with F1, BLEU, and Tool Accuracy on 400 tasks, showing oracle retrieval F1 53.8 versus best passive hybrid retrieval F1 30.7 and confirming strong memory dependence in 91.3 percent of tasks.
BENCHMARK
Average F1 on Mem2ActBench across Qwen2.5 model sizes for representative memory frameworks.
KEY INSIGHT
Mem2ActBench shows that oracle retrieval boosts F1 from 30.7 to 53.8, a 23.1 point gain over the best passive hybrid retriever.
This is surprising because many assume larger Qwen2.5 backbones solve memory issues, yet Mem2ActBench reveals retrieval quality, not model scale, dominates performance.
WHY IT MATTERS
Mem2ActBench enables precise diagnosis of where long term memory pipelines fail, separating retrieval misses, mid context loss, and parameter grounding errors in tool use.
With Mem2ActBench, builders can now stress test agent memory managers on realistic, interruption heavy sessions and design retrieval and grounding strategies explicitly targeting mid context and default value failures.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…