AuthorsHaonan Bian, Zhiyuan Yao, Sen Hu et al.
TL;DR
RealMem uses a three-stage Project Foundation Construction + Multi-Agent Dialogue Generation + Memory and Schedule Management pipeline to expose a large Oracle gap (QA 0.804 vs 0.567 for MemoryOS).
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing memory benchmarks focus on casual or task-oriented dialogue, ignoring long-term project-oriented interactions with evolving goals and interleaved sessions.
In these settings, even advanced memory systems like MemoryOS and Graph Memory cannot close the gap to the Oracle QA Score of 0.804, leading to broken project states and inconsistent assistance.
HOW IT WORKS
RealMem centers on Project Foundation Construction, Multi-Agent Dialogue Generation, and Memory and Schedule Management to build coherent, multi-session project dialogues with dynamic attributes.
You can think of RealMem like a project management OS: blueprints define the plan, agents simulate users and assistants, and memory modules act as a shared timeline and notebook.
This design lets RealMem stress-test proactive alignment, temporal reasoning, and fine-grained state tracking in ways a plain context window or static QA benchmark cannot.
DIAGRAM
This diagram shows how RealMem simulates each dialogue session via multi-agent interaction and then extracts and updates memory and schedules.
DIAGRAM
This diagram shows how RealMem evaluates different memory systems using retrieval metrics and QA-based generation under memory-only and session-based contexts.
PROCESS
Project Foundation Construction
RealMem initializes user personas, project goals, and project attributes, then builds a Project Blueprint, Event List, and Session Summaries to scaffold long-term interactions.
Multi-Agent Dialogue Generation
RealMem uses a User Agent and Assistant Agent, guided by the interleaved session queue, static project background, memory points, and schedules to simulate realistic multi-session dialogues.
Memory and Schedule Management
RealMem employs a Memory Extraction Agent, Schedule Agent, and Deduplication Agent to convert dialogues into structured memory points and schedule tables while cleaning redundant entries.
Hybrid Evaluation on RealMem
RealMem feeds queries to systems like Graph Memory, Mem0, A-mem, and MemoryOS, measuring Recall@k, NDCG@k, Mem Recall, Mem Helpful, and QA Score under memory-only and session-based contexts.
KEY CONTRIBUTIONS
RealMem benchmark for long-term project-oriented interaction
RealMem provides over 2,000 cross-session dialogues across eleven scenarios, with Project Foundation Construction ensuring coherent project skeletons and dynamic project attributes for realistic memory stress tests.
Three-stage synthesis pipeline
RealMem formalizes Project Foundation Construction, Multi-Agent Dialogue Generation, and Memory and Schedule Management into a closed-loop pipeline where memory points and schedules evolve alongside dialogue.
Comprehensive evaluation of agent memory systems
RealMem systematically evaluates Mem0, A-mem, MemoryOS, and Graph Memory using QA Score, Mem Recall, Mem Helpful, Recall@k, and NDCG@k, revealing large gaps to the Oracle (Recall 0.993).
RESULTS
QA-4o
0.567 score
+0.070 over Mem0 in memory-only setting
Mem Recall
0.606 score
+0.077 over Mem0 in memory-only setting
R@10
0.6344 score
Graph Memory vs Mem0 on session-level retrieval
Avg. Retrieve Memory Time (s)
0.151 seconds
MemoryOS retrieval latency vs 0.013 for A-mem
On RealMem, MemoryOS achieves QA-4o 0.567 and Mem Helpful 0.606 in the memory-only setting, while Graph Memory reaches R@10 0.6344 on session-level retrieval. These results show RealMem can separate retrieval precision, generation quality, and efficiency for long-term project memory systems.
BENCHMARK
On RealMem, MemoryOS achieves QA-4o 0.567 and Mem Helpful 0.606 in the memory-only setting, while Graph Memory reaches R@10 0.6344 on session-level retrieval. These results show RealMem can separate retrieval precision, generation quality, and efficiency for long-term project memory systems.
BENCHMARK
QA-4o scores with Value = Memory on RealMem.
KEY INSIGHT
RealMem shows that A-mem achieves the highest Recall@20 (0.7235) but lags in NDCG and QA performance compared to Graph Memory and MemoryOS.
This is surprising because developers often assume higher recall is always better, yet RealMem reveals that noisy high-recall retrieval can hurt downstream QA more than precise ranking.
WHY IT MATTERS
RealMem unlocks a way to rigorously test whether memory systems can handle dynamic project states, proactive alignment, and temporal reasoning across fragmented sessions.
Builders can now benchmark agent memory architectures against realistic multi-project workflows, identifying whether improvements in retrieval, scheduling, or memory compression actually translate into better long-term collaboration.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…