AuthorsZhen Tao, Jinxiang Zhao, Peng Liu et al.
TL;DR
MemConflict formalizes dynamic, static, and conditional memory conflicts and shows six long-term memory systems have divergent strengths, with MemOS reaching 0.5539 average Answer Accuracy.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing evaluations mainly assess outcome-level performance or temporal updating, providing limited insight into how systems retrieve and rank valid memories under conflicting alternatives.
Long-term conversational agents and memory systems can return correct answers while relying on outdated, contradictory, or inapplicable memories, undermining reliability and governance.
HOW IT WORKS
MemConflict’s core mechanism combines user profile initialization, timeline simulation and conflict construction, multi-session dialogue generation, query and label construction, and a two-level evaluation protocol.
You can think of MemConflict like a controlled lab for memory: it builds a user’s life log, then injects conflicting "cards" into a catalog and checks which card each system pulls and ranks.
This design lets MemConflict test what conflict-aware retrieval and ranking enable that a plain context window cannot, separating missing memories, misranking, and ineffective use of retrieved evidence.
DIAGRAM
This diagram shows how MemConflict issues queries at conflict-bearing sessions and evaluates both answers and supporting-memory retrieval.
DIAGRAM
This diagram shows how MemConflict constructs the benchmark and runs controlled evaluations with different conflict types and factors.
PROCESS
Timeline Simulation and Conflict Construction
MemConflict simulates the user’s timeline and inserts dynamic updates Δ_t, static information P_t, conditional bindings B_t, and distractors E_t into specific sessions.
Multi-Session Dialogue Generation
MemConflict generates a synopsis z_t for each session and expands it into dialogue G_t, ensuring Δ_t, P_t, B_t, and E_t are naturally verbalized and correctly attributed.
Query and Label Construction
After a conflict-bearing session, MemConflict builds conflict-targeted query q_j over history prefix H_t and defines gold answer y* and gold memory m* for the target attribute.
Two-Level Evaluation Protocol
MemConflict feeds H_t and q_j to each memory system, then computes Answer Accuracy, UOCS or CRS, and memory metrics SEH@K and SRS from the returned answers and memories.
KEY CONTRIBUTIONS
Diagnostic framework for memory conflicts
MemConflict formalizes dynamic, static, and conditional conflicts along temporal, factual, and contextual validity, and evaluates six systems with metrics like AA, UOCS, CRS, SEH@K, and SRS.
Controlled multi-session benchmark construction
MemConflict constructs long-horizon histories averaging 52.33 sessions and 203,910.83 tokens per instance, with cross-session conflicts and 32.83 semantically similar distractors per user.
Two-level evaluation of answers and memories
MemConflict combines black-box Answer Accuracy with white-box Support Evidence Hit@3 and Support Rank Score, revealing uneven strengths across dynamic, static, and conditional conflicts.
RESULTS
Average Answer Accuracy
0.5539
+0.2717 over LangMem
Support Evidence Hit@3
0.6710
MemOS vs Letta at 0.6202
Support Rank Score
0.5879
MemOS vs A-Mem at 0.4828
Dynamic AA range
0.1224–0.4966
shows large spread across six systems
On the MemConflict benchmark with 124.33 questions per instance, MemOS achieves 0.5539 average Answer Accuracy, 0.6710 SEH@3, and 0.5879 SRS across conflict types. These numbers show MemConflict can separate systems like MemOS from LangMem at 0.2822 AA and Mem0 at 0.3612 AA, exposing conflict-specific reliability gaps.
BENCHMARK
On the MemConflict benchmark with 124.33 questions per instance, MemOS achieves 0.5539 average Answer Accuracy, 0.6710 SEH@3, and 0.5879 SRS across conflict types. These numbers show MemConflict can separate systems like MemOS from LangMem at 0.2822 AA and Mem0 at 0.3612 AA, exposing conflict-specific reliability gaps.
BENCHMARK
Average Answer Accuracy (AA) over dynamic, static, and conditional conflicts.
KEY INSIGHT
MemConflict shows that static conflicts yield the lowest Answer Accuracy, even though the underlying facts never change and should be easier to preserve.
This is counterintuitive because many builders assume temporal updates are hardest, but MemConflict reveals that later false mentions can more easily override stable user information than true updates.
WHY IT MATTERS
MemConflict unlocks conflict-aware, retrieval-aware evaluation of long-term memory systems, letting you see whether valid memories are preserved, retrieved, and ranked correctly under competition.
With MemConflict, builders can now design and tune memory modules for temporal validity, factual correctness, and contextual applicability separately, instead of relying on coarse answer-only benchmarks.
Guilin Zhang, Wei Jiang et al.
· 2026
A-MAC scores candidate memories using Utility, Confidence, Novelty, Recency, and Type Prior combined by a learned linear admission policy with Algorithm 1 A-MAC Memory Admission. On the LoCoMo benchmark, A-MAC achieves F1 0.583 and 2644 ms latency, improving F1 by 0.042 and reducing latency by 1187 ms compared to A-mem.
Robbyant Team, Zelin Gao et al.
arXiv 2026 · 2026
LingBot-World combines a Data Engine, Fundamental World Model, Action-Conditioned World Model, and Post-Training causal adaptation to turn a 28B-parameter video generator into a real-time interactive world simulator. On the VBench benchmark, LingBot-World achieves a dynamic degree of 0.8857 versus 0.7612 for Yume-1.5, while also improving imaging quality to 0.6683.
Manoj Madushanka Perera, Adnan Mahmood et al.
· 2026
AgenticAI-DialogGen chains ChatPreprocessor, KnowledgeExtractor, TopicAnalyzer, KnowledgeGraphBuilder, PersonaGenerator, DuelingChat Agent, ConversationValidator, ConversationRefiner, QAGeneration, and PostProcessing to turn raw multi-session chats into topic-guided, persona-grounded conversations with explicit short- and long-term memories. On the TGC / KG memory QA benchmark, Mistral-7B fine-tuned within AgenticAI-DialogGen achieves 87.36 F1, compared to GPT-4’s 83.77 F1 in a zero-shot setting on the same task.
Answers use this explainer on Memory Papers.
Checking…