AuthorsPeijun Qing, Fobo Shi, Soroush Vosoughi
TL;DR
UTILMEM uses evidence-sensitive question filtering plus adversarial distractor haystacks to show that NaiveRAG only reaches 59.3 NR with 49.2 DR despite strong factual recall.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing long-term memory benchmarks mostly test pointwise factual recall, not whether agents can use distributed, implicit, and noisy evidence across sessions.
UTILMEM shows that systems with strong recall still fail on dense multi-session reasoning, implicit retrieval, and distractor filtering, leading to degraded long-form guidance and plans.
HOW IT WORKS
UTILMEM builds on four core components: Evidence Synthesis, Question Generation, Evidence-Sensitivity Filtering, and Haystack Construction to create 1,717 utilization-focused instances.
You can think of UTILMEM as a memory stress lab: it loads the “disk” with realistic histories, then tests how well agents pull the right “cards” from a noisy catalog.
This design lets UTILMEM expose failures that a plain context window or recall-only benchmark cannot, especially under adversarial distractors and long-form composition.
DIAGRAM
This diagram shows how UTILMEM evaluates memory systems by comparing noisy-history answers against oracle-evidence answers with rubric LLM judges.
DIAGRAM
This diagram shows UTILMEM’s four-stage construction pipeline from source datasets to adversarial haystack histories.
PROCESS
Evidence Synthesis
UTILMEM groups source items into multi-session bundles and rewrites them into dialogues, ensuring persona consistency and deduplication across Evidence Synthesis.
Question Generation
UTILMEM runs Question Generation over each bundle to create deterministic, multi-session dependent questions that use retrieval-compatible phrasing tied to the conversation text.
Evidence-Sensitivity Filtering
UTILMEM applies Evidence-Sensitivity Filtering via sampled removal paths and dual LLM judges, keeping only questions where answer quality strictly degrades from E(1) to E(3).
Haystack Construction
UTILMEM performs Haystack Construction by synthesizing adversarial distractors, merging cross-domain sessions, and sorting them into full histories H that stress retrieval robustness.
KEY CONTRIBUTIONS
Benchmark for memory utilization
UTILMEM introduces 1,717 instances across five domains that jointly target dense multi-session reasoning, implicit retrieval, long-form composition, and distractor filtering.
Scalable synthesis pipeline
UTILMEM’s pipeline combines Evidence Synthesis, Question Generation, Evidence-Sensitivity Filtering, and Haystack Construction to produce evidence-sensitive, adversarially noisy tasks.
Empirical gap between access and use
UTILMEM shows systems like Mem0 and Mem0+Graph drop to 18.9 NR with 97.8 DR, revealing that strong factual recall does not guarantee robust memory utilization.
RESULTS
Normalized Robustness
59.3 NR
+41.9 over Mem0+Graph
Degradation Rate
49.2 DR
-48.6 vs Mem0+Graph
Instances
1,717 tasks
UTILMEM benchmark size across five domains
Domains
5 domains
Learning, Finance, Mental, Fitness, Document
UTILMEM evaluates memory utilization over 1,717 tasks spanning Learning Support, Finance Guidance, Mental Wellness, Fitness Coaching, and Document Analysis. The main result shows NaiveRAG + Qwen3-Embedding-8B retains 59.3% of the oracle robustness score with 49.2% of instances degrading, while Mem0+Graph collapses to 18.9 NR and 97.8 DR, highlighting the cost of compression-heavy memory.
BENCHMARK
UTILMEM evaluates memory utilization over 1,717 tasks spanning Learning Support, Finance Guidance, Mental Wellness, Fitness Coaching, and Document Analysis. The main result shows NaiveRAG + Qwen3-Embedding-8B retains 59.3% of the oracle robustness score with 49.2% of instances degrading, while Mem0+Graph collapses to 18.9 NR and 97.8 DR, highlighting the cost of compression-heavy memory.
BENCHMARK
Average Normalized Robustness (NR) across all UTILMEM domains for retrieval-only and structured memory systems.
KEY INSIGHT
UTILMEM shows that NaiveRAG with simple turn-level retrieval reaches 59.3 NR, while graph-augmented Mem0+Graph drops to 18.9 NR despite added structure.
This is counterintuitive because builders often assume more structured memory and compression help, but UTILMEM reveals that write-time fact extraction can destroy crucial utilization details.
WHY IT MATTERS
UTILMEM unlocks a way to stress-test long-term agents on evidence integration, not just recall, under realistic multi-session noise and adversarial distractors.
With UTILMEM, builders can now diagnose whether new memory architectures truly help agents use past interactions for planning, analysis, and coaching, rather than just storing more facts.
Guilin Zhang, Wei Jiang et al.
· 2026
A-MAC scores candidate memories using Utility, Confidence, Novelty, Recency, and Type Prior combined by a learned linear admission policy with Algorithm 1 A-MAC Memory Admission. On the LoCoMo benchmark, A-MAC achieves F1 0.583 and 2644 ms latency, improving F1 by 0.042 and reducing latency by 1187 ms compared to A-mem.
Robbyant Team, Zelin Gao et al.
arXiv 2026 · 2026
LingBot-World combines a Data Engine, Fundamental World Model, Action-Conditioned World Model, and Post-Training causal adaptation to turn a 28B-parameter video generator into a real-time interactive world simulator. On the VBench benchmark, LingBot-World achieves a dynamic degree of 0.8857 versus 0.7612 for Yume-1.5, while also improving imaging quality to 0.6683.
Manoj Madushanka Perera, Adnan Mahmood et al.
· 2026
AgenticAI-DialogGen chains ChatPreprocessor, KnowledgeExtractor, TopicAnalyzer, KnowledgeGraphBuilder, PersonaGenerator, DuelingChat Agent, ConversationValidator, ConversationRefiner, QAGeneration, and PostProcessing to turn raw multi-session chats into topic-guided, persona-grounded conversations with explicit short- and long-term memories. On the TGC / KG memory QA benchmark, Mistral-7B fine-tuned within AgenticAI-DialogGen achieves 87.36 F1, compared to GPT-4’s 83.77 F1 in a zero-shot setting on the same task.
Answers use this explainer on Memory Papers.
Checking…