AuthorsShweta Mishra, Shashank Mishra
TL;DR
MERIT benchmark + leak-checked dependent tasks + cost-aware metrics show memory lifts task success from 0.00 to up to 1.00 while some architectures collapse on updates.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
MERIT shows agents sometimes hold the correct fact in context yet fail to act on it, with Ignore Rate 0.45–0.53 across implementations.
This breaks tool-using agents on dependent tasks, where stale or unused memory leads to confidently wrong actions like incorrect refunds or misapplied configuration fixes.
HOW IT WORKS
MERIT’s core mechanism is a grid of memory conditions C0–C5, episodic tool-use tasks, controlled memory corruption, and cost-adjusted metrics like CAMU and MUR.
Think of MERIT as a lab where agent memory is swapped like hardware modules, while tasks and tools stay fixed, revealing how each memory behaves under stress.
This design lets MERIT expose when architectures like retrieval-only memory fail on updated facts, something a plain context window or conversational QA benchmark cannot reveal.
DIAGRAM
This diagram shows how MERIT structures arcs into plant, update, and probe episodes to enforce leak-free memory-dependent tasks.
DIAGRAM
This diagram shows how MERIT’s evaluation grid spans domains, difficulty tiers, memory conditions, and computes CAMU and Ignore Rate.
PROCESS
Environment and Arc Generation
MERIT creates arcs of 4–6 episodes in domains like customer support, IT operations, and personal assistant with deterministic seeded worlds.
Plant and Update Episodes
MERIT runs plant episodes to establish gold facts and optional update episodes to revise them, while users forbid acting on these facts.
Leak Check and Probe Episodes
MERIT’s leak check ensures gold values are absent from probe inputs and tools, then runs probe episodes that require memory to succeed.
Metric and Cost Computation
MERIT computes TSR, MUR, Ignore Rate, SMH, and CAMU over 23,440 scored episodes with full token and dollar metering.
KEY CONTRIBUTIONS
Task Suite with Controllable Memory Dependency
MERIT provides episodic tool-use tasks in three domains with a difficulty ladder ending in updated-fact recall, verified by an automated leak check over 23,440 episodes.
Controlled Memory Corruption and Stale Harm
MERIT injects stale, contradictory, and distractor records at known rates ρ∈{0.1,0.3} and measures Stale-Memory Harm, revealing up to +0.25 TSR drop for hybrid memory.
New Metrics: MUR, Ignore Rate, CAMU
MERIT introduces Memory Utilization Rate, Ignore Rate up to 0.53, and Cost-Adjusted Marginal Utility, showing full replay is 2.7–3.9× less economical than the best memory.
RESULTS
Dependent-task TSR (C0 floor)
0.000
-0.55 to -1.00 vs all memory conditions on easy tier
Dependent-task TSR (C2 hard)
0.30–0.95
spread across models and domains for embedding retrieval
Dependent-task TSR (C3 hard)
0.70–1.00
+0.28 to +0.44 over C2 across agent models
CAMU best condition
2.7–3.9× marginal utility per dollar
vs full replay which reaches only 1,222–1,741 pts/$
MERIT evaluates memory on episodic tool-use tasks in customer support, IT operations, and personal assistant domains. These results show MERIT’s overwrite-style memories maintain 0.70–1.00 TSR on updated facts while embedding retrieval varies between 0.30 and 0.95, and that cost-aware metrics reveal full replay is never the economical choice.
BENCHMARK
MERIT evaluates memory on episodic tool-use tasks in customer support, IT operations, and personal assistant domains. These results show MERIT’s overwrite-style memories maintain 0.70–1.00 TSR on updated facts while embedding retrieval varies between 0.30 and 0.95, and that cost-aware metrics reveal full replay is never the economical choice.
BENCHMARK
Task Success Rate on D1 commerce hard tier, comparing MERIT memory architectures.
KEY INSIGHT
MERIT finds that hybrid memory combining a fact store with retrieval scores worse than the fact store alone on updated facts, dropping to 0.50–0.80 vs 0.75–1.00.
This breaks the common assumption that adding retrieval to structured memory is always helpful, showing retrieval can re-import staleness the overwrite path had eliminated.
WHY IT MATTERS
MERIT gives practitioners a way to measure when memory actually changes agent actions, including Ignore Rate and cost-adjusted marginal utility across architectures.
Builders can now choose between replay, retrieval, summarization, and fact stores based on task success and dollars, rather than conversational recall scores alone, especially for updated operational facts.
Guilin Zhang, Wei Jiang et al.
· 2026
A-MAC scores candidate memories using Utility, Confidence, Novelty, Recency, and Type Prior combined by a learned linear admission policy with Algorithm 1 A-MAC Memory Admission. On the LoCoMo benchmark, A-MAC achieves F1 0.583 and 2644 ms latency, improving F1 by 0.042 and reducing latency by 1187 ms compared to A-mem.
Robbyant Team, Zelin Gao et al.
arXiv 2026 · 2026
LingBot-World combines a Data Engine, Fundamental World Model, Action-Conditioned World Model, and Post-Training causal adaptation to turn a 28B-parameter video generator into a real-time interactive world simulator. On the VBench benchmark, LingBot-World achieves a dynamic degree of 0.8857 versus 0.7612 for Yume-1.5, while also improving imaging quality to 0.6683.
Manoj Madushanka Perera, Adnan Mahmood et al.
· 2026
AgenticAI-DialogGen chains ChatPreprocessor, KnowledgeExtractor, TopicAnalyzer, KnowledgeGraphBuilder, PersonaGenerator, DuelingChat Agent, ConversationValidator, ConversationRefiner, QAGeneration, and PostProcessing to turn raw multi-session chats into topic-guided, persona-grounded conversations with explicit short- and long-term memories. On the TGC / KG memory QA benchmark, Mistral-7B fine-tuned within AgenticAI-DialogGen achieves 87.36 F1, compared to GPT-4’s 83.77 F1 in a zero-shot setting on the same task.
Answers use this explainer on Memory Papers.
Checking…