AuthorsArseniy Varlamov, Rishat Zinnatullin, Elisei Rykov et al.
TL;DR
MemToC uses correctness-controlled tool returns plus cross-fitted SFT and DPO to raise correct-answer retention up to 31.6% without reducing 93.1% correct-tool following on Llama-3.1-8B-Instruct.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
MemToC shows that instruction-tuned models retain a verified-correct answer against an incorrect tool in only 6.5–17.1% of eligible cases.
Across five open-weight 7–9B models, tool returns strongly dominate parametric memory, causing harmful deference and silent conflicts in agentic tool-augmented LLMs.
HOW IT WORKS
MemToC centers on ToolHop pool construction and filtering, Distractor generation, Correctness-conditioned evaluation, and Annotator quality control to build 6,504 controlled episodes.
Think of MemToC like a lab test bench: verified answers are the ground truth, tools are noisy sensors, and parametric memory is a second sensor whose reliability is measured separately.
This design lets MemToC test correctness-conditioned arbitration that a plain context window cannot, distinguishing appropriate tool use from indiscriminate source preference across four source-correctness cases.
DIAGRAM
This diagram shows how MemToC runs closed-book and tool-conditioned passes, assigns source-correctness cases, and scores arbitration behavior.
DIAGRAM
This diagram shows how MemToC filters ToolHop chains, constructs distractors, and performs quality control to obtain 542 benchmark questions.
PROCESS
Closed-book baseline
MemToC prompts the model without tool access, collects the closed-book answer m, and compares it to the verified answer g to determine memory correctness.
Tool-conditioned pass
MemToC presents the same question with a controlled tool return r, including correct, near distractor, far distractor, error, or no-return conditions, and records the final response y.
Correctness-case assignment
MemToC assigns each episode to both-correct, memory-only, tool-only, or neither-correct cases by comparing m and r against g before scoring y.
Correctness-conditioned evaluation
MemToC scores retention, correct-tool following, incorrect-tool following, and tool-error abstention, producing metrics like conflict-resolved accuracy for each model and adaptation.
KEY CONTRIBUTIONS
MemToC benchmark
MemToC introduces 542 quality-controlled factual questions and 6,504 episodes with executable calls, verified answers, and controlled correct and incorrect returns to study arbitration across four source-correctness cases.
Correctness-conditioned evaluation
MemToC separates source correctness from source preference, showing instruction-tuned models retain correct memory only 6.5–17.1% of the time but follow correct tools in 86.0–93.1% of cases.
Cross-fitted SFT and DPO mitigation
MemToC uses chain-level cross-fitting over ToolHop to compare SFT and DPO, finding both objectives improve retention on the same two backbones while preserving correct-tool following.
RESULTS
Correct-answer retention
31.6%
+14.5 points over Llama-3.1-8B-Instruct baseline 17.1%
Correct-tool following
93.1%
baseline Llama-3.1-8B-Instruct, SFT keeps 92.8% and DPO keeps 92.1%
Incorrect-tool following
78.4%
ranges 78.4–86.0% across instruction-tuned models when both sources are wrong
Tool-error abstention
99.5%
gemma-2-9b-it abstains on 99.5% structured errors vs 73.9% for Mistral-7B-Instruct-v0.3
MemToC evaluates arbitration on ToolHop-derived questions, measuring how often models retain correct memory versus follow tools. These numbers show MemToC can both expose extreme tool dominance and quantify selective gains from SFT and DPO without sacrificing correct-tool following.
BENCHMARK
MemToC evaluates arbitration on ToolHop-derived questions, measuring how often models retain correct memory versus follow tools. These numbers show MemToC can both expose extreme tool dominance and quantify selective gains from SFT and DPO without sacrificing correct-tool following.
BENCHMARK
Correct-answer retention on MemToC conflict cases for different adaptations.
KEY INSIGHT
MemToC finds that instruction-tuned models repeat the tool return in 78.4–86.0% of cases where both memory and tool are wrong.
This is surprising because developers often assume tools add reliability, but MemToC shows models can over-defer to tools even when they provide clearly incorrect values.
WHY IT MATTERS
MemToC gives practitioners a way to measure correctness-conditioned arbitration, not just tool-use rates, across executable tools and parametric memory.
With MemToC, builders can design and fine-tune agents that selectively resist wrong tools while still following correct ones, and evaluate trade-offs in abstention and robustness across prompts and frames.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…