AuthorsRyuichi Sumida, Koji Inoue, Tatsuya Kawahara
TL;DR
MemUse reframes memory evaluation around user-cued natural integration, showing a 71-point retrieval–integration gap (78.8% Direct QA vs 7.9% conversational reference).
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
MemUse shows existing-benchmark Direct QA accuracy climbing from 19.7% to 70.1% across seven memory conditions, yet user satisfaction remains unchanged.
This means long-context and RAG capacity boosts recall-on-demand, but diary-style human AI conversation still feels the same to users, breaking a core evaluation assumption.
HOW IT WORKS
MemUse builds on Summary-only, LC-k%, RAG-k%, and the MEMUSE benchmark to compare elicited retrieval against natural integration in a 4-month GPT-4.1-mini deployment.
Think of Direct QA as checking what’s in disk, while MEMUSE tests whether the assistant actually pulls the right files into RAM when the user casually references them.
This integration-aware framing lets MemUse reveal cases where information is retrievable but not conversationally used, something a plain context window and standard QA benchmarks cannot expose.
DIAGRAM
This diagram shows how MemUse scores a single user-cued memory moment under Natural Integration, Direct QA, and Reference using the same reconstructed context.
DIAGRAM
This diagram shows how MemUse runs the 4-month deployment, detects memory moments, and compares existing benchmarks with MEMUSE metrics.
PROCESS
Randomized memory condition assignment
MemUse randomly assigns each session to Summary-only, LC-10/50/100%, or RAG-10/50/100% so GPT-4.1-mini sees different prior-context provisioning regimes.
Importance scoring and context construction
MemUse uses a RoBERTa importance model trained on LUFY to select top k percent prior turns, then builds long-context or RAG pools on top of the shared summary.
User interaction and satisfaction rating
Users chat with the diary companion Luke, then MemUse collects a 1–7 satisfaction score per session to link memory behavior with user experience.
Memory moment detection and MEMUSE scoring
MemUse detects explicit probes and re provisions, reconstructs context, and scores Natural Integration, Direct QA, and Reference for each of the 72 MEMUSE instances.
KEY CONTRIBUTIONS
Four month randomized diary deployment
MemUse runs 40 users over 1,872 sessions and seven memory conditions, with Direct QA varying from 19.7% to 70.1% while satisfaction remains flat.
MEMUSE benchmark of real memory moments
MemUse introduces a 72 instance benchmark of user-cued memory moments, decomposed into 316 fact questions and scored for Natural Integration and Reference.
Evidence for retrieval integration dissociation
MemUse shows GPT-4.1-mini at LC-100% reaches 78.8% MEMUSE Direct QA but only 7.9% Reference, a 71 point gap between retrieval and conversational use.
RESULTS
Existing QA Acc
19.7% to 70.1%
+50.4 points over Summary-only when moving to LC-100% capacity
MEMUSE Natural Integration
22.2% at LC-100%
flat across Summary to LC-100% despite large retrieval gains
MEMUSE Direct QA
78.8% at LC-100%
71.0 points above MEMUSE Reference on the same instances
MEMUSE Reference
7.9% at LC-100%
facts almost never surface in natural replies despite high Direct QA
MemUse evaluates memory on long-term diary conversations and MEMUSE’s 72 user-cued moments, showing that capacity-driven gains on existing QA benchmarks do not translate into Natural Integration or satisfaction. The 71-point gap between MEMUSE Direct QA and Reference proves MemUse’s claim that retrieval and conversational integration are dissociable capabilities.
BENCHMARK
MemUse evaluates memory on long-term diary conversations and MEMUSE’s 72 user-cued moments, showing that capacity-driven gains on existing QA benchmarks do not translate into Natural Integration or satisfaction. The 71-point gap between MEMUSE Direct QA and Reference proves MemUse’s claim that retrieval and conversational integration are dissociable capabilities.
BENCHMARK
Scores on MEMUSE Direct QA and MEMUSE Reference for LC-100% compared to Summary-only.
KEY INSIGHT
MemUse shows that boosting Direct QA from 19.7% to 70.1% across memory conditions does not change user satisfaction at all.
This breaks the assumption that better benchmark recall automatically improves user experience, revealing that conversational integration, not elicited retrieval, is what users actually care about.
WHY IT MATTERS
MemUse gives the field a way to measure whether assistants naturally weave past conversations into replies, instead of just passing fact-check style QA tests.
With MemUse, builders can design and tune memory systems against integration-aware metrics and satisfaction, rather than chasing higher Direct QA scores that may never be felt by users.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…