AuthorsAnkit Goyal, Jaideep Ray
TL;DR
Does Your Agent’s Memory Survive a Model Upgrade? shows that fixed-schema KG-fixed memories migrate almost losslessly while NOTES and mixed RAG embeddings can lose up to 13.28pp and 7.0pp of accuracy.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Does Your Agent’s Memory Survive a Model Upgrade? shows that NOTES accuracy can change asymmetrically by +9.91 or −13.28 percentage points after a writer swap.
When NOTES or mixed RAG embeddings drift, an upgraded agent silently forgets past experience, degrading retrieval and answer accuracy even though the memory store appears intact.
HOW IT WORKS
Does Your Agent’s Memory Survive a Model Upgrade? factorizes memory into LC-RAW, RAG, NOTES, and KG-fixed plus distinct writer W, reader R, and embedder E roles.
Think of LC-RAW as a full log on disk, RAG as an indexed card catalog, NOTES as compressed sticky notes, and KG-fixed as a structured database schema.
This separation lets Does Your Agent’s Memory Survive a Model Upgrade? pinpoint whether failures arise during writing, retrieval, or reading, something a plain context window cannot reveal.
DIAGRAM
This diagram shows how Does Your Agent’s Memory Survive a Model Upgrade? routes a query through inherited memory when the reader model or embedding index has changed.
DIAGRAM
This diagram shows how Does Your Agent’s Memory Survive a Model Upgrade? generates synthetic histories, builds four memory formats, and runs migration, embedding, repair, and diagnostic tests.
PROCESS
Synthetic histories with randomized codes
Does Your Agent’s Memory Survive a Model Upgrade? creates 48 scripted histories with randomized answer codes and 160 questions to avoid pretraining leakage and enable exact scoring.
Memory formats and their trade offs
Does Your Agent’s Memory Survive a Model Upgrade? converts each history into LC-RAW, RAG chunks, NOTES summaries under a byte budget, and KG-fixed subject predicate object claims.
Measure the migration
Does Your Agent’s Memory Survive a Model Upgrade? computes Retained Performance After Swap and Cost to Recover by swapping writers and readers and varying embedding and repair settings.
Agent memory migration tests
Does Your Agent’s Memory Survive a Model Upgrade? runs writer swap, embedding upgrade, repair with or without raw history, and diagnostic evidence injections to localize where information is lost.
KEY CONTRIBUTIONS
Controlled comparison of four memory formats
Does Your Agent’s Memory Survive a Model Upgrade? compares LC-RAW, RAG, NOTES, and KG-fixed under identical histories, showing KG-fixed accuracy changes by only +0.0004 ± 0.0020 after a writer swap.
Separate measurements for each migration direction
Does Your Agent’s Memory Survive a Model Upgrade? reports asymmetric NOTES portability, with Llama reading Qwen notes gaining +9.91pp but Qwen reading Llama notes losing −13.28pp.
Tests of model swaps, mixed indexes, and repair
Does Your Agent’s Memory Survive a Model Upgrade? shows a 50 or 50 mixed index recovers only 4.96pp of an 11.90pp re embedding gain and that raw history repair beats store only repair by 8.90pp.
RESULTS
KG fixed own store accuracy
0.9878
+0.1422 over NOTES Qwen own store
NOTES writer swap drop
13.28 pp
Qwen own store 0.4719 to 0.3391 with Llama notes
Embedding full re embed gain
11.90 pp
over old embedding index accuracy 0.4257
Mixed index partial gain
4.96 pp
forfeits about 7.0pp of potential re embedding improvement
On 48 synthetic histories with 160 questions each, Does Your Agent’s Memory Survive a Model Upgrade? measures exact match accuracy under controlled migrations. The results show KG-fixed is highly portable while NOTES and mixed RAG embeddings can lose large portions of performance.
BENCHMARK
On 48 synthetic histories with 160 questions each, Does Your Agent’s Memory Survive a Model Upgrade? measures exact match accuracy under controlled migrations. The results show KG-fixed is highly portable while NOTES and mixed RAG embeddings can lose large portions of performance.
BENCHMARK
Exact match accuracy across memory formats before and after writer swap.
KEY INSIGHT
Does Your Agent’s Memory Survive a Model Upgrade? finds that a 50 or 50 mixed embedding index recovers only 4.96pp of an 11.90pp full re embedding gain.
This is surprising because identical 1024 dimensional vectors suggest compatibility, yet mixing spaces silently breaks retrieval, contradicting the intuition that gradual embedding migrations are harmless.
WHY IT MATTERS
Does Your Agent’s Memory Survive a Model Upgrade? gives builders concrete metrics like RPAS and CTR to reason about memory portability across LC-RAW, RAG, NOTES, and KG-fixed.
Armed with these numbers, practitioners can design memory systems that survive model and embedding upgrades, retain raw histories for repair, and avoid silent failures from mixed embedding indexes.
Jingbiao Mei, Jinghong Chen et al.
arXiv 2026 · 2026
ATM-Bench structurally evaluates long-term multimodal personal memory using Memory Ingestion, Retrieval, and Answer Generation with Schema-Guided Memory and Descriptive Memory variants. On ATM-Bench-Hard, Oracle with SGM reaches 47.3% QS while the best full system stays under 20% accuracy, revealing a large gap.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Answers use this explainer on Memory Papers.
Checking…