AuthorsJingbiao Mei, Jinghong Chen, Guangyu Yang et al.
TL;DR
ATM-Bench + Schema-Guided Memory (SGM) lifts Oracle QS on ATM-Bench-Hard from 25.6% to 47.3% (+21.7 points) while exposing <20% accuracy for current systems.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
ATM-Bench shows that state-of-the-art memory systems attain under 20% accuracy on the challenging ATM-Bench-Hard split.
When long-term multimodal, multi-source memories and personalized references are involved, assistants mis-ground answers, mishandle updates, and frequently hallucinate.
HOW IT WORKS
ATM-Bench formalizes personal memory QA into Memory Ingestion, Retrieval, and Answer Generation, contrasting Descriptive Memory and Schema-Guided Memory representations.
Think of ATM-Bench as a personal data card catalog: SGM turns scattered images, videos, and emails into well-labeled cards instead of one long narrative.
This structured design lets ATM-Bench test multi-source grounding, temporal updates, and referential reasoning that a plain context window cannot reliably capture.
DIAGRAM
This diagram shows how ATM-Bench queries move through Memory Ingestion, Retrieval, and Answer Generation during evaluation.
DIAGRAM
This diagram shows how ATM-Bench computes QS, Recall@10, and Joint@10 for each memory system.
PROCESS
Memory Ingestion
ATM-Bench runs Memory Ingestion to convert raw images, videos, and emails into Descriptive Memory or Schema-Guided Memory items for unified access.
Memory Organization
ATM-Bench optionally applies Piled Memory or Linked Memory organization, using graph link functions and update functions to relate and refine items.
Retrieval
ATM-Bench uses embedding-based Retrieval with all-MiniLM-L6-v2 or Qwen3-Embedding to select top k evidence items for each query.
Answer Generation
ATM-Bench evaluates Answer Generation via single-pass RAG (ATM-RAG, Self-RAG) or agentic answerers (Mem0Agentic), then scores with Question Type Score.
KEY CONTRIBUTIONS
ATM-Bench Dataset
ATM-Bench introduces 12k memory items and 1,038 question–answer pairs with cross-validated evidence, spanning approximately four years of multimodal personal data.
Memory Ingestion Formalism
ATM-Bench formulates Memory Ingestion, Retrieval, and Answer Generation, enabling systematic comparison of Piled Memory versus Linked Memory organizations.
Schema-Guided Memory
ATM-Bench proposes Schema-Guided Memory, which raises Oracle QS on ATM-Bench-Hard from 25.6% with DM to 47.3% with SGM.
RESULTS
QS
51.0%
+9.0 points over ATM-RAG DM on ATM-Bench
R@10
68.7%
same retrieval budget k=10 across systems
Joint@10
48.6
+7.3 over ATM-RAG DM on ATM-Bench
Oracle QS Hard
47.3%
+21.7 over Oracle DM on ATM-Bench-Hard
ATM-Bench evaluates long-term personalized referential memory QA with QS, Recall@10, and Joint@10 on ATM-Bench and ATM-Bench-Hard. The 51.0% QS for ATM-RAG with Schema-Guided Memory on ATM-Bench and 47.3% Oracle QS on ATM-Bench-Hard show how structured memory changes the ceiling and floor for current systems.
BENCHMARK
ATM-Bench evaluates long-term personalized referential memory QA with QS, Recall@10, and Joint@10 on ATM-Bench and ATM-Bench-Hard. The 51.0% QS for ATM-RAG with Schema-Guided Memory on ATM-Bench and 47.3% Oracle QS on ATM-Bench-Hard show how structured memory changes the ceiling and floor for current systems.
BENCHMARK
QS score on ATM-Bench for ATM-RAG and related RAG baselines.
KEY INSIGHT
Even with gold evidence, GPT-5 only reaches 74.7% QS on ATM-Bench-Hard, while smaller Qwen3-VL-2B achieves just 34.1%.
This is surprising because frontier multimodal LLMs are strong on many QA benchmarks, yet ATM-Bench-Hard exposes unresolved challenges in multi-evidence personal memory reasoning.
WHY IT MATTERS
ATM-Bench unlocks a way to stress-test assistants on realistic long-term multimodal personal memories with explicit referential, temporal, and spatial grounding.
Builders can now design and compare memory architectures like Schema-Guided Memory, Linked Memory graphs, and agentic RAG workflows against a demanding, privacy-preserving benchmark.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…