AuthorsDi Wu, Hongwei Wang, Wenhao Yu et al.
TL;DR
LongMemEval uses a three-stage indexing–retrieval–reading memory design with fact-augmented keys and time-aware queries to expose 30–60% accuracy drops in long-context LLMs on 115k-token chat histories.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
LongMemEval shows long-context LLMs drop 30%–60% accuracy on LONGMEMEVALS when reading full 115k token histories instead of only evidence sessions.
This means commercial chat assistants and long-context LLMs fail at personalized QA, breaking tasks like counseling, planning, and secretarial support over sustained interactions.
HOW IT WORKS
LongMemEval structures long-term memory into three stages, indexing, retrieval, and reading, controlled by four points: value, key, query, and reading strategy.
You can think of LongMemEval like a library: indexing is cataloging books, retrieval is searching the catalog, and reading is a researcher skimming and synthesizing notes.
By decomposing sessions into rounds, expanding keys with user facts, and using time-aware query expansion, LongMemEval enables behaviors that a plain context window cannot, especially for temporal reasoning and multi-session aggregation.
DIAGRAM
This diagram shows how LongMemEval feeds multi session chat histories into a memory augmented assistant during online evaluation.
DIAGRAM
This diagram shows how LongMemEval constructs attribute based questions, evidence sessions, and long chat histories for evaluation.
PROCESS
Indexing
LongMemEval runs indexing by converting each timestamped chat session into key value items, using session or round granularity and optional summaries or user facts.
Retrieval
LongMemEval performs retrieval with dense encoders over keys, optionally using fact augmented key expansion and time aware query expansion to narrow the search space.
Reading
LongMemEval invokes a reader LLM like GPT-4o or Llama 3.1 with Chain of Note and structured JSON format to extract notes and reason over retrieved items.
Question Answering
LongMemEval evaluates question answering by scoring assistant responses with a gpt 4o evaluator, measuring accuracy across information extraction, multi session reasoning, temporal reasoning, knowledge updates, and abstention.
KEY CONTRIBUTIONS
LongMemEval benchmark design
LongMemEval introduces 500 manually curated questions over about 50k sessions, covering information extraction, multi session reasoning, temporal reasoning, knowledge updates, and abstention with freely extensible histories up to 1.5M tokens.
Unified memory framework
LongMemEval formulates long term memory as three stages, indexing, retrieval, and reading, with four control points over value, key, query, and reading strategy that subsume nine existing memory augmented systems.
Memory design optimizations
LongMemEval proposes session decomposition into rounds, fact augmented key expansion, and time aware query expansion, improving recall@5 by up to 9.4% and boosting downstream accuracy by 5.4 absolute points.
RESULTS
Accuracy
0.9184
+0.3411 over Coze GPT-4o in commercial assistant comparison
Accuracy
0.5773
vs Offline Reading GPT-4o 0.9184 for ChatGPT memory mode
Accuracy
0.3299
Coze GPT-4o vs Offline Reading GPT-4o 0.9184 reveals 64% drop
Accuracy
0.6400
GPT-4o LONGMEMEVALS with Chain of Note vs 0.9240 oracle evidence only
LongMemEval reports these metrics on LONGMEMEVALS and a 97 question subset for commercial systems, testing long term interactive memory under realistic multi session histories. The main result shows long-context LLMs and commercial assistants lose 30%–60% accuracy when forced to read full 115k token histories instead of oracle evidence, proving that naive long context is insufficient for robust memory.
BENCHMARK
LongMemEval reports these metrics on LONGMEMEVALS and a 97 question subset for commercial systems, testing long term interactive memory under realistic multi session histories. The main result shows long-context LLMs and commercial assistants lose 30%–60% accuracy when forced to read full 115k token histories instead of oracle evidence, proving that naive long context is insufficient for robust memory.
BENCHMARK
Accuracy on LongMemEval style questions with short multi session histories and offline reading.
BENCHMARK
Accuracy on LONGMEMEVALS under full history reading vs oracle evidence sessions.
KEY INSIGHT
LongMemEval shows that GPT-4o drops from 0.924 accuracy with oracle evidence sessions to 0.640 on LONGMEMEVALS when reading full 115k token histories.
This is surprising because long context LLMs are marketed as solving context length issues, yet LongMemEval reveals they still lose around 30% accuracy purely from context sprawl.
WHY IT MATTERS
LongMemEval unlocks a realistic way to stress test long term interactive memory, including temporal reasoning, knowledge updates, and abstention, over histories up to 1.5M tokens.
Builders can now benchmark and iterate memory systems with explicit control over value granularity, fact augmented keys, and time aware queries, rather than relying on opaque long context behavior.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…