AuthorsDi Wu, Zixiang Ji, Asmi Kawatkar et al.
TL;DR
LongMemEval-V2 uses AgentRunbook-C’s scaffolded coding-agent memory to turn 115M-token web-agent histories into reusable experience, reaching 72.5% accuracy vs 48.5% for the strongest RAG baseline.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing memory benchmarks mostly use short traces or user chats, leaving open whether memory systems internalize environment-specific experience over 25M–115M tokens.
Without effective long-term memory, web agents fail to recall interface affordances, workflow knowledge, and environment gotchas, limiting their ability to act as experienced colleagues in specialized environments.
HOW IT WORKS
LongMemEval-V2 introduces AgentRunbook-R, AgentRunbook-C, raw state slice pools, state transition event pools, and procedure and hint note pools to structure noisy trajectories into queryable memory.
Think of AgentRunbook-R as a smart card catalog over trajectories and AgentRunbook-C as a coding agent librarian navigating a large file archive.
This file-centric, agent-controlled memory in LongMemEval-V2 lets agents selectively inspect, summarize, and reuse experience far beyond a plain context window’s limits.
DIAGRAM
This diagram shows how LongMemEval-V2 streams haystack trajectories into memory and then queries AgentRunbook for compact evidence before reader answering.
DIAGRAM
This diagram shows how LongMemEval-V2 evaluates AgentRunbook-R and AgentRunbook-C across small and medium tiers, including ablations and latency measurement.
PROCESS
Trajectory Collection
LongMemEval-V2 collects web-agent trajectories from WebArena and WorkArena using AgentLab and Codex, forming haystacks up to 498 trajectories and 114.8M tokens.
Insert API
LongMemEval-V2 calls Insert on each trajectory, letting AgentRunbook-R build raw state slice pools, state transition event pools, and procedure and hint note pools or AgentRunbook-C store files.
Query API
LongMemEval-V2 invokes Query with a question, prompting AgentRunbook-R’s LLM controller or AgentRunbook-C’s coding agent sandbox to assemble a compact multimodal evidence context.
Reader Answering
LongMemEval-V2 truncates the memory context to 200k tokens and passes it with the question to the Qwen3.5-9B reader, computing accuracy and measuring query latency.
KEY CONTRIBUTIONS
LongMemEval-V2 benchmark
LongMemEval-V2 introduces 451 questions over 100–498 trajectory haystacks and 25M–115M tokens, covering static recall, dynamic tracking, workflows, gotchas, and premise awareness.
AgentRunbook-R memory design
LongMemEval-V2 proposes AgentRunbook-R with raw state slice pools, state transition event pools, and procedure and hint note pools, reaching 58.6% accuracy on LME-V2-Small with 26.9s latency.
AgentRunbook-C coding agent memory
LongMemEval-V2 develops AgentRunbook-C, a scaffolded coding-agent memory that attains 72.5% average accuracy and is 32% faster than the off-the-shelf Codex baseline at query time.
RESULTS
Overall accuracy LME-V2-Medium
70.1%
+24.2 over RAG query slice+notes
Static accuracy LME-V2-Medium
78.8%
+29.9 over RAG query slice+notes
Workflow accuracy LME-V2-Medium
61.3%
+17.9 over RAG query slice+notes
Query latency LME-V2-Medium
139.9s
45.9s faster than Codex at 185.8s
LongMemEval-V2 evaluates memory systems on LME-V2-Small and LME-V2-Medium, which test long-term web-agent experience across up to 115M tokens. AgentRunbook-C’s 70.1% overall accuracy on LME-V2-Medium shows that scaffolded coding-agent memory can turn massive trajectories into reliable environment knowledge.
BENCHMARK
LongMemEval-V2 evaluates memory systems on LME-V2-Small and LME-V2-Medium, which test long-term web-agent experience across up to 115M tokens. AgentRunbook-C’s 70.1% overall accuracy on LME-V2-Medium shows that scaffolded coding-agent memory can turn massive trajectories into reliable environment knowledge.
BENCHMARK
Overall accuracy (%) of memory methods on LME-V2-Medium under the context gathering formulation.
BENCHMARK
Overall accuracy (%) and latency (s) comparison for key memory methods on LME-V2-Small.
KEY INSIGHT
LongMemEval-V2 shows that an off-the-shelf Codex coding agent reaches 69.3% accuracy in pilot studies and up to 69.9% on LME-V2-Small, despite no memory-specific training.
This is surprising because typical expectations favor specialized RAG pipelines, yet AgentRunbook-C’s scaffolded coding agent surpasses both Codex and RAG by up to 24.8 points.
WHY IT MATTERS
LongMemEval-V2 unlocks memory systems where agents act as experienced colleagues, recalling workflows, gotchas, and dynamic state changes across hundreds of trajectories.
Builders can now stress-test and design memory modules like AgentRunbook-C that treat trajectories as files, using coding agents to manage long-term experience beyond fixed-context RAG.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Okan Bursa
· 2026
Adaptive RAG Memory (ARM) augments a standard retriever–generator stack with a Dynamic Embedding Layer and Remembrance Engine that track usage statistics and apply selective remembrance and decay to embeddings. On a lightweight retrieval benchmark, ARM achieves NDCG@5 ≈ 0.9401 and Recall@5 = 1.000 with 22M parameters, matching larger baselines like gte-small while providing the best efficiency among ultra-efficient models.
Yijie Zhong, Yunfan Gao, Haofen Wang
· 2026
HingeMem combines Boundary Guided Long-Term Memory, Dialogue Boundary Extraction, Memory Construction, Query Adaptive Retrieval, Hyperedge Rerank, and Adaptive Stop to segment dialogues into element-indexed hyperedges and plan query-specific retrieval. On LOCOMO, HingeMem achieves 63.9 overall F1 and 75.1 LLM-as-a-Judge score, surpassing the best baseline Zep (56.9 F1) by 7.0 F1 without using category-specific QA formats.
Answers use this explainer on Memory Papers.
Checking…