Memory Papers
Survey

Evaluating Very Long-Term Conversational Memory of LLM Agents

AuthorsAdyasha Maharana, Dong-Ho Lee, Sergey Tulyakov et al.

2024
arXivPDF

TL;DR

LOCOMO uses LLM generative agents with temporal event graphs and human editing to expose long-term memory failures, where GPT-3.5-turbo-16k reaches only 37.8 F1 vs 87.9 for humans on QA.

SharePost on XLinkedIn

Read our summary here, or open the publisher PDF on the next tab.

THE PROBLEM

LLMs Fail on Very Long Conversations: Humans at 87.9 F1 vs GPT-3.5-16k at 37.8

Existing long-term dialogue work evaluates at around 1K tokens over five sessions, while LOCOMO conversations average 9,209.2 tokens over 19.3 sessions.

In such very long contexts, LOCOMO shows GPT-3.5-turbo-16k reaching only 37.8 F1 overall on QA, far below human 87.9 F1, with especially poor temporal reasoning and adversarial robustness.

HOW IT WORKS

LOCOMO Generative Pipeline and Evaluation Framework

LOCOMO constructs dialogues using Persona, Temporal Event Graph, Virtual Agent Architecture, Image Sharing and Image Reaction, and Human Verification and Editing to simulate realistic long-term multi-modal interactions.

Conceptually, LOCOMO treats personas and event graphs like a life log on disk, while the Virtual Agent Architecture acts as RAM that reflects, summarizes, and retrieves memories across months of conversation.

This KEY_MECHANISM of combining event-grounded agents with human-edited long-range consistency enables tests of memory, temporal, and causal understanding that a plain context window or short-session datasets cannot provide.

DIAGRAM

Virtual Agent Memory and Response Flow in LOCOMO

This diagram shows how LOCOMO agents use short-term and long-term memory plus event graphs to reflect and respond across sessions.

DIAGRAM

LOCOMO Evaluation Pipeline Across Three Tasks

This diagram shows how LOCOMO uses its dataset to run question answering, event summarization, and multi-modal dialogue generation experiments.

PROCESS

How LOCOMO Handles a Very Long-Term Conversation Session

  1. 01

    Persona

    LOCOMO starts by expanding an MSC persona into a rich Persona statement with objectives, habits, and relationships using gpt-3.5-turbo.

  2. 02

    Temporal Event Graph

    LOCOMO applies text-davinci-003 to the Persona to build a Temporal Event Graph of up to 25 events over 6 to 12 months with causal links.

  3. 03

    Virtual Agent Architecture

    Within each session, LOCOMO uses the Virtual Agent Architecture to store observations in long-term memory and summaries wk in short-term memory while conditioning on relevant events.

  4. 04

    Human Verification and Editing

    After automated generation, LOCOMO applies Human Verification and Editing to fix long-range inconsistencies, adjust event grounding, and remove or substitute about 19% of images.

KEY CONTRIBUTIONS

Key Contributions

  • 01

    LOCOMO Very Long-Term Dialog Dataset

    LOCOMO introduces 50 conversations averaging 304.9 turns, 19.3 sessions, and 9,209.2 tokens, built via Persona, Temporal Event Graph, and Virtual Agent Architecture plus Human Verification and Editing.

  • 02

    Three-Task Long-Term Memory Benchmark

    LOCOMO defines Question Answering Task, Event Summarization Task, and Multi-Modal Dialogue Generation Task to probe single-hop, multi-hop, temporal, commonsense, and adversarial reasoning over months-long dialogs.

  • 03

    Analysis of Long Context LLMs and RAG

    LOCOMO shows GPT-3.5-turbo-16k reaching 37.8 F1 overall on QA and 39.9 FactScore F1 on event summarization, while RAG over Observation units yields 41.4 F1, still far below human 87.9 F1.

RESULTS

By the Numbers

QA Overall F1

37.8

+5.4 over GPT-3.5-turbo 4K (22.4 F1) and +5.7 over GPT-3.5-turbo-16k 12K (33.5 F1)

Human QA F1

87.9

+50.1 over GPT-3.5-turbo-16k 16K (37.8 F1) showing large memory gap

Event Sum FactScore F1

45.9

+6.0 over GPT-3.5-turbo-16k 16K (39.9 F1) for incremental summarization

RAG Observation Overall F1

41.4

+19.0 over non-RAG GPT-3.5-turbo (22.4 F1) when using top 5 observations

These metrics are on the LOCOMO benchmark, which tests very long-term conversational memory over up to 35 sessions. The MAIN_RESULT shows that even GPT-3.5-turbo-16k and RAG setups struggle badly compared to humans, especially on temporal and adversarial QA, highlighting the difficulty of long-range dialog understanding.

BENCHMARK

By the Numbers

These metrics are on the LOCOMO benchmark, which tests very long-term conversational memory over up to 35 sessions. The MAIN_RESULT shows that even GPT-3.5-turbo-16k and RAG setups struggle badly compared to humans, especially on temporal and adversarial QA, highlighting the difficulty of long-range dialog understanding.

BENCHMARK

Question Answering Overall F1 on LOCOMO

Overall F1 on LOCOMO QA across humans, base LLMs, long-context LLMs, and RAG with dialog retrieval.

KEY INSIGHT

The Counterintuitive Finding

LOCOMO shows GPT-3.5-turbo-16k adversarial QA F1 collapsing to 2.1%, while GPT-4-turbo with only 4K context reaches 70.2%.

This is surprising because longer context is expected to help, yet LOCOMO reveals that extended windows can amplify hallucinations and misattributions instead of improving robustness.

WHY IT MATTERS

What this unlocks for the field

LOCOMO unlocks a realistic way to stress-test conversational agents over 300-turn, 9K-token, multi-modal dialogues grounded in Temporal Event Graphs and Persona.

Builders can now benchmark memory modules, RAG designs, and long-context LLMs on truly long-term interactions, moving beyond short-session datasets and exposing failure modes that were previously invisible.

~12 min read← Back to papers

Related papers

SurveyAgent Memory

Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

Dongming Jiang, Yi Li et al.

arXiv 2026 · 2026

Anatomy of Agentic Memory organizes agentic memory into four structures using components like Lightweight Semantic Memory, Entity-Centric and Personalized Memory, Episodic and Reflective Memory, and Structured and Hierarchical Memory. Anatomy of Agentic Memory then reports comparative results such as Nemori’s 0.781 semantic judge score on LoCoMo versus SimpleMem’s 0.298, and latency differences like 1.129s for Nemori versus 32.372s for MemoryOS.

arXiv:2602.19320Read explainer
SurveyBenchmarkAgent MemoryLong-Term MemoryMemory Architecture

A Survey on the Security of Long-Term Memory in LLM Agents: Toward Mnemonic Sovereignty

Zehao Lin, Chunyu Li, Kai Chen

· 2026

Mnemonic Sovereignty analyzes long term Write, Store, Retrieve, Execute, Share, and Forget Rollback phases against integrity, confidentiality, availability, and governance objectives for agent memory. Mnemonic Sovereignty’s lifecycle matrix shows most of the ~70 works cluster on write and retrieve integrity, leaving store, availability, and governance primitives like write gate validation and post deletion verification almost entirely unexplored.

arXiv:2604.16548Read explainer
SurveyRAGAgent Memory

Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers

Pengfei Du

· 2026

Memory for Autonomous LLM Agents decomposes agent memory into a POMDP-grounded write–manage–read loop, a three-dimensional taxonomy, and five mechanism families spanning context compression, retrieval stores, reflection, hierarchical virtual context, and policy-learned management. Memory for Autonomous LLM Agents synthesizes results like Voyager’s 15.3× tech-tree speedup and MemoryArena’s 80%→45% drop to show that memory architecture often matters more than backbone choice.

arXiv:2603.07670Read explainer

Questions about this paper?

Paper: Evaluating Very Long-Term Conversational Memory of LLM Agents

Answers use this explainer on Memory Papers.

Checking…