REMem: Reasoning with Episodic Memory in Language Agent

AuthorsYiheng Shu, Saisri Padmaja Jonnalagedda, Xiang Gao et al.

2026

TL;DR

REMem uses a hybrid gist plus fact memory graph with agentic tools to reach 93.1% EM on Test of Time, +26.2 over HippoRAG 2.

SharePost on XLinkedIn

Read our summary here, or open the publisher PDF on the next tab.

THE PROBLEM

Episodic agents lack event structure and temporal grounding

REMem observes that existing language agents focus on semantic memory and often ignore spatiotemporal context, losing coherent event structure and situational elements.

This means tools like RAG systems and graph-based memories struggle with episodic recollection and reasoning, limiting multi-event queries over long interaction histories.

HOW IT WORKS

REMem — Hybrid memory graph plus agentic inference

REMem builds episodic memory by combining Gist Extraction, Fact Extraction, Graph Construction, and Agentic Inference into a unified, time-aware memory system.

You can think of REMem like a library card catalog plus a timeline: gists are human-readable cards, facts are relational links, and the graph is the shelving system.

This design lets REMem perform temporal filtering, multi-hop graph exploration, and ordinal reasoning that a plain context window or flat embedding index cannot support.

DIAGRAM

Agentic inference flow over the memory graph

This diagram shows how REMem's agentic inference iteratively retrieves and reasons over the hybrid memory graph using curated tools.

DIAGRAM

Evaluation pipeline and ablation design

This diagram shows how REMem is evaluated across four benchmarks and how ablations remove gists, facts, synonymy edges, or tools.

PROCESS

How REMem Handles an Episodic Query

  1. 01

    Indexing

    In Indexing, REMem processes interaction histories with Gist Extraction, Fact Extraction, and Graph Construction to build a hybrid memory graph grounded in time.

  2. 02

    Gist Extraction

    During Gist Extraction, REMem converts episodes into concise, timestamped gists that capture participants, actions, locations, intentions, and quantities as atomic events.

  3. 03

    Fact Extraction

    In Fact Extraction, REMem derives schemaless triples with temporal qualifiers, anchoring who did what to whom along a precise timeline for later reasoning.

  4. 04

    Agentic Inference

    During Agentic Inference, REMem uses semantic retrieve, lexical retrieve, find gist contexts, and find entity contexts to iteratively collect evidence and call output answer.

KEY CONTRIBUTIONS

Key Contributions

  • 01

    Hybrid memory graph for episodic events

    REMem introduces a hybrid memory graph that integrates Gist Extraction, Fact Extraction, and Graph Construction to encode time-aware gists and time-scoped facts in one structure.

  • 02

    Agentic retriever with curated tools

    REMem designs Agentic Inference with semantic retrieve, lexical retrieve, find gist contexts, and find entity contexts to support iterative retrieval and temporal graph exploration.

  • 03

    Comprehensive episodic evaluation

    REMem is evaluated on LoCoMo, REALTALK, Complex TR, and Test of Time, achieving a 3.4% gain in episodic recollection and 13.4% gain in episodic reasoning over strong baselines.

RESULTS

By the Numbers

LLM-J

89.6% LLM-J

+8.7 over HippoRAG 2 on Complex-TR

EM

93.1% EM

+26.2 over HippoRAG 2 on Test of Time

F1

42.4% F1

+3.4 over HippoRAG 2 on LoCoMo

LLM-J

65.3% LLM-J

+9.5 over HippoRAG 2 on REALTALK

On LoCoMo and REALTALK, which test conversational episodic recollection, REMem improves LLM-J scores by up to 9.5 points over HippoRAG 2. On Complex-TR and Test of Time, which stress temporal reasoning, REMem-I reaches 89.6% LLM-J and 93.1% EM, demonstrating robust multi-step episodic reasoning.

BENCHMARK

By the Numbers

On LoCoMo and REALTALK, which test conversational episodic recollection, REMem improves LLM-J scores by up to 9.5 points over HippoRAG 2. On Complex-TR and Test of Time, which stress temporal reasoning, REMem-I reaches 89.6% LLM-J and 93.1% EM, demonstrating robust multi-step episodic reasoning.

BENCHMARK

Episodic reasoning performance on Test of Time

EM on Test of Time semantic split.

BENCHMARK

Episodic recollection on LoCoMo

LLM-J on LoCoMo episodic recollection benchmark.

KEY INSIGHT

The Counterintuitive Finding

REMem-I achieves 93.1% EM on Test of Time, while Full-Context with the entire corpus reaches only 79.7% despite having maximal input.

This is surprising because many assume more context always helps, but REMem shows that structured episodic memory plus tools can beat naive full-context reasoning.

WHY IT MATTERS

What this unlocks for the field

REMem unlocks reliable episodic recollection and temporal reasoning over long interaction histories using a hybrid memory graph and agentic tools.

Builders can now create language agents that answer timeline, duration, and cross-session questions with explicit event structure instead of brittle, flat retrieval.

~12 min read← Back to papers

Related papers

Agent MemoryLong-Term Memory

Adaptive Memory Admission Control for LLM Agents

Guilin Zhang, Wei Jiang et al.

· 2026

A-MAC scores candidate memories using Utility, Confidence, Novelty, Recency, and Type Prior combined by a learned linear admission policy with Algorithm 1 A-MAC Memory Admission. On the LoCoMo benchmark, A-MAC achieves F1 0.583 and 2644 ms latency, improving F1 by 0.042 and reducing latency by 1187 ms compared to A-mem.

Long-Term Memory

Advancing Open-source World Models

Robbyant Team, Zelin Gao et al.

arXiv 2026 · 2026

LingBot-World combines a Data Engine, Fundamental World Model, Action-Conditioned World Model, and Post-Training causal adaptation to turn a 28B-parameter video generator into a real-time interactive world simulator. On the VBench benchmark, LingBot-World achieves a dynamic degree of 0.8857 versus 0.7612 for Yume-1.5, while also improving imaging quality to 0.6683.

BenchmarkBenchmarkLong-Term Memory

AgenticAI-DialogGen: Topic-Guided Conversation Generation for Fine-Tuning and Evaluating Short- and Long-Term Memories of LLMs

Manoj Madushanka Perera, Adnan Mahmood et al.

· 2026

AgenticAI-DialogGen chains ChatPreprocessor, KnowledgeExtractor, TopicAnalyzer, KnowledgeGraphBuilder, PersonaGenerator, DuelingChat Agent, ConversationValidator, ConversationRefiner, QAGeneration, and PostProcessing to turn raw multi-session chats into topic-guided, persona-grounded conversations with explicit short- and long-term memories. On the TGC / KG memory QA benchmark, Mistral-7B fine-tuned within AgenticAI-DialogGen achieves 87.36 F1, compared to GPT-4’s 83.77 F1 in a zero-shot setting on the same task.

Questions about this paper?

Paper: REMem: Reasoning with Episodic Memory in Language Agent

Answers use this explainer on Memory Papers.

Checking…