Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning

AuthorsSikuan Yan, Xiufeng Yang, Zuchao Huang et al.

2025

TL;DR

Memory-R1 uses RL trained Memory Manager and Answer Agent to learn memory operations and distillation, reaching 45.02 F1 and 37.51 BLEU-1 on LoCoMo vs 35.04 and 27.99 for MemoryOS.

SharePost on X LinkedIn

Read our summary here, or open the publisher PDF on the next tab.

THE PROBLEM

LLM agents forget beyond finite context windows and mismanage CRUD memories

Memory-R1 targets stateless LLMs whose memory is bounded by a finite context window, which prevents maintaining knowledge across long conversations and evolving tasks.

Existing memory-augmented agents misinterpret updates, for example issuing DELETE plus ADD instead of a single UPDATE, fragmenting knowledge and degrading downstream question answering.

HOW IT WORKS

Memory-R1 — RL trained Memory Manager and Answer Agent

Memory-R1 centers on a Memory Manager, Answer Agent, Memory Distillation policy, and RL fine-tuning with PPO and GRPO to control ADD, UPDATE, DELETE, and NOOP operations.

You can think of Memory-R1 as giving an LLM both RAM and disk, plus a learned operating system that decides what to write, overwrite, or ignore in its memory bank.

This RL driven control lets Memory-R1 consolidate facts, filter noisy retrievals, and support long horizon reasoning that a plain context window cannot sustain.

DIAGRAM

Multi session QA flow with Memory-R1

This diagram shows how Memory-R1 processes multi session dialogues, updates memory, retrieves 60 candidates via RAG, and distills them for question answering.

DIAGRAM

RL training loop for Memory-R1 components

This diagram shows how Memory-R1 trains the Memory Manager and Answer Agent with PPO and GRPO using exact match rewards from downstream QA.

PROCESS

How Memory-R1 Handles a Multi Session Dialogue Question

01
RL Fine tuning for Memory Manager
Memory-R1 uses RL Fine-tuning for Memory Manager with PPO and GRPO so the Memory Manager learns ADD, UPDATE, DELETE, and NOOP operations that maximize downstream QA rewards.
02
RL Fine Tuning for Answer Agent
Memory-R1 applies RL Fine-Tuning for Answer Agent, training the Answer Agent with PPO or GRPO on questions plus 60 retrieved memories to maximize exact match rewards.
03
Memory Distillation policy
The Answer Agent in Memory-R1 uses a Memory Distillation policy to filter noisy retrieved memories and focus on the most relevant entries before generating answers.
04
Generalization and Scalability
Memory-R1 evaluates Generalization and Scalability by training on LoCoMo only, then testing zero shot on MSC and LongMemEval across Qwen-2.5 3B, 7B, and 14B backbones.

KEY CONTRIBUTIONS

Key Contributions

01
Memory-R1 framework for memory augmented LLMs
Memory-R1 introduces a Memory Manager and Answer Agent trained with PPO and GRPO to operate over ADD, UPDATE, DELETE, and NOOP, achieving 45.02 F1 and 37.51 BLEU-1 on LoCoMo.
02
Data efficient RL fine tuning strategy
Memory-R1 uses a data efficient fine-tuning strategy with PPO and GRPO that reaches strong performance using only 152 training question–answer pairs from LoCoMo.
03
In depth analysis of RL choices and memory design
Memory-R1 provides ablations over Memory Manager, Answer Agent, Memory Distillation, and reward design, showing for example that GRPO converges faster while PPO and GRPO reach similar final rewards.

RESULTS

By the Numbers

45.02

+9.98 over MemoryOS

BLEU-1

37.51

+9.52 over MemoryOS

LLM-as-a-Judge

62.74

+14.54 over MemoryOS

Training QA pairs

152

LoCoMo train split size used to train Memory-R1

On the LoCoMo benchmark, which tests long multi session dialogue reasoning, Memory-R1-GRPO with LLaMA-3.1-8B-Instruct reaches 45.02 F1, 37.51 BLEU-1, and 62.74 LLM-as-a-Judge, showing that reinforcement learning over memory operations and distillation yields large gains over MemoryOS.

BENCHMARK

By the Numbers

BENCHMARK

LoCoMo overall results with LLaMA-3.1-8B-Instruct

F1 on LoCoMo overall for Memory-R1 and key memory baselines.

KEY INSIGHT

The Counterintuitive Finding

Using only 152 training QA pairs, Memory-R1-GRPO still reaches 45.02 F1 and 62.74 LLM-as-a-Judge on LoCoMo, beating MemoryOS by large margins.

This is surprising because many practitioners assume RL for LLM agents requires massive datasets, yet Memory-R1 shows substantial gains with extremely sparse supervision.

WHY IT MATTERS

What this unlocks for the field

Memory-R1 unlocks learned memory operations and memory distillation for LLM agents, enabling adaptive ADD, UPDATE, DELETE, and NOOP decisions driven by downstream rewards.

Builders can now deploy agents that maintain coherent long term memories and robustly answer cross session questions without hand tuned heuristics or massive supervised traces.

~13 min read← Back to papers

Related papers

SurveyAgent Memory

Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

Dongming Jiang, Yi Li et al.

arXiv 2026 · 2026

Anatomy of Agentic Memory organizes agentic memory into four structures using components like Lightweight Semantic Memory, Entity-Centric and Personalized Memory, Episodic and Reflective Memory, and Structured and Hierarchical Memory. Anatomy of Agentic Memory then reports comparative results such as Nemori’s 0.781 semantic judge score on LoCoMo versus SimpleMem’s 0.298, and latency differences like 1.129s for Nemori versus 32.372s for MemoryOS.

arXiv:2602.19320 Read explainer

SurveyBenchmarkAgent MemoryLong-Term MemoryMemory Architecture

A Survey on the Security of Long-Term Memory in LLM Agents: Toward Mnemonic Sovereignty

Zehao Lin, Chunyu Li, Kai Chen

· 2026

Mnemonic Sovereignty analyzes long term Write, Store, Retrieve, Execute, Share, and Forget Rollback phases against integrity, confidentiality, availability, and governance objectives for agent memory. Mnemonic Sovereignty’s lifecycle matrix shows most of the ~70 works cluster on write and retrieve integrity, leaving store, availability, and governance primitives like write gate validation and post deletion verification almost entirely unexplored.

arXiv:2604.16548 Read explainer

SurveyRAGAgent Memory

Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers

Pengfei Du

· 2026

Memory for Autonomous LLM Agents decomposes agent memory into a POMDP-grounded write–manage–read loop, a three-dimensional taxonomy, and five mechanism families spanning context compression, retrieval stores, reflection, hierarchical virtual context, and policy-learned management. Memory for Autonomous LLM Agents synthesizes results like Voyager’s 15.3× tech-tree speedup and MemoryArena’s 80%→45% drop to show that memory architecture often matters more than backbone choice.

arXiv:2603.07670 Read explainer

Questions about this paper?

Answers use this explainer on Memory Papers.

Checking…