Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation

AuthorsZhanghao Hu, Qinglin Zhu, Di Liang et al.

2026

TL;DR

xMemory decouples interaction history into memory components and revisable groups, yielding 38.71 BLEU and 50.00 F1 on LoCoMo with GPT-5 nano, +2.06 BLEU and +1.83 F1 over Nemori.

SharePost on XLinkedIn

Read our summary here, or open the publisher PDF on the next tab.

THE PROBLEM

Agent Memory Is Redundant And Blurs Subtle Updates

Standard RAG for agent memory returns redundant context because interaction streams contain many highly correlated or near duplicate spans.

When answers hinge on small updates, constraints, or factual differences, flat top k similarity retrieval misses decisive evidence and produces wrong or vague responses.

HOW IT WORKS

xMemory: Decoupling Before Aggregation

xMemory builds a hierarchical memory using segments, memory components, groups, and a guidance objective that balances sparsity and semantic faithfulness during dynamic maintenance.

Think of xMemory as a card catalog for agent memory: it first files fine grained evidence cards, then shelves related cards into revisable sections instead of compressing whole books.

This decoupling before aggregation lets xMemory surface decisive evidence units and adaptively expand context, something a plain context window or fixed top k RAG cannot achieve.

DIAGRAM

Top Down Adaptive Retrieval Pipeline

This diagram shows how xMemory performs Stage I backbone selection and Stage II adaptive text expansion when answering a query.

DIAGRAM

LoCoMo Evaluation Setup And Ablations

This diagram shows how xMemory is evaluated on LoCoMo with Qwen3 8B, including baselines and retrieval stage ablations.

PROCESS

How xMemory Handles a Long Term Agent Query

  1. 01

    Building a Revisable Memory Structure

    xMemory first segments raw messages into local segments, then decouples them into memory components and aggregates components into groups guided by the sparsity plus semantic faithfulness objective.

  2. 02

    Dynamic Maintenance

    As new memory arrives, xMemory attaches components to compatible groups, and uses split and merge operations driven by the guidance objective to keep the hierarchy coherent and compact.

  3. 03

    Stage I High Level Backbone Selection

    Given a query, xMemory retrieves relevant groups and memory components via kNN links, greedily building a complementary backbone using coverage and query similarity as defined in Equation 4.

  4. 04

    Stage II Adaptive Text Expansion

    xMemory then expands from selected components to linked segments and original messages only when predictive entropy shows positive uncertainty reduction, stopping when extra text no longer helps.

KEY CONTRIBUTIONS

Key Contributions

  • 01

    Decoupling Before Aggregation Principle

    xMemory formalises decoupling before aggregation for agent memory, extracting memory components from segments before grouping, which raises LoCoMo average BLEU F1 to 34.48 43.98 with Qwen3 8B.

  • 02

    Revisable Hierarchical Memory Structure

    xMemory constructs a revisable hierarchy of segments, memory components, and groups, maintained by a sparsity plus semantic faithfulness guidance objective with split and merge operations.

  • 03

    Top Down Adaptive Retrieval

    xMemory introduces two stage retrieval that first selects a high level backbone of groups and components, then adaptively expands to segments and original messages based on uncertainty reduction.

RESULTS

By the Numbers

Average BLEU LoCoMo

38.71

+2.06 over Nemori with GPT-5 nano

Average F1 LoCoMo

50.00

+1.83 over Nemori with GPT-5 nano

Tokens per query LoCoMo

6581.20

-2573.56 vs Nemori with GPT-5 nano

F1 PerLTQA Qwen3

47.08

+4.28 over MemoryOS on PerLTQA with Qwen3 8B

On LoCoMo, which tests multi session dialogue memory, xMemory reaches 38.71 BLEU and 50.00 F1 with GPT-5 nano versus Nemori’s 36.65 BLEU and 48.17 F1. On PerLTQA personalised long term memory, xMemory achieves 47.08 F1 with Qwen3 8B, showing that decoupled evidence and adaptive retrieval improve both accuracy and token efficiency.

BENCHMARK

By the Numbers

On LoCoMo, which tests multi session dialogue memory, xMemory reaches 38.71 BLEU and 50.00 F1 with GPT-5 nano versus Nemori’s 36.65 BLEU and 48.17 F1. On PerLTQA personalised long term memory, xMemory achieves 47.08 F1 with Qwen3 8B, showing that decoupled evidence and adaptive retrieval improve both accuracy and token efficiency.

BENCHMARK

LoCoMo Average Performance With GPT-5 nano

Average BLEU on LoCoMo across all categories using GPT-5 nano.

BENCHMARK

PerLTQA F1 With Qwen3 8B

Token level F1 on PerLTQA using Qwen3 8B.

KEY INSIGHT

The Counterintuitive Finding

xMemory performs substantial retroactive restructuring, changing 44.91 percent of component group assignments, yet still improves average F1 to 43.98 on LoCoMo.

This is surprising because many memory systems assume fixed clusters are safer, but xMemory shows that revisable groups and dynamic split merge operations can increase stability and accuracy together.

WHY IT MATTERS

What this unlocks for the field

xMemory unlocks evidence oriented agent memory where fine grained memory components and revisable groups give dense, non redundant context for long horizon reasoning.

Builders can now design agents that maintain multi session, personalised histories with efficient, adaptive retrieval, avoiding both full context dumping and over compressed summaries that hide decisive details.

~12 min read← Back to papers

Related papers

RAG

A Dynamic Retrieval-Augmented Generation System with Selective Memory and Remembrance

Okan Bursa

· 2026

Adaptive RAG Memory (ARM) augments a standard retriever–generator stack with a Dynamic Embedding Layer and Remembrance Engine that track usage statistics and apply selective remembrance and decay to embeddings. On a lightweight retrieval benchmark, ARM achieves NDCG@5 ≈ 0.9401 and Recall@5 = 1.000 with 22M parameters, matching larger baselines like gte-small while providing the best efficiency among ultra-efficient models.

RAGLong-Term Memory

HingeMem: Boundary Guided Long-Term Memory with Query Adaptive Retrieval for Scalable Dialogues

Yijie Zhong, Yunfan Gao, Haofen Wang

· 2026

HingeMem combines Boundary Guided Long-Term Memory, Dialogue Boundary Extraction, Memory Construction, Query Adaptive Retrieval, Hyperedge Rerank, and Adaptive Stop to segment dialogues into element-indexed hyperedges and plan query-specific retrieval. On LOCOMO, HingeMem achieves 63.9 overall F1 and 75.1 LLM-as-a-Judge score, surpassing the best baseline Zep (56.9 F1) by 7.0 F1 without using category-specific QA formats.

Questions about this paper?

Paper: Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation

Answers use this explainer on Memory Papers.

Checking…