Memory Papers
Memory Architecture

LMEB: Long-horizon Memory Embedding Benchmark

AuthorsXinping Zhao, Xinshuo Hu, Jiaxin Xu et al.

2026
arXivPDF

TL;DR

LMEB uses a unified IR-style memory format and four memory types to show that the best embedding setup only reaches 61.41 NDCG@10 on long-horizon retrieval.

SharePost on XLinkedIn

Read our summary here, or open the publisher PDF on the next tab.

THE PROBLEM

Long-horizon memory retrieval is missing from current benchmarks

Current text embedding benchmarks mainly focus on traditional passage retrieval and fail to assess long-horizon memory retrieval tasks involving fragmented, context-dependent, and temporally distant information.

This gap leaves memory-augmented agentic systems and evolving environments without a way to test whether their embeddings can support reliable long-term memory retrieval.

HOW IT WORKS

LMEB: Long-horizon Memory Embedding Benchmark

LMEB organizes 22 datasets into four memory types — LMEB-Episodic, LMEB-Dialogue, LMEB-Semantic, and LMEB-Procedural — all converted into a unified IR-style schema of queries, corpus, qrels, and candidates.

Think of LMEB as a standardized card catalog for long-term memories: episodic and dialogue memories are like dated conversation logs, while semantic and procedural memories are stable reference manuals and skill playbooks.

This unified memory taxonomy and IR-style format let LMEB probe retrieval behaviors that a plain context window and traditional passage benchmarks like MTEB cannot reveal.

DIAGRAM

LMEB Memory Taxonomy Across Abstraction and Time

This diagram shows how LMEB categorizes episodic, dialogue, semantic, and procedural memory along abstraction level and temporal dependency.

DIAGRAM

LMEB Evaluation Pipeline and Correlation with MTEB

This diagram shows how LMEB runs retrieval evaluation, aggregates NDCG@10, and compares scores with MTEB to reveal orthogonality.

PROCESS

How LMEB Handles a Long-horizon Memory Retrieval Task

  1. 01

    Unified Data Format Conversion

    LMEB converts raw episodic, dialogue, semantic, and procedural datasets into queries jsonl, corpus jsonl, qrels tsv, and optional candidates jsonl so all memory types share one schema.

  2. 02

    Zero-shot Evaluation Setup

    LMEB emphasizes generalization by running 193 retrieval tasks in a zero-shot setting, using LMEB-Episodic, LMEB-Dialogue, LMEB-Semantic, and LMEB-Procedural without task-specific fine tuning.

  3. 03

    Task Instruction Configuration

    LMEB evaluates models both w inst and w o inst, optionally attaching task instructions to queries to test instruction sensitivity across memory types and datasets.

  4. 04

    Metric Aggregation and Correlation Analysis

    LMEB aggregates NDCG10 as Mean Dataset and Mean Type scores, then computes Pearson and Spearman correlations with MTEB retrieval to analyze orthogonality and generalization.

KEY CONTRIBUTIONS

Key Contributions

  • 01

    Long-horizon Memory Embedding Benchmark

    LMEB introduces 22 datasets and 193 zero-shot retrieval tasks across LMEB-Episodic, LMEB-Dialogue, LMEB-Semantic, and LMEB-Procedural, providing a standardized framework for long-term memory retrieval.

  • 02

    Unified IR-style Data Format

    LMEB defines a unified IR-style schema with queries, corpus, qrels, and candidates, enabling consistent preprocessing, indexing, and evaluation across heterogeneous memory sources.

  • 03

    Orthogonality to MTEB Retrieval

    LMEB shows Pearson and Spearman correlations of -0.115 and -0.130 with MTEB retrieval, proving that traditional passage benchmarks do not capture long-horizon memory retrieval capabilities.

RESULTS

By the Numbers

N@10 Mean Dataset

61.41

+1.31 over NV-Embed-v2 w inst

R@10 Mean Dataset

73.27

vs NV-Embed-v2 w inst at 71.61

Pearson Correlation LMEB MTEB

-0.115

shows near-zero linear relation between LMEB and MTEB retrieval scores

Spearman Correlation LMEB MTEB

-0.130

indicates ranking on LMEB does not track ranking on MTEB retrieval

These numbers come from LMEB’s NDCG@10 Mean Dataset scores and correlation analysis with MTEB eng v2 retrieval. They show LMEB’s top configuration is challenging and that LMEB evaluates capabilities distinct from traditional passage retrieval.

BENCHMARK

By the Numbers

These numbers come from LMEB’s NDCG@10 Mean Dataset scores and correlation analysis with MTEB eng v2 retrieval. They show LMEB’s top configuration is challenging and that LMEB evaluates capabilities distinct from traditional passage retrieval.

BENCHMARK

LMEB Mean Dataset N@10 with Instructions Across Embedding Models

NDCG@10 Mean Dataset on LMEB in the w inst setting.

KEY INSIGHT

The Counterintuitive Finding

In the w o inst setting, the 560M bge-m3 Dense reaches a Mean Dataset N@10 of 56.83, beating the 9B bge-multilingual-gemma2 at 45.10.

This breaks the assumption that larger embedding models automatically perform better on long-horizon memory retrieval, highlighting the importance of architecture and training data.

WHY IT MATTERS

What this unlocks for the field

LMEB unlocks a way to systematically stress test memory embeddings on episodic, dialogue, semantic, and procedural retrieval instead of just passage search.

Builders of memory-augmented agents can now choose and tune embeddings based on real long-horizon performance, rather than assuming MTEB retrieval scores will transfer.

~10 min read← Back to papers

Related papers

Memory Architecture

A Control Architecture for Training-Free Memory Use

Yanzhen Lu, Muchen Jiang et al.

· 2026

TAG routes low-confidence steps to uncertainty-based routing, filters them with guarded acceptance with rollback, chooses between bank selection across rule and exemplar memory, and prunes via evidence-based retirement inside a unified control loop. On SVAMP and ASDiv, TAG reaches 81.0% and 85.2% accuracy, improving over the 74.0% and 77.5% no-memory baselines while a compute-matched Retry baseline stays flat.

arXiv:2604.18206Read explainer

Questions about this paper?

Paper: LMEB: Long-horizon Memory Embedding Benchmark

Answers use this explainer on Memory Papers.

Checking…