AgentIR: A Workload-Adaptive Cascade Retrieval Substrate for Long-Term Conversational Memory

THE PROBLEM

Agents Need Sub 10 ms Retrieval on Millions of Growing Memories

HOW IT WORKS

AgentIR — A Cascade Retrieval Substrate for Agent Memory

DIAGRAM

AgentIR Query-Time Cascade Retrieval Flow

This diagram shows how AgentIR routes each query through BM25, the temporal index, and optionally the dense channel using a confidence-triggered cascade.

DIAGRAM

AgentIR Evaluation and Scaling Pipeline

This diagram summarizes how AgentIR is evaluated on BEIR, LongMemEval, and synthetic scaling workloads, including baselines and metrics.

PROCESS

How AgentIR Handles a Long-Term Agent Session

1. 01

   Agent Memory Workload Characterization

   AgentIR first models agent memory as tuples with dense embedding, role, session, agent, tool type, timestamp, and importance weight to capture recency and heterogeneity.

2. 02

   Temporal Partitioned Index Construction

   AgentIR builds a time-partitioned sparse index in 7 day windows so that recency biased queries only touch a few recent partitions with O(log 1/ε) work.

3. 03

   Agent Aware Fusion and Routing

   At query time, AgentIR runs SIMD BM25, optionally dense HNSW, and applies agent aware RRF with recency and importance bonuses chosen by the Cascade Router.

4. 04

   Workload Conditional Cascade Execution

   AgentIR’s cascade router uses BM25 confidence or a TF IDF classifier to skip the 52 ms dense channel on easy queries, adapting thresholds per workload like LongMemEval or LoCoMo.

KEY CONTRIBUTIONS

Key Contributions

• 01

  Sub Linear Temporal Partitioned Index

  AgentIR’s temporal partitioned index achieves O(log 1/ε) expected work, so 1234× corpus growth from 4,052 to 4,998,640 records costs only 3.6× latency growth and yields 1769× speedup over sequential at 5M records.

• 02

  Heterogeneous CPU GPU BM25 Pipeline

  AgentIR’s SIMD Accelerated BM25 with GPU CSR scoring matches Lucene quality within ±0.020 nDCG@10 and runs 1.8–29× faster than Pyserini 8T and 11× faster than PISA 1T BlockMax WAND on BEIR.

• 03

  Workload Conditional Fusion and Cascade Router

  AgentIR’s agent aware fusion and cascade router pick BM25, Dense, RRF, or agent_rrf per query, skipping dense on 63% of LongMemEval queries for 2.67× speedup and auto tuning to 100% skips on LoCoMo for 132× speedup and +0.089 Hit@5.

RESULTS

By the Numbers

On BEIR, AgentIR’s BM25 core matches or beats Pyserini Lucene quality while being up to 29× faster, and on LongMemEval AgentIR’s agent aware fusion reaches R@10=0.978 with a cascade router that cuts hybrid latency by 2.67× without losing LLM judged accuracy.

BENCHMARK

By the Numbers

On BEIR, AgentIR’s BM25 core matches or beats Pyserini Lucene quality while being up to 29× faster, and on LongMemEval AgentIR’s agent aware fusion reaches R@10=0.978 with a cascade router that cuts hybrid latency by 2.67× without losing LLM judged accuracy.

BENCHMARK

LongMemEval LLM Judged Strict Accuracy with Different Fusion Strategies

Strict LLM judged accuracy on LongMemEval using gpt-4o-mini as both answerer and judge.

BENCHMARK

BEIR BM25 Latency Comparison on NFCorpus

Average per query latency on NFCorpus comparing AgentIR to Pyserini and PISA.

KEY INSIGHT

WHY IT MATTERS

What this unlocks for the field