AuthorsHan Zhang, Zihao Tang, Xin Yu et al.
TL;DR
RHELM uses the LOOP pLan rOllout evOlve Prune pipeline plus heterogeneous external sources to expose that even Claude Opus 4.5 only reaches 38.1 average score on realistic long term memory.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing benchmarks report that RHELM’s best model, Claude Opus 4.5, achieves an average of only 38.1 with external sources.
This means long term personal assistants still fail at multi source aggregation, misleading queries, and realistic temporal reasoning, even with million token contexts.
HOW IT WORKS
RHELM combines Profile Generation, LOOP pLan rOllout evOlve Prune, External Sources, Dialogue Synthesis, and Question Curation to simulate year long, heterogeneous user histories.
You can think of RHELM like a life simulator with a planner as the brain, outcome narratives as experiences, and external documents as a personal archive.
This design lets RHELM test memory behaviors that a plain context window cannot, including implicit state conflicts, cross source aggregation, and evolving user preferences.
DIAGRAM
This diagram shows how RHELM’s LOOP module iteratively plans events, rolls out outcomes, evolves profiles, and prunes stale information along a timeline.
DIAGRAM
This diagram shows how RHELM feeds dialogue and external sources into RAG baselines, long context models, and memory frameworks for evaluation.
PROCESS
Profile Generation
RHELM initializes a six dimensional persona using Profile Generation, storing Identity, Personality, Traits, Relationships, Belongings, and Current Status in a strict JSON schema.
LOOP pLan rOllout evOlve Prune
RHELM runs LOOP to PLAN events, ROLLOUT outcomes with probability p, EVOLVE the profile, and PRUNE outdated attributes over the simulated year.
Dialogue Synthesis
RHELM applies Dialogue Synthesis to convert daily outcome bullets into realistic multi turn conversations across five dialogue categories aligned with the persona.
Question Curation
RHELM performs Question Curation to generate 1,305 QA pairs spanning seven categories and 27 challenging characteristics, including Memory Conditioned Misleading Queries.
KEY CONTRIBUTIONS
RHELM benchmark for long term memory
RHELM introduces a realistic benchmark with Profile Generation and LOOP, yielding 10 personas, 629 days, 11,764 turns, and 2,180 external sources with 500k 1M tokens per persona.
Taxonomy of 27 memory characteristics
RHELM defines seven query categories and 27 challenging characteristics, such as Implicit State Conflict and Structural Navigation, to stress test nuanced memory reasoning.
Comprehensive evaluation across memory paradigms
RHELM evaluates RAG baselines, long context models, and memory frameworks like MemGPT, Mem0, and MemU, showing best average scores of only 38.1 with external data.
RESULTS
Avg score with external sources
38.1
+2.6 over Gemini 2.5 Pro (35.5)
Avg score without external sources
36.2
+3.6 over Gemini 2.5 Pro (32.6)
Misleading accuracy Claude Opus 4.5
41.5
+18.4 over GPT 4.1 mini k=20 (23.1)
Total QA pairs
1305
Seven categories across dialogue, external, and mixed queries
RHELM evaluates memory on 1,305 QA pairs spanning Fact, Temporal, Aggregation, Hallucination, Misleading, External Source, and Mixed categories. The 38.1 average score for Claude Opus 4.5 on RHELM shows that realistic, heterogeneous, evolving memory remains far from solved.
BENCHMARK
RHELM evaluates memory on 1,305 QA pairs spanning Fact, Temporal, Aggregation, Hallucination, Misleading, External Source, and Mixed categories. The 38.1 average score for Claude Opus 4.5 on RHELM shows that realistic, heterogeneous, evolving memory remains far from solved.
BENCHMARK
Average score across all RHELM categories with external sources enabled.
KEY INSIGHT
RHELM shows that adding external sources can reduce RAG performance, for example GPT 4.1 mini k=50 drops from 28.9 to 32.6 average but hurts some categories.
This is surprising because many assume more context always helps, yet RHELM reveals that naive retrieval over 500k 1M token histories can confuse systems and lower accuracy.
WHY IT MATTERS
RHELM unlocks a way to stress test assistants on realistic, year long, heterogeneous memories with misleading, hallucination, and mixed source queries.
Builders can now design and debug memory architectures, retrieval strategies, and safety behaviors against a benchmark that actually reflects evolving users, documents, and implicit constraints.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Okan Bursa
· 2026
Adaptive RAG Memory (ARM) augments a standard retriever–generator stack with a Dynamic Embedding Layer and Remembrance Engine that track usage statistics and apply selective remembrance and decay to embeddings. On a lightweight retrieval benchmark, ARM achieves NDCG@5 ≈ 0.9401 and Recall@5 = 1.000 with 22M parameters, matching larger baselines like gte-small while providing the best efficiency among ultra-efficient models.
Zhanghao Hu, Qinglin Zhu et al.
· 2026
xMemory organises agent memory via segments, memory components, groups, and dynamic maintenance with a sparsity plus semantic faithfulness objective. On LoCoMo with GPT-5 nano, xMemory reaches 38.71 BLEU and 50.00 F1 versus Nemori’s 36.65 BLEU and 48.17 F1, while cutting tokens per query from 9154.76 to 6581.20.
Answers use this explainer on Memory Papers.
Checking…