AuthorsWei-Chieh Huang, Weizhi Zhang, Yuchen Wu et al.
TL;DR
Harness the Memory uses a unified evaluation harness across 11 memory substrates to show regime-dependent Pareto fronts and retrieval-depth reversals in 26 metrics.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing memory agents mostly evaluate on LoCoMo and LongMemEval, with 62% of benchmark–system pairs concentrated on these two dialogue-centric datasets.
This leaves agent-centric tasks and efficiency dimensions underexplored, so long-horizon agents cannot tell which memory substrate to use under different operating regimes.
HOW IT WORKS
Harness the Memory builds a Unified Evaluation Harness that swaps External Memory Substrate Families and Internal Memory Substrate Families while keeping backbones, prompts, and auxiliary LLM fixed.
Think of it like a standardized motherboard where different memory modules—vectors, graphs, hierarchies, refinements, weights, activations—can be plugged in and stress-tested under identical workloads.
This harness lets Harness the Memory expose when substrate choices flip across regimes, revealing retrieval-depth failures and scalability limits that a plain context window or single memory design cannot capture.
DIAGRAM
This diagram shows how Harness the Memory routes user queries through agents and different memory substrates during evaluation.
DIAGRAM
This diagram shows how Harness the Memory evaluates substrates across four benchmarks, three backbones, and 26 metrics.
PROCESS
Unified Evaluation Harness Setup
Harness the Memory first configures the Unified Evaluation Harness with fixed prompts, auxiliary GPT-4O-MINI, and vLLM inference across chosen backbones.
Memory Substrate Configuration
Harness the Memory then selects one of the External Memory Substrate Families or Internal Memory Substrate Families, instantiating methods M1–M11 with shared hyperparameters.
Retrieval Depth Sweep
Harness the Memory runs a Retrieval Depth Sweep, varying top-k values on LoCoMo and ALFWorld to attribute performance changes to retrieval breadth rather than scaffolding.
Scalability Study on MAB Conflict Resolution
Harness the Memory performs a Scalability Study, sweeping conversation length from 6K to 262K tokens on MAB Conflict Resolution to measure quality and latency scaling.
KEY CONTRIBUTIONS
Holistic Harness Evaluation of Agent Memory
Harness the Memory delivers the first Unified Evaluation Harness over 11 substrates spanning External Memory Substrate Families and Internal Memory Substrate Families, three backbones, and four benchmarks with 26 metrics.
Cross-Regime Performance–Latency Analysis
Harness the Memory maps each substrate onto performance–latency landscapes, showing disjoint Pareto fronts between LoCoMo, LongMemEval-S, ALFWorld, and BigCodeBench-Hard using the Retrieval Depth Sweep.
Diagnostic Retrieval and Scalability Studies
Harness the Memory runs a Retrieval Depth Sweep and Scalability Study on MAB Conflict Resolution, revealing attention shifts and history-length routing needs across External Memory Substrate Families and Internal Memory Substrate Families.
RESULTS
LLM-judge P4 on LoCoMo (QWEN3-32B-AWQ, M5)
0.683 score
+0.196 over M4 Structural
Per-query latency E15 on LoCoMo (QWEN3-32B-AWQ, M5)
29.22 s/query
≈4.5× M2 Flat at 7.31 s/query
Task success rate TSR on ALFWorld-unseen (QWEN3-32B-AWQ, M7)
32.1 %
+9.7 percentage points over NoMem at 22.4 %
Pass@1 on BigCodeBench-Hard (QWEN3-32B-AWQ, M2)
19.6 %
+2.0 percentage points over NoMem at 17.6 %
Harness the Memory evaluates substrates on LoCoMo, LongMemEval-S, ALFWorld-unseen, and BigCodeBench-Hard. These results show that structural graphs (M5) excel on long-context QA, refinement strategies (M7) lead embodied planning, and flat indices (M2) give cheap gains on hard code tasks.
BENCHMARK
Harness the Memory evaluates substrates on LoCoMo, LongMemEval-S, ALFWorld-unseen, and BigCodeBench-Hard. These results show that structural graphs (M5) excel on long-context QA, refinement strategies (M7) lead embodied planning, and flat indices (M2) give cheap gains on hard code tasks.
BENCHMARK
Comparison of memory substrates on LoCoMo using LLM-judge P4 scores.
BENCHMARK
TSR comparison for selected substrates and NoMem on ALFWorld-unseen.
KEY INSIGHT
Harness the Memory shows that increasing retrieval depth k improves LoCoMo QA but reduces ALFWorld task success, with M7 dropping from 32.1% to about 25% TSR as k grows.
This breaks the common assumption that more retrieved memory is always helpful, revealing that retrieval breadth must be tuned to whether answers live in retrieved history or current observations.
WHY IT MATTERS
Harness the Memory unlocks regime-aware memory routing, where agents can choose between flat, structural, hierarchical, refinement, weight, and activation substrates based on task and history length.
Builders can now design multi-substrate or multi-agent memory systems that trade read breadth for write depth, composing substrates so long-horizon agents remain efficient and reliable across QA, planning, and code tasks.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…