AuthorsMihir Shriniwas Arya
TL;DR
RECON benchmarks provenance-graph memory over 50K–100K token narratives, revealing an Oracle ceiling of 54.6% Accuracy while the best non-Oracle reaches only 22.4%.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
RECON shows that even the strongest non-Oracle system reaches only 22.4% Accuracy, while the Oracle ceiling is 54.6%, on 1,414 clean questions.
When agents mis-handle cascading invalidations and counterfactual dependencies in 50K–100K token cases, they hallucinate stale conclusions and miss which downstream inferences should break.
HOW IT WORKS
RECON builds each case via attribute-conditioned blueprint synthesis, then encodes it in a provenance DAG and proof-trace grounding, and finally generates questions with deterministic task synthesis.
You can think of RECON like a database-backed investigation: the blueprint is the schema, the provenance DAG is the dependency graph, and narration is just a human-readable view.
This design lets RECON test whether agents can follow explicit derivation histories and invalidations across long contexts, something a plain context window or simple state-machine memory cannot.
DIAGRAM
This diagram shows how an external agent interacts with RECON case files and questions during evaluation.
DIAGRAM
This diagram shows how RECON generates deterministic cases and validates 1,604 questions before release.
PROCESS
Attribute-conditioned blueprint synthesis
RECON uses attribute-conditioned blueprint synthesis to expand a catalog D with production rules P and attributes A into a deterministic case blueprint B.
Skeleton expansion and temporal enforcement
RECON converts the blueprint into a timestamped skeleton, enforcing lexicographic temporal monotonicity τ(v) and validating causal edges for consistency.
Provenance DAG and proof-trace grounding
RECON induces a provenance DAG where edges reflect attribute dependencies and attaches per-question proof traces as minimal subgraphs for answer derivation.
Deterministic task synthesis
RECON algorithmically generates six task types from the provenance DAG, then narrates cases and runs a six-stage validation pipeline before releasing questions.
KEY CONTRIBUTIONS
The RECON benchmark
RECON provides 1,604 questions across 24 case files of 50K–100K tokens in three domains, with six memory-intensive tasks grounded in provenance DAGs.
Deterministic generation pipeline
RECON uses attribute-conditioned blueprint synthesis, skeleton expansion, and provenance DAG and proof-trace grounding to derive all gold answers in code, not via LLMs.
Empirical study of long-context and memory systems
RECON evaluates long-context LLMs, RAG variants, and memory agents, finding the best non-Oracle Accuracy at 22.4% while the Oracle reaches 54.6% Accuracy.
RESULTS
Accuracy
54.6%
+32.2 over gemini-2.5-pro
Score
0.654
vs gemini-2.5-pro Score 0.265
Best non-Oracle Accuracy
22.4%
Gemini-2.5-Pro on long-context cases
Supermemory Cascade Score
0.708
+0.347 over long-context family mean on Cascade Propagation
RECON evaluates 1,414 contamination-filtered questions over 50K–100K token cases, testing six memory tasks. The Oracle’s 54.6% Accuracy versus 22.4% for Gemini-2.5-Pro shows that even with perfect structure, compositional reasoning over provenance graphs remains challenging.
BENCHMARK
RECON evaluates 1,414 contamination-filtered questions over 50K–100K token cases, testing six memory tasks. The Oracle’s 54.6% Accuracy versus 22.4% for Gemini-2.5-Pro shows that even with perfect structure, compositional reasoning over provenance graphs remains challenging.
BENCHMARK
Overall Score on RECON across long-context, Oracle, RAG, and memory systems.
KEY INSIGHT
RECON shows that even with the full structured dependency graph, the Oracle only reaches 54.6% Accuracy, far from perfect reasoning.
This is surprising because many assume that once retrieval is solved, reasoning is easy, but RECON’s Oracle–LLM gaps and Oracle ceiling contradict that assumption.
WHY IT MATTERS
RECON unlocks a way to stress-test agents on cascading invalidations, source conflicts, and counterfactuals in 50K–100K token narratives with deterministic ground truth.
Builders can now benchmark memory architectures and long-context models on provenance-aware reasoning, not just fact retrieval, guiding designs that track how conclusions depend on evolving evidence.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…