AuthorsSarthak Singh
TL;DR
DreamBench-SWE uses executable multi-session SWE traps to profile memory hygiene, showing B5 at 89/180 vs hybrid probe at 95/180 without significant separation.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Multi-session software agents often misremember earlier sessions, overgeneralize feedback, or reuse stale architecture facts, causing hidden failures despite local task success.
On DreamBench-SWE, the no-memory baseline B0 achieves only 21/180 (0.117) S3 Pass@1, showing that later tasks depend on non-inferable earlier evidence and break without reliable memory.
These failures matter for real repositories where agents must respect generated-file boundaries, scoped reviewer preferences, and flaky test diagnostics to avoid harmful code edits.
HOW IT WORKS
DreamBench-SWE couples a fixed wake agent with a raw episode log, derived memory store, maintenance pipeline, and retrieval gate to study memory hygiene under executable traps.
You can think of the raw episode log as disk, the derived memory store as an indexed card catalog, and the retrieval gate as a cautious librarian deciding which cards reach RAM.
This design lets DreamBench-SWE test what the maintenance pipeline and retrieval gate enable beyond a plain context window, by forcing agents to reproduce hidden CSPRNG tokens and avoid stale or overscoped memories under code oracles.
DIAGRAM
This diagram shows how DreamBench-SWE sequences S1, S2, and S3 sessions to create non-inferable memory traps scored by hidden executable oracles.
DIAGRAM
This diagram shows how DreamBench-SWE evaluates different memory conditions like B0, B5, and the reference probe hybrid across 60 traps and 3 seeds.
PROCESS
Trajectory logger
DreamBench-SWE logs each wake agent run as a trajectory with task metadata, tool calls, diffs, memory reads, and outcomes into the raw episode log.
Sleep scheduler
The sleep scheduler decides when offline maintenance runs over raw episodes, triggering the maintenance pipeline after S1 and S2 sessions.
Maintenance pipeline
The maintenance pipeline performs typed consolidation, contradiction repair, counterfactual replay, and stale suppression into the derived memory store.
Retrieval gate
Before S3, the retrieval gate filters and ranks candidate memories from the raw episode log and derived memory store under fixed token budgets and scope rules.
KEY CONTRIBUTIONS
DreamBench-SWE dynamic memory hygiene benchmark
DreamBench-SWE introduces 60 three-session SWE traps with hidden non-inferable tokens and executable oracles, yielding 1890 condition-level result files and a clean admission funnel.
Reference probe wake sleep read architecture
DreamBench-SWE defines a training-free reference probe with a raw episode log, derived memory store, maintenance pipeline, and retrieval gate around a fixed SWE agent.
Executable hygiene diagnostics and successor audit
DreamBench-SWE reports oracle metrics like RepeatedErrorRate 1512/2646 and a v2.1 audit where B5-MEM0-LIT reaches 97/180 S3 passes and separates from B0 after Holm correction.
RESULTS
S3 Pass@1 B5
89/180 (0.494)
+0.1227 over B0 no memory
S3 Pass@1 hybrid probe
95/180 (0.528)
+0.034 over B5 verbatim baseline
IrrelevantImportRate
1434/3024 (0.474)
overall hygiene diagnostic across conditions
RepeatedErrorRate
1512/2646 (0.571)
shows frequent repeated failures despite memory
On the DreamBench-SWE v2 confirmatory fold with 60 traps and 3 seeds, B5 achieves 89/180 S3 Pass@1 while the reference probe hybrid reaches 95/180, but the clustered P1 test yields p=0.518 and fails to reject. Hygiene metrics like IrrelevantImportRate 0.474 and RepeatedErrorRate 0.571 show that DreamBench-SWE exposes memory-related pathologies beyond simple success rates.
BENCHMARK
On the DreamBench-SWE v2 confirmatory fold with 60 traps and 3 seeds, B5 achieves 89/180 S3 Pass@1 while the reference probe hybrid reaches 95/180, but the clustered P1 test yields p=0.518 and fails to reject. Hygiene metrics like IrrelevantImportRate 0.474 and RepeatedErrorRate 0.571 show that DreamBench-SWE exposes memory-related pathologies beyond simple success rates.
BENCHMARK
S3 Pass@1 on 60 DreamBench-SWE traps (3 seeds, 180 cells per condition).
BENCHMARK
S3 Pass@1 on the v2.1 successor audit for four admissible conditions.
KEY INSIGHT
DreamBench-SWE shows that the hybrid reference probe scores 95/180 S3 passes while B5 scores 89/180, yet the clustered P1 test yields p=0.518 and no rejection.
This is surprising because many would expect a sophisticated typed-plus-raw maintenance pipeline to clearly beat a simple verbatim event-memory baseline, but DreamBench-SWE reveals that strong recall-verbatim traps can favor B5.
The v2.1 audit further surprises by showing B5-MEM0-LIT at 97/180 passes, numerically above both B5 and the hybrid probe without a confirmed superiority claim.
WHY IT MATTERS
DreamBench-SWE gives the field an executable, contamination-audited way to measure memory hygiene in multi-session SWE agents, not just long-context recall.
Builders can now stress-test external memory systems against hidden CSPRNG tokens, stale architecture facts, and scoped reviewer feedback, and reason about mechanisms like raw-evidence invariants and retrieval gates before deploying agents to real repositories.
The benchmark also highlights that simple verbatim memory baselines can be competitive, encouraging more nuanced design and evaluation of complex maintenance pipelines.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…