AuthorsWujiang Xu, Yu Wang, Kai Mei et al.
TL;DR
MemGym unifies five long-horizon agent environments behind a shared memory–reasoning interface and adds the MEMRM reward model with AUROC 0.985 for cheap memory evaluation.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing memory benchmarks mostly test personalized dialogue recall, not dynamic memory formation during long-horizon agent execution in coding or web environments.
Agent gyms like SWE-Gym and WebArena only report end-task success, conflating memory failures with reasoning, retrieval, and tool-use errors, and making memory design iteration expensive.
HOW IT WORKS
MemGym centers a shared BaseMemoryManager, BaseAgent, BaseRunner, and BaseMemoryEnvironment that wrap the prompt to a fixed reasoning model across five environments.
Think of MemGym like a computer with standardized RAM and disk: environments plug into the same memory module, while MEMRM acts as a fast critic instead of a slow hardware test.
This design lets MemGym isolate memory gain under a fixed reasoner and use MEMRM to replace full Docker rollouts with calibrated scalar rewards that a plain context window cannot provide.
DIAGRAM
This diagram shows how MemGym runs the env.reset → manage_context → agent.act → env.step loop with memory-isolated scoring.
DIAGRAM
This diagram shows how MemGym collects trajectories, generates SAFE/HARMFUL labels, and trains MEMRM as a lightweight reward model.
PROCESS
Per-step loop env.reset manage_context agent.act env.step
MemGym initializes the environment with env.reset, then uses BaseMemoryManager.manage_context and BaseAgent.act before each env.step to keep memory operations explicit.
Trajectory Recorder and condensation_event logging
MemGym’s Trajectory Recorder logs full turns plus condensation_event metadata so every compression, forgotten index, and summary is reconstructible without rerunning the episode.
Replay-and-fork harness with ObservationReplayRunner
MemGym replays tool actions via ObservationReplayRunner and compute_auto_fork_step, saving about 10× policy_call_savings while reconstructing repository or web state at compaction.
Counterfactual replay and MEMRM QLoRA training
MemGym runs text-only and Docker-snapshotted replays to label SAFE or HARMFUL compressions, then fine-tunes MEMRM with QLoRA on 18.6K triples for fast memory-quality prediction.
KEY CONTRIBUTIONS
Five tracks behind one memory interface
MemGym unifies τ2-bench, SWE-Gym, WebArena-Infinity, MEMGYM-DR, and MEMGYM-CODEQA under BaseMemoryEnvironment and BaseMemoryManager, enabling memory-isolated scoring across 17 (track, strategy) cells.
Controllable ablation verified synthetic pipelines
MemGym builds MEMGYM-CODEQA and MEMGYM-DR with length controllable instances, multi-hop retention_span facts, and multi-criterion verifiers that reject shortcuts and parametric leakage.
MEMRM a scalar gate that replaces a Docker rollout
MemGym trains MEMRM, a Qwen3-1.7B QLoRA reward model on 18.6K labels, achieving AUROC 0.985 on SWE-Gym IID and turning minutes long Docker evaluations into sub second scalar reads.
RESULTS
AUROC
0.985
SWE-Gym IID MEMRM gate quality vs random 0.5
ECE
0.009
calibration error for MEMRM on SWE-Gym IID
Coverage
26.5%
strategy OOD covered subset under selection rule
Coverage
20.4%
scenario OOD WebArena V2 covered subset
MemGym evaluates MEMRM on SWE-Gym compression events, where AUROC 0.985 and ECE 0.009 show near perfect ranking and calibration. Strategy and scenario OOD coverages of 26.5% and 20.4% indicate MemGym can safely deploy MEMRM on characterized subsets beyond training data.
BENCHMARK
MemGym evaluates MEMRM on SWE-Gym compression events, where AUROC 0.985 and ECE 0.009 show near perfect ranking and calibration. Strategy and scenario OOD coverages of 26.5% and 20.4% indicate MemGym can safely deploy MEMRM on characterized subsets beyond training data.
BENCHMARK
AUROC of MEMRM on SWE-Gym IID and two OOD settings.
KEY INSIGHT
MemGym shows that adding summary memory to SWE-Gym yields a resolve-rate change of 0.0 percentage points for Sonnet 4.5 despite 1.47× compression.
This is surprising because many assume more memory always helps coding agents, but MemGym reveals the binding constraint is reasoning over the file system, not the memory module.
WHY IT MATTERS
MemGym lets researchers iterate on memory strategies with memory-isolated scores and MEMRM’s sub second AUROC 0.985 gate instead of costly Docker rollouts.
Builders can now prototype, compare, and train agent memory modules across coding, dialogue, search, and web regimes without entangling results with reasoning or infrastructure limits.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…