AuthorsDarshan Deshpande, Varun Gangal, Hersh Mehta et al.
TL;DR
MEMTRACK uses a containerized multi platform agent environment with event timelines and sequential questions to show GPT 5 only reaches 60% Correctness on long horizon memory tasks.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
MEMTRACK reports that the best performing GPT 5 configuration reaches only a 60% Correctness score on its long horizon memory tasks.
This means modern LLM agents struggle to track cross platform event timelines, leading to wrong SWE decisions and repeated, inefficient tool use.
HOW IT WORKS
MEMTRACK builds a Containerized Environment with Docker Network, Linear, Slack, Git, and a Live Notification server plus an Async Process Monitor.
You can think of MEMTRACK like a realistic enterprise lab, where Slack, tickets, and Git history act as long term memory spread across different rooms.
This design lets MEMTRACK stress test memory acquisition, selection, and conflict resolution in ways a plain context window or single thread conversation cannot.
DIAGRAM
This diagram shows how an evaluated agent in MEMTRACK interacts with multi platform tools and memory backends over time to answer sequential SWE questions.
DIAGRAM
This diagram shows how MEMTRACK generates timelines, injects events into the containerized environment, monitors agent trajectories, and computes Correctness, Efficiency, and Redundancy.
PROCESS
Task Generation
MEMTRACK uses Task Generation to create SWE workflows by combining closed Git issues, simulated Slack conversations, and Linear tickets into a coherent event history.
Inject Events into Environment
An Event Parser and Timeline Generator load events onto Slack, Linear, Git, and the Containerized Environment, ensuring agents only see events via tools not as a raw timeline.
Agent Execution Monitoring
During runs, Execution Monitoring and an Async Process Monitor track tool calls, memory usage, and planning while MEMTRACK injects sequential questions without revealing their total count.
Performance Evaluation
Finally, Performance Evaluation computes Correctness, Efficiency, and Redundancy using brief phrase outputs and an LLM as judge, producing an Evaluation Report for each instance.
KEY CONTRIBUTIONS
MEMTRACK multi platform agent environment
MEMTRACK introduces a Containerized Environment with Slack, Linear, Git, and a Docker Network plus Live Notification servers to simulate realistic enterprise SWE workflows across 47 instances.
Scalable event history creation
MEMTRACK combines a Bottom Up Approach, Top Down Approach, and a hybrid protocol using agents and CLAUDE 4 SONNET to synthesize ecologically valid, noisy, and conflicting event timelines.
New memory evaluation metrics
MEMTRACK defines Correctness, Efficiency, and Redundancy to capture how well agents use memory tools beyond simple QA accuracy, revealing GPT 5 only reaches 60% Correctness.
RESULTS
Correctness
60% score
best GPT 5 configuration on MEMTRACK
Per Instance Event Count
39.9 mean
up to 115 events per timeline
Event Tokens
4.01K mean
up to 11.1K tokens per timeline
Platform Entropy
0.668 mean
up to 0.989 indicating highly cross platform timelines
MEMTRACK is a 47 instance benchmark of multi platform SWE workflows with long event histories, large token counts, and high platform entropy. The 60% Correctness result shows MEMTRACK exposes substantial weaknesses in long term memory and cross platform reasoning for GPT 5 and similar agents.
BENCHMARK
MEMTRACK is a 47 instance benchmark of multi platform SWE workflows with long event histories, large token counts, and high platform entropy. The 60% Correctness result shows MEMTRACK exposes substantial weaknesses in long term memory and cross platform reasoning for GPT 5 and similar agents.
BENCHMARK
Distribution of MEMTRACK instance level metrics such as mean event count, mean event tokens, and mean platform entropy.
KEY INSIGHT
Even with dedicated memory backends like Zep and Mem0, MEMTRACK shows GPT 5 only achieves a 60% Correctness score on its tasks.
This is surprising because many assume powerful long context LLMs plus memory tools should nearly solve enterprise SWE workflows, but MEMTRACK reveals persistent reasoning and retention failures.
WHY IT MATTERS
MEMTRACK unlocks a way to stress test long term memory, conflict resolution, and cross platform reasoning in realistic multi tool agent environments.
Builders can now benchmark memory augmented agents on ecologically valid SWE workflows, guiding design of better memory acquisition, selection, and forgetting mechanisms instead of relying on toy conversational tests.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…