AuthorsNatchanon Pollertlam, Witchayut Kornsuwannawit
TL;DR
Total Recall at What Cost benchmarks Mem0, Hindsight, and Mastra OM against rolling and full-history baselines, showing 18–69% cost-model error and break-even points ranging from turn 0 to never within 400 turns.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Total Recall at What Cost shows that a separable cost model misses memory-system serving cost by 18–69 percent while predicting baselines within 2.9–6.5 percent.
This means long-running conversational agents with Mem0, Hindsight, or Mastra OM cannot estimate serving cost from conversation length and message size alone, risking large budget miscalculations.
HOW IT WORKS
Total Recall at What Cost builds a benchmark around Mem0, Hindsight, and Mastra Observational Memory, plus Rolling window and Full history baselines, all driven by a shared synthetic dialogue grid.
You can think of Total Recall at What Cost like profiling an operating system: Ingest, Retrieval, and Answer stages are metered separately, revealing how internal memory pipelines, not just transcript size, dominate cost.
This design lets Total Recall at What Cost expose when memory systems actually beat a plain context window, computing turn level break even points instead of assuming monotonic token savings.
DIAGRAM
This diagram shows how Total Recall at What Cost replays each synthetic conversation turn through ingest, retrieval, and answer stages while logging per turn token cost.
DIAGRAM
This diagram shows how Total Recall at What Cost combines the cost grid with LoCoMo accuracy to compute cost per correct answer.
PROCESS
Design grid
Total Recall at What Cost selects five core (N, L) cells plus extras, defining conversation length and per turn token size for every system.
Synthetic dialogue generation
Total Recall at What Cost uses gemma 4 26B A4B to generate cached two speaker dialogues, trimmed and padded to match the grid token targets.
Replay through systems and baselines
Total Recall at What Cost replays each cached dialogue through Mem0, Hindsight, Mastra OM, Rolling window, and Full history while logging per turn token usage.
Separable cost model and break even analysis
Total Recall at What Cost fits log C plus one equals a plus p log L plus one plus q log t plus one and computes turn level break even against full history.
KEY CONTRIBUTIONS
Controlled benchmark of agentic memory serving cost
Total Recall at What Cost measures Mem0, Hindsight, and Mastra OM against Rolling window and Full history across two backbones and conversations up to 400 turns, with 8 repetitions per grid cell.
Separable per turn cost model with validation
Total Recall at What Cost introduces a log log cost model with exponents p and q, showing R squared up to 0.999 and LOOCV MAPE as low as 0.029 for baselines but 0.184–0.685 for memory systems.
Break even and cost accuracy trade off analysis
Total Recall at What Cost computes break even turns from 0 to never within 400 turns and reports LoCoMo accuracies from 0.214 to 0.541 with matched cost per correct answer.
RESULTS
LOOCV MAPE full history
0.029–0.053
up to 0.436 lower than Hindsight LOOCV MAPE 0.461–0.478
LOOCV MAPE Mem0
0.184–0.222
0.224–0.294 below Mastra OM LOOCV MAPE 0.408–0.685
Accuracy Mem0 LoCoMo
0.214–0.516
span of 0.302 across backbones and reasoning levels
Cost per correct Mastra OM
0.028–0.979 USD
0.427 lower than Hindsight 0.455–1.607 at best settings
Total Recall at What Cost evaluates on a 665 question LoCoMo subset, showing that window baselines are predictable while Mem0, Hindsight, and Mastra OM have large cost model errors and widely varying cost per correct answers.
BENCHMARK
Total Recall at What Cost evaluates on a 665 question LoCoMo subset, showing that window baselines are predictable while Mem0, Hindsight, and Mastra OM have large cost model errors and widely varying cost per correct answers.
BENCHMARK
Cost per correct answer ˆC divided by Acc for Mem0, Mastra OM, and Hindsight across backbones at the 100 turn, 100 token reference cell.
KEY INSIGHT
Total Recall at What Cost finds that Hindsight can cost up to 3.3 times the full transcript at small messages, despite being a memory system designed to save tokens.
This breaks the common assumption that any agentic memory will eventually be cheaper than resubmitting the full history, showing that internal pipelines can dominate serving cost.
WHY IT MATTERS
Total Recall at What Cost gives practitioners a concrete way to reason about when Mem0, Hindsight, or Mastra OM actually reduce serving cost for long running agents.
Armed with break even turns, LOOCV errors, and cost per correct answers, builders can now choose memory systems and backbones jointly instead of relying on vague token saving claims.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…