DolphinBench: Mapping the Pareto Frontier of Agent Memory

AuthorsSoumil Rathi, Deshraj Yadav, Taranjeet Singh

2026

TL;DR

DolphinBench uses solvability-verified action tasks plus cost and latency reporting to map a 70.67% accuracy Pareto frontier for agent memory systems.

SharePost on X LinkedIn

Read our summary here, or open the publisher PDF on the next tab.

THE PROBLEM

Memory benchmarks miss retrieval decisions and real costs

DolphinBench targets benchmarks where a direct question “acts as a signal” and “the evaluation effectively begins only after the most difficult step”. Existing memory benchmarks also “rarely require anything beyond accuracy”, letting systems “make unreasonable cost time tradeoffs” without reporting latency or monetary cost.

This failure mode affects long-running agents that must “keep track of information across project histories and past sessions that outgrow their context windows”. As a result, memory systems can look strong on QA scores while remaining unusable in real deployments due to hidden cost and latency.

HOW IT WORKS

DolphinBench — action-based tasks with verified solvability

DolphinBench’s core mechanism is a staged pipeline of Personas, History Planning, History Generation, Test Construction, and Test Verification that builds multi-year, tool-grounded histories. DolphinBench then issues tasks whose correct completion depends on specific earlier facts, grading via deterministic checks and LLM judges.

You can think of DolphinBench like a carefully written project diary plus a test suite: the histories are the diary, and the tasks are end-to-end integration tests that only pass if the agent “knows to remember”. Instead of reading a static transcript, agents must act in simulated apps, similar to executing workflows against a controlled sandbox.

This design lets DolphinBench test retrieval decisions, not just recall, and compare accuracy, cost, and latency together, something a plain context window benchmark or pure QA dataset cannot provide.

DIAGRAM

Agent interaction and grading flow in DolphinBench

This diagram shows how an agent interacts with DolphinBench histories, tools, and grading checks during task evaluation.

DIAGRAM

DolphinBench construction and verification pipeline

This diagram shows how DolphinBench builds personas, histories, and tests, then verifies solvability with and without oracle messages.

PROCESS

How DolphinBench Handles a Memory-Dependent Task

01
History Generation
DolphinBench uses History Generation to turn quarterly and weekly plans into user messages aligned with app records, ensuring each persona’s history reaches roughly 500k tokens.
02
Test Construction
During Test Construction, DolphinBench proposes tasks tied to specific facts, writes user instructions, and designs grading checks that require using those earlier facts.
03
Test Verification
In Test Verification, DolphinBench runs GPT-5.6-Luna twice with oracle messages and twice without them, requiring two passes and two failures per task.
04
Grading and Metrics
In Grading and Metrics, DolphinBench applies deterministic and LLM-based checks, then aggregates accuracy, total cost, and median latency across all 600 tasks.

KEY CONTRIBUTIONS

Key Contributions

01
Action-based memory evaluation
DolphinBench replaces QA with action tasks across three personas and 600 tests, where success depends on using history to act in simulated apps.
02
Cost latency accuracy triad
DolphinBench requires every submission to report accuracy, total dollar cost, and median task latency, exposing tradeoffs like 70.67% accuracy at $96.21 and 37.69 seconds.
03
Solvability-certified tests
DolphinBench verifies each test by requiring GPT-5.6-Luna to pass twice with oracle messages and fail twice without them, ensuring tasks are both solvable and memory-dependent.

RESULTS

By the Numbers

Accuracy (%)

70.67%

+5.00 over GPT-5.6-Luna built-in memory with Hermes

Total cost ($)

96.21

vs 61.48 for GPT-5.6-Luna built-in memory with Hermes

Median latency (s)

37.69

6.66 seconds faster than GPT-5.6-Luna built-in memory with Hermes

Tasks completed

600

200 per persona across Morgan, Alex, and Riley

On DolphinBench’s 600 action-based tasks, Hermes with GPT-5.6-Luna and Mem0 reaches 70.67% accuracy while reporting $96.21 total cost and 37.69 seconds median latency. This proves DolphinBench can surface concrete accuracy–cost–latency tradeoffs for agent memory configurations.

BENCHMARK

By the Numbers

BENCHMARK

Results on DolphinBench: comparing the accuracy, cost, and latency of agents using different memory systems

Accuracy (%) on DolphinBench’s 600 tasks for Hermes with GPT-5.6-Luna across memory systems.

KEY INSIGHT

The Counterintuitive Finding

With Hermes and GPT-5.6-Luna, DolphinBench shows Mem0 reaching 70.67% accuracy while reducing median latency from 44.35 to 37.69 seconds. At the same time, total cost rises from $61.48 to $96.21.

This is surprising because we might expect higher accuracy to always require slower responses, yet DolphinBench reveals a configuration that is both more accurate and faster than built-in memory, breaking the simple “accuracy versus speed” assumption.

WHY IT MATTERS

What this unlocks for the field

DolphinBench unlocks a way to evaluate agent memory as end-to-end behavior, including retrieval decisions, action correctness, and real-world cost and latency. Builders can now choose memory systems along a measured Pareto frontier, rather than relying on QA scores, and design agents whose memory is tuned to their deployment’s accuracy, budget, and responsiveness constraints.

~12 min read← Back to papers

Related papers

SurveyAgent Memory

Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

Dongming Jiang, Yi Li et al.

arXiv 2026 · 2026

Anatomy of Agentic Memory organizes agentic memory into four structures using components like Lightweight Semantic Memory, Entity-Centric and Personalized Memory, Episodic and Reflective Memory, and Structured and Hierarchical Memory. Anatomy of Agentic Memory then reports comparative results such as Nemori’s 0.781 semantic judge score on LoCoMo versus SimpleMem’s 0.298, and latency differences like 1.129s for Nemori versus 32.372s for MemoryOS.

arXiv:2602.19320 Read explainer

SurveyBenchmarkAgent MemoryLong-Term MemoryMemory Architecture

A Survey on the Security of Long-Term Memory in LLM Agents: Toward Mnemonic Sovereignty

Zehao Lin, Chunyu Li, Kai Chen

· 2026

Mnemonic Sovereignty analyzes long term Write, Store, Retrieve, Execute, Share, and Forget Rollback phases against integrity, confidentiality, availability, and governance objectives for agent memory. Mnemonic Sovereignty’s lifecycle matrix shows most of the ~70 works cluster on write and retrieve integrity, leaving store, availability, and governance primitives like write gate validation and post deletion verification almost entirely unexplored.

arXiv:2604.16548 Read explainer

SurveyRAGAgent Memory

Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers

Pengfei Du

· 2026

Memory for Autonomous LLM Agents decomposes agent memory into a POMDP-grounded write–manage–read loop, a three-dimensional taxonomy, and five mechanism families spanning context compression, retrieval stores, reflection, hierarchical virtual context, and policy-learned management. Memory for Autonomous LLM Agents synthesizes results like Voyager’s 15.3× tech-tree speedup and MemoryArena’s 80%→45% drop to show that memory architecture often matters more than backbone choice.

arXiv:2603.07670 Read explainer

Questions about this paper?

Answers use this explainer on Memory Papers.

Checking…