AuthorsHyojeong Yu, Hyukhun Koh, Minsung Kim et al.
TL;DR
PRAGMA uses a controlled conversational history pipeline and a four-way query taxonomy to show that even oracle evidence access leaves grounding below 86.00 on trajectory guidance.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
PRAGMA shows that No-Context responses have near-zero grounding, for example 0.00 grounding on Trajectory-Align queries.
In long-term personalized guidance, systems ignore distributed evidence, causing misaligned recommendations and failures on corrective reasoning over evolving user trajectories.
HOW IT WORKS
PRAGMA builds on a User Schema, annotated Evidence, long Sessions, and diverse Queries to stress-test memory-grounded guidance.
Think of PRAGMA as a carefully scripted life log plus a card catalog: events and trajectories are stored, indexed, and then probed with tricky guidance questions.
This design lets PRAGMA expose failures that a plain context window cannot, especially when evidence is temporally scattered and user assumptions are memory-misaligned.
DIAGRAM
This diagram shows how PRAGMA organizes guidance queries by memory dynamics and alignment with conversational evidence.
DIAGRAM
This diagram shows how PRAGMA builds synthetic users, generates conversations, and annotates evidence for evaluation.
PROCESS
Query Design
PRAGMA first classifies the user request into Event-Align, Event-Correct, Traj-Align, or Traj-Correct using the Query Taxonomy to control memory dynamics.
Conversation History Construction
PRAGMA expands Evidence Events and Trajectory states into Event Sessions and Trajectory Sessions, interleaving Filler Sessions to create 160K-token histories per user.
Evidence Annotation
PRAGMA annotates which Sessions contain necessary evidence for each query, enabling session-level Recall and Exact Recall evaluation of retrieval systems.
Response Evaluation
PRAGMA scores generated guidance with alignment and grounding rubrics, using LLM judges to assess whether responses match trajectories and cite annotated evidence.
KEY CONTRIBUTIONS
PRAGMA benchmark introduction
PRAGMA provides 100 users, 400 queries, and about 160K conversational tokens per user, covering Event-Align, Event-Correct, Traj-Align, and Traj-Correct guidance scenarios.
Controlled construction pipeline
PRAGMA uses a controlled pipeline with User Schema, Event Sessions, Trajectory Sessions, and Filler Sessions plus human validation to avoid recency shortcuts and shallow trajectories.
Joint retrieval and guidance evaluation
PRAGMA evaluates retrieval Recall and Exact Recall alongside alignment and grounding, revealing that Oracle-Summary can reach 99.00 alignment but Oracle-Session lags far behind.
RESULTS
Alignment Traj-Align Oracle Summary
99.00
+21.00 over Oracle Session gpt-5-mini
Grounding Traj-Align Oracle Summary
86.00
+69.00 over No-Context gpt-5-mini
Alignment Event-Align Oracle Summary
99.00
+15.00 over Oracle Session gpt-5-mini
Grounding Event-Align Oracle Summary
97.00
+58.00 over Oracle Session gpt-5-mini
These metrics come from PRAGMA’s Oracle-Session and Oracle-Summary settings on Event-Align and Traj-Align queries. They show that compressed evidence summaries in PRAGMA dramatically improve guidance quality compared to raw session context, quantifying the memory utilization gap.
BENCHMARK
These metrics come from PRAGMA’s Oracle-Session and Oracle-Summary settings on Event-Align and Traj-Align queries. They show that compressed evidence summaries in PRAGMA dramatically improve guidance quality compared to raw session context, quantifying the memory utilization gap.
BENCHMARK
Alignment scores on Traj-Align queries in PRAGMA for different evidence settings.
KEY INSIGHT
On Traj-Align queries, PRAGMA’s Oracle-Summary reaches 99.00 alignment and 86.00 grounding, while Oracle-Session manages only 78.00 alignment and 17.00 grounding.
This is surprising because gold evidence sessions are fully available, yet PRAGMA shows systems still fail to synthesize them without explicit summarization.
WHY IT MATTERS
PRAGMA unlocks a way to separately stress-test retrieval, memory preservation, and evidence utilization in personalized guidance over long conversations.
With PRAGMA, builders can design memory architectures that optimize not just recall but generation-usable summaries, moving beyond naive long-context conditioning.
Jingbiao Mei, Jinghong Chen et al.
arXiv 2026 · 2026
ATM-Bench structurally evaluates long-term multimodal personal memory using Memory Ingestion, Retrieval, and Answer Generation with Schema-Guided Memory and Descriptive Memory variants. On ATM-Bench-Hard, Oracle with SGM reaches 47.3% QS while the best full system stays under 20% accuracy, revealing a large gap.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Answers use this explainer on Memory Papers.
Checking…