AuthorsShuochen Liu, Junyi Zhu, Long Shu et al.
TL;DR
PERMA uses an event-driven persona state with Clean, Noise, and style-aligned long-context variants to expose where memory agents fail to maintain coherent personalization over 1.8M-token interaction histories.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing personalization benchmarks interleave sparse preference snippets with irrelevant dialogue, reducing evaluation to needle in a haystack retrieval.
Under these settings, agents treat preferences as static facts, ignore event-driven evolution, and fail to maintain a coherent persona state across domains and sessions.
HOW IT WORKS
PERMA builds a timeline of Interaction Events, maintains a dynamic Persona State, and evaluates dual-process Memory Systems under Clean, Noise, and style-aligned long-context settings.
You can think of PERMA as a long-term user diary plus an indexed card catalog, where events are diary pages and persona states are curated summary cards.
This event-driven construction lets PERMA probe evolving preferences and cross-domain synthesis in ways a plain context window or static RAG corpus cannot.
DIAGRAM
This diagram shows how PERMA sequences Emergence and Supplement events and inserts Type 1, Type 2, and Type 3 tasks along the interaction timeline.
DIAGRAM
This diagram shows how PERMA constructs datasets, injects noise, and runs MCQ plus interactive evaluations with memory systems and standalone LLMs.
PROCESS
Timeline generation
PERMA uses a timeline generation agent (fplan) to turn domain summaries into ordered Interaction Events with Emergence, Supplement, and Task types plus dependencies.
Dialogue generation
PERMA prompts a dialogue generation agent Pg to expand each event into multi-turn dialogues and extract explicit or implicit preferences Φt into the evolving Persona State.
Noise injection
PERMA injects text variability using five in-session noise types and aligns user queries with WildChat idiolects to stress Memory Systems under realistic prompt biases.
Task insertion and evaluation
PERMA inserts Type 1, Type 2, and Type 3 tasks, then runs MCQ and interactive evaluation to test how well agents use the Persona State across temporal depth and domains.
KEY CONTRIBUTIONS
Event-driven persona state benchmark
PERMA builds 808 events, 3.5k Clean dialogs, and 324k Clean tokens across 10 users and 20 domains, modeling evolving preferences via Interaction Events and a dynamic Persona State.
Realistic noise and style alignment
PERMA injects five prompt bias noise types and aligns queries with WildChat idiolects, creating Clean, In-session Noise, and Style-aligned Long-context datasets totaling 1,165.4k tokens.
Decoupled memory evaluation protocol
PERMA separately scores retrieval (BERT-f1, Memory Score) and generation (MCQ, Completion), revealing how Memory Systems like MemOS trade token compression for persona consistency.
RESULTS
MCQ Acc. (Clean single, MemOS)
0.811
+0.109 over RAG (0.702)
Memory Score (Clean single, MemOS)
2.27
+0.38 over Mem0 (1.89)
Context Token (Clean single, MemOS)
709.1 tokens
vs 34,078.6 tokens for standalone LLMs
Completion (Clean single, EverMemOS)
0.846
+0.016 over MemOS (0.830) with 3,230.5ms search duration
PERMA’s Clean single-domain benchmark tests task completion, preference consistency, and informational confidence under multi-session personalization. These results show that MemOS, evaluated within PERMA, can compress long histories while improving MCQ accuracy and Memory Score compared to vanilla RAG and raw-context standalone LLMs.
BENCHMARK
PERMA’s Clean single-domain benchmark tests task completion, preference consistency, and informational confidence under multi-session personalization. These results show that MemOS, evaluated within PERMA, can compress long histories while improving MCQ accuracy and Memory Score compared to vanilla RAG and raw-context standalone LLMs.
BENCHMARK
MCQ Acc. on PERMA Clean single-domain tasks for GPT-4o-mini-based memory agents.
KEY INSIGHT
Under Noise, MemOS increases MCQ Acc. from 0.811 to 0.853 by retrieving more tokens (709.1 to 1,486.7), despite added in-session ambiguity.
This is surprising because we usually expect noisy prompts and longer retrieved contexts to hurt accuracy, not improve persona-state synthesis over multi-session histories.
WHY IT MATTERS
PERMA unlocks a way to stress-test memory agents on evolving, cross-domain persona states instead of static preference recall in toy dialogues.
Builders can now design and tune Memory Systems that explicitly manage persona drift, noise robustness, and long-context efficiency, using PERMA’s event-driven benchmarks as a realistic target.
Guilin Zhang, Wei Jiang et al.
· 2026
A-MAC scores candidate memories using Utility, Confidence, Novelty, Recency, and Type Prior combined by a learned linear admission policy with Algorithm 1 A-MAC Memory Admission. On the LoCoMo benchmark, A-MAC achieves F1 0.583 and 2644 ms latency, improving F1 by 0.042 and reducing latency by 1187 ms compared to A-mem.
Robbyant Team, Zelin Gao et al.
arXiv 2026 · 2026
LingBot-World combines a Data Engine, Fundamental World Model, Action-Conditioned World Model, and Post-Training causal adaptation to turn a 28B-parameter video generator into a real-time interactive world simulator. On the VBench benchmark, LingBot-World achieves a dynamic degree of 0.8857 versus 0.7612 for Yume-1.5, while also improving imaging quality to 0.6683.
Manoj Madushanka Perera, Adnan Mahmood et al.
· 2026
AgenticAI-DialogGen chains ChatPreprocessor, KnowledgeExtractor, TopicAnalyzer, KnowledgeGraphBuilder, PersonaGenerator, DuelingChat Agent, ConversationValidator, ConversationRefiner, QAGeneration, and PostProcessing to turn raw multi-session chats into topic-guided, persona-grounded conversations with explicit short- and long-term memories. On the TGC / KG memory QA benchmark, Mistral-7B fine-tuned within AgenticAI-DialogGen achieves 87.36 F1, compared to GPT-4’s 83.77 F1 in a zero-shot setting on the same task.
Answers use this explainer on Memory Papers.
Checking…