Memory Papers
Memory Architecture

OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents

AuthorsYulin Hu, Zimo Long, Jiahe Guo et al.

2026
arXivPDF

TL;DR

OP-Bench + Self-ReCheck filters user memories before use, cutting over-personalization scores by 29% while BASE memory-free agents stay strongest on OP-Bench.

SharePost on XLinkedIn

Read our summary here, or open the publisher PDF on the next tab.

THE PROBLEM

Memory hijacking: OP-Bench shows 26.2%–61.1% score drops with memory

OP-Bench reveals that adding memory causes over-personalization, with OP-Bench scores dropping 26.2%–61.1% compared to BASE agents without memory.

These failures appear in long-horizon personalized dialogue, where memory-augmented systems become irrelevant, repetitive, or sycophantic, degrading factual accuracy and user comfort.

HOW IT WORKS

OP-Bench and Self-ReCheck: structured OP tasks plus memory filtering

OP-Bench builds tasks from Irrelevance, Repetition, and Sycophancy categories using LoCoMo dialogues, then scores outputs with an LLM-based evaluator and attention analysis.

Think of OP-Bench as a stress test for a memory-augmented agent’s “social RAM,” while Self-ReCheck acts like a smart cache that only forwards truly relevant memory lines.

This combination lets OP-Bench expose memory hijacking and Self-ReCheck selectively gate memories, enabling behavior that a plain context window or naive RAG pipeline cannot control.

DIAGRAM

OP-Bench query construction pipeline from LoCoMo

This diagram shows how OP-Bench constructs evaluation queries from LoCoMo dialogues across three stages.

DIAGRAM

Evaluation setup: OP-Bench plus LoCoMo across models and memory methods

This diagram shows how OP-Bench is paired with LoCoMo to evaluate multiple LLMs and six memory-augmentation methods.

PROCESS

How OP-Bench Handles a Long-Horizon Personalized Dialogue Session

  1. 01

    Data Preprocessing

    OP-Bench preprocesses LoCoMo dialogues, treating speaker_a as user and speaker_b as assistant to derive structured profiles and topics.

  2. 02

    Task Construction

    OP-Bench builds Irrelevance, Sycophancy, and Repetition tasks using LLM-based generation, including Fully Irrelevant and Baiting queries plus multi-question repetition sets.

  3. 03

    Human Review

    OP-Bench sends each instance through double human review, accepting only items with annotator consensus or senior adjudication.

  4. 04

    Model Evaluation

    OP-Bench evaluates 36 configurations of LLMs and memory methods, then applies Self-ReCheck to filter memories and recompute OP-Bench and LoCoMo scores.

KEY CONTRIBUTIONS

Key Contributions

  • 01

    Formalizing Over-Personalization Types

    OP-Bench formalizes Irrelevance, Repetition, and Sycophancy into three categories and six subcategories, then builds 1,700 instances across 20 users from LoCoMo.

  • 02

    OP-Bench Benchmark and Analysis

    OP-Bench evaluates 36 configurations of GPT-4o-mini, Gemini-2.5-flash, DeepSeek-v3.2, Qwen3-235B-A22B, Qwen3-32B, and Qwen3-8B with BASE, RAG, LDAgent, Mem0, MemU, and MEMOS.

  • 03

    Self-ReCheck Memory Filtering

    OP-Bench introduces Self-ReCheck, a lightweight LLM-based memory filter that reduces over-personalization by 29% on Qwen3-8B while improving personalization by 3% on LoCoMo.

RESULTS

By the Numbers

AVG.(OP) GPT-4o-mini BASE

83.10 score

+27.14 over GPT-4o-mini RAG

AVG.(OP) Gemini BASE

70.55 score

+18.45 vs Gemini-2.5-flash RAG

AVG.(OP) Qwen3-32B BASE

72.91 score

+25.18 context: drop vs Qwen3-32B RAG

AVG.(OP) Qwen3-8B BASE

73.80 score

+27.42 context: drop vs Qwen3-8B RAG

OP-Bench scores measure how well agents avoid over-personalization across Irrelevance, Repetition, and Sycophancy. BASE memory-free agents achieve the highest OP-Bench scores, while memory methods cause 26.2%–61.1% relative drops, proving OP-Bench exposes memory hijacking effects.

BENCHMARK

By the Numbers

OP-Bench scores measure how well agents avoid over-personalization across Irrelevance, Repetition, and Sycophancy. BASE memory-free agents achieve the highest OP-Bench scores, while memory methods cause 26.2%–61.1% relative drops, proving OP-Bench exposes memory hijacking effects.

BENCHMARK

Evaluation results on OP-Bench (higher scores indicate less over-personalization)

AVG.(OP) scores across memory methods for GPT-4o-mini from Table 2.

BENCHMARK

Performance of memory-augmented methods on LoCoMo multi-hop categories using GPT-4o-mini

Overall F1 on LoCoMo for GPT-4o-mini with different memory methods from Table 3.

KEY INSIGHT

The Counterintuitive Finding

OP-Bench shows that sophisticated memory systems like MEMOS and MemU, which summarize and structure events, suffer the largest OP-Bench score drops up to 61.1%.

This is counterintuitive because developers expect richer memory to help personalization, yet OP-Bench reveals that more powerful memory modules can actually amplify over-personalization and memory hijacking.

WHY IT MATTERS

What this unlocks for the field

OP-Bench unlocks a way to quantify when personalization becomes intrusive, letting builders tune memory systems against Irrelevance, Repetition, and Sycophancy explicitly.

With OP-Bench and Self-ReCheck, developers can now deploy memory-augmented agents that retain long-term personalization benefits while algorithmically filtering memories to respect immediate user intent.

~12 min read← Back to papers

Related papers

Memory Architecture

A Control Architecture for Training-Free Memory Use

Yanzhen Lu, Muchen Jiang et al.

· 2026

TAG routes low-confidence steps to uncertainty-based routing, filters them with guarded acceptance with rollback, chooses between bank selection across rule and exemplar memory, and prunes via evidence-based retirement inside a unified control loop. On SVAMP and ASDiv, TAG reaches 81.0% and 85.2% accuracy, improving over the 74.0% and 77.5% no-memory baselines while a compute-matched Retry baseline stays flat.

arXiv:2604.18206Read explainer

Questions about this paper?

Paper: OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents

Answers use this explainer on Memory Papers.

Checking…