AuthorsLingyang Zeng, Guangze Chen, Kaichen Yu et al.
TL;DR
Setoka uses a psychology-grounded four-level hierarchy plus a psychometrics-based generation pipeline to expose that accuracy drops from 0.85 (SM) to 0.24 (PT).
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing memory benchmarks mainly assess whether an agent can retrieve information explicitly stated in conversational histories, ignoring deeper user understanding.
When behavior pattern and personality trait tasks require integrating heterogeneous and fragmented information dispersed over time, performance declines even further, limiting personalized assistance quality.
HOW IT WORKS
Setoka combines User Understanding Hierarchy, correlation-aware personality trait sampling, psychological-scale-based behavior pattern generation, and an event-grounded generation tree to generate traceable multi-level user data.
Think of Setoka like a layered memory system: abstract Personality Traits guide Behavior Patterns, which expand into Episodic Memory events and concrete Semantic Memory records, all grounded in psychometrics.
This design lets Setoka test selection, linking, aggregation, and generalization across heterogeneous sources, which a plain context window or single-dialogue benchmark cannot evaluate.
DIAGRAM
This diagram shows how Setoka organizes Semantic Memory, Episodic Memory, Behavior Patterns, and Personality Traits into a nested evidence hierarchy.
DIAGRAM
This diagram shows how Setoka generates personality traits, behavior patterns, events, and heterogeneous records for each synthetic user.
PROCESS
Memory Construction
Setoka applies the preprocessing operator P to serialize heterogeneous user corpus D into a normalized memory store M, aligning with the User Understanding Hierarchy.
Retrieval
Setoka uses the retriever R over M to obtain an evidence set Ê tailored to the query level, supporting selection, linking, aggregation, or generalization.
Answer Generation
Setoka feeds the query q and retrieved evidence Ê into the answer generator G to produce a response â for the UUQA instance.
Level Specific Evaluation
Setoka scores â against reference a using Vℓ, employing an LLM judge for SM EM BP and Kendall rank correlation for PT trait rankings.
KEY CONTRIBUTIONS
Psychology grounded user understanding framework
Setoka formalizes a four level User Understanding Hierarchy with Semantic Memory, Episodic Memory, Behavior Pattern, and Personality Trait, each with explicit operators σ λ γ ρ.
Psychometrics based generation pipeline
Setoka introduces correlation aware personality trait sampling and psychological scale based behavior pattern generation to create coherent personas grounded in Big Five BFI 2 data.
Comprehensive evaluation over heterogeneous data
Setoka instantiates 10 synthetic users with 23 schemas over 3 data models and evaluates 3 language models with 5 memory systems plus DBQuery on UUQA.
RESULTS
SM level average
37 (scaled score)
drops 14 points to EM level average 23
EM best score
46 (scaled score)
+19 over DBQuery EM 27 with DeepSeek V4 Flash
BP level average
16 (scaled score)
10 points lower than SM level average 37
PT best score
24 (scaled score)
only 17 points above random like PT level average 7
Setoka reports LLM judge similarity for SM EM BP and Kendall rank correlation for PT on UUQA over 10 users, revealing monotonic accuracy decay from concrete memories to abstract traits.
BENCHMARK
Setoka reports LLM judge similarity for SM EM BP and Kendall rank correlation for PT on UUQA over 10 users, revealing monotonic accuracy decay from concrete memories to abstract traits.
BENCHMARK
LLM judge similarity scores on Semantic Memory queries using DeepSeek V4 Flash.
KEY INSIGHT
On Setoka, the best configuration scores 0.85 on Semantic Memory but only 0.24 on Personality Traits, despite using the same backbone.
This is surprising because many assume strong LLMs with memory systems can generalize user traits easily, yet Setoka shows trait ranking is only marginally above chance.
WHY IT MATTERS
Setoka gives researchers a controlled way to test selection, linking, aggregation, and generalization for hierarchical user understanding over heterogeneous data.
Builders can now benchmark memory systems on realistic multi source personas, designing mechanisms for cross source integration and abstraction beyond simple fact retrieval.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…