AuthorsLuanbo Wan, Weizhi Ma
TL;DR
StoryBench uses interactive fiction with Immediate Feedback and Self Recovery modes to expose long-term memory failures, showing Doubao1.5-pro reaching 80.98% Overall Acc while still missing many story completions.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
StoryBench targets the gap where existing benchmarks fail to jointly test knowledge retention, sequential reasoning, and flexibility in realistic, evolving narratives.
Without StoryBench, long-term memory in multi-session dialogue and interactive fiction remains under-evaluated, so LLM agents break coherence and fail at self-recovery in complex branching stories.
HOW IT WORKS
StoryBench combines Dynamic Narrative and Multi-Turn Decision-Making, Two Task Modes for Evaluating LTM, and Tailored Metrics for Assessing LTM Models over a DAG of scenes and choices.
You can think of StoryBench like a branching video game where each choice writes to long-term memory, and later turns must read and reconcile those earlier saves.
StoryBench enables evaluation of long-horizon causal reasoning, self-correction, and multi-solution navigation that a plain context window benchmark with static QA cannot capture.
DIAGRAM
This diagram shows how StoryBench runs a multi-turn interactive fiction episode, alternating scene descriptions, choices, and feedback or self-recovery prompts.
DIAGRAM
This diagram shows how StoryBench constructs the dataset from The Invisible Guardian and evaluates four LLMs under Immediate Feedback and Self Recovery modes.
PROCESS
Dynamic Narrative and Multi-Turn Decision-Making
StoryBench loads a path in the Dynamic Narrative and Multi-Turn Decision-Making DAG, streaming scene nodes and choice nodes from The Invisible Guardian into the interaction.
Two Task Modes for Evaluating LTM
StoryBench assigns either Immediate Feedback or Self Recovery, controlling whether the LLM receives per-turn correctness signals or must later backtrack errors.
Tailored Metrics for Assessing LTM Models
StoryBench records decision correctness, retries, and difficulty labels, computing Overall Accuracy, First-Try Accuracy, Hard Accuracy, Retry Count, and Longest Consecutive Correct Sequence.
Experiments and Results
StoryBench runs Experiments and Results over ten trials per mode for Doubao1.5-pro, GPT-4o, Claude 3.5 Sonnet, and Deepseek-R1, aggregating Success Count and efficiency metrics.
KEY CONTRIBUTIONS
A Dynamic Multi-turn Evaluation Framework
StoryBench introduces a Dynamic Narrative and Multi-Turn Decision-Making framework with Immediate Feedback and Self Recovery modes, evaluating 80+ branching story paths in interactive fiction.
A Novel Dataset for Long-Term Memory Evaluation
StoryBench builds a Data Source and Annotation pipeline over The Invisible Guardian, yielding 311 scene nodes and 86 choice nodes organized as a directed acyclic graph.
Reliable and Robust Experimental Analysis
StoryBench runs repeated Experiments and Results on Doubao1.5-pro, GPT-4o, Claude 3.5 Sonnet, and Deepseek-R1 with ten trials per mode, reporting metrics like Overall Acc and Success Count.
RESULTS
Overall Acc (%)
80.98
+9.10 over GPT-4o in Immediate Feedback
First-Try Acc (%)
79.14
+10.93 over GPT-4o in Immediate Feedback
Hard Acc (%)
74.47
+7.53 over GPT-4o in Immediate Feedback
Success Count
8.00
highest value for Claude 3.5 Sonnet in Immediate Feedback
StoryBench evaluates LLMs on The Invisible Guardian interactive fiction benchmark, testing long-term knowledge retention and sequential reasoning. The Immediate Feedback results show Doubao1.5-pro’s 80.98% Overall Acc yet only 3.00 Success Count, while StoryBench reveals Claude 3.5 Sonnet’s 8.00 Success Count despite slightly lower accuracy.
BENCHMARK
StoryBench evaluates LLMs on The Invisible Guardian interactive fiction benchmark, testing long-term knowledge retention and sequential reasoning. The Immediate Feedback results show Doubao1.5-pro’s 80.98% Overall Acc yet only 3.00 Success Count, while StoryBench reveals Claude 3.5 Sonnet’s 8.00 Success Count despite slightly lower accuracy.
BENCHMARK
Overall Acc (%) for StoryBench Immediate Feedback mode.
KEY INSIGHT
StoryBench shows that in Self Recovery mode, First-Try Acc for Doubao1.5-pro reaches 83.05% despite Overall Acc dropping to 69.66%.
This is surprising because removing Immediate Feedback should hurt first attempts, yet StoryBench reveals that feedback can disrupt long-horizon coherence rather than always helping.
WHY IT MATTERS
StoryBench unlocks a way to stress-test long-term memory, causal backtracking, and self-recovery in realistic branching narratives instead of static long-context QA.
With StoryBench, builders can design agents that must survive multi-turn interactive fiction, debug failure chains, and benchmark memory-augmented systems under controlled, replayable story graphs.
Guilin Zhang, Wei Jiang et al.
· 2026
A-MAC scores candidate memories using Utility, Confidence, Novelty, Recency, and Type Prior combined by a learned linear admission policy with Algorithm 1 A-MAC Memory Admission. On the LoCoMo benchmark, A-MAC achieves F1 0.583 and 2644 ms latency, improving F1 by 0.042 and reducing latency by 1187 ms compared to A-mem.
Robbyant Team, Zelin Gao et al.
arXiv 2026 · 2026
LingBot-World combines a Data Engine, Fundamental World Model, Action-Conditioned World Model, and Post-Training causal adaptation to turn a 28B-parameter video generator into a real-time interactive world simulator. On the VBench benchmark, LingBot-World achieves a dynamic degree of 0.8857 versus 0.7612 for Yume-1.5, while also improving imaging quality to 0.6683.
Manoj Madushanka Perera, Adnan Mahmood et al.
· 2026
AgenticAI-DialogGen chains ChatPreprocessor, KnowledgeExtractor, TopicAnalyzer, KnowledgeGraphBuilder, PersonaGenerator, DuelingChat Agent, ConversationValidator, ConversationRefiner, QAGeneration, and PostProcessing to turn raw multi-session chats into topic-guided, persona-grounded conversations with explicit short- and long-term memories. On the TGC / KG memory QA benchmark, Mistral-7B fine-tuned within AgenticAI-DialogGen achieves 87.36 F1, compared to GPT-4’s 83.77 F1 in a zero-shot setting on the same task.
Answers use this explainer on Memory Papers.
Checking…