AuthorsYingtie Lei, Zhongwei Wan, Jiankun Zhang et al.
TL;DR
SkillEvolBench uses frozen skill libraries plus Raw-Trajectory controls to show a lossy abstraction bottleneck where distilled skills often trail raw episodic reuse on ESR 37.6% vs 44.7% ARSR.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
SkillEvolBench shows that Raw-Trajectory achieves mean ESR 37.6% and ARSR 44.7%, while distilled skills frequently underperform on these deployment metrics.
This means episodic agent work in realistic environments becomes brittle task-local memory instead of durable procedural skills, breaking robustness under context shift and composition.
HOW IT WORKS
SkillEvolBench connects Episodic Task Attempts, Structured Verifier Feedback, Skill Author, and an Environment-specific Shared Skill Library to evolve skills before frozen deployment.
You can think of SkillEvolBench like a software team’s playbook: noisy incident logs are periodically distilled into reusable runbooks instead of replaying old chat transcripts.
This design lets SkillEvolBench test selective procedural abstraction that a plain context window or naive trajectory replay cannot, especially under adversarial shortcuts and multi-skill composition.
DIAGRAM
This diagram shows how SkillEvolBench moves a family through canonical, enriched, variant, context-shift, adversarial, and composition roles.
DIAGRAM
This diagram shows how SkillEvolBench initializes libraries, runs acquisition, freezes skills, evaluates deployment, replays, and then resets per environment.
PROCESS
Initialization and Skill Conditions
SkillEvolBench sets Sc(e, f) for each environment and family, choosing between experience-based self-generated, zero-shot, or gap-exposed curated skills.
Acquisition From Episodic Evidence to Skill Updates
SkillEvolBench runs canonical, enriched, and variant roles, compacts trajectories into ˜τ and pairs them with Structured Verifier Feedback vr for the Skill Author.
Frozen Deployment and Replay
SkillEvolBench freezes the Environment-specific Shared Skill Library, then measures ESR, CSSR, ARSR, and CompSR plus replay LSR and RSR without further updates.
Capacity Diagnostic With Tier-3 Forcing
SkillEvolBench activates Always+Tier3, forcing new scripts, references, or assets per revision to test whether larger libraries improve frozen deployment.
KEY CONTRIBUTIONS
SkillEvolBench benchmark design
SkillEvolBench introduces 180 tasks across six environments and five task families each, with role-conditioned arcs and an Environment-specific Shared Skill Library plus Skill Author loop.
Procedural vs episodic reuse analysis
SkillEvolBench contrasts curated and self-generated skills against No-Skill and Raw-Trajectory, showing Raw-Trajectory mean ESR 37.6% and ARSR 44.7% as strong baselines.
Capacity and cost diagnostics
SkillEvolBench uses Always and Always+Tier3 policies to show that larger skill libraries often add cost without improving ESR, revealing procedural clutter and episode-specific drift.
RESULTS
RSR
48.2%
+? vs No-Skill (Raw-Trajectory strongest replay baseline)
ESR
37.6%
Raw-Trajectory deployment success across models and harnesses
ARSR
44.7%
Raw-Trajectory adversarial robustness under frozen deployment
CompSR
25.7%
Raw-Trajectory composition success vs skill-based conditions
SkillEvolBench evaluates across 180 tasks and six environments, decomposing deployment into ESR, CSSR, ARSR, and CompSR. The MAIN_RESULT shows Raw-Trajectory dominates many skill variants on ESR 37.6%, ARSR 44.7%, and CompSR 25.7%, revealing a lossy abstraction bottleneck.
BENCHMARK
SkillEvolBench evaluates across 180 tasks and six environments, decomposing deployment into ESR, CSSR, ARSR, and CompSR. The MAIN_RESULT shows Raw-Trajectory dominates many skill variants on ESR 37.6%, ARSR 44.7%, and CompSR 25.7%, revealing a lossy abstraction bottleneck.
BENCHMARK
ESR across SkillEvolBench for Raw-Trajectory and representative skill variants.
KEY INSIGHT
SkillEvolBench finds Raw-Trajectory achieves mean ESR 37.6% and ARSR 44.7%, while many distilled skill variants score lower on these deployment metrics.
This is surprising because curated and self-generated skills were expected to compress and improve episodic evidence, yet SkillEvolBench shows abstraction often discards useful contextual and procedural cues.
WHY IT MATTERS
SkillEvolBench gives researchers a controlled way to measure when episodic experience becomes durable procedural knowledge instead of task-local patches.
With SkillEvolBench, builders can systematically stress-test new skill authoring, retrieval, and capacity strategies under context shift, adversarial shortcuts, and multi-skill composition before deploying agentic systems.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…