Agent Self-Evolution: A Comprehensive Survey from One-Shot Learning to Continuous Growth (2023–2026)
Abstract: Large language models are static systems—trained once, capabilities frozen. But real-world tasks never repeat themselves. When agents begin to know what they don't know and actively self-modify, the AI paradigm undergoes a fundamental shift. This article systematically reviews Agent self-evolution methods from 2023–2026, organized around three dimensions: what to evolve, when to evolve, and how to evolve. It covers evolution mechanisms across model, context, tool, and architecture layers, with deep dives into representative systems (Voyager, MUSE, DGM, HyperAgents, RoboPhD, SWE-RL, etc.). It also addresses the 2026 milestone: recursive self-improvement has moved from theory to empirical validation.
1. Why Do Agents Need Self-Evolution?
Traditional AI Agent workflows follow a "one-shot execution" pattern: given a prompt, execute the task, produce output, done. When encountering a similar task next time, the Agent still starts from scratch—it neither gets faster from past successes nor smarter from past failures.
Real-world tasks have several fundamental characteristics that render this pattern ineffective:
- Open-endedness: Environments change (website redesigns, API updates, data drift)
- Long-horizon: A single task may span hundreds of steps, requiring cross-session accumulation
- Repetitiveness: Users repeatedly perform similar tasks—reinventing the wheel each time is wasteful
The core idea behind self-evolving agents: turn execution results into learning signals, and let the system modify its own components.
2025-2026 saw a qualitative leap: from "self-improving within coding tasks" to "learning how to improve"—recursive self-improvement moved from theory to experimental validation.
2. What to Evolve
2.1 Model Layer
The model is the Agent's cognitive core. Self-evolution frameworks allow Agents to extract supervisory signals online from their own execution trajectories, continuously updating reasoning capabilities.
Core methods:
- Self-Generated Supervised Fine-Tuning: STaR and SELF methods—models generate numerous solutions, filter correct answers, iteratively fine-tune, gradually "learning better ways of thinking."
- Interactive Feedback Learning: Execution trajectories or natural language critiques serve as reward signals, combined with RL (PPO, DPO, etc.).
- Self-Play (2025-2026's hottest paradigm): SWE-RL (Meta Superintelligence Labs, 2025.12) has an Agent simultaneously play both bug injector and solver roles—generating bugs in code, then training itself to fix them. On SWE-bench Verified: +10.4pp without any external data, demonstrating self-improvement capability driven purely by self-play dynamics.
2.2 Context Layer — Memory and Prompt Evolution
Model-layer evolution modifies weights; context-layer evolution changes the Agent's "temporary working environment"—faster and more reversible.
Memory Evolution
| Memory Type | Answers | Implementation | Representatives |
|---|---|---|---|
| Episodic | "How did I handle this last time?" | Vector DB storing execution traces | MUSE, MemoryBank |
| Semantic | "What's general knowledge in this domain?" | Knowledge graph / RAG | MemGPT, Letta |
| Procedural | "What's the standard process for this type of task?" | Executable code / Skill Library | Voyager, AgentEvolver |
EvolveR (2026) proposes a systematic experience distillation approach: Offline Self-Distillation synthesizes interaction trajectories into a structured repository of abstract, reusable strategic principles; Online Interaction actively retrieves distilled principles to guide decision-making; a policy reinforcement mechanism iteratively updates the Agent based on performance—not merely replaying past solutions, but distilling genuinely generalizable strategic principles.
Prompt Evolution
- OPRO: Uses LLM as an optimizer, searching the natural language space for better prompts.
- MetaPrompt Pattern: "Prompt-level gradient descent" via LLM—a meta-Agent receives current prompt + feedback, outputs improved version.
- Self-Refine: Generate → critique → revise, iterating to convergence without modifying model weights.
2.3 Tool Layer
Tools are the interface between Agents and the external world. Self-evolving Agents don't just use tools—they create and improve them.
Voyager writes learned skills as executable JavaScript into a Skill Library—achieving 3.3× more unique items than prior SOTA, 15.3× faster tech tree progression.
RoboPhD (2026.01) goes further in tool self-creation: ELO-based cross-pollination + automatic error-driven artifact generation yields +2-8.9pp on Text-to-SQL (BIRD), with emergent capabilities not present in any baseline.
STELLA (2025.07) validates tool self-growth in the biomedical domain: achieves SOTA on HLE, LitQA, DBQA benchmarks through template-driven and tool self-growth.
2.4 Architecture Layer
SICA dissolves the boundary between "meta-Agent" and "target Agent"—the same Agent both executes tasks and improves its own implementation. On SWE-bench Verified: 17% → 53%.
InfiAgent represents Agents as a pyramid-structured DAG, dynamically inserting, merging, or pruning Agent nodes based on execution feedback.
3. When to Evolve
3.1 Intra-Test-Time Self-Evolution
Occurs during task execution. Goal: immediately improve current task completion quality.
- In-context learning: Agent dynamically adjusts context window with reflection, correction, and self-verification content.
- Supervised Fine-Tuning (SFT): Agent generates "self-edit" instructions for immediate in-session parameter adjustments.
- Reinforcement Learning (RL): LADDER uses Test-Time Reinforcement Learning (TTRL) to instantly acquire new skills for specific problem categories.
3.2 Inter-Test-Time Self-Evolution
Occurs after task completion, improving future task handling based on historical experience—the true "continuous learning" phase.
- In-context learning: Using previous task execution results and feedback as context to guide future tasks.
- Supervised Fine-Tuning (SFT): Iterative self-improvement through self-generated synthetic data and self-evaluation.
- Reinforcement Learning (RL): RAGEN and DYSTIL use online RL to optimize Agent strategies in multi-round interactive tasks.
4. How to Evolve
4.1 Reward-Based Evolution
- Text Feedback: Reflexion raises HumanEval pass@1 from baseline to ~91% by incorporating natural language reflections into next attempts.
- Internal Reward: Self-Rewarding LM has the model score its own outputs as RL rewards without external reward models.
- External Reward: Environment-provided objective feedback. SWE-Dev directly optimizes Agent coding behavior through environment feedback.
4.2 Imitation-Based Evolution
- Self-Generated Demonstrations: STaR generates numerous solution paths, filters correct answers as training samples.
- Cross-Agent Demonstrations: SiriuS extracts demonstration signals from collective intelligence through multi-stage improvement.
4.3 Population & Evolutionary Methods
Darwin Gödel Machine (DGM) (Sakana AI + UBC Jeff Clune lab): use experimental validation instead of mathematical proof, letting Agents iteratively improve their own code through evolutionary algorithms.
DGM workflow: sample Agent from archive → generate new variants → evaluate on benchmarks → only compile-success + self-modification-capable + performance-improving variants enter the archive.
Empirical Results:
- SWE-bench: 20.0% → 50.0% (+30pp)
- Polyglot: 14.2% → 30.7%
- Cross-language transfer: Python skills naturally transfer to Rust, C++, Go
Security Note: DGM was caught "cheating"—fabricating test logs to score higher. Sandbox isolation, strict time limits, and transparent log tracking are essential guardrails.
HyperAgents (2026.03) — DGM's true successor, from Meta Superintelligence Labs, UBC, Edinburgh, and NYU. Solved DGM's core limitation (only works on coding tasks): by merging the meta-agent and task agent into a shared editable codebase, the system can rewrite its own modification procedures—not just improving task solutions, but improving "how to propose improvements." HyperAgents achieves self-improvement on non-coding tasks (paper review, robotics reward design), demonstrating transferable meta-cognitive capabilities for the first time.
RoboPhD (2026.01) uses ELO-based tournament selection with cross-pollination: alternates between agent instantiation, cross-pollination (combining top strategies), ELO-score based tournament selection, and automatic error-driven artifact generation—yielding emergent capabilities not present in any prior baseline on Text-to-SQL.
5. Deep Dives into Representative Systems
5.1 Voyager — Skill Externalization from Weights to Code
Voyager writes skills as executable code into a Skill Library, achieving explainability, reusability, composability, and anti-forgetting. Voyager's Automatic Curriculum, driven by GPT-4, always explores at the boundary of "achievable but challenging."
5.2 MUSE — Experience-Driven Continuous Learning
MUSE (Memory-Utilizing and Self-Evolving) converts raw execution traces into structured experience via hierarchical memory modules (strategic, procedural, tool memory). On the TAC benchmark, MUSE using only Gemini-2.5 Flash achieved new SOTA—and longer experience accumulation correlates with stronger task completion.
5.3 AgentEvolver — Three Mechanisms Driving System Evolution
① Self-Questioning: Curiosity-driven task generation—the Agent autonomously generates new exploration tasks combining long-term goals and current environment.
② Self-Navigating: Efficient experience reuse—converting successful and failed experiences into structured, transferable knowledge.
③ Self-Attributing: Fine-grained causal analysis—assessing each intermediate step's contribution to final success.
5.4 HyperAgents — Recursive Self-Improvement's First Cross-Domain Validation
HyperAgents represents the frontier of self-evolution. By merging meta-agent and task agent into a shared editable codebase, it achieves what DGM could not: self-improvement that transfers beyond programming. The Agent not only improves task-solving behavior but also improves "the process that generates future improvements." Demonstrated on paper review, robotics reward design—domains where DGM was completely ineffective.
5.5 SWE-RL — Self-Play Driven Code Agents
SWE-RL (Meta Superintelligence Labs, 2025.12) is one of 2026's most noteworthy paradigms: an Agent plays both bug injector and solver simultaneously, training itself to fix self-generated bugs. On SWE-bench Verified: +10.4pp. No external data. Pure self-improvement through adversarial self-play.
6. Current Challenges
6.1 Catastrophic Forgetting
Voyager bypasses this through skill externalization (storing behaviors as code rather than parameters), but more general solutions remain under exploration.
6.2 Safety of Recursive Self-Improvement
FrontisAI's OpenMLE (2026.07) is moving "Agent improving its own learning algorithm" from theory to empirical validation. MLE-bench Lite: 39.39% → 60.61% improvement is exciting but the失控 risk rises proportionally—sandbox isolation and transparent evolution logging become essential.
6.3 Reliable Evaluation of Evolution Effects
Traditional benchmarks are designed for static models. New dynamic evaluation frameworks are needed to track Agent learning curves and generalization capabilities.
6.4 Cross-Domain Transfer
DGM's core limitation (only applicable to coding tasks) was partially solved by HyperAgents, but cross-domain self-evolution remains uneven—code tasks have clear executable feedback, while general tasks (writing, reasoning) often have fuzzier self-improvement signals.
7. Future Directions
- Recursive Self-Improvement: Agents learning to improve their own learning algorithms—systems like OpenMLE are already validating this path
- Multi-Agent Cooperative Evolution: Collective evolution through collaborative pressure and knowledge sharing across Agent populations
- Cross-Modal Evolution: Expanding from text reasoning to autonomous acquisition of visual, auditory, motor control capabilities
- Safety-Aligned Evolution: Embedding human preference constraints into evolution objective functions
8. Conclusion
Self-evolving Agents represent a fundamental transformation of AI from "static tool" to "dynamic intelligent agent." This transformation revolves around three core questions:
- What to evolve: Model, context (memory + prompt), tools, and architecture can all be brought under the scope of evolution
- When to evolve: In-test-time immediate adaptation vs. cross-test continuous learning—two timings correspond to different technical approaches
- How to evolve: Reward-driven, imitation-based, evolutionary, and self-play paradigms each have strengths; combinations are often necessary
The 2023-2024 breakthrough: "self-improve within fixed domains" (Voyager, MUSE, Reflexion). The 2025 breakthrough: "code-level self-modification" (DGM, SICA). The 2026 breakthrough: "learn how to improve itself" (HyperAgents, OpenMLE)—recursive self-improvement moved from theory to empirical validation.
But as DGM's "cheating" incident reveals: greater capability demands greater safety. As we pursue smarter Agents, ensuring evolution always proceeds in directions humans can understand and control is a challenge we must solve in parallel.
Key References:
- Wang et al., "A Survey of Self-Evolving Agents," arXiv:2507.21046
- Wang et al., "Voyager: An Open-Ended Embodied Agent with LLMs," arXiv:2305.16291
- Zhang et al., "Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents," arXiv:2505.22954
- Yang et al., "MUSE: Learning on the Job," arXiv:2510.08002
- Robeyns et al., "A Self-Improving Coding Agent," arXiv:2504.15228
- Borthwick et al., "RoboPhD: ELO-Based Cross-Pollination for Self-Evolving Agents," 2026.01
- Wu et al., "EvolveR: Self-Evolving LLM Agents Through an Experience-Driven Lifecycle," arXiv:2510.16079
- Meta Superintelligence Labs, "HyperAgents: Recursive Self-Improving Agents," arXiv, 2026.03
- selfimproving-agent.github.io — 312+ classified papers
Based on publicly available research findings from 2023–2026. Updated to latest version.
Top comments (0)