Beyond the Prompt: How to Build an AI Agent That Actually Learns From Its Mistakes

We’ve all seen the standard pattern of AI development over the past couple of years. You write a system prompt, wrap it in an API call, hook up a vector database for Retrieval-Augmented Generation (RAG), and call it an "autonomous agent."

But let’s be honest: these systems aren't actually autonomous. They are stateless function evaluators. They receive a prompt, generate a response, and instantly forget everything that just happened. If they make a mistake on step three of a task, they are highly likely to repeat that exact same mistake on step five. They don’t learn, they don’t adapt, and they don’t grow. They are stuck in a digital version of Groundhog Day.

To build truly autonomous software, we need a profound architectural shift. We need systems that can observe their own performance, write their own journals, and build their own custom tools.

Enter Hermes: an open-source, self-learning cognitive architecture.

In this deep dive, we will break down the core theoretical architecture of Hermes. We’ll look at real production code to see how it implements persistent memory, executes a closed learning loop, and achieves continuous self-improvement through reflection.

(The concepts and code demonstrated here are drawn from my ebook Hermes Agent, The Self-Evolving AI Workforce)

---

The Core Theoretical Architecture: The Master Craftsman Paradigm

To understand why Hermes represents a shift in agentic AI, consider the analogy of a master craftsman.

A novice apprentice reads a manual, builds a single chair, and immediately forgets the nuances of the wood they used. A master craftsman, however, keeps a detailed journal. In this journal, they document every project: what type of joinery worked, how a specific varnish reacted to humidity, the client’s unique aesthetic preferences, and the shortcuts that saved hours of labor.

Before starting a new project, the master reviews this journal. After completing it, they write a new entry, updating their techniques. Over decades, this journal becomes an extension of their mind.

This is exactly how Hermes operates. Instead of treating memory as a cold database query, Hermes integrates a structured, self-authored "journal" directly into an LLM reasoning loop. It is built on three interlocking theoretical pillars:

1. Persistent Memory as a Continuous Learning Journal: A living, semantic record of facts and user preferences.
2. The Closed Learning Loop: Real-time, in-context learning driven by tool execution outcomes and guardrails.
3. Self-Improvement through Reflection: An asynchronous meta-process where the agent analyzes its own transcripts to build, update, and prune its own codebase of "skills."

Let’s explore how these three pillars are implemented in code.

---

Pillar 1: Persistent Memory as a Continuous Learning Journal

Most agent architectures rely on vector databases to handle memory. While vector search is great for retrieving raw documents, it is terrible for maintaining an agent's identity, behavioral guidelines, or deep user relationships. If a user says, "I prefer concise code reviews with bullet points," that shouldn't be a vector chunk that might get retrieved if the cosine similarity is high enough. It should be a core parameter of how the agent behaves.

Hermes solves this by using a structured, LLM-authored plain-text journal split into two files:

• MEMORY.md: Factual knowledge about the environment, tasks, and accumulated domain knowledge.
• USER.md: A deep, evolving model of the user's persona, work style, and expectations.

This memory is initialized directly when the agent starts:

# From run_agent.py - Memory initialization
if not skip_memory:
    mem_config = _agent_cfg.get("memory", {})
    self._memory_enabled = mem_config.get("memory_enabled", False)
    self._user_profile_enabled = mem_config.get("user_profile_enabled", False)
    self._memory_nudge_interval = int(mem_config.get("nudge_interval", 10))
    if self._memory_enabled or self._user_profile_enabled:
        from tools.memory_tool import MemoryStore
        self._memory_store = MemoryStore(
            memory_char_limit=mem_config.get("memory_char_limit", 2200),
            user_char_limit=mem_config.get("user_char_limit", 1375),
        )
        self._memory_store.load_from_disk()

Proactive, Asynchronous Introspection

Instead of waiting for the user to explicitly say "remember this," Hermes autonomously decides what is worth preserving. It does this using a nudge interval (memory.nudge_interval). Every $N$ turns, the agent triggers a background review process guided by the following system prompt:

_MEMORY_REVIEW_PROMPT = (
    "Review the conversation above and consider saving to memory if appropriate.\n\n"
    "Focus on:\n"
    "1. Has the user revealed things about themselves — their persona, desires, "
    "preferences, or personal details worth remembering?\n"
    "2. Has the user expressed expectations about how you should behave, their work "
    "style, or ways they want you to operate?\n\n"
    "If something stands out, save it using the memory tool. "
    "If nothing is worth saving, just say 'Nothing to save.' and stop."
)

Crucially, this review does not block the user interface. A self-improving agent must not sacrifice real-time performance for the sake of learning. Hermes spawns this review in a background thread, cloning its state so the user never experiences latency:

def _spawn_background_review(self, messages_snapshot, review_memory=False, review_skills=False):
    """Spawn a background thread to review the conversation for memory/skill saves."""
    def _run_review():
        review_agent = AIAgent(
            model=self.model,
            max_iterations=16,
            quiet_mode=True,
            platform=self.platform,
            provider=self.provider,
            api_mode=_parent_runtime.get("api_mode"),
            base_url=_parent_runtime.get("base_url"),
            api_key=_parent_runtime.get("api_key"),
            credential_pool=getattr(self, "_credential_pool", None),
            parent_session_id=self.session_id,
            enabled_toolsets=["memory", "skills"],
        )
        review_agent._memory_store = self._memory_store
        review_agent.run_conversation(user_message=prompt, conversation_history=messages_snapshot)
        # ... collect actions and print summary
    t = threading.Thread(target=_run_review, daemon=True, name="bg-review")
    t.start()

By directly inheriting the parent’s _memory_store, the background agent writes directly to disk. On the very next turn, this updated markdown is injected straight into the system prompt:

# From _build_system_prompt - Memory injection
if self._memory_store:
    if self._memory_enabled:
        mem_block = self._memory_store.format_for_system_prompt("memory")
        if mem_block:
            prompt_parts.append(mem_block)
    if self._user_profile_enabled:
        user_block = self._memory_store.format_for_system_prompt("user")
        if user_block:
            prompt_parts.append(user_block)

---

Pillar 2: The Closed Learning Loop

The second pillar of Hermes is the Closed Learning Loop. Traditional reinforcement learning requires complex external reward functions, and supervised learning requires massive labeled datasets. Hermes bypasses both by using its own tool execution outcomes as a real-time feedback signal.

The loop consists of four rapid-fire stages:

1. Observation: The agent receives the user's message and current context.
2. Action: The agent calls a tool (e.g., writing a file, running a script).
3. Outcome: The tool executes and returns a raw result (either data or an error trace).
4. Refinement: The agent inspects the tool output and immediately adjusts its strategy.

This execution loop is managed inside run_conversation():

while (api_call_count < self.max_iterations and self.iteration_budget.remaining > 0) or self._budget_grace_call:
    # 1. Build API messages from current history
    # 2. Call the model (streaming or non-streaming)
    # 3. Parse tool calls from the response
    # 4. Execute tool calls, append results to messages
    # 5. Continue loop

Because the raw tool outcomes—including stack traces and bash errors—are appended directly to the conversation history, the LLM uses its in-context learning capabilities to debug itself on the fly.

Guardrails and Resource Constraints

To prevent the agent from getting stuck in infinite loops (e.g., repeatedly reading the same directory when a search fails), Hermes uses a ToolCallGuardrailController. If the model repeats a non-progressing action, the guardrail intervenes and injects a synthetic warning:

# From run_agent.py - Guardrail observation after tool execution
function_result = self._append_guardrail_observation(
    function_name, function_args, function_result, failed=is_error,
)

Additionally, Hermes enforces an IterationBudget. This class acts as a form of "learning pressure," forcing the agent to be highly strategic with its tool usage:

class IterationBudget:
    """Thread-safe iteration counter for an agent."""
    def __init__(self, max_total: int):
        self.max_total = max_total
        self._used = 0
        self._lock = threading.Lock()
    def consume(self) -> bool:
        with self._lock:
            if self._used >= self.max_total:
                return False
            self._used += 1
            return True
    def refund(self) -> None:
        with self._lock:
            if self._used > 0:
                self._used -= 1

If an action is highly efficient (like programmatically running code), the budget can be refunded. If the agent wastes LLM calls on repetitive tasks, it quickly runs out of budget and is forced to summarize and exit. This design teaches the agent to prioritize high-leverage tools.

Context Compression: Lossy Semantic Preservation

As the conversation grows, it risks overflowing the LLM's context window. Instead of hard-truncating old messages (which causes catastrophic forgetting), Hermes uses a ContextCompressor.

An auxiliary LLM reviews the middle turns of the conversation and summarizes them into a highly dense semantic paragraph, discarding raw execution logs but retaining the core decisions, discoveries, and outcomes:

# From _compress_context
messages, active_system_prompt = self._compress_context(
    messages, system_message, approx_tokens=approx_tokens,
    task_id=effective_task_id,
)

---

Pillar 3: Self-Improvement Through Reflection

While the closed learning loop handles short-term adaptation within a single conversation, reflection is how Hermes achieves long-term, structural self-improvement.

After a conversation ends, Hermes reviews its performance to extract procedural knowledge. It doesn't just ask "What did I learn about this user?"—it asks "What did I learn about how to solve this class of problem?"

These procedural insights are compiled into Skills. Skills are structured directories stored in ~/.hermes/skills/ containing:

• SKILL.md: A step-by-step markdown guide explaining how to perform a task.
• references/: Session-specific details or condensed knowledge bases.
• templates/: Starter files and boilerplates.
• scripts/: Statically re-runnable scripts.

The background agent is pushed to actively seek out these learning opportunities via the _SKILL_REVIEW_PROMPT:

_SKILL_REVIEW_PROMPT = (
    "Review the conversation above and update the skill library. Be "
    "ACTIVE — most sessions produce at least one skill update, even if "
    "small. A pass that does nothing is a missed learning opportunity, "
    "not a neutral outcome.\n\n"
    # ... detailed guidance on when to update/create skills
    "If you notice two existing skills that overlap, note it in your "
    "reply — the background curator handles consolidation at scale.\n\n"
    "'Nothing to save.' is a real option but should NOT be the "
    "default. If the session ran smoothly with no corrections and "
    "produced no new technique, just say 'Nothing to save.' and stop. "
    "Otherwise, act."
)

This prompt programs a growth mindset directly into the agent. It treats mistakes, user corrections, and unexpected workarounds not as failures, but as first-class signals to update its procedural skills.

The Skill Lifecycle and the Curator

If an agent continuously creates skills, its library will eventually suffer from bloat, duplicate skills, and outdated APIs. To solve this, Hermes includes a background system service called the Curator (agent/curator.py).

The Curator runs periodically to analyze the skill library, merge overlapping skills, and archive stale ones:

# From AGENTS.md - Curator invariants
# Invariants:
# - Curator only touches skills with created_by: "agent" provenance —
#   bundled + hub-installed skills are off-limits.
# - Never deletes; max destructive action is archive.
# - Pinned skills are exempt from every auto-transition and from the
#   LLM review pass.

By enforcing these safety invariants, Hermes ensures that self-improvement never results in the accidental deletion of core system tools or user-pinned workflows.

---

The Complete Self-Learning Cycle

When you put these three pillars together, you get a continuous, self-reinforcing learning loop:

[1. Interaction] ──> User talks to Agent, executing tools in real-time.
       │
       ▼
[2. Closed-Loop Adaptation] ──> Agent debugs tool errors within the turn.
       │
       ▼
[3. Persistent Memory] ──> Background thread saves facts to MEMORY.md / USER.md.
       │
       ▼
[4. Skill Update] ──> Reflection creates/updates procedural guides in SKILL.md.
       │
       ▼
[5. Curator Maintenance] ──> Background curator merges, prunes, and archives skills.
       │
       ▼
[6. Next Turn] ──> Updated memory and skills are injected into the next prompt.

This cycle allows the agent to grow more capable, more personalized, and highly specialized the more you use it—without requiring a single gradient update, fine-tuning run, or manual code deployment.

---

The Shift to Truly Stateful AI

The era of the stateless AI wrapper is drawing to a close. To build applications that can truly operate as autonomous software engineers, research assistants, or personal chiefs of staff, we must build systems that remember, reflect, and adapt.

By combining plain-text semantic journals, real-time tool guardrails, and asynchronous skill curation, the Hermes architecture provides a blueprint for the next generation of agentic AI. It moves us away from static prompt engineering and toward dynamic, evolving systems that learn from their environments.

---

Let's Discuss

1. Memory Storage: Hermes uses structured Markdown files (MEMORY.md/USER.md) rather than a vector database for its core memory. What are the architectural trade-offs of this approach when scaling to millions of tokens of interaction history?
2. Self-Writing Code: If an agent has the power to autonomously write and update its own "skills" (including executable scripts), what security protocols and sandboxing techniques would you implement to prevent catastrophic system failures or malicious exploits?

The concepts and code demonstrated here are drawn directly from the comprehensive roadmap laid out in the ebook Hermes Agent, The Self-Evolving AI Workforce: details link, you can find also my programming ebooks with AI here: Programming & AI eBooks.