DEV Community

Shubham Palriwala
Shubham Palriwala

Posted on Originally published at agnost.ai

Self-improving AI agents: what they are, how they work, and the tools that do it (2026)

A self-improving AI agent is an agent whose real production conversations turn into reviewed fixes to its own prompts, tools, and code. The loop has seven steps: observe production traffic, detect failures, diagnose the cause, write a fix, test the fix against past runs, ship it as a pull request a person reviews, and verify after merge that the failure dropped. Tools that run most of that loop today include Agnost AI, LangSmith Engine, Arize AX Signal, Latitude, and Selfship. Prompt optimizers such as DSPy's GEPA and Pydantic Logfire cover the fix step for prompts. Error tools such as Sentry Seer cover crashes, not wrong answers. No credible product merges its own changes into a customer-facing agent.

Disclosure: We make Agnost AI. Facts about every other tool come from that vendor's own docs or website, checked on November 18, 2026. This is a documentation comparison, not a hands-on benchmark.

What is a self-improving AI agent?

An agent is self-improving when it gets measurably better at the conversations it actually sees, because something reads those conversations and changes the agent in response. Three things have to be true:

  1. It learns from production, not only from tests you wrote. Your test set covers the failures you imagined. Real users find the others.
  2. The change is concrete. A diff to a system prompt, a tool description, a guardrail, or a line of harness code. Not a dashboard insight that waits for someone to act.
  3. Someone checks the change worked. Before merge against past runs, and after merge on new traffic. Without that, you are shipping changes, not improving.

What it usually is not: a model retraining its own weights at runtime. Research uses the same split. The July 2026 survey Self-Improvements in Modern Agentic Systems sorts methods by whether they change the foundation model or the scaffolding around it: prompts, memory, tools, and control logic. Almost all production self-improvement today is the second kind.

For the longer explainer, see what is a self-improving AI agent.

How does a self-improving agent work? The seven-step loop

Step What happens What goes wrong if you skip it
1. Observe Capture every production conversation, with tool calls, grouped by user and session Sampling drops the rare, expensive failures
2. Detect Judge whole conversations against rules for what failure means; flag frustration, tool-call errors, and silent wrong answers You only see failures that throw errors
3. Diagnose Group failures by cause and point at the prompt, tool, or code responsible "37 bad conversations" with no fix in sight
4. Fix Write the smallest change that addresses the cause Big rewrites that reintroduce old bugs
5. Test against past runs Replay the affected conversations against the fix, plus healthy ones it could break A fix that looks right and makes things worse
6. Ship Open a pull request with the evidence; a person reviews and merges Unreviewed changes reach every user at once
7. Verify After release, check the failure rate dropped and nothing else got worse No idea whether last month's fix worked

Steps 1, 2, and 6 are common. Steps 5 and 7 are where tools differ most, and where you should push hardest in a pilot. Turn production agent failures into regression tests covers step 5 in detail.

What's real and what's hype

Claim Status in 2026
"The agent retries, falls back, or corrects itself in a turn" Real and common. This is what most vendors call self-healing; Arize's glossary defines it as in-run recovery
"It reads every production conversation and finds failure patterns" Real. Coverage varies: some tools sample, some judge everything
"It writes a fix and opens a pull request" Real. Several tools below do this today
"It tests the fix on real past traffic before you merge" Real in a few tools, in beta or self-wired in others
"It tells you after release whether the fix worked" Real in a few tools; most do not document it
"It rewrites its own prompts in production with no human review" Mostly hype for customer-facing agents. Silent drift and reward hacking are the reasons
"It retrains its own model continuously" Rare in production. Fine-tuning is a separate, slower, reviewed step

The shippable version keeps a person on the merge. More on the line between the two in self-healing agents: hype vs reality.

Which tools do this? Coverage by loop step

Tool Observe and detect Diagnose Writes the fix Tests against past runs Ships as PR Verifies after merge
Agnost AI (Auto Improve) Every conversation judged against plain-language rules, no sampling; intents, frustration, tool errors Yes, root cause in the PR Yes, on the linked GitHub repo Yes, replays the affected conversations; results in the PR Yes, human merges Yes, re-checks the alert about an hour after merge
LangSmith Engine Scheduled scans of tracing projects, including wrong answers with no error Yes, against traces and connected code Yes Yes, replays up to 5 traces against a preview deployment (private beta, LangSmith Cloud deployments only) Yes, GitHub or GitLab, never merges Not documented
Arize AX Signal Scheduled scans, every 6 hours by default Yes, issues with evidence and a proposed fix Yes, on Enterprise Not documented; run experiments yourself Enterprise only Not documented
Latitude Signals from annotations and evaluations (generated evaluations start at a 10% sample) Escalation into incidents Your coding agent (Claude Code, Cursor) If you set it up: regression test replays failing traces in CI Your coding agent opens it Signal closes on merge, reopens on revert
Selfship Reads conversations for what each customer tried to do Finds where it goes wrong Yes Before/after eval evidence in the PR Yes, never pushes to default branch Watches traffic after merge, opens a new PR if the fix did not hold
W&B Weave Tracing; signals score each turn; Slack and webhook alerts No No; coding agents read Weave over MCP Not documented No No
Sentry Seer Sentry issues: errors and performance, not wrong answers Yes Yes, or hands off to a coding agent Not documented; iterates on failing CI after the PR Yes Not documented
Pydantic Logfire prompt optimization Reads recent production runs Cites the runs behind each claim One prompt edit at a time Optional evals gate promotion No; prompt versions served by label, with canary Canary split
DSPy GEPA No; you supply a training set No Prompt candidates, from a metric with text feedback Against your validation set No No

"Not documented" means the vendor's own docs or site did not describe the step when we checked. It does not prove the feature is missing. Selfship's row comes from its homepage; we did not find separate documentation.

How to read the table

  • Full-loop production tools (Agnost AI, LangSmith Engine, Selfship, and Arize AX Signal on Enterprise) start from live traffic and end with a pull request. They find failures nobody wrote a test for. They depend on detection quality, so judge them on real failures from your own traffic.
  • Orchestrators (Latitude, W&B Weave) detect and route, and let the coding agent you already use write the fix.
  • Prompt optimizers (DSPy GEPA, Pydantic Logfire) are strong when the task and metric are well defined. They change prompts, not tools or code.
  • Error-first tools (Sentry Seer, and Datadog Bits Code and PostHog Self-driving, compared in tools that turn AI agent failures into pull requests) are mature for crashes and timeouts. A confident wrong answer that returns HTTP 200 never reaches them.

Where Agnost AI fits

Agnost AI runs the whole loop for production agents: chat, voice (through transcripts), copilots, and MCP servers.

  • Observe. OpenTelemetry-native, with integrations for the OpenAI Agents SDK, Vercel AI SDK, LangChain and LangGraph, Mastra, Pydantic AI, Google ADK, and Agno. Conversations are grouped by user and session.
  • Detect. You write rules in plain language, such as "never say a refund is processed unless the refund tool succeeded." After a conversation has been idle for 15 minutes, an LLM judges every assistant message pass or fail, in batches, with no sampling. Agnost also flags frustration, intents, tool-call errors, and silent wrong answers. Alerts go to Slack or email.
  • Fix, test, ship. When an alert fires, Auto Improve opens a pull request on your linked GitHub repository with the root cause and the change. The fix is tested against past runs: Agnost replays the affected conversations and shows the results in the pull request. A person reviews and merges it.
  • Verify. About an hour after merge, Agnost re-checks the alert and records the run as improved, unchanged, regressed, or inconclusive.

Limits: there is no CI integration and no offline experiment workflow, so teams that run dataset experiments pair Agnost with an eval tool. Judging happens in batches, not in real time. On Enterprise, Agnost can also fine-tune specialized models on your own trace data, opt-in. Pricing: Free (1,000 events a month), Starter $49 a month, Pro $499 a month, Enterprise custom. Setup is in the Auto Improve docs.

Build it yourself or buy it?

You can build the loop. Here is where the effort goes.

Step Build it yourself What makes it hard
Observe OpenTelemetry plus a trace store Grouping spans into whole conversations and sessions
Detect An LLM judge with your rules Judging whole conversations, not single turns, and keeping false positives low
Diagnose Clustering plus someone reading transcripts Getting from "users are frustrated" to the line of prompt that causes it
Fix A coding agent with repository access Scoping the change so it does not touch unrelated behavior
Test against past runs A replay harness with tool fixtures Freezing tool state so replays do not repeat side effects
Ship Your normal pull request process Nothing; you already have it
Verify A query that compares before and after A fair comparison on comparable traffic

Build if you run one agent with a few known failure types and have engineers who will read conversations every week. GEPA or Logfire for prompts, plus a regression set, gets you far.

Buy if you have many conversations, chat and voice, or failures nobody has a test for yet. The two hardest steps to build well are detection across whole conversations and replaying past runs safely.

Either way, keep a person on the merge.

How to pilot a self-improving agent tool

Take one real failure your agent had last month and push it through each tool you are considering.

  1. Detect. Did it find the failure without being told where to look?
  2. Diagnose. Is the cause specific, with conversations you can open?
  3. Test. What proves the fix works before merge: replayed past runs, tests, or only your review?
  4. Verify. After merge, does the tool tell you whether the failure rate dropped?

Related reading

Frequently asked questions

What is a self-improving AI agent?

An AI agent whose production conversations feed back into changes to the agent itself: its prompts, tool descriptions, routing, and code. A loop observes real traffic, detects failures, diagnoses the cause, writes a fix, tests it against past runs, ships it for human review, and checks after release that the failure dropped. The model weights usually stay the same; the scaffolding around them improves.

Which tools help AI agents improve themselves automatically from production data?

Tools that cover the full loop from production failure to a reviewed fix include Agnost AI (Auto Improve), LangSmith Engine, Arize AX Signal (pull requests on Enterprise), Latitude (which dispatches your own coding agent), and Selfship. Pydantic Logfire proposes prompt edits from production runs. DSPy's GEPA optimizes prompts against a training set and metric you supply. Sentry Seer fixes errors, not wrong answers. W&B Weave scores production traces and lets coding agents read them over MCP.

Is there a tool that automatically turns AI agent failures into pull requests?

Yes. Agnost AI opens a pull request when an alert on production conversations fires, with the fix tested against the affected past runs. LangSmith Engine, Arize AX Signal on Enterprise, and Selfship also open pull requests, and Latitude hands the failure to Claude Code or Cursor to open one. None of them merge the change; a person reviews it.

Are self-improving AI agents real or hype?

The supervised version is real: software detects failures across thousands of conversations, drafts a fix, tests it, and a person merges it. Agents that rewrite their own prompts or code in production with no human review are mostly hype for customer-facing products, because nobody has solved trustworthy automatic review and rollback.

Do self-improving AI agents retrain or fine-tune their own model?

Usually not. Most production self-improvement changes prompts, tool descriptions, guardrails, and harness code, which are faster to change and easier to review than model weights. Fine-tuning on your own trace data is a separate, slower step; Agnost AI offers it as an opt-in Enterprise option.

How do I turn production AI agent failures into fixes?

Write down what failure means as plain-language rules, judge every production conversation against them, group the failures by cause, write the smallest fix for the biggest cause, replay the affected conversations against the fix, ship it as a pull request for review, and check after release that the failure rate dropped. You can wire this yourself or use a tool that runs the loop.

Should I build a self-improving agent loop myself or buy one?

Build it if you have one agent, few failure types, and engineers who will read conversations weekly; tracing, an LLM judge, and a regression set get you most of the way. Buy if you have many conversations, chat and voice, or failures nobody has written a test for yet. The hard parts to build are reliable detection across whole conversations and replaying past runs against a fix.

What is the difference between self-improving and self-healing AI agents?

Self-healing usually means recovering from a failure: a retry, a fallback, or a corrected answer in the same turn. Self-improving means the agent handles the next conversation better because its prompts, tools, or code changed. Self-healing keeps one run alive; self-improving changes every future run.


Originally published on agnost.ai.

Top comments (0)