A self-improving AI agent is an agent whose real production conversations turn into reviewed fixes to its own prompts, tools, and code. The loop has seven steps: observe production traffic, detect failures, diagnose the cause, write a fix, test the fix against past runs, ship it as a pull request a person reviews, and verify after merge that the failure dropped. Tools that run most of that loop today include Agnost AI, LangSmith Engine, Arize AX Signal, Latitude, and Selfship. Prompt optimizers such as DSPy's GEPA and Pydantic Logfire cover the fix step for prompts. Error tools such as Sentry Seer cover crashes, not wrong answers. No credible product merges its own changes into a customer-facing agent.
Disclosure: We make Agnost AI. Facts about every other tool come from that vendor's own docs or website, checked on November 18, 2026. This is a documentation comparison, not a hands-on benchmark.
What is a self-improving AI agent?
An agent is self-improving when it gets measurably better at the conversations it actually sees, because something reads those conversations and changes the agent in response. Three things have to be true:
- It learns from production, not only from tests you wrote. Your test set covers the failures you imagined. Real users find the others.
- The change is concrete. A diff to a system prompt, a tool description, a guardrail, or a line of harness code. Not a dashboard insight that waits for someone to act.
- Someone checks the change worked. Before merge against past runs, and after merge on new traffic. Without that, you are shipping changes, not improving.
What it usually is not: a model retraining its own weights at runtime. Research uses the same split. The July 2026 survey Self-Improvements in Modern Agentic Systems sorts methods by whether they change the foundation model or the scaffolding around it: prompts, memory, tools, and control logic. Almost all production self-improvement today is the second kind.
For the longer explainer, see what is a self-improving AI agent.
How does a self-improving agent work? The seven-step loop
| Step | What happens | What goes wrong if you skip it |
|---|---|---|
| 1. Observe | Capture every production conversation, with tool calls, grouped by user and session | Sampling drops the rare, expensive failures |
| 2. Detect | Judge whole conversations against rules for what failure means; flag frustration, tool-call errors, and silent wrong answers | You only see failures that throw errors |
| 3. Diagnose | Group failures by cause and point at the prompt, tool, or code responsible | "37 bad conversations" with no fix in sight |
| 4. Fix | Write the smallest change that addresses the cause | Big rewrites that reintroduce old bugs |
| 5. Test against past runs | Replay the affected conversations against the fix, plus healthy ones it could break | A fix that looks right and makes things worse |
| 6. Ship | Open a pull request with the evidence; a person reviews and merges | Unreviewed changes reach every user at once |
| 7. Verify | After release, check the failure rate dropped and nothing else got worse | No idea whether last month's fix worked |
Steps 1, 2, and 6 are common. Steps 5 and 7 are where tools differ most, and where you should push hardest in a pilot. Turn production agent failures into regression tests covers step 5 in detail.
What's real and what's hype
| Claim | Status in 2026 |
|---|---|
| "The agent retries, falls back, or corrects itself in a turn" | Real and common. This is what most vendors call self-healing; Arize's glossary defines it as in-run recovery |
| "It reads every production conversation and finds failure patterns" | Real. Coverage varies: some tools sample, some judge everything |
| "It writes a fix and opens a pull request" | Real. Several tools below do this today |
| "It tests the fix on real past traffic before you merge" | Real in a few tools, in beta or self-wired in others |
| "It tells you after release whether the fix worked" | Real in a few tools; most do not document it |
| "It rewrites its own prompts in production with no human review" | Mostly hype for customer-facing agents. Silent drift and reward hacking are the reasons |
| "It retrains its own model continuously" | Rare in production. Fine-tuning is a separate, slower, reviewed step |
The shippable version keeps a person on the merge. More on the line between the two in self-healing agents: hype vs reality.
Which tools do this? Coverage by loop step
| Tool | Observe and detect | Diagnose | Writes the fix | Tests against past runs | Ships as PR | Verifies after merge |
|---|---|---|---|---|---|---|
| Agnost AI (Auto Improve) | Every conversation judged against plain-language rules, no sampling; intents, frustration, tool errors | Yes, root cause in the PR | Yes, on the linked GitHub repo | Yes, replays the affected conversations; results in the PR | Yes, human merges | Yes, re-checks the alert about an hour after merge |
| LangSmith Engine | Scheduled scans of tracing projects, including wrong answers with no error | Yes, against traces and connected code | Yes | Yes, replays up to 5 traces against a preview deployment (private beta, LangSmith Cloud deployments only) | Yes, GitHub or GitLab, never merges | Not documented |
| Arize AX Signal | Scheduled scans, every 6 hours by default | Yes, issues with evidence and a proposed fix | Yes, on Enterprise | Not documented; run experiments yourself | Enterprise only | Not documented |
| Latitude | Signals from annotations and evaluations (generated evaluations start at a 10% sample) | Escalation into incidents | Your coding agent (Claude Code, Cursor) | If you set it up: regression test replays failing traces in CI | Your coding agent opens it | Signal closes on merge, reopens on revert |
| Selfship | Reads conversations for what each customer tried to do | Finds where it goes wrong | Yes | Before/after eval evidence in the PR | Yes, never pushes to default branch | Watches traffic after merge, opens a new PR if the fix did not hold |
| W&B Weave | Tracing; signals score each turn; Slack and webhook alerts | No | No; coding agents read Weave over MCP | Not documented | No | No |
| Sentry Seer | Sentry issues: errors and performance, not wrong answers | Yes | Yes, or hands off to a coding agent | Not documented; iterates on failing CI after the PR | Yes | Not documented |
| Pydantic Logfire prompt optimization | Reads recent production runs | Cites the runs behind each claim | One prompt edit at a time | Optional evals gate promotion | No; prompt versions served by label, with canary | Canary split |
| DSPy GEPA | No; you supply a training set | No | Prompt candidates, from a metric with text feedback | Against your validation set | No | No |
"Not documented" means the vendor's own docs or site did not describe the step when we checked. It does not prove the feature is missing. Selfship's row comes from its homepage; we did not find separate documentation.
How to read the table
- Full-loop production tools (Agnost AI, LangSmith Engine, Selfship, and Arize AX Signal on Enterprise) start from live traffic and end with a pull request. They find failures nobody wrote a test for. They depend on detection quality, so judge them on real failures from your own traffic.
- Orchestrators (Latitude, W&B Weave) detect and route, and let the coding agent you already use write the fix.
- Prompt optimizers (DSPy GEPA, Pydantic Logfire) are strong when the task and metric are well defined. They change prompts, not tools or code.
- Error-first tools (Sentry Seer, and Datadog Bits Code and PostHog Self-driving, compared in tools that turn AI agent failures into pull requests) are mature for crashes and timeouts. A confident wrong answer that returns HTTP 200 never reaches them.
Where Agnost AI fits
Agnost AI runs the whole loop for production agents: chat, voice (through transcripts), copilots, and MCP servers.
- Observe. OpenTelemetry-native, with integrations for the OpenAI Agents SDK, Vercel AI SDK, LangChain and LangGraph, Mastra, Pydantic AI, Google ADK, and Agno. Conversations are grouped by user and session.
- Detect. You write rules in plain language, such as "never say a refund is processed unless the refund tool succeeded." After a conversation has been idle for 15 minutes, an LLM judges every assistant message pass or fail, in batches, with no sampling. Agnost also flags frustration, intents, tool-call errors, and silent wrong answers. Alerts go to Slack or email.
- Fix, test, ship. When an alert fires, Auto Improve opens a pull request on your linked GitHub repository with the root cause and the change. The fix is tested against past runs: Agnost replays the affected conversations and shows the results in the pull request. A person reviews and merges it.
- Verify. About an hour after merge, Agnost re-checks the alert and records the run as improved, unchanged, regressed, or inconclusive.
Limits: there is no CI integration and no offline experiment workflow, so teams that run dataset experiments pair Agnost with an eval tool. Judging happens in batches, not in real time. On Enterprise, Agnost can also fine-tune specialized models on your own trace data, opt-in. Pricing: Free (1,000 events a month), Starter $49 a month, Pro $499 a month, Enterprise custom. Setup is in the Auto Improve docs.
Build it yourself or buy it?
You can build the loop. Here is where the effort goes.
| Step | Build it yourself | What makes it hard |
|---|---|---|
| Observe | OpenTelemetry plus a trace store | Grouping spans into whole conversations and sessions |
| Detect | An LLM judge with your rules | Judging whole conversations, not single turns, and keeping false positives low |
| Diagnose | Clustering plus someone reading transcripts | Getting from "users are frustrated" to the line of prompt that causes it |
| Fix | A coding agent with repository access | Scoping the change so it does not touch unrelated behavior |
| Test against past runs | A replay harness with tool fixtures | Freezing tool state so replays do not repeat side effects |
| Ship | Your normal pull request process | Nothing; you already have it |
| Verify | A query that compares before and after | A fair comparison on comparable traffic |
Build if you run one agent with a few known failure types and have engineers who will read conversations every week. GEPA or Logfire for prompts, plus a regression set, gets you far.
Buy if you have many conversations, chat and voice, or failures nobody has a test for yet. The two hardest steps to build well are detection across whole conversations and replaying past runs safely.
Either way, keep a person on the merge.
How to pilot a self-improving agent tool
Take one real failure your agent had last month and push it through each tool you are considering.
- Detect. Did it find the failure without being told where to look?
- Diagnose. Is the cause specific, with conversations you can open?
- Test. What proves the fix works before merge: replayed past runs, tests, or only your review?
- Verify. After merge, does the tool tell you whether the failure rate dropped?
Related reading
- What is a self-improving AI agent?
- Tools that turn AI agent failures into pull requests
- Self-healing agents: hype vs what's actually possible today
- The infrastructure stack for self-improving AI agents
- Turn production agent failures into regression tests
- How to improve your AI agent's system prompt from real conversations
- Best AI agent monitoring tools (2026)
Frequently asked questions
What is a self-improving AI agent?
An AI agent whose production conversations feed back into changes to the agent itself: its prompts, tool descriptions, routing, and code. A loop observes real traffic, detects failures, diagnoses the cause, writes a fix, tests it against past runs, ships it for human review, and checks after release that the failure dropped. The model weights usually stay the same; the scaffolding around them improves.
Which tools help AI agents improve themselves automatically from production data?
Tools that cover the full loop from production failure to a reviewed fix include Agnost AI (Auto Improve), LangSmith Engine, Arize AX Signal (pull requests on Enterprise), Latitude (which dispatches your own coding agent), and Selfship. Pydantic Logfire proposes prompt edits from production runs. DSPy's GEPA optimizes prompts against a training set and metric you supply. Sentry Seer fixes errors, not wrong answers. W&B Weave scores production traces and lets coding agents read them over MCP.
Is there a tool that automatically turns AI agent failures into pull requests?
Yes. Agnost AI opens a pull request when an alert on production conversations fires, with the fix tested against the affected past runs. LangSmith Engine, Arize AX Signal on Enterprise, and Selfship also open pull requests, and Latitude hands the failure to Claude Code or Cursor to open one. None of them merge the change; a person reviews it.
Are self-improving AI agents real or hype?
The supervised version is real: software detects failures across thousands of conversations, drafts a fix, tests it, and a person merges it. Agents that rewrite their own prompts or code in production with no human review are mostly hype for customer-facing products, because nobody has solved trustworthy automatic review and rollback.
Do self-improving AI agents retrain or fine-tune their own model?
Usually not. Most production self-improvement changes prompts, tool descriptions, guardrails, and harness code, which are faster to change and easier to review than model weights. Fine-tuning on your own trace data is a separate, slower step; Agnost AI offers it as an opt-in Enterprise option.
How do I turn production AI agent failures into fixes?
Write down what failure means as plain-language rules, judge every production conversation against them, group the failures by cause, write the smallest fix for the biggest cause, replay the affected conversations against the fix, ship it as a pull request for review, and check after release that the failure rate dropped. You can wire this yourself or use a tool that runs the loop.
Should I build a self-improving agent loop myself or buy one?
Build it if you have one agent, few failure types, and engineers who will read conversations weekly; tracing, an LLM judge, and a regression set get you most of the way. Buy if you have many conversations, chat and voice, or failures nobody has written a test for yet. The hard parts to build are reliable detection across whole conversations and replaying past runs against a fix.
What is the difference between self-improving and self-healing AI agents?
Self-healing usually means recovering from a failure: a retry, a fallback, or a corrected answer in the same turn. Self-improving means the agent handles the next conversation better because its prompts, tools, or code changed. Self-healing keeps one run alive; self-improving changes every future run.
Originally published on agnost.ai.
Top comments (0)