Teaching a CI/CD Engine to Heal Itself

How I built rule-based auto-remediation and AI-powered failure diagnosis into PipelineOS.

A build fails.

You hit "Re-run."

It passes.

Nothing changed in the code. Nobody fixed anything. So what actually failed? And why did you have to be the one to click that button?

Every developer has experienced this. You push code, wait for the pipeline, get a red, and immediately assume it's just a transient network hiccup. You restart the job, and magically, it works.

That manual click is the duct tape holding modern CI/CD together. While building PipelineOS, I decided I didn't want to be the retry mechanism anymore. I wanted the pipeline to understand why it failed and heal itself.

1. Why "Re-run" Isn't a Strategy

Most CI/CD systems treat every failure as equal, whether it's a syntax error in your codebase or a temporary DNS hiccup. But in reality, failures fall into two categories:

Deterministic (Code issues): Compilation errors, failing unit tests, missing dependencies. Hitting re-run will never fix these.
Transient (Infrastructure issues): NPM registry timeouts, Docker Hub rate limits (429 Too Many Requests), flaky internal network proxies, or temporary Git clone failures.
When humans manually retry transient failures, we waste time, context-switch out of deep work, and mask underlying infrastructure instability. A smart CI/CD engine shouldn't bother a developer with an NPM timeout; it should handle it automatically.

2. Designing a Rule Engine

To solve this, I designed an Auto-Remediation Rule Engine. The goal was to build a deterministic system that recognizes known failure patterns and executes safe recovery actions.

Here is the conceptual flow:

When a stage fails, PipelineOS doesn't immediately mark the run as dead. Instead, the Rule Engine scans the final output logs against a registry of known regex signatures (e.g., npm ERR! network timeout or ECONNRESET). If a match is found, the engine executes the prescribed recovery action—such as applying an exponential backoff sleep and automatically retrying the specific stage.

3. Where AI Fits

Rule engines are incredible for known, recurring infrastructure flakes. But what happens when a build fails for a reason the rule engine has never seen before?
This is where the architecture becomes interesting. In PipelineOS, deterministic rules come first; AI comes second.

Instead of searching StackOverflow for obscure GCC compiler errors, the developer is immediately presented with a concise, AI-generated diagnosis directly in the UI.

4. The Decision Boundary

The important design decision isn't simply "use AI." It's deciding where AI should and shouldn't be involved.

Known failure modes stay deterministic and cheap. Unknown failures get richer contextual analysis. Potentially destructive actions remain securely behind explicit safety boundaries.

5. What the AI Actually Receives

You can't build good CI intelligence from unstructured logs alone. In the previous article, I described why real-time observability became a first-class part of PipelineOS (PIP-35). That architecture turned out to be more important than I initially expected. Before a system can intelligently diagnose a failure, it needs structured information about what actually happened.
PIP-35 gave PipelineOS that context. PIP-36 and PIP-37 build on top of it. When a failure is routed to the AI, it receives a deeply structured context:

AI Diagnosis Context

Pipeline:   kpm-clinic
Stage:      deploy
Exit code:  1
Duration:   3m 46s
CPU:        42.3% avg
Memory:     968.9 MiB peak

Recent logs:
[2026-08-01 10:14] ...

By injecting this precise context alongside the final logs, the LLM has significantly more structured context to reason about the failure instead of relying on raw logs alone.

6. Balancing Automation with Safety

Whenever you introduce AI or automated recovery into an execution engine, the first question engineers ask is: Is AI executing arbitrary shell commands on my infrastructure?
The answer is absolutely not.
Safety requires strict boundaries:

1. Approval Gates: AI can diagnose and suggest fixes, but it cannot mutate the source code or execute arbitrary recovery commands without explicit human approval.
2. Allowlists: The deterministic Rule Engine only acts on pre-approved, strictly scoped recovery actions, such as retrying a stage or applying a predefined recovery policy.
3. Confidence-Based Auto-Disable: Every automated remediation rule tracks its own historical success rate and attempt count. If a regex match is too broad, it could trigger a false positive retry loop. PipelineOS evaluates attempts >= minAttempts and disables any rule where the success rate falls below the configured disableBelowSuccessRate threshold.

7. Real Examples in the Wild

I tested the architecture against scenarios like these:
Scenario A: The Transient Network Flake
The Error: npm ERR! network timeout at: https://registry.npmjs.org/...
The Flow:

1. The stage exits with code 1.
2. The Rule Engine matches the signature npm ERR! network timeout.
3. The engine pauses for 10 seconds.
4. The engine automatically retries the isolated stage.
5. Result: Success. The developer was never notified of the failure, and the pipeline stayed green.

Scenario B: The Genuine Configuration Error
The Error: AccessDenied: User: arn:aws:iam::123:user/deployer is not authorized to perform: s3:PutObject on resource...
The Flow:

1. The stage exits with code 1.
2. The Rule Engine scans for known signatures and finds no matches.
3. The structured context is routed to the AI Diagnosis Engine.
4. Result: The UI displays a clear explanation: "The build failed because the deployment IAM user lacks the s3:PutObject permission. Suggested action: Add the s3:PutObject policy to the deployer role before retrying."

8. Lessons Learned

Building this self-healing architecture taught me a few crucial lessons:

1. Deterministic automation wins for infrastructure: You don't need a heavy language model to realize that a 503 Service Unavailable from a Docker registry just needs a 30-second retry.
2. False positives are dangerous: If your remediation rules are too greedy, you will mask actual, deterministic code bugs by endlessly retrying them. Tracking success rates and disabling ineffective rules is mandatory.
3. AI excels at context, not action: Using AI to execute arbitrary code is risky and often non-deterministic. But using AI to summarize 10,000 lines of obscure stack traces into a 2-sentence explanation is a massive productivity multiplier.

9. PipelineOS Evolution

It is deeply rewarding to look back and see how PipelineOS has grown from a basic state machine into a robust, observable distributed system. Each layer exists because the next layer depends on it:

10. What's Next: The Learning Loop

This architecture fundamentally changes how we interact with CI/CD. But it's only the beginning.
The next step is turning successful remediation history into increasingly reliable recommendations. Rather than treating every failure independently, PipelineOS can use historical outcomes to rank fixes by confidence and continuously improve its remediation decisions.
The goal isn't to let the model blindly rewrite its own rules. Instead, successful remediation outcomes become evidence that can improve how existing fixes are ranked and eventually help generate candidates for new deterministic rules.
If you're interested in distributed systems, CI/CD internals, or practical AI applications in developer tools, I'd love to hear your thoughts or feedback on this architecture!
PipelineOS is an open-source, self-hosted CI/CD runtime built for developers who value simplicity, visibility, and control. Check out our architecture and contribute on GitHub!

Comments

[Rahan Judes Michael]

Thankyou karthik sir for this valuable content ❤️