The engineering journey behind OdinRun: building a self-hosted, observable, self-healing, and AI-assisted CI/CD runtime.
A few months ago, this project was called PipelineOS.
It started as an experiment in building a CI/CD execution engine from scratch: a control plane, a Docker-based runner, a simple dashboard, and a way to execute pipeline stages reliably.
Since then, the project grew far beyond that original scope. It gained durable persistence, event-driven execution, real-time observability, runtime telemetry, automated remediation, AI-assisted diagnosis, recovery planning, and an interactive execution debugger.
Along the way, the project also outgrew its original name.
PipelineOS is now OdinRun.
This is the story of that evolution—and what I learned building a complete CI/CD platform layer by layer.
1. The First Problem: Reliable Execution
The first major lesson was simple: A CI/CD platform is only as reliable as its execution engine.
When the project began under the name PipelineOS, the original runner was an ~810-line monolith. It was responsible for everything: API communication, Docker container lifecycle management, log streaming, metric collection, retries, auto-remediation, and task orchestration.
It was a classic dumping ground. As a result, subtle bugs plagued the runtime:
- Docker attach stream corruption
- Unenforced stage timeouts
- Zombie containers left behind during SIGTERM shutdowns
- Flaky runner registration logic
PIP-33 became the first major reliability milestone. I decomposed the monolithic runner into clean, single-responsibility components:
- Container Runner: Exclusively manages the Docker daemon lifecycle, container creation, execution, and cleanup.
- Stage Runner: Manages individual stage semantics—retries, log scrubbing, workspace setup, and environment isolation.
- API Client: Handles control plane communication, heartbeat registration, and task claiming.
- Executor: Acts purely as an orchestration engine coordinating the lower-level runners. Decomposing the monolith immediately brought testability, isolation, and predictable error handling to the runtime.
2. Fixing the Things That Should Never Have Been Broken
Some of the most valuable engineering work wasn't a shiny new feature—it was fixing fundamental runtime behaviors that should never have been broken in the first place.
Docker stdout / stderr Demultiplexing
Docker's raw multiplexed stream (/containers/{id}/attach) prefixes stdout and stderr frames with an 8-byte header ([stream_type, 0, 0, 0, size1, size2, size3, size4]). The original runner treated this as raw text, scattering binary headers into output logs and corrupting log streams. The engine now explicitly demultiplexes the Docker stream into separate, clean channels.
Enforcing Stage Timeouts
While pipeline YAML configurations allowed developers to specify timeouts, the old runner ignored them. If a subprocess hung indefinitely, the runner hung indefinitely. Under PIP-33, stage execution was wrapped in a hard timeout context—forcefully terminating runaway steps before they drain compute fleet resources.
Graceful Container Cleanup
When a runner receives a termination signal (SIGTERM/SIGINT), it shouldn't abandon active Docker containers. The engine maintains an in-memory active container registry and executes an emergency teardown sequence during shutdown to guarantee zero zombie containers.
Resource Boundaries
Stages can now be configured with explicit CPU cores and memory limits (memory_limit: 512m, cpu_quota: 1.5), preventing a single rogue step from causing Out-Of-Memory (OOM) crashes across shared runner hosts.
These aren't glamorous features, but they form the mandatory foundation of an execution engine you can trust.
3. From REST Calls to Domain Events
Once stage execution became reliable, a new architectural bottleneck surfaced: Tight coupling between execution and transport.
Whenever a stage changed state, created logs, or recorded metrics, the runner made synchronous HTTP REST calls back to the API:
Stage State Change ──> HTTP POST /api/runs/:id/status
Log Output ──> HTTP POST /api/runs/:id/logs
Metric Snapshot ──> HTTP POST /api/runs/:id/metrics
This HTTP chatter choked the runner, introduced network latency spikes, and made it impossible to attach new capabilities (like live streaming or remediation) without modifying the core execution loop.
The solution was transitioning to an In-Process Domain Event
Architecture:
Instead of execution logic invoking HTTP endpoints directly, the runner emits strongly-typed domain events to an in-process bus. Independent consumers subscribe to these events asynchronously.
The execution engine no longer cares who consumes the events—whether it's an S3 uploader, a WebSocket streamer, or an auto-remediation rule engine.
4. Making Execution Durable
Real-time execution is useless if your build history vanishes when a node restarts.
Through PIP-31, PIP-32, and PIP-34, the platform established clean persistence abstractions:
- PIP-31 & PIP-32: Pluggable database persistence abstraction (supporting both lightweight SQLite for single-node self-hosting and MongoDB for multi-tenant scale).
- PIP-34: Durable log and artifact storage abstraction.
By placing storage operations behind abstract interfaces (ILogStorage, IArtifactStorage), the runtime remains completely agnostic to the underlying infrastructure. Whether artifacts are stored on a local NVMe drive during local testing or uploaded to AWS S3 / MinIO in production, zero changes are required in the core engine.
5. The Moment OdinRun Stopped Being a Black Box
Then came PIP-35 one of the most satisfying milestones in the project.
A modern CI/CD engine shouldn't just print Build Failed after 5 minutes. It should expose what is happening inside the execution container in real time.
OdinRun began capturing rich telemetry metrics directly from the Docker daemon during stage runs:
- Stage lifecycle state transitions
- CPU utilization percentage (calculated via delta math across container CPU stats)
- Memory consumption & peak working sets
- Network ingress/egress bytes
- Real-time stdout/stderr log streams
- Stage execution duration & runner heartbeats
By decoupling telemetry capture from presentation via Server-Sent Events (SSE) and WebSockets, developers can watch CPU graphs spike, memory usage climb, and logs stream instantly in the UI.
The platform could finally see itself running.
6. But Seeing a Failure Isn't the Same as Fixing It
Once real-time observability was in place, the next logical question emerged: What should the system do when a failure occurs?
This led to PIP-36: Rule-Based Auto-Remediation.
Most pipeline retries are triggered by human developers clicking "Re-run" on transient infrastructure glitches (NPM timeouts, Docker registry rate limits, flaky DNS queries). OdinRun automates this through a deterministic Rule Engine:
Key Principle: Deterministic Rules First, AI Second
If a failure matches a known regex pattern with a proven recovery policy (e.g. npm ERR! network timeout -> exponential backoff sleep & retry), the engine handles it deterministically. No heavy LLMs needed.
7. Teaching the System to Understand Unknown Failures
What happens when a build fails for a reason the rule engine has never seen before?
That brought the system to PIP-37: AI Failure Diagnosis.
Instead of dumping 10,000 raw lines of unformatted terminal output into a Large Language Model, OdinRun leverages the structured context built during PIP-35:
AI Diagnosis Context Packet
Pipeline: kpm-clinic
Stage: deploy
Exit code: 1
Duration: 3m 46s
CPU: 42.3% avg
Memory: 968.9 MiB peak
Captured Error Logs:
AccessDenied: User: arn:aws:iam::123:user/deployer is not authorized
to perform: s3:PutObject on resource arn:aws:s3:::prod-bucket/app
Because the LLM receives structured execution context (CPU/Memory profile, exit code, stage duration, and scrubbed log snippets), it can deliver concise, actionable root-cause analysis directly in the UI:
AI Diagnosis: The deploy stage failed due to missing IAM permissions. The deployer IAM user lacks the s3:PutObject policy on arn:aws:s3:::prod-bucket/app.
Suggested Fix: Attach s3:PutObject to the deployment IAM role before retrying.
8. AI Doesn't Get Unlimited Control
Safety in automated execution engines is non-negotiable. An AI engine inside a CI/CD platform should never be allowed to execute arbitrary shell commands on production infrastructure.
OdinRun strictly enforces a Decision Boundary:
- Deterministic Rule Engine: Can automatically execute pre-approved, safe recovery policies (like stage retries or workspace cleanup).
- AI Engine: Acts strictly as an analytical advisor. It generates root-cause hypotheses and suggested fixes, requiring human approval for any mutating action.
- Confidence Auto-Disable: Rules track their own historical success rates. If a rule's success rate drops below a configured threshold (e.g. <20% over 10 attempts), it is automatically disabled to prevent infinite retry loops.
9. The Learning Loop
PIP-37B introduced historical outcome tracking to create a feedback loop for recovery actions:
Failure ──> Diagnosis ──> Recovery ──> Outcome Record ──> Historical Confidence ──> Better Future Decisions
The goal isn't to let an AI model blindly rewrite the CI runtime. Instead, every recovery outcome (whether successful or failed) is recorded as structured evidence.
Over time, this historical evidence ranks effective remediation rules higher, isolates failing rules, and surfaces candidates for developers to turn into permanent, deterministic rules.
10. Making Execution Understandable: Visual Timelines
At this point, OdinRun could reliably execute, observe, diagnose, and auto-remediate. But scrolling through text logs to understand complex, multi-stage pipelines was still painful.
PIP-38 introduced the Visual Execution Timeline:
0s 30s 60s 90s 120s
│──────────│──────────│──────────│──────────│
lint ██████████ (Passed)
build ███████████████ (Passed)
test ██████ (Failed) ──> [Retry #1] ──> ██████ (Passed)
By mapping stage lifecycle domain events onto a visual timeline, developers can immediately spot execution bottlenecks, stage overlaps, retries, and failure points at a glance.
11. From Engineering Prototype to Product
A powerful backend architecture is useless if the developer interface is clunky. The final UX pass turned OdinRun into a cohesive developer dashboard:
- Dashboard: Real-time control center for fleet runner health, active runs, and system throughput.
- Run Details: Primary debugging surface bringing together stage logs, live SSE telemetry graphs, execution timeline, remediation history, and AI diagnosis.
- Remediation Rules: Interface to manage deterministic regex rules, set backoff policies, and inspect rule confidence scores.
- AI History: Searchable repository of past AI diagnoses and suggested fixes.
- Runner Management: Fleet-level view of registered runner agents, active job allocations, and system capacity.
12. The Architecture I Ended Up With
What started as a simple execution script evolved into a decoupled, layered CI/CD platform:
13. The Evolution Roadmap
Looking back, the evolution of the project can be summarized as a sequence of core engineering questions:
14. What I Learned Building It
- Reliability Comes Before Intelligence It's tempting to jump straight to building cool AI features. But AI built on top of an unreliable, non-deterministic execution engine is just an expensive way to hallucinate about broken builds. Fix container streams, timeouts, and process cleanup first.
- Observability Changes System Architecture Once execution emits structured domain events, adding streaming, visual timelines, automated retries, and AI diagnostics becomes simple. Observability isn't a UI feature; it's a foundational architectural property.
- Deterministic Automation is Underrated Not every problem needs an LLM. If a build fails due to a 503 Service Unavailable from a package registry, a deterministic regex rule with exponential backoff is faster, cheaper, safer, and 100% predictable. Save AI for interpretation of unknown failures.
- Protect Your Architectural Boundaries Separating Execution from Transport, Persistence from Database, and Diagnosis from Action allowed OdinRun to scale from SQLite to MongoDB, and local storage to S3, without touching the runner execution loop.
- Infrastructure Code Demands Empirical Proof You can't guess how Docker streams behave or how signal handling works under load. Building infrastructure forces you to read raw logs, inspect process trees, and write empirical tests.
15. From PipelineOS to OdinRun
There is one more important milestone that isn't represented by a PIP number.
The project was renamed.
The project originally began under the name PipelineOS. At that stage, the name accurately described what I was building: an operating layer for executing CI/CD pipelines.
But as the architecture evolved, the scope of the project changed.
It was no longer just a pipeline executor. It had become a complete execution and recovery system combining:
- CI/CD execution
- containerized runners
- durable state
- event-driven architecture
- real-time observability
- runtime telemetry
- automated remediation
- AI-assisted diagnosis
- recovery planning
- execution debugging
- developer-focused visualization
The name OdinRun represents that broader direction.
Why OdinRun?
The rename wasn't intended to erase the history of the project.
PipelineOS is the name under which the architecture was designed and developed. OdinRun is the identity the project is moving forward with.
The existing architecture, milestones, and engineering decisions remain part of the same project.
So the evolution is:
In other words, OdinRun isn't a completely new project. It is the next identity of the project that began as PipelineOS.
The earlier technical milestones—from persistence and the runner engine through observability and intelligence—form the foundation of OdinRun. The original project documentation describes PipelineOS as a self-hosted CI/CD runtime with Docker-backed execution and a React-based dashboard, while the later milestones expanded that foundation substantially.
16. Final Thoughts
Building PipelineOS—and eventually evolving it into OdinRun—started as an exploration of CI/CD internals.
It became an exercise in:
- distributed systems
- container orchestration
- event-driven architecture
- persistence
- observability
- automated recovery
- developer tooling
- applied AI
The most important lesson was that these aren't isolated problems. They form a dependency chain:
You can't skip directly to the end.
A system can't intelligently diagnose execution it can't observe. It can't safely remediate failures it can't understand. And it can't build useful historical intelligence without durable execution data.
That's what made this project interesting to build.
It wasn't about simply adding AI to CI/CD. It was about gradually building the infrastructure that makes intelligent CI/CD possible.
PipelineOS was where the journey started.
OdinRun is where it continues.
OdinRun is an open-source, self-hosted CI/CD runtime built around execution reliability, observability, automated remediation, and developer control.
Top comments (0)