Building a CI/CD Engine That Doesn't Feel Like a Black Box

Designing Real-Time Observability with Docker, Domain Events, and WebSockets

Every developer has experienced this:

You push code.

CI starts.

Five minutes later...

"Build Failed."

Why?

Somewhere inside thousands of log lines generated by an ephemeral Docker container that no longer exists, there is a tiny, obscure error message. You refresh the page, dig through the static text, and try to reverse-engineer what the build agent was actually doing when it died.

That frustration is one of the reasons I started building PipelineOS. Instead of treating observability as an afterthought—something you bolt onto the database after the execution engine is finished—I made it a first-class part of the runtime itself.

Why I Didn't Use Existing Tools

Before diving into the architecture, I should answer the obvious question: Why build another CI/CD engine? Why not just use GitHub Actions, GitLab CI, Jenkins, Tekton, or Argo?

Those tools are incredible. GitHub Actions revolutionized how we think about integrated workflows, and Tekton provides an industrial-grade, Kubernetes-native execution layer. I didn't build PipelineOS because I think the industry is doing it wrong. I built it because I wanted to understand how these massive distributed systems actually work under the hood.

I wanted to learn what it takes to securely isolate workloads inside Docker, how to reliably stream multiplexed logs over WebSockets, and how to design a distributed job queue that doesn't lose state when a runner node crashes. When you use a managed platform, you are shielded from the brutal reality of system state machines, zombie process recovery, and stream demuxing. By building PipelineOS from the ground up, I forced myself to solve these exact engineering problems.

1. Why Observability Matters

In modern engineering, waiting on pipelines is one of the biggest momentum killers.

When developers push code, they need immediate, granular feedback. If an npm install stalls because of a network proxy issue, or a C++ compilation spikes memory and gets silently killed by the Linux OOM killer, developers shouldn't have to wait 15 minutes for the pipeline to time out before they realize something is wrong.
Poor visibility creates poor developer experience:

• Debugging blind: Parsing thousands of lines of raw text logs hours after the container has been destroyed.
• Delayed feedback: Reloading a dashboard 20 times to see if a status has moved from Pending to Running.
• Lack of infrastructure insight: When a build runs slowly, you have no idea if the cause is a poorly optimized Dockerfile or a starved CI/CD runner host running out of CPU credits.

Observability isn't just about collecting debug logs when things break; it’s about reducing cognitive load and giving engineers real-time confidence in their automation.

2. What "Real-Time Observability" Actually Means

In CI/CD, most platforms equate "observability" to simple streaming text logs. But true real-time visibility in a modern distributed build runtime requires multi-dimensional awareness:

• Stage Lifecycle Transitions: Exact state machines reporting millisecond-accurate transitions (Pending → Claimed → Running → Success / Failed).
• Runtime Resource Metrics: Continuous CPU profiling, memory utilization peaks, network RX/TX bandwidth, and block I/O per container.
• Runner Health & Heartbeats: Detecting daemon crashes and host resource exhaustion before builds turn into orphaned zombies.
• Execution Timelines: Waterfall charts and cost estimations based on live compute duration.

3. The Architecture

To achieve this level of granular visibility without overloading the backend, you have to separate event generation from ingestion and transport.

Here is how the PipelineOS observability data plane flows from a git push all the way to a developer's React dashboard:

This decoupled architecture ensures that the execution engine (Docker container) doesn't care who is watching it, and the transport layer (WebSockets) doesn't care how the events were generated.
By treating observability as a stream of immutable domain events rather than scattered API calls, the same architecture can support future features like distributed runners, replayable execution timelines, and AI-driven analysis without changing the execution engine itself.

4. The Biggest Engineering Challenges

While building PipelineOS, I explored several architectural trade-offs and engineering challenges.

A. Docker Telemetry
To get live CPU and memory metrics out of a running stage, PipelineOS periodically polls snapshots of the Docker Engine API using the stats({ stream: false }) endpoint. But calculating true CPU utilization across multiple cores isn't trivial.

You can't just read a cpu_usage_percent field. You have to calculate the delta between the container's CPU cycles and the host system's CPU cycles, factored across the number of online CPU cores:

// Calculating CPU percentage across online cores
const cpuDelta = cpuTotal - prevCpu;
const systemDelta = systemTotal - prevSystem;
const onlineCpus = cpuStats.online_cpus || 1;

if (cpuDelta > 0 && systemDelta > 0) {
  return (cpuDelta / systemDelta) * onlineCpus * 100;
}

B. Event-Driven Architecture (Reducing Runner Complexity)
In early prototypes, the Runner Agent made direct REST API calls every time a stage changed status: POST /status, POST /logs, POST /metrics. This tightly coupled the runner's execution logic with the backend's HTTP routing.

I solved this by decomposing the executor and introducing an InProcessEventBus. Now, the container runner simply publishes immutable domain events (like StageStarted or StageMetricsUpdated) to the local bus. It doesn't know or care if those events are written to a file, sent to an API, or dropped.

C. Bulk Ingestion: Slashing Network Chatter
When a container installs heavy Node dependencies, it can generate hundreds of log chunks per second. If the Runner made a separate HTTP POST request for every single log chunk and telemetry poll, it would exhaust its TCP connection pool and effectively DDoS the control plane API.

To solve this, I implemented Bulk Ingestion. The internal event bus batches all domain events and flushes them to a single endpoint (POST /internal/events).

Before: ~200 HTTP requests/sec during a noisy build

After: 1 batched request every 2000 ms

This drastically reduces HTTP overhead, lowers TCP handshake latency, and turns erratic, spiky traffic into a smooth, predictable ingestion pipeline.

D. WebSockets vs. Server-Sent Events (SSE)
Once the Control Plane ingests these bulk events, it needs to stream them to the user's browser. PipelineOS supports both WebSockets and Server-Sent Events (SSE).

While WebSockets provide full-duplex, low-latency communication, Server-Sent Events (SSE) provide unidirectional streaming over standard HTTP/1.1 (or HTTP/2). Because SSE is just a long-lived HTTP request, it automatically handles reconnections and plays nicely with standard reverse proxies (like NGINX).

5. Runtime Telemetry & Operational Health

Monitoring the code that builds your code is just as critical as monitoring production.

In PipelineOS, runtime telemetry extends beyond container CPU and network I/O. The Runner Agent constantly emits RunnerHeartbeat events. This automated pulse allows the control plane to actively monitor live workers.

Runner Health DashboardActive heartbeats ensuring the runner hasn't crashed or run out of memory.

If a runner hardware node crashes mid-build or experiences a power interruption, the heartbeat drops. The control plane’s Staleness Detection Service sweeps the database, identifies the orphaned "zombie" run, and reclaims or cleanly fails the pipeline so developers aren't stuck staring at a dashboard waiting for a build that will never finish

6. Project Evolution Timeline

It is deeply rewarding to look back and see how PipelineOS has grown from a basic state machine into a robust, observable distributed system. Here is a timeline of our recent milestones:

7. Lessons Learned

Building this architecture reinforced a few deep engineering truths:

• **Observability is an architectural bedrock, not a bolt-on feature. **If you don't design your runtime around emitting atomic events from day one, adding real-time visibility later requires rewriting half your execution loops.
• **Event-driven architectures dramatically simplify live streaming. **When your backend internally communicates via immutable events, streaming to WebSockets or SSE becomes nothing more than attaching a simple .on() listener.
• Separating transport from business logic pays compound interest. By isolating the transport from event ingestion, adding future real-time features requires zero changes to the core Docker execution engine.

8. What's Next: From Visibility to Intelligence

Building a CI/CD platform has taught me that execution is only the beginning.
Reliable systems need durability.
Durable systems need observability.
And observable systems finally become capable of intelligence.
That's the direction PipelineOS is heading next. In the coming milestones, we will move beyond just displaying failures to understanding them—applying automated remediation rules, dynamically allocating resources based on historical patterns, and using intelligence to keep developers in their flow state.

If you're interested in CI/CD internals, distributed systems, or building developer tools from scratch, I'd love to hear your thoughts or feedback on the architecture.
PipelineOS is an open-source, self-hosted CI/CD runtime built for developers who value simplicity, visibility, and control. Check out our architecture and contribute on GitHub!

Comments

[Rahan Judes Michael]

Karthik sir eager to see more blog from you really usefull 🫶❤️