DEV Community

Chukwudum Agbasi
Chukwudum Agbasi

Posted on

DevOps Magic: Building a Self-Healing, Policy-Driven Deployment Engine

Modern DevOps isn't just about moving code; it’s about creating a "clinical" environment where infrastructure is predictable, self-validating, and resilient. For my recent project, SwiftDeploy, I built a CLI tool that doesn't just deploy containers—it diagnoses the host environment and enforces strict policy guardrails before a single container is birthed.

Here is the technical deep dive into how I built a self-generating infrastructure stack with Open Policy Agent (OPA) integration.

  1. The Design: Infrastructure as Logic Most CI/CD pipelines rely on static YAML files. SwiftDeploy takes a different approach: it treats infrastructure as a dynamic output of a manifest.

How it works:
The tool uses a manifest.yaml as the "source of truth." When you run ./swiftdeploy init, the script acts as a compiler:

It parses service definitions (images, ports, environment variables) using yq.

It injects these variables into .template files using envsubst.

It generates a perfectly tailored docker-compose.yml and nginx.conf on the fly.

By "writing" its own infrastructure files, SwiftDeploy eliminates configuration drift. If the manifest changes, the infrastructure regenerates to match, ensuring that what you see in your config is exactly what runs in your stack.

  1. The Guardrails: OPA and the "Pre-Flight" Check

In a production environment, isolation isn't just a preference—it's a requirement. I integrated Open Policy Agent (OPA) to act as the "Medical Board" for my deployments.

The Logic of Isolation:
We use OPA to evaluate two distinct policy sets:

Infrastructure Policy: Checks host health (CPU load, Disk space, Memory). If the host is "feverish" (e.g., CPU load > 2.0), OPA blocks the deployment to prevent a cascading failure.

Canary Safety: During promotion from Stable to Canary, OPA analyzes live metrics. If the error rate exceeds 5%, the promotion is "quarantined."

Why Rego?
Using Rego (OPA's query language) allows us to write policies like this:

Code snippet
package infrastructure
default allow = false
allow {

input.cpu_load < data.infrastructure.max_cpu_load
input.disk_free_gb > data.infrastructure.min_disk_free_gb

}
This decouples "how to deploy" from "when it is safe to deploy."`

  1. *The Chaos: Watching the System Break * A deployment tool is only as good as its visibility. To test SwiftDeploy, I intentionally injected a "Slow State" and an "Error State" into the Python backend.

The Failure Scenario:
I updated the app to return 500 Internal Server Error on 20% of traffic. I then monitored the stack using the built-in status view.

The Status View Capture:

Plaintext
UPTIME: 124.5s
REQUESTS: 50 total

POLICY COMPLIANCE

metrics: error_rate=20.0% p99_latency=450ms
checking canary_safety policy...
BLOCK canary_safety: error_rate 20.0% exceeds maximum 5%
Because the status loop was feeding real-time metrics into OPA, the Promote command was automatically locked. The system "knew" it was sick before I did.

  1. Lessons Learne Building this journey from Stage 2 through 4B taught me three critical lessons:

Environment Parity is Hard: Moving from Linux-based logic to a Windows/Git Bash environment revealed how much we rely on specific binaries like free or /proc. Python is the ultimate "bridge" for cross-platform hardware stats.

Fail-Safe is the Only Way: My policy engine was designed to "Block if OPA is unavailable." This saved me multiple times when the OPA container hadn't fully mapped its ports to the host.

Observability is Part of Deployment: A deployment doesn't end when the container is Up. It ends when the metrics prove the container is healthy.

Replicate the Work

If you want to build your own policy-driven CLI:

Tooling: Bash, Docker, OPA, and yq.

Step 1: Create templates for your YAML.

Step 2: Use curl to POST your system stats to OPA’s Data API.

Step 3: Only trigger docker compose up if the API returns {"result": true}.

Infrastructure should be smart enough to say "No" to a bad deployment. SwiftDeploy makes sure it does.

Top comments (0)