How hard can it be to build a CI/CD system?

How hard can it be to build a CI/CD system?

That question stuck with me long enough that I actually started building one. Not because someone asked me to. Not because I spotted a market gap. Just because the question wouldn't go away.

The trigger was Concourse CI. I've been using it for a while and what I love about it is the resource abstraction, an interface that anything external has to follow. Check for new versions, pull them, push back. As a Go developer this kind of clean interface resonates with me. Everything in the pipeline is just something that implements that contract.

But the operational overhead is significant. And I needed CI for my own side projects anyway, games and open source tools that require custom environments GitHub Actions can't provide. So I started building.

What I wanted

A single binary that could also scale horizontally when needed. Start with nothing, grow when you need to.

You start like this:

./pikoci server \
  --db-system mem \
  --pubsub-system mem \
  --run-worker \
  --pipeline-config pipeline.hcl

That's a complete CI/CD system. In memory, no files, no external services. When you want persistence, add --db-system sqlite. When you need distributed workers, add NATS and start workers on other machines. The pipeline config never changes.

The interesting parts

Four pluggable abstractions

PikoCI has four concepts you define in HCL and can source from a URL: resource types, runners, service types, and secret types. Each follows the same pattern: define the type once, instantiate it with params.

A resource_type defines how to watch something for changes and fetch it. A resource is an instance of it:

resource_type "git" {
  source = "pikoci://git"  # built-in
}

resource "git" "my-app" {
  params {
    url  = "https://github.com/org/app"
    name = "app"
  }
  check_interval = "@every 1m"
}

A runner_type defines where tasks execute. The docker and exec runners are built-in, no declaration needed. Here's what the docker runner looks like under the hood, in case you want to define your own:

# this is what pikoci://docker looks like
# define your own to customize or replace it
runner_type "docker" {
  run {
    path = "docker"
    args = [
      "run", "--rm",
      "-v", "$WORKDIR:/workdir",
      "-w", "/workdir",
      "$image",
      "/bin/sh", "-ec", "$cmd",
    ]
  }
}

job "test" {
  get "git" "my-app" { trigger = true }
  task "run-tests" {
    run "docker" {
      image = "golang:1.25"
      cmd   = "cd app && make test"
      args  = ["-v", "/cache/go:/root/go/pkg/mod"]
    }
  }
}

A secret_type defines where credentials come from. Secrets are bound to variables and referenced anywhere in the pipeline:

secret_type "vault" {
  source = "pikoci://vault"
}

variable "db_password" {
  secret "vault" {
    path = "secret/data/db"
    key  = "password"
  }
}

A service_type defines processes that run alongside your tasks, started before, stopped after, guaranteed regardless of outcome. This is the feature I'm most proud of:

service_type "postgres" {
  # source = "pikoci://postgres" — or define inline
  start "exec" {
    path = "/bin/sh"
    args = ["-ec", "docker run -d --name db -p 5432:5432 postgres:16"]
  }
  ready_check "exec" {
    path    = "/bin/sh"
    args    = ["-ec", "pg_isready -h localhost"]
    timeout = "30s"
  }
  stop "exec" {
    path = "/bin/sh"
    args = ["-ec", "docker rm -f db"]
  }
}

job "integration" {
  service "postgres" {}
  get "git" "my-app" { trigger = true }
  task "test" {
    run "exec" {
      path = "make"
      args = ["integration-test"]
    }
  }
}

No Docker-in-Docker. No docker-compose alongside CI. The service stops regardless of whether the job passed or failed.

All four types are sourceable from a URL. The built-ins use pikoci://, the same mechanism as anything you host yourself. If you need a runner that executes jobs in Kubernetes or Azure, write it once, host it anywhere, reference it by URL.

Putting it all together

Here's a small pipeline that uses all four abstractions at once:

resource_type "git" {
  source = "pikoci://git"
}

resource "git" "my-app" {
  params {
    url  = "https://github.com/org/app"
    name = "app"
  }
  check_interval = "@every 1m"
}

secret_type "vault" {
  source = "pikoci://vault"
}

variable "db_password" {
  secret "vault" {
    path = "secret/data/db"
    key  = "password"
  }
}

service_type "postgresql" {
  source = "pikoci://postgresql"
}

job "test" {
  get "git" "my-app" { trigger = true }
  service "postgresql" {
    version  = "17"
    port     = "5432"
    password = var.db_password
  }
  task "run-tests" {
    run "docker" {
      image = "golang:1.25"
      cmd   = "cd app && make integration-test"
      args  = ["-v", "/cache/go:/root/go/pkg/mod"]
    }
  }
}

A git resource watches for changes, a Vault secret feeds the Postgres password, Postgres starts as a service, and the task runs inside Docker. All four abstractions, one pipeline.

Running pipelines locally

pikoci run --pipeline-config pipeline.hcl --job test

Any job, on your laptop, no server required. Override resources with local paths, inject secrets via --var. The same pipeline that runs in CI runs on your laptop.

The queue decision

Workers don't connect directly to the server, they subscribe to a queue. This means workers can be behind NAT, on a laptop, in a different data center, or completely ephemeral. The server never needs to know where workers are. It made distributed workers trivially easy to add. Start a worker anywhere with network access to the queue and it just works.

Where it is now

PikoCI deploys itself. The pipeline runs PR checks, mock tests, and integration tests against six different database and queue backends: MariaDB, PostgreSQL, NATS, RabbitMQ, Kafka, and Vault, all running as services. Then it builds multi-arch Docker images and redeploys itself with zero downtime.

Those six backends are not just targets, they are the pluggable abstractions themselves. The same PikoCI binary connects to any of them depending on how you start it. Testing against all of them is how I make sure the abstractions actually hold.

All of it is publicly visible at ci.pikoci.com/teams/main/pipelines/pikoci. No account needed.

It's Apache 2.0, written in Go, and very much something I use for my own projects every day.

If you try it and something is broken, I want to know. If something is missing, I also want to know.

pikoci.com · github.com/pikoci/pikoci

Comments

[Harjot Singh]

"How hard can it be" is the perfect setup, because CI/CD is the canonical iceberg - the happy path (clone, run steps, report status) is a weekend; the 90% you don't see is where it lives. Reliable isolated execution environments, caching that's correct and not stale, secrets handling, concurrency and queueing, retries vs idempotency, log streaming, artifact storage, and the brutal one: making failures reproducible and debuggable instead of "works in CI, fails locally" or vice versa. The orchestration of untrusted, side-effecting steps with clean isolation is genuinely hard distributed-systems work dressed up as "just run my scripts."

Building one is the best way to appreciate why the boring infrastructure layer is where all the real engineering hides - which is exactly the space I work in. Moonshift, the thing I build, is a multi-agent pipeline that takes a prompt to a deployed SaaS, and the same lesson holds: the orchestration (isolation, retries, reproducibility, gating each step) is the hard part, not the happy path. Multi-model routing keeps a build ~$3 flat, first run free no card. Really enjoy this kind of "I went down the rabbit hole" post. What surprised you most as the hidden-hard part - the isolation/sandboxing, or the caching correctness? Cache invalidation in CI is the bug that's eaten the most of my life.

[Francesc Gil]

You nailed it, the happy path is deceptive. The iceberg analogy is exactly right.

The hardest part for me was the scheduler. Making sure multiple server instances don't double-trigger the same job and getting graceful shutdown right so in-flight jobs survive a redeploy. None of that is glamorous but all of it is where the real work hides. And a lot of coordination issues that where found thanks to PikoCI deploying PiloCI.

Caching has been interesting too, I implemented a cache flag on resources and resource types that gives the script access to a persistent $CACHE_DIR between runs. For git that means fetch instead of clone on subsequent pulls. The correctness problem you're describing is exactly why I kept it simple, a flag that opts in, a directory that persists, nothing clever. A stale cache that silently passes a build that should fail is worse than no cache at all.

The isolation problem largely solved itself, the exec runner runs directly on the worker, the Docker runner gives you container isolation, and the service lifecycle (start/ready_check/stop) gives you clean ephemeral dependencies without Docker-in-Docker. Worker tags(still not implemented) add another layer, you can dedicate specific workers to specific jobs, isolating sensitive workloads at the infrastructure level. Since PikoCI is self-hosted, the sandboxing question is really about trusting your own pipeline configs, same trust model as any other CI tool.

Moonshift sounds interesting, multi-agent pipelines with the same orchestration challenges but at the LLM layer. The gating and retries problem must be fascinating when the steps are non-deterministic.