How hard can it be to build a CI/CD system?

Francesc Gil — Thu, 28 May 2026 22:31:06 +0000

That question stuck with me long enough that I actually started building one. Not because someone asked me to. Not because I spotted a market gap. Just because the question wouldn't go away.

The trigger was Concourse CI. I've been using it for a while and what I love about it is the resource abstraction, an interface that anything external has to follow. Check for new versions, pull them, push back. As a Go developer this kind of clean interface resonates with me. Everything in the pipeline is just something that implements that contract.

But the operational overhead is significant. And I needed CI for my own side projects anyway, games and open source tools that require custom environments GitHub Actions can't provide. So I started building.

What I wanted

A single binary that could also scale horizontally when needed. Start with nothing, grow when you need to.

You start like this:

./pikoci server \
  --db-system mem \
  --pubsub-system mem \
  --run-worker \
  --pipeline-config pipeline.hcl

That's a complete CI/CD system. In memory, no files, no external services. When you want persistence, add --db-system sqlite. When you need distributed workers, add NATS and start workers on other machines. The pipeline config never changes.

The interesting parts

Four pluggable abstractions

PikoCI has four concepts you define in HCL and can source from a URL: resource types, runners, service types, and secret types. Each follows the same pattern: define the type once, instantiate it with params.

A resource_type defines how to watch something for changes and fetch it. A resource is an instance of it:

resource_type "git" {
  source = "pikoci://git"  # built-in
}

resource "git" "my-app" {
  params {
    url  = "https://github.com/org/app"
    name = "app"
  }
  check_interval = "@every 1m"
}

A runner_type defines where tasks execute. The docker and exec runners are built-in, no declaration needed. Here's what the docker runner looks like under the hood, in case you want to define your own:

# this is what pikoci://docker looks like
# define your own to customize or replace it
runner_type "docker" {
  run {
    path = "docker"
    args = [
      "run", "--rm",
      "-v", "$WORKDIR:/workdir",
      "-w", "/workdir",
      "$image",
      "/bin/sh", "-ec", "$cmd",
    ]
  }
}

job "test" {
  get "git" "my-app" { trigger = true }
  task "run-tests" {
    run "docker" {
      image = "golang:1.25"
      cmd   = "cd app && make test"
      args  = ["-v", "/cache/go:/root/go/pkg/mod"]
    }
  }
}

A secret_type defines where credentials come from. Secrets are bound to variables and referenced anywhere in the pipeline:

secret_type "vault" {
  source = "pikoci://vault"
}

variable "db_password" {
  secret "vault" {
    path = "secret/data/db"
    key  = "password"
  }
}

A service_type defines processes that run alongside your tasks, started before, stopped after, guaranteed regardless of outcome. This is the feature I'm most proud of:

service_type "postgres" {
  # source = "pikoci://postgres" — or define inline
  start "exec" {
    path = "/bin/sh"
    args = ["-ec", "docker run -d --name db -p 5432:5432 postgres:16"]
  }
  ready_check "exec" {
    path    = "/bin/sh"
    args    = ["-ec", "pg_isready -h localhost"]
    timeout = "30s"
  }
  stop "exec" {
    path = "/bin/sh"
    args = ["-ec", "docker rm -f db"]
  }
}

job "integration" {
  service "postgres" {}
  get "git" "my-app" { trigger = true }
  task "test" {
    run "exec" {
      path = "make"
      args = ["integration-test"]
    }
  }
}

No Docker-in-Docker. No docker-compose alongside CI. The service stops regardless of whether the job passed or failed.

All four types are sourceable from a URL. The built-ins use pikoci://, the same mechanism as anything you host yourself. If you need a runner that executes jobs in Kubernetes or Azure, write it once, host it anywhere, reference it by URL.

Putting it all together

Here's a small pipeline that uses all four abstractions at once:

resource_type "git" {
  source = "pikoci://git"
}

resource "git" "my-app" {
  params {
    url  = "https://github.com/org/app"
    name = "app"
  }
  check_interval = "@every 1m"
}

secret_type "vault" {
  source = "pikoci://vault"
}

variable "db_password" {
  secret "vault" {
    path = "secret/data/db"
    key  = "password"
  }
}

service_type "postgresql" {
  source = "pikoci://postgresql"
}

job "test" {
  get "git" "my-app" { trigger = true }
  service "postgresql" {
    version  = "17"
    port     = "5432"
    password = var.db_password
  }
  task "run-tests" {
    run "docker" {
      image = "golang:1.25"
      cmd   = "cd app && make integration-test"
      args  = ["-v", "/cache/go:/root/go/pkg/mod"]
    }
  }
}

A git resource watches for changes, a Vault secret feeds the Postgres password, Postgres starts as a service, and the task runs inside Docker. All four abstractions, one pipeline.

Running pipelines locally

pikoci run --pipeline-config pipeline.hcl --job test

Any job, on your laptop, no server required. Override resources with local paths, inject secrets via --var. The same pipeline that runs in CI runs on your laptop.

The queue decision

Workers don't connect directly to the server, they subscribe to a queue. This means workers can be behind NAT, on a laptop, in a different data center, or completely ephemeral. The server never needs to know where workers are. It made distributed workers trivially easy to add. Start a worker anywhere with network access to the queue and it just works.

Where it is now

PikoCI deploys itself. The pipeline runs PR checks, mock tests, and integration tests against six different database and queue backends: MariaDB, PostgreSQL, NATS, RabbitMQ, Kafka, and Vault, all running as services. Then it builds multi-arch Docker images and redeploys itself with zero downtime.

Those six backends are not just targets, they are the pluggable abstractions themselves. The same PikoCI binary connects to any of them depending on how you start it. Testing against all of them is how I make sure the abstractions actually hold.

All of it is publicly visible at ci.pikoci.com/teams/main/pipelines/pikoci. No account needed.

It's Apache 2.0, written in Go, and very much something I use for my own projects every day.

If you try it and something is broken, I want to know. If something is missing, I also want to know.

pikoci.com · github.com/pikoci/pikoci

DEV Community: Francesc Gil

How hard can it be to build a CI/CD system?

What I wanted

The interesting parts

Putting it all together

The queue decision

Where it is now