DEV Community: Francesc Gil

PikoCI v0.5.0: worker tagging, in_parallel, Linux packages, and more

Francesc Gil — Tue, 16 Jun 2026 12:14:39 +0000

Six months into building PikoCI and v0.5.0 is the biggest release yet. This one is almost entirely about workers: routing work to the right place, running things in parallel, and knowing what your workers are doing.

Worker tagging

You can now route jobs and resource checks to specific workers using tags. Add tags = ["gpu"] to a job in your pipeline HCL, start a worker with --tags gpu, and that job will only run on workers with the matching tag. Matching uses AND logic: all job tags must be present on the worker.

job "train-model" {
  tags = ["gpu"]
  task "train" {
    run "exec" {
      path = "python"
      args = ["train.py"]
    }
  }
}

Workers can also be made exclusive. The --exclusive-tags flag restricts a worker to only handle tagged work, keeping it free for the jobs that need it.

Worker health monitoring

Workers now send periodic heartbeats to the server. A new admin dashboard shows all connected workers with status, platform, version, and uptime. If a worker goes stale (no heartbeat for 90 seconds), a warning appears and admins can remove it from the UI or via the API.

There is also a pikoci_workers Prometheus gauge with a status label if you want to alert on missing workers.

in_parallel step

Run multiple steps concurrently within a job. Useful for running tests against multiple configurations simultaneously, or for parallelizing independent tasks.

job "test" {
  in_parallel {
    limit     = 3
    fail_fast = true

    task "test-postgres" {
      run "exec" { path = "make" args = ["test-postgres"] }
    }
    task "test-mysql" {
      run "exec" { path = "make" args = ["test-mysql"] }
    }
    task "test-sqlite" {
      run "exec" { path = "make" args = ["test-sqlite"] }
    }
  }
}

limit caps how many run simultaneously. fail_fast cancels remaining steps on the first failure.

pipeline validate

Validate a pipeline HCL file without a running server.

pikoci pipeline validate pipeline.hcl
pikoci pipeline validate pipeline.hcl --var env=staging

Useful for pre-commit hooks and CI checks on the pipeline config itself. Catches syntax errors, structural errors, and invalid cross-references, all reported at once with line numbers.

Linux packages

PikoCI now ships as .deb and .rpm packages alongside the existing binaries. Both amd64 and arm64. A SHA256SUMS file is included in every release.

fs resource type

A built-in resource type for watching local files and directories. Triggers jobs when file contents change via SHA256 hash comparison.

resource_type "fs" {
  source = "pikoci://fs"
}

resource "fs" "config" {
  params {
    path = "/etc/myapp/config.yaml"
  }
}

Useful for local development workflows and watching config files.

Worker communication now uses gRPC

Workers connect to the server via a persistent gRPC stream instead of the previous queue-based system. Job dispatch is instant, cancellation signals reach the worker in milliseconds instead of waiting for a polling interval, and log streaming happens on the same connection. No external queue needed for distributed workers.

A dedicated post covering the full migration from queues to gRPC is coming soon.

Other fixes

New jobs no longer replay the entire version backlog on creation: they record a baseline at creation time and only see newer versions. Downstream jobs now trigger immediately when upstream builds succeed, instead of waiting for the next scheduler tick. Cross-reference validation catches invalid references at save time with suggestions for typos.

github.com/pikoci/pikoci · pikoci.com · Apache 2.0

v0.5.0 release notes

Services without Docker-in-Docker: how PikoCI handles test dependencies

Francesc Gil — Tue, 09 Jun 2026 08:13:24 +0000

If you've ever needed a database for your integration tests in CI, you've probably encountered one of these solutions:

Docker-in-Docker. A separate docker-compose file you run alongside CI. A pre-provisioned shared database that everyone fights over. Or just skipping integration tests in CI entirely.

None of these are great. Docker-in-Docker requires privileged containers and is notoriously fragile. docker-compose alongside CI is manual, error-prone, and hard to clean up. Shared databases cause flaky tests. Skipping integration tests defeats the purpose.

PikoCI takes a different approach: services as a first-class concept.

What services are

A service in PikoCI is an ephemeral process that runs alongside your job's tasks. Services start where they are defined in the job plan, if you define them first, they start first; if you define them after a get step, they start after the get. They always stop unconditionally when the job ends, regardless of where they were defined or whether tasks succeeded or failed.

You define a service_type once and reference it from any job:

service_type "postgres" {
  params = ["version"]

  start "exec" {
    path = "/bin/sh"
    args = ["-ec", <<-EOT
      NAME="pikoci-${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-pg"
      docker rm -f $NAME 2>/dev/null || true
      docker run -d --name $NAME -p 5432:5432 \
        -e POSTGRES_PASSWORD=test \
        postgres:$param_version
    EOT
    ]
  }

  ready_check "exec" {
    path     = "/bin/sh"
    args     = ["-ec", "docker exec pikoci-${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-pg pg_isready"]
    interval = "2s"
    timeout  = "30s"
  }

  stop "exec" {
    path = "/bin/sh"
    args = ["-ec", "docker rm -f pikoci-${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-pg 2>/dev/null || true"]
  }
}

job "integration" {
  get "git" "app" { trigger = true }

  service "postgres" {
    version = "16"
  }

  task "test" {
    run "exec" {
      path = "make"
      args = ["integration-test"]
    }
  }
}

PikoCI starts Postgres, waits for pg_isready to return 0, runs your tasks, then stops and removes the container. Your task connects to localhost:5432 like it would anywhere else.

The lifecycle

Services start in the order they appear in the job plan. You control placement:

Define a get step first if you need the code pulled before services start
Define services wherever makes sense for your job
After all steps complete, success or failure, stop runs for every started service

The stop step always runs. If your task panics, if the job is cancelled, if the worker gets a SIGTERM, stop still runs. No orphaned containers.

No Docker-in-Docker

Services don't run inside a container. They run on the worker host directly. Your tasks can run inside Docker containers via the docker runner, and they connect to services on localhost because everything is on the same host network.

This is the key difference from Docker-in-Docker. With DinD you're running a Docker daemon inside a container, which requires --privileged, has kernel-level implications, and is generally fragile. With PikoCI services, Docker is just a process manager, the worker runs docker run, the container starts on the host network, your tasks connect normally.

Orphan prevention

If a worker crashes mid-job, the stop block never runs and Docker containers keep running. The next job tries to start on the same port and fails.

The solution is stable container names with pre-start cleanup. Use $BUILD_PIPELINE_NAME and $BUILD_JOB_NAME, stable across runs, and always clean up at the start of the start block:

NAME="pikoci-${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-postgres"
docker rm -f $NAME 2>/dev/null || true   # kill orphan if exists
docker run -d --name $NAME ...           # start fresh

The || true means cleanup never fails the job.

Not just Docker

Services aren't Docker-specific. You can start any process: a local daemon, a background script, anything the worker can run.

service_type "redis" {
  start "exec" {
    path = "/bin/sh"
    args = ["-ec", "redis-server --daemonize yes --dir $WORKDIR"]
  }

  stop "exec" {
    path = "/bin/sh"
    args = ["-ec", "redis-cli shutdown"]
  }
}

No ready check needed, Redis starts fast enough that the job can proceed immediately after start completes.

Sourceable and reusable

Like all PikoCI abstractions, service_type definitions are sourceable from a URL. Write a service type once, host it in a git repo or anywhere with an HTTPS endpoint, and reference it from any pipeline:

service_type "postgres" {
  #source = "https://raw.githubusercontent.com/myorg/pikoci-services/main/postgres.hcl"
  source = "pikoci://postgresql"
}

The built-in types use the same mechanism, pikoci://postgres is just a URL that ships with the binary. Community-built service types work identically.

What if my worker runs in Docker?

If your worker runs as a Docker container, services that use docker run need the Docker socket mounted so the worker spawns sibling containers on the host instead of nested ones:

docker run -v /var/run/docker.sock:/var/run/docker.sock pikoci-worker

Alternatively, if your worker's Docker image already has the service preinstalled (Postgres, MySQL, Redis) you can start it as a local process directly, no socket needed. It's a less common setup but works cleanly for fixed environments.

The simplest approach is to run the worker as a plain process on the host. Then any service can use Docker freely, just the worker running containers, no nesting involved. That's how the PikoCI dogfooding pipeline runs its workers.

PikoCI uses this itself

The pipeline that tests PikoCI runs integration tests against six backends simultaneously using services: MariaDB, PostgreSQL, NATS, RabbitMQ, Kafka, and Vault. All six are defined in the job plan, all six stop when the job ends. The pipeline is publicly visible at ci.pikoci.com/teams/main/pipelines/pikoci, no account needed.

This is how integration tests should work in CI. Not Docker-in-Docker, not shared test databases, not skipped tests. Just services that start where you need them and stop when the job ends.

pikoci.com · github.com/pikoci/pikoci · docs.pikoci.com/Services

How hard can it be to build a CI/CD system?

Francesc Gil — Thu, 28 May 2026 22:31:06 +0000

How hard can it be to build a CI/CD system?

That question stuck with me long enough that I actually started building one. Not because someone asked me to. Not because I spotted a market gap. Just because the question wouldn't go away.

The trigger was Concourse CI. I've been using it for a while and what I love about it is the resource abstraction, an interface that anything external has to follow. Check for new versions, pull them, push back. As a Go developer this kind of clean interface resonates with me. Everything in the pipeline is just something that implements that contract.

But the operational overhead is significant. And I needed CI for my own side projects anyway, games and open source tools that require custom environments GitHub Actions can't provide. So I started building.

What I wanted

A single binary that could also scale horizontally when needed. Start with nothing, grow when you need to.

You start like this:

./pikoci server \
  --db-system mem \
  --pubsub-system mem \
  --run-worker \
  --pipeline-config pipeline.hcl

That's a complete CI/CD system. In memory, no files, no external services. When you want persistence, add --db-system sqlite. When you need distributed workers, add NATS and start workers on other machines. The pipeline config never changes.

The interesting parts

Four pluggable abstractions

PikoCI has four concepts you define in HCL and can source from a URL: resource types, runners, service types, and secret types. Each follows the same pattern: define the type once, instantiate it with params.

A resource_type defines how to watch something for changes and fetch it. A resource is an instance of it:

resource_type "git" {
  source = "pikoci://git"  # built-in
}

resource "git" "my-app" {
  params {
    url  = "https://github.com/org/app"
    name = "app"
  }
  check_interval = "@every 1m"
}

A runner_type defines where tasks execute. The docker and exec runners are built-in, no declaration needed. Here's what the docker runner looks like under the hood, in case you want to define your own:

# this is what pikoci://docker looks like
# define your own to customize or replace it
runner_type "docker" {
  run {
    path = "docker"
    args = [
      "run", "--rm",
      "-v", "$WORKDIR:/workdir",
      "-w", "/workdir",
      "$image",
      "/bin/sh", "-ec", "$cmd",
    ]
  }
}

job "test" {
  get "git" "my-app" { trigger = true }
  task "run-tests" {
    run "docker" {
      image = "golang:1.25"
      cmd   = "cd app && make test"
      args  = ["-v", "/cache/go:/root/go/pkg/mod"]
    }
  }
}

A secret_type defines where credentials come from. Secrets are bound to variables and referenced anywhere in the pipeline:

secret_type "vault" {
  source = "pikoci://vault"
}

variable "db_password" {
  secret "vault" {
    path = "secret/data/db"
    key  = "password"
  }
}

A service_type defines processes that run alongside your tasks, started before, stopped after, guaranteed regardless of outcome. This is the feature I'm most proud of:

service_type "postgres" {
  # source = "pikoci://postgres" — or define inline
  start "exec" {
    path = "/bin/sh"
    args = ["-ec", "docker run -d --name db -p 5432:5432 postgres:16"]
  }
  ready_check "exec" {
    path    = "/bin/sh"
    args    = ["-ec", "pg_isready -h localhost"]
    timeout = "30s"
  }
  stop "exec" {
    path = "/bin/sh"
    args = ["-ec", "docker rm -f db"]
  }
}

job "integration" {
  service "postgres" {}
  get "git" "my-app" { trigger = true }
  task "test" {
    run "exec" {
      path = "make"
      args = ["integration-test"]
    }
  }
}

No Docker-in-Docker. No docker-compose alongside CI. The service stops regardless of whether the job passed or failed.

All four types are sourceable from a URL. The built-ins use pikoci://, the same mechanism as anything you host yourself. If you need a runner that executes jobs in Kubernetes or Azure, write it once, host it anywhere, reference it by URL.

Putting it all together

Here's a small pipeline that uses all four abstractions at once:

resource_type "git" {
  source = "pikoci://git"
}

resource "git" "my-app" {
  params {
    url  = "https://github.com/org/app"
    name = "app"
  }
  check_interval = "@every 1m"
}

secret_type "vault" {
  source = "pikoci://vault"
}

variable "db_password" {
  secret "vault" {
    path = "secret/data/db"
    key  = "password"
  }
}

service_type "postgresql" {
  source = "pikoci://postgresql"
}

job "test" {
  get "git" "my-app" { trigger = true }
  service "postgresql" {
    version  = "17"
    port     = "5432"
    password = var.db_password
  }
  task "run-tests" {
    run "docker" {
      image = "golang:1.25"
      cmd   = "cd app && make integration-test"
      args  = ["-v", "/cache/go:/root/go/pkg/mod"]
    }
  }
}

A git resource watches for changes, a Vault secret feeds the Postgres password, Postgres starts as a service, and the task runs inside Docker. All four abstractions, one pipeline.

Running pipelines locally

pikoci run --pipeline-config pipeline.hcl --job test

Any job, on your laptop, no server required. Override resources with local paths, inject secrets via --var. The same pipeline that runs in CI runs on your laptop.

The queue decision

Workers don't connect directly to the server, they subscribe to a queue. This means workers can be behind NAT, on a laptop, in a different data center, or completely ephemeral. The server never needs to know where workers are. It made distributed workers trivially easy to add. Start a worker anywhere with network access to the queue and it just works.

Where it is now

PikoCI deploys itself. The pipeline runs PR checks, mock tests, and integration tests against six different database and queue backends: MariaDB, PostgreSQL, NATS, RabbitMQ, Kafka, and Vault, all running as services. Then it builds multi-arch Docker images and redeploys itself with zero downtime.

Those six backends are not just targets, they are the pluggable abstractions themselves. The same PikoCI binary connects to any of them depending on how you start it. Testing against all of them is how I make sure the abstractions actually hold.

All of it is publicly visible at ci.pikoci.com/teams/main/pipelines/pikoci. No account needed.

It's Apache 2.0, written in Go, and very much something I use for my own projects every day.

If you try it and something is broken, I want to know. If something is missing, I also want to know.

pikoci.com · github.com/pikoci/pikoci