DEV Community: Francesc Gil

Why PikoCI uses HCL instead of YAML

Francesc Gil — Tue, 21 Jul 2026 12:31:54 +0000

Why PikoCI uses HCL instead of YAML

Almost every CI/CD tool uses YAML. GitHub Actions, GitLab CI, Concourse, Drone, Woodpecker, all YAML. Before I wrote a single line of PikoCI's code, the first decision I made was that it would not be YAML. I spent time writing pipelines in HCL just to see how they would look compared to the YAML ones I was writing in Concourse. They looked better. That was enough.

What's wrong with YAML

YAML's problems are well documented but worth stating plainly for CI/CD specifically.

No types. Everything is a string unless you're careful. true is a boolean. "true" is a string. yes is also a boolean. no is a boolean too. 0755 is an integer (octal). These are actual footguns that show up in pipeline config regularly.

Indentation is syntax. One extra space and your pipeline silently does something different. No error, just wrong behavior. In a format that humans edit frequently under time pressure, whitespace-sensitive syntax is a constant source of mistakes.

No expressions. If you want to construct a string from two variables in YAML you need template syntax bolted on top, like ${{ }} in GitHub Actions, or just shell interpolation inside a string value. The language itself has no way to express computation.

Anchors and aliases. YAML has a reuse mechanism but it's awkward (&anchor and *alias) and most CI tools don't fully support it anyway. You end up copying and pasting configuration.

Why HCL

HCL is HashiCorp Configuration Language. If you've used Terraform you already know it. It was designed specifically for configuration files that humans write and machines parse, exactly the use case for CI/CD pipeline definitions.

Real types. Strings, numbers, booleans, lists, maps, all distinct. true is always a boolean. "true" is always a string. No ambiguity.

Expressions built in. String interpolation with ${}, function calls like upper(), join(), format(), all part of the language without bolting on a template layer.

Variables with types and defaults. Declare what a pipeline accepts, what type each parameter is, and what the default is. The parser validates on load, so you get an error immediately, not at runtime.

variable "go_image" {
  type    = string
  default = "golang:1.24"
}

Job generation from expressions. for_each and matrix generate job instances from sets or maps, no template engine, no copy-paste:

job "validate" {
  for_each = toset(["lint", "vet", "test"])

  get "git" "app" { trigger = true }
  task "run" {
    run "docker" {
      image = var.go_image
      cmd   = "make ${each.value}"
    }
  }
}

This creates three jobs (validate--lint, validate--vet, validate--test) each running its respective make target. In YAML you'd copy the job block three times.

Readable without a reference. HCL reads more like a configuration file and less like a serialization format. Block structure maps naturally to the concepts it represents: a job is a block, a task is a block inside it, a run is a block inside that.

Side by side

The same pipeline (get from git, run tests, deploy on success) in YAML (GitHub Actions style) and HCL (PikoCI).

YAML:

on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run tests
        run: go test ./...

  deploy:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Deploy
        run: ./deploy.sh
        env:
          ENV: production
          VERSION: ${{ github.sha }}

HCL (PikoCI):

resource "git" "app" {
  params {
    url    = "https://github.com/myorg/myapp"
    branch = "main"
  }
}

job "test" {
  get "git" "app" { trigger = true }

  task "run-tests" {
    run "exec" {
      path = "go"
      args = ["test", "./..."]
    }
  }
}

job "deploy" {
  get "git" "app" {
    trigger = true
    passed  = ["test"]
  }

  task "deploy" {
    run "exec" {
      path = "./deploy.sh"
    }
  }
}

A few things stand out in the comparison.

The HCL version makes the dependency between test and deploy explicit via passed = ["test"], not just a needs: list but an actual resource version constraint. The deploy job only runs when the same git version that passed the test job is available. This is a fundamentally different model: jobs don't just "wait for each other", they share versioned resources.

The resource is declared once and referenced by both jobs. In YAML each job does its own checkout, there's no concept of a shared resource with versioning.

The HCL version is more verbose upfront, you declare the resource explicitly instead of relying on implicit checkout. That explicitness pays off when you have 10 jobs sharing the same resource, or when you need to pin a version across the whole pipeline.

More declarative, easier to maintain

HCL is declarative in a way that YAML isn't. In YAML-based CI/CD you describe what commands to run and in what order, it's closer to a script. In HCL you describe what things are and how they relate to each other. The jobs, resources, and dependencies are named, typed, and structured entities, not a sequence of shell commands glued together with indentation.

This makes a real difference as pipelines grow. A 10-job pipeline in YAML is already hard to navigate. A 10-job pipeline in HCL reads like a map, you can see the structure at a glance, jump to any job by name, and understand how things connect without reading the whole file top to bottom.

Variables, resource declarations, and job definitions all live in clearly separated blocks. When something breaks you know where to look. When you want to change the Go image across all jobs you change one variable, not ten image: lines scattered through the file.

Tooling

Two things PikoCI ships that make working with HCL pipelines smoother:

pikoci pipeline validate validates a pipeline file without a running server. Catches syntax errors, type mismatches, invalid references, and unknown fields. Good for pre-commit hooks or CI checks on the pipeline config itself.

pikoci pipeline validate pipeline.hcl
pikoci pipeline validate pipeline.hcl --var environment=production

pikoci pipeline edit opens a browser-based editor for a local HCL file. The editor validates as you type and shows errors inline. Edit, validate, save, no round-trip to git required for quick fixes. You can also edit pipelines directly from the PikoCI web UI.

pikoci pipeline edit pipeline.hcl

The honest tradeoff

HCL has one real disadvantage: it's less universally known than YAML. Every developer knows YAML. HCL is familiar to Terraform users but not everyone has used Terraform.

In practice this hasn't been a problem, HCL is readable enough that most engineers understand a PikoCI pipeline without having seen HCL before. And the Terraform ecosystem is large enough that "if you know Terraform you already know this" covers a significant part of the DevOps audience.

github.com/pikoci/pikoci · pikoci.com · Apache 2.0

Why I replaced queue-based worker communication with gRPC

Francesc Gil — Thu, 16 Jul 2026 13:12:52 +0000

Replacing Queues with gRPC in PikoCI

When I started building PikoCI, I kept coming back to a simplification: at its core, a CI/CD system is just jobs and workers waiting to be told to do them. Strip away everything else and what you have is a notification problem: the server knows a job is ready, a worker needs to find out.

That's a textbook pub/sub pattern. So I built PikoCI on queues from the start.

Why queues were the right call

I used go-cloud's pubsub abstraction, which let me support multiple backends (NATS, Kafka, RabbitMQ, or an in-memory queue for local development) without rewriting the worker logic for each one. People could plug in whatever they already run, or use nothing at all for local setups.

Queues also solve worker registration cleanly. A worker just subscribes to a topic. It doesn't need to be reachable by the server, doesn't need a stable address, doesn't need anything beyond outbound access to the queue. Adding a worker anywhere is trivial.

But there was a recurring question I kept arguing with myself about: was the queue actually buying me anything over a direct connection? Running a queue is one more thing to deploy, one more thing to operate, for self-hosters who just want a binary. I kept using it anyway because few CI/CD tools use this pattern and I wanted to see it through.

Where it started breaking down

The cracks showed up gradually, mostly while implementing features that needed more from the queue than "deliver this message."

Ordering wasn't guaranteed. go-cloud's in-memory pubsub doesn't guarantee FIFO delivery. I found this out while building worker tagging, when jobs started arriving out of order in ways that broke the scheduler's assumptions.

Retrying was painful. If a worker needed to re-queue a job (say, after a transient failure) there was no guarantee it would come back to the front of the line. It could end up anywhere, with no way to reason about when it would actually run.

Tagging didn't map across backends. Worker tagging needed dynamic routing: a worker with tags like gpu and linux should only receive matching jobs. NATS does this naturally with subject-based routing. Kafka and SQS don't, not without real workarounds.

Cancellation was slow. Workers had no way to learn that a build had been cancelled other than polling the server. That meant a cancelled build could keep running for seconds before the worker noticed, wasteful and visibly wrong in the UI.

Log streaming was a separate problem entirely. Build logs needed their own mechanism to get from worker to server, completely unrelated to the job dispatch queue. Two communication paths to build and maintain for what is fundamentally one relationship: server talks to worker.

Each of these pushed me toward solving the problem outside the queue rather than with it. The message stopped carrying the job payload entirely and became just a ping, a doorbell. The worker would get notified, then immediately turn around and ask the server over HTTP: "what do I need to do?" The server's scheduler held all the actual logic about what was next, regardless of which queue backend was configured underneath.

Considering NATS-only

Before dropping queues entirely, I seriously considered the opposite move: drop every backend except NATS and the in-memory option. NATS could actually do what I needed. Subject-based routing handled tagging naturally, and it's about as lightweight as a queue gets, which fit PikoCI's "easy to deploy" goal better than Kafka or RabbitMQ ever would.

But getting there required NATS-specific configuration: JetStream settings, subject hierarchies tuned for tagging, ordering guarantees that the generic abstraction didn't give me for free. That's the part I didn't like. It stopped being "bring whatever queue you already run" and became "run NATS, configured exactly this way, or tagging won't work." That's not a pluggable abstraction anymore, that's a hard dependency wearing an abstraction's clothes.

And even with NATS configured perfectly, I'd still need the worker to ask the scheduler what to do next, because the scheduler is the only thing that actually knows the answer. The queue would just be there to make the connection happen. At that point I was about to add an entire piece of required infrastructure whose only job was to say "hey, ping."

The moment it clicked

By the time I was deep into the tagging work, the pattern was obvious: the worker was already asking the scheduler directly for its next job. The queue's only remaining job was to say "hey, something happened, go ask." That's not nothing, but it's also not worth running a separate piece of infrastructure for.

If the server already knows which workers exist, it can just tell them directly. No doorbell needed.

Why gRPC

gRPC gives me a persistent bidirectional stream between server and worker. The worker opens one connection and keeps it open. Over that single connection:

The server pushes jobs to the worker the moment they're ready. No asking, no doorbell
The server pushes a cancellation message instantly when a build is cancelled, milliseconds instead of the multi-second polling interval cancellation needed before
The worker streams build logs back in real time
Heartbeats flow both ways, so the server always knows which workers are alive

One connection, one mental model, no external infrastructure to run or reason about. Workers still only need outbound network access; that part of the queue model's appeal didn't have to be sacrificed.

What this replaced

Before:
  go-cloud pubsub (NATS / Kafka / RabbitMQ / in-memory)
  --pubsub-system flag
  Queue message as a doorbell, worker asks server what's next
  Worker polls GET /api/builds/:id/status for cancellation
  Separate mechanism for log streaming

After:
  gRPC bidirectional stream on a dedicated port
  No --pubsub-system flag, removed entirely
  Server pushes jobs directly on the open stream
  Server pushes CancelJob message on the same stream
  Logs stream on the same connection

The architecture today

Two ports. Port 8080 serves the HTTP JSON API: everything the frontend and CLI talk to, unchanged. Port 9090 serves gRPC, workers only. The two transports never mix.

Multi-server deployments still use Postgres LISTEN/NOTIFY to coordinate which server instance dispatches a job; that part didn't change. What changed is how the chosen server instance gets the job to the worker: a direct push on an open stream instead of a queue message.

The migration was a clean break. v0.5.0 dropped the --pubsub-system flag entirely. Workers needed to update to the matching version, but since PikoCI ships as a single binary that's both server and worker, upgrading one means upgrading the other. No transition period with both protocols running in parallel.

The result

Removed an entire category of optional infrastructure. Self-hosted PikoCI users no longer need to think about which queue backend to run, or whether to run one at all. Cancellation now takes effect in milliseconds instead of seconds. Log streaming and job dispatch share one connection instead of two separate mechanisms.

Shipped in v0.5.0.

github.com/pikoci/pikoci · pikoci.com · Apache 2.0

PikoCI v0.7.0: Approval gates, audit log, OIDC, secrets masking, and build reports

Francesc Gil — Wed, 08 Jul 2026 18:37:08 +0000

PikoCI is an open-source CI/CD server written in Go. A single binary, no dependencies, pipeline-as-code in HCL. v0.7.0 is the security and compliance release. A lot shipped in this one.

Approval gates

Jobs can now require human approval before continuing. Add an approve block to any job and the pipeline pauses with a purple "Waiting for Approval" status until someone with Maintain access approves or rejects it.

job "deploy" {
  get "git" "main" { passed = ["test"] }

  approve "deploy to production" {
    approvals = 2
    notify "discord" "deploy-alerts" {}
  }

  task "deploy" {
    run "exec" { path = "./deploy.sh" }
  }
}

Approve or reject via the UI, CLI (pikoci builds approve), or API. Each approval is recorded with who approved and when. If you need two people to sign off before a production deploy, set approvals = 2.

Audit log

Every significant action on a team now has a permanent append-only record: pipeline changes, manual triggers, approvals, secret modifications, membership changes. 16 event types in total, filterable by user, event type, and date range. Available via the UI, CLI, and API. Read access and above can view it.

OIDC and OAuth2

Single sign-on with GitHub, Google, GitLab, Keycloak, Microsoft, or any OIDC provider. Configure providers from the admin UI with built-in templates, link existing accounts, and optionally disable local username/password auth entirely.

Secrets masking in build logs

Secret values are now automatically replaced with *** in build log output, both in real-time streaming and stored logs. If a task accidentally echoes a secret, it never appears in the UI. Values shorter than 3 characters are skipped to avoid masking noise.

Build report export

Export a complete JSON report for any build containing the full status, each step's result, logs, approval records, and resource version data. Available from the UI (Export button on the build detail page), CLI (pikoci builds report my-pipeline deploy 42), and API.

Useful for compliance, incident review, or just keeping a record of what deployed and when.

Secret chaining

Secret path and key fields can now reference other variables, resolved in dependency order with cycle detection. This means you can build secrets that depend on other secrets. Useful for patterns like using a vault token to fetch another secret's path.

Worker team isolation

Team-scoped worker tokens restrict workers to only run builds for their team. Global workers automatically defer to team workers when available, so teams with dedicated infrastructure get their own capacity without affecting others.

Role rename

The role names have been updated to match GitHub's model: Viewer→Read, Operator→Write, Maintainer→Maintain. The database migrates automatically on upgrade.

Session security

Two new session security improvements: configurable session expiration via --session-lifetime (e.g. 24h, 7d, 30d), and automatic session invalidation when a password is changed. All existing sessions for that user are revoked immediately. Passwords now also enforce a minimum 8-character length.

Performance

N+1 query elimination across pipeline, job list, and resource list endpoints. Build polling now fetches all active builds in a single request instead of one HTTP call per build every 2 seconds. Modulepreload hints collapse 4 browser discovery rounds to 1 parallel fetch.

Release notes · pikoci.com · github.com/pikoci/pikoci · Apache 2.0

PikoCI v0.6: New pipeline views, version tracking, and goodbye Backbone

Francesc Gil — Thu, 25 Jun 2026 15:27:23 +0000

PikoCI is a self-hosted CI/CD system, one binary, any database, runs anywhere. v0.6 is the biggest UI release yet: a complete frontend rewrite and new ways to explore pipeline state.

Goodbye Backbone, hello Preact

The frontend has been migrated from BackboneJS + jQuery + Underscore to Preact + HTM. No build step, no bundler, no npm. Same philosophy as the rest of PikoCI, just a binary. The virtual DOM eliminates the flickering that came from Backbone's full re-renders.

New pipeline views

The pipeline page now has two view modes: Graph and List, switchable via a toolbar.

Graph view is the existing DAG view with two new toggles via a gear icon:

Hide intermediate resources removes resource nodes between jobs and draws direct job-to-job edges. Trigger resources stay visible. Useful for complex pipelines where the resource nodes add visual noise.

Group parallel jobs collapses jobs sharing identical upstream parents into a single compact node, showing a colored status dot per job. One click to expand and see each job individually.

List view shows jobs in a left panel and build detail on the right. Jobs are displayed in a tree reflecting the pipeline DAG. Parallel siblings are grouped under collapsible headers. Clicking a job loads its build history on the right. A resource selector at the top lets you switch between trigger resources. View preference, selected resource, and collapsed groups all persist across reloads.

Resources panel

A new Resources button on the pipeline toolbar opens a slide-out panel showing all pipeline resources: name, type, latest version, build status, last check time, and a Check Now button. Available from both views. Version list is expandable per resource with Track, Trigger, and Pin actions.

Version tracking

Track a specific resource version through the pipeline to see which jobs it passed through and their build status.

Available from the resource versions page, the Resources panel, and a version dropdown in the list view. When tracking, both graph and list views scope to only that version's path. A banner shows progress.

Resource version status dots

Resource version rows now show a colored status dot indicating the aggregate build status of all jobs that consumed that version. No more clicking into each build to know if a version is good.

Share and embed

A new share panel on the pipeline toolbar provides copyable URLs for SVG and PNG pipeline images, plus a Markdown snippet for embedding pipeline badges in READMEs. Public pipelines can be embedded without authentication.

Other improvements

Deterministic retries: Retrying a build now reuses the resource versions from the original build instead of fetching latest. This means retries reproduce the same build conditions, making failures easier to debug.

Correct parallel step durations: in_parallel sub-step durations now display correctly instead of raw nanoseconds.

Smart polling: Polling pauses when the browser tab is hidden, reducing wasted network requests by around 50%.

v0.6.1 and v0.6.2 included minor fixes for graceful shutdown timeouts and build recovery after server restarts. See the release notes for details.

Release notes · pikoci.com · github.com/pikoci/pikoci · Apache 2.0

PikoCI v0.5.0: worker tagging, in_parallel, Linux packages, and more

Francesc Gil — Tue, 16 Jun 2026 12:14:39 +0000

Six months into building PikoCI and v0.5.0 is the biggest release yet. This one is almost entirely about workers: routing work to the right place, running things in parallel, and knowing what your workers are doing.

Worker tagging

You can now route jobs and resource checks to specific workers using tags. Add tags = ["gpu"] to a job in your pipeline HCL, start a worker with --tags gpu, and that job will only run on workers with the matching tag. Matching uses AND logic: all job tags must be present on the worker.

job "train-model" {
  tags = ["gpu"]
  task "train" {
    run "exec" {
      path = "python"
      args = ["train.py"]
    }
  }
}

Workers can also be made exclusive. The --exclusive-tags flag restricts a worker to only handle tagged work, keeping it free for the jobs that need it.

Worker health monitoring

Workers now send periodic heartbeats to the server. A new admin dashboard shows all connected workers with status, platform, version, and uptime. If a worker goes stale (no heartbeat for 90 seconds), a warning appears and admins can remove it from the UI or via the API.

There is also a pikoci_workers Prometheus gauge with a status label if you want to alert on missing workers.

in_parallel step

Run multiple steps concurrently within a job. Useful for running tests against multiple configurations simultaneously, or for parallelizing independent tasks.

job "test" {
  in_parallel {
    limit     = 3
    fail_fast = true

    task "test-postgres" {
      run "exec" { path = "make" args = ["test-postgres"] }
    }
    task "test-mysql" {
      run "exec" { path = "make" args = ["test-mysql"] }
    }
    task "test-sqlite" {
      run "exec" { path = "make" args = ["test-sqlite"] }
    }
  }
}

limit caps how many run simultaneously. fail_fast cancels remaining steps on the first failure.

pipeline validate

Validate a pipeline HCL file without a running server.

pikoci pipeline validate pipeline.hcl
pikoci pipeline validate pipeline.hcl --var env=staging

Useful for pre-commit hooks and CI checks on the pipeline config itself. Catches syntax errors, structural errors, and invalid cross-references, all reported at once with line numbers.

Linux packages

PikoCI now ships as .deb and .rpm packages alongside the existing binaries. Both amd64 and arm64. A SHA256SUMS file is included in every release.

fs resource type

A built-in resource type for watching local files and directories. Triggers jobs when file contents change via SHA256 hash comparison.

resource_type "fs" {
  source = "pikoci://fs"
}

resource "fs" "config" {
  params {
    path = "/etc/myapp/config.yaml"
  }
}

Useful for local development workflows and watching config files.

Worker communication now uses gRPC

Workers connect to the server via a persistent gRPC stream instead of the previous queue-based system. Job dispatch is instant, cancellation signals reach the worker in milliseconds instead of waiting for a polling interval, and log streaming happens on the same connection. No external queue needed for distributed workers.

A dedicated post covering the full migration from queues to gRPC is coming soon.

Other fixes

New jobs no longer replay the entire version backlog on creation: they record a baseline at creation time and only see newer versions. Downstream jobs now trigger immediately when upstream builds succeed, instead of waiting for the next scheduler tick. Cross-reference validation catches invalid references at save time with suggestions for typos.

github.com/pikoci/pikoci · pikoci.com · Apache 2.0

v0.5.0 release notes

Services without Docker-in-Docker: how PikoCI handles test dependencies

Francesc Gil — Tue, 09 Jun 2026 08:13:24 +0000

If you've ever needed a database for your integration tests in CI, you've probably encountered one of these solutions:

Docker-in-Docker. A separate docker-compose file you run alongside CI. A pre-provisioned shared database that everyone fights over. Or just skipping integration tests in CI entirely.

None of these are great. Docker-in-Docker requires privileged containers and is notoriously fragile. docker-compose alongside CI is manual, error-prone, and hard to clean up. Shared databases cause flaky tests. Skipping integration tests defeats the purpose.

PikoCI takes a different approach: services as a first-class concept.

What services are

A service in PikoCI is an ephemeral process that runs alongside your job's tasks. Services start where they are defined in the job plan, if you define them first, they start first; if you define them after a get step, they start after the get. They always stop unconditionally when the job ends, regardless of where they were defined or whether tasks succeeded or failed.

You define a service_type once and reference it from any job:

service_type "postgres" {
  params = ["version"]

  start "exec" {
    path = "/bin/sh"
    args = ["-ec", <<-EOT
      NAME="pikoci-${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-pg"
      docker rm -f $NAME 2>/dev/null || true
      docker run -d --name $NAME -p 5432:5432 \
        -e POSTGRES_PASSWORD=test \
        postgres:$param_version
    EOT
    ]
  }

  ready_check "exec" {
    path     = "/bin/sh"
    args     = ["-ec", "docker exec pikoci-${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-pg pg_isready"]
    interval = "2s"
    timeout  = "30s"
  }

  stop "exec" {
    path = "/bin/sh"
    args = ["-ec", "docker rm -f pikoci-${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-pg 2>/dev/null || true"]
  }
}

job "integration" {
  get "git" "app" { trigger = true }

  service "postgres" {
    version = "16"
  }

  task "test" {
    run "exec" {
      path = "make"
      args = ["integration-test"]
    }
  }
}

PikoCI starts Postgres, waits for pg_isready to return 0, runs your tasks, then stops and removes the container. Your task connects to localhost:5432 like it would anywhere else.

The lifecycle

Services start in the order they appear in the job plan. You control placement:

Define a get step first if you need the code pulled before services start
Define services wherever makes sense for your job
After all steps complete, success or failure, stop runs for every started service

The stop step always runs. If your task panics, if the job is cancelled, if the worker gets a SIGTERM, stop still runs. No orphaned containers.

No Docker-in-Docker

Services don't run inside a container. They run on the worker host directly. Your tasks can run inside Docker containers via the docker runner, and they connect to services on localhost because everything is on the same host network.

This is the key difference from Docker-in-Docker. With DinD you're running a Docker daemon inside a container, which requires --privileged, has kernel-level implications, and is generally fragile. With PikoCI services, Docker is just a process manager, the worker runs docker run, the container starts on the host network, your tasks connect normally.

Orphan prevention

If a worker crashes mid-job, the stop block never runs and Docker containers keep running. The next job tries to start on the same port and fails.

The solution is stable container names with pre-start cleanup. Use $BUILD_PIPELINE_NAME and $BUILD_JOB_NAME, stable across runs, and always clean up at the start of the start block:

NAME="pikoci-${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-postgres"
docker rm -f $NAME 2>/dev/null || true   # kill orphan if exists
docker run -d --name $NAME ...           # start fresh

The || true means cleanup never fails the job.

Not just Docker

Services aren't Docker-specific. You can start any process: a local daemon, a background script, anything the worker can run.

service_type "redis" {
  start "exec" {
    path = "/bin/sh"
    args = ["-ec", "redis-server --daemonize yes --dir $WORKDIR"]
  }

  stop "exec" {
    path = "/bin/sh"
    args = ["-ec", "redis-cli shutdown"]
  }
}

No ready check needed, Redis starts fast enough that the job can proceed immediately after start completes.

Sourceable and reusable

Like all PikoCI abstractions, service_type definitions are sourceable from a URL. Write a service type once, host it in a git repo or anywhere with an HTTPS endpoint, and reference it from any pipeline:

service_type "postgres" {
  #source = "https://raw.githubusercontent.com/myorg/pikoci-services/main/postgres.hcl"
  source = "pikoci://postgresql"
}

The built-in types use the same mechanism, pikoci://postgres is just a URL that ships with the binary. Community-built service types work identically.

What if my worker runs in Docker?

If your worker runs as a Docker container, services that use docker run need the Docker socket mounted so the worker spawns sibling containers on the host instead of nested ones:

docker run -v /var/run/docker.sock:/var/run/docker.sock pikoci-worker

Alternatively, if your worker's Docker image already has the service preinstalled (Postgres, MySQL, Redis) you can start it as a local process directly, no socket needed. It's a less common setup but works cleanly for fixed environments.

The simplest approach is to run the worker as a plain process on the host. Then any service can use Docker freely, just the worker running containers, no nesting involved. That's how the PikoCI dogfooding pipeline runs its workers.

PikoCI uses this itself

The pipeline that tests PikoCI runs integration tests against six backends simultaneously using services: MariaDB, PostgreSQL, NATS, RabbitMQ, Kafka, and Vault. All six are defined in the job plan, all six stop when the job ends. The pipeline is publicly visible at ci.pikoci.com/teams/main/pipelines/pikoci, no account needed.

This is how integration tests should work in CI. Not Docker-in-Docker, not shared test databases, not skipped tests. Just services that start where you need them and stop when the job ends.

pikoci.com · github.com/pikoci/pikoci · docs.pikoci.com/Services

How hard can it be to build a CI/CD system?

Francesc Gil — Thu, 28 May 2026 22:31:06 +0000

How hard can it be to build a CI/CD system?

That question stuck with me long enough that I actually started building one. Not because someone asked me to. Not because I spotted a market gap. Just because the question wouldn't go away.

The trigger was Concourse CI. I've been using it for a while and what I love about it is the resource abstraction, an interface that anything external has to follow. Check for new versions, pull them, push back. As a Go developer this kind of clean interface resonates with me. Everything in the pipeline is just something that implements that contract.

But the operational overhead is significant. And I needed CI for my own side projects anyway, games and open source tools that require custom environments GitHub Actions can't provide. So I started building.

What I wanted

A single binary that could also scale horizontally when needed. Start with nothing, grow when you need to.

You start like this:

./pikoci server \
  --db-system mem \
  --pubsub-system mem \
  --run-worker \
  --pipeline-config pipeline.hcl

That's a complete CI/CD system. In memory, no files, no external services. When you want persistence, add --db-system sqlite. When you need distributed workers, add NATS and start workers on other machines. The pipeline config never changes.

The interesting parts

Four pluggable abstractions

PikoCI has four concepts you define in HCL and can source from a URL: resource types, runners, service types, and secret types. Each follows the same pattern: define the type once, instantiate it with params.

A resource_type defines how to watch something for changes and fetch it. A resource is an instance of it:

resource_type "git" {
  source = "pikoci://git"  # built-in
}

resource "git" "my-app" {
  params {
    url  = "https://github.com/org/app"
    name = "app"
  }
  check_interval = "@every 1m"
}

A runner_type defines where tasks execute. The docker and exec runners are built-in, no declaration needed. Here's what the docker runner looks like under the hood, in case you want to define your own:

# this is what pikoci://docker looks like
# define your own to customize or replace it
runner_type "docker" {
  run {
    path = "docker"
    args = [
      "run", "--rm",
      "-v", "$WORKDIR:/workdir",
      "-w", "/workdir",
      "$image",
      "/bin/sh", "-ec", "$cmd",
    ]
  }
}

job "test" {
  get "git" "my-app" { trigger = true }
  task "run-tests" {
    run "docker" {
      image = "golang:1.25"
      cmd   = "cd app && make test"
      args  = ["-v", "/cache/go:/root/go/pkg/mod"]
    }
  }
}

A secret_type defines where credentials come from. Secrets are bound to variables and referenced anywhere in the pipeline:

secret_type "vault" {
  source = "pikoci://vault"
}

variable "db_password" {
  secret "vault" {
    path = "secret/data/db"
    key  = "password"
  }
}

A service_type defines processes that run alongside your tasks, started before, stopped after, guaranteed regardless of outcome. This is the feature I'm most proud of:

service_type "postgres" {
  # source = "pikoci://postgres" — or define inline
  start "exec" {
    path = "/bin/sh"
    args = ["-ec", "docker run -d --name db -p 5432:5432 postgres:16"]
  }
  ready_check "exec" {
    path    = "/bin/sh"
    args    = ["-ec", "pg_isready -h localhost"]
    timeout = "30s"
  }
  stop "exec" {
    path = "/bin/sh"
    args = ["-ec", "docker rm -f db"]
  }
}

job "integration" {
  service "postgres" {}
  get "git" "my-app" { trigger = true }
  task "test" {
    run "exec" {
      path = "make"
      args = ["integration-test"]
    }
  }
}

No Docker-in-Docker. No docker-compose alongside CI. The service stops regardless of whether the job passed or failed.

All four types are sourceable from a URL. The built-ins use pikoci://, the same mechanism as anything you host yourself. If you need a runner that executes jobs in Kubernetes or Azure, write it once, host it anywhere, reference it by URL.

Putting it all together

Here's a small pipeline that uses all four abstractions at once:

resource_type "git" {
  source = "pikoci://git"
}

resource "git" "my-app" {
  params {
    url  = "https://github.com/org/app"
    name = "app"
  }
  check_interval = "@every 1m"
}

secret_type "vault" {
  source = "pikoci://vault"
}

variable "db_password" {
  secret "vault" {
    path = "secret/data/db"
    key  = "password"
  }
}

service_type "postgresql" {
  source = "pikoci://postgresql"
}

job "test" {
  get "git" "my-app" { trigger = true }
  service "postgresql" {
    version  = "17"
    port     = "5432"
    password = var.db_password
  }
  task "run-tests" {
    run "docker" {
      image = "golang:1.25"
      cmd   = "cd app && make integration-test"
      args  = ["-v", "/cache/go:/root/go/pkg/mod"]
    }
  }
}

A git resource watches for changes, a Vault secret feeds the Postgres password, Postgres starts as a service, and the task runs inside Docker. All four abstractions, one pipeline.

Running pipelines locally

pikoci run --pipeline-config pipeline.hcl --job test

Any job, on your laptop, no server required. Override resources with local paths, inject secrets via --var. The same pipeline that runs in CI runs on your laptop.

The queue decision

Workers don't connect directly to the server, they subscribe to a queue. This means workers can be behind NAT, on a laptop, in a different data center, or completely ephemeral. The server never needs to know where workers are. It made distributed workers trivially easy to add. Start a worker anywhere with network access to the queue and it just works.

Where it is now

PikoCI deploys itself. The pipeline runs PR checks, mock tests, and integration tests against six different database and queue backends: MariaDB, PostgreSQL, NATS, RabbitMQ, Kafka, and Vault, all running as services. Then it builds multi-arch Docker images and redeploys itself with zero downtime.

Those six backends are not just targets, they are the pluggable abstractions themselves. The same PikoCI binary connects to any of them depending on how you start it. Testing against all of them is how I make sure the abstractions actually hold.

All of it is publicly visible at ci.pikoci.com/teams/main/pipelines/pikoci. No account needed.

It's Apache 2.0, written in Go, and very much something I use for my own projects every day.

If you try it and something is broken, I want to know. If something is missing, I also want to know.

pikoci.com · github.com/pikoci/pikoci