If you've ever needed a database for your integration tests in CI, you've probably encountered one of these solutions:
Docker-in-Docker. A separate docker-compose file you run alongside CI. A pre-provisioned shared database that everyone fights over. Or just skipping integration tests in CI entirely.
None of these are great. Docker-in-Docker requires privileged containers and is notoriously fragile. docker-compose alongside CI is manual, error-prone, and hard to clean up. Shared databases cause flaky tests. Skipping integration tests defeats the purpose.
PikoCI takes a different approach: services as a first-class concept.
What services are
A service in PikoCI is an ephemeral process that runs alongside your job's tasks. Services start where they are defined in the job plan, if you define them first, they start first; if you define them after a get step, they start after the get. They always stop unconditionally when the job ends, regardless of where they were defined or whether tasks succeeded or failed.
You define a service_type once and reference it from any job:
service_type "postgres" {
params = ["version"]
start "exec" {
path = "/bin/sh"
args = ["-ec", <<-EOT
NAME="pikoci-${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-pg"
docker rm -f $NAME 2>/dev/null || true
docker run -d --name $NAME -p 5432:5432 \
-e POSTGRES_PASSWORD=test \
postgres:$param_version
EOT
]
}
ready_check "exec" {
path = "/bin/sh"
args = ["-ec", "docker exec pikoci-${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-pg pg_isready"]
interval = "2s"
timeout = "30s"
}
stop "exec" {
path = "/bin/sh"
args = ["-ec", "docker rm -f pikoci-${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-pg 2>/dev/null || true"]
}
}
job "integration" {
get "git" "app" { trigger = true }
service "postgres" {
version = "16"
}
task "test" {
run "exec" {
path = "make"
args = ["integration-test"]
}
}
}
PikoCI starts Postgres, waits for pg_isready to return 0, runs your tasks, then stops and removes the container. Your task connects to localhost:5432 like it would anywhere else.
The lifecycle
Services start in the order they appear in the job plan. You control placement:
- Define a
getstep first if you need the code pulled before services start - Define services wherever makes sense for your job
- After all steps complete, success or failure,
stopruns for every started service
The stop step always runs. If your task panics, if the job is cancelled, if the worker gets a SIGTERM, stop still runs. No orphaned containers.
No Docker-in-Docker
Services don't run inside a container. They run on the worker host directly. Your tasks can run inside Docker containers via the docker runner, and they connect to services on localhost because everything is on the same host network.
This is the key difference from Docker-in-Docker. With DinD you're running a Docker daemon inside a container, which requires --privileged, has kernel-level implications, and is generally fragile. With PikoCI services, Docker is just a process manager, the worker runs docker run, the container starts on the host network, your tasks connect normally.
Orphan prevention
If a worker crashes mid-job, the stop block never runs and Docker containers keep running. The next job tries to start on the same port and fails.
The solution is stable container names with pre-start cleanup. Use $BUILD_PIPELINE_NAME and $BUILD_JOB_NAME, stable across runs, and always clean up at the start of the start block:
NAME="pikoci-${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-postgres"
docker rm -f $NAME 2>/dev/null || true # kill orphan if exists
docker run -d --name $NAME ... # start fresh
The || true means cleanup never fails the job.
Not just Docker
Services aren't Docker-specific. You can start any process: a local daemon, a background script, anything the worker can run.
service_type "redis" {
start "exec" {
path = "/bin/sh"
args = ["-ec", "redis-server --daemonize yes --dir $WORKDIR"]
}
stop "exec" {
path = "/bin/sh"
args = ["-ec", "redis-cli shutdown"]
}
}
No ready check needed, Redis starts fast enough that the job can proceed immediately after start completes.
Sourceable and reusable
Like all PikoCI abstractions, service_type definitions are sourceable from a URL. Write a service type once, host it in a git repo or anywhere with an HTTPS endpoint, and reference it from any pipeline:
service_type "postgres" {
#source = "https://raw.githubusercontent.com/myorg/pikoci-services/main/postgres.hcl"
source = "pikoci://postgresql"
}
The built-in types use the same mechanism, pikoci://postgres is just a URL that ships with the binary. Community-built service types work identically.
What if my worker runs in Docker?
If your worker runs as a Docker container, services that use docker run need the Docker socket mounted so the worker spawns sibling containers on the host instead of nested ones:
docker run -v /var/run/docker.sock:/var/run/docker.sock pikoci-worker
Alternatively, if your worker's Docker image already has the service preinstalled (Postgres, MySQL, Redis) you can start it as a local process directly, no socket needed. It's a less common setup but works cleanly for fixed environments.
The simplest approach is to run the worker as a plain process on the host. Then any service can use Docker freely, just the worker running containers, no nesting involved. That's how the PikoCI dogfooding pipeline runs its workers.
PikoCI uses this itself
The pipeline that tests PikoCI runs integration tests against six backends simultaneously using services: MariaDB, PostgreSQL, NATS, RabbitMQ, Kafka, and Vault. All six are defined in the job plan, all six stop when the job ends. The pipeline is publicly visible at ci.pikoci.com/teams/main/pipelines/pikoci, no account needed.
This is how integration tests should work in CI. Not Docker-in-Docker, not shared test databases, not skipped tests. Just services that start where you need them and stop when the job ends.
pikoci.com · github.com/pikoci/pikoci · docs.pikoci.com/Services
Top comments (0)