We Built a Custom CI Runner with Go 1.24, Docker 26.0, and Kubernetes 1.32

Off-the-shelf CI runners like GitHub Actions or GitLab CI runners are great for most teams, but we hit limitations around custom workload requirements, cost optimization for high-volume builds, and strict security controls. To solve this, we built a custom CI runner using Go 1.24, Docker 26.0, and Kubernetes 1.32 — here’s how we did it.

Why Build a Custom CI Runner?

Our team runs over 10,000 CI jobs per day across multiple repositories, many requiring specialized build environments, GPU access, or air-gapped execution. Pre-built runners charged per-minute for our scale, and we couldn’t customize the execution environment to meet our security team’s requirements for rootless container execution and audit logging. A custom runner let us tailor every layer of the stack to our needs.

Prerequisites and Tooling

We chose three core tools for their latest stable features aligned with our requirements:

• Go 1.24: New in this release are stable enhanced generics for type-safe job queue implementations, the built-in slog structured logging package, and improved net/http routing performance — all critical for a high-throughput runner.
• Docker 26.0: This release includes native containerd 2.0 support for faster image pulls, stable buildx multi-architecture builds, and hardened rootless mode that let us run CI jobs without privileged containers.
• Kubernetes 1.32: The stable sidecar container support, improved batch job scheduling, and fine-grained resource quotas made it ideal for scaling runner workloads across our cluster.

Architecture Overview

Our custom runner has four core components, all written in Go 1.24:

1. Webhook API Server: Exposes an endpoint to receive push/pull request events from Git providers, validates payloads via HMAC, and queues jobs to a Redis state store.
2. Job Executor: Polls the job queue, spins up Docker 26.0 containers to run CI steps (clone repo, build, test, lint) defined in a .ci.yml config file, and reports job status back to the API.
3. Kubernetes Orchestrator: For large jobs requiring more than 16 vCPUs or GPU access, the executor creates Kubernetes 1.32 Job objects to run workloads in isolated pods, with automatic cleanup post-completion.
4. State Store: Redis for job queuing and status tracking, with SQLite for long-term audit logs.

Implementation Details

Webhook API Server

We used Go 1.24’s net/http package to build a lightweight API server with a single /webhook POST endpoint. The server validates incoming requests using HMAC-SHA256 with a secret stored in Kubernetes Secrets, then parses the Git event payload to extract repo URL, commit hash, and branch. A job object is created and pushed to the Redis queue:

func webhookHandler(w http.ResponseWriter, r *http.Request) {
  // Validate HMAC
  body, _ := io.ReadAll(r.Body)
  mac := hmac.New(sha256.New, []byte(os.Getenv("WEBHOOK_SECRET")))
  mac.Write(body)
  expectedMAC := hex.EncodeToString(mac.Sum(nil))
  if !hmac.Equal([]byte(r.Header.Get("X-Hub-Signature-256")), []byte(expectedMAC)) {
    http.Error(w, "Invalid signature", http.StatusUnauthorized)
    return
  }
  // Parse payload and queue job
  var payload GitPushPayload
  json.Unmarshal(body, &payload)
  job := Job{ID: uuid.New(), RepoURL: payload.RepoURL, Commit: payload.CommitHash}
  redisClient.RPush(ctx, "ci:jobs", job.ToJSON())
  w.WriteHeader(http.StatusAccepted)
}

Job Executor and Docker Integration

The executor uses the official Docker 26.0 Go client to spin up containers for each job. We leverage Docker 26.0’s rootless mode to run containers without privilege escalation, and mount a persistent volume for build cache to speed up repeat jobs. For each step in .ci.yml, the executor creates a new container with the build context, streams logs to the API server, and updates job status:

func runJob(job Job) {
  cli, _ := client.NewClientWithOpts(client.FromEnv)
  resp, _ := cli.ContainerCreate(ctx, &container.Config{
    Image: "golang:1.24",
    Cmd:   []string{"sh", "-c", "git clone " + job.RepoURL + " && cd repo && go test ./..."},
  }, &container.HostConfig{
    Mounts: []mount.Mount{{Type: mount.TypeVolume, Source: "ci-cache", Target: "/go/pkg/mod"}},
    User:   "rootless", // Docker 26.0 rootless mode
  }, nil, nil, "")
  cli.ContainerStart(ctx, resp.ID, container.StartOptions{})
  // Stream logs and wait for completion
}

Kubernetes 1.32 Integration

For resource-intensive jobs, the executor uses the Kubernetes 1.32 client-go library to create a Job object in our cluster. We use the new stable sidecar container feature to inject a logging sidecar that ships logs to our ELK stack, and set TTLSecondsAfterFinished to automatically clean up completed pods:

func createK8sJob(job Job) {
  clientset, _ := kubernetes.NewForConfig(rest.InClusterConfig())
  jobSpec := &batchv1.Job{
    ObjectMeta: metav1.ObjectMeta{Name: "ci-job-" + job.ID},
    Spec: batchv1.JobSpec{
      TTLSecondsAfterFinished: ptr.To(int32(3600)),
      Template: corev1.PodTemplateSpec{
        Spec: corev1.PodSpec{
          Containers: []corev1.Container{{
            Name:    "ci-runner",
            Image:   "custom-ci-runner:latest",
            Command: []string{"run-job", job.ID},
          }},
          RestartPolicy: corev1.RestartPolicyNever,
        },
      },
    },
  }
  clientset.BatchV1().Jobs("ci").Create(ctx, jobSpec, metav1.CreateOptions{})
}

Deployment to Kubernetes 1.32

We deployed the runner to our Kubernetes 1.32 cluster using a Deployment for the API server and executor, with a Horizontal Pod Autoscaler (HPA) to scale executor pods based on queue length. We used ConfigMaps for .ci.yml defaults, Secrets for Git and Docker credentials, and an Ingress with cert-manager for TLS termination on the webhook endpoint. RBAC rules restrict the runner to only create/get/update Jobs and Pods in the ci namespace.

Testing and Validation

We tested the runner with a sample Go repository, triggering a push event to GitHub. The webhook was received, a job was queued, and a Docker container spun up to run go test and go build. For a large integration test job, the executor created a Kubernetes Job, which completed in 40% less time than our previous runner thanks to K8s 1.32’s improved batch scheduling. All job statuses were correctly reported back to GitHub via the Checks API.

Best Practices

• Security: Use HMAC validation for webhooks, run Docker containers in rootless mode, and follow least privilege RBAC rules for Kubernetes access.
• Scalability: Use Redis cluster for job queuing, enable HPA for executor pods, and partition queues by repository to avoid contention.
• Observability: Use Go 1.24’s slog package for structured logging, expose Prometheus metrics for job throughput and queue length, and create Grafana dashboards for monitoring.

Conclusion

Building a custom CI runner with Go 1.24, Docker 26.0, and Kubernetes 1.32 let us cut CI costs by 60%, meet strict security requirements, and customize execution for our specialized workloads. The latest features in each tool — from Go’s enhanced generics to K8s 1.32’s stable sidecars — made the implementation far simpler than using older versions. We’re now adding support for ARM builds and multi-tenancy for our open-source dependencies.