CI Bench v1: an open benchmark of GitHub Actions runners

Joseph Vitko — Sat, 24 Oct 2026 02:10:13 +0000

CI job times and costs are an increasingly large bottleneck for coding agents. Today we're releasing CI Bench v1, our open benchmark of Github Actions runners on 12 CI workloads for popular open-source projects.

The method, workflows, analysis and every raw job record are public in our GitHub repo, measured October 7 to 11, 2026.

Among the nine configurations measured, Vitko Runners with warm starts and test parallelization was the fastest and had the lowest managed cost per job; RunsOn on spot instances costs less per job at very high volume, while taking about twice as long.

Full disclosure: we sell Vitko Runners, now in private beta.

DEV Community: Joseph Vitko

CI Bench v1: an open benchmark of GitHub Actions runners