I maintain CI Bench, an open benchmark of GitHub Actions runners.
Disclosure first: I'm the founder of Vitko, which sells one of the products it measures (Vitko Runners). That's why I made every workflow, raw job record and line of method public, so you can check the numbers or re-run them.
In my previous post I described how the benchmark works. This one is the results.
What we measured
Nine runner configurations, twelve real CI jobs from open-source projects (fd, flask, guava, pydantic, vite, vue, a monorepo and others), measured October 7 to 11, 2026:
- GitHub-hosted:
ubuntu-latest(2 vCPU) and a 4 vCPU larger runner - Managed services: Blacksmith, Ubicloud, Vitko Runners (and Vitko with warm starts and test splitting)
- Self-hosted: RunsOn, plus two setups we ran ourselves on Google Cloud and on AWS (one VM per job)
Ranked by speed
Speed against ubuntu-latest, as the geometric mean of median job time over the 12 workloads:
| Rank | Configuration | Speed vs ubuntu-latest |
|---|---|---|
| 1 | Vitko Runners (warm starts + test splitting) | 2.3× faster |
| 2 | Vitko Runners | 1.7× faster |
| 3 | GitHub-hosted larger runner (4 vCPU) | 1.6× faster |
| 4 | Blacksmith | 1.6× faster |
| 5 | Self-hosted on Google Cloud | 1.5× faster |
| 6 | RunsOn | 1.2× faster |
| 7 | Self-hosted on AWS (one VM per job) | 1.2× faster |
| 8 | Ubicloud | 1.1× faster |
| 9 | GitHub-hosted ubuntu-latest
|
baseline |
The ranking hides a lot: Blacksmith was fastest on the burst workload, and the self-hosted Google Cloud setup tied for fastest on flask. The per-workload table is on the speed ranking page.
Ranked by cost per job
Price per minute isn't cost per job, because a faster runner finishes sooner. At 100,000 jobs a month, using list prices dated October 5:
| Rank | Configuration | Cost per job vs ubuntu-latest |
|---|---|---|
| 1 | RunsOn (self-hosted, priced all-in) | 88% lower |
| 2 | Vitko Runners | 84% lower |
| 3 | Vitko Runners (warm starts + test splitting) | 84% lower |
| 4 | Ubicloud | 83% lower |
| 5 | Self-hosted on AWS | 62% lower |
| 6 | Self-hosted on Google Cloud | 59% lower |
| 7 | Blacksmith | 52% lower |
| 8 | GitHub-hosted ubuntu-latest
|
baseline |
| 9 | GitHub-hosted larger runner (4 vCPU) | 39% higher |
Self-hosted costs depend on volume: RunsOn is 88% lower at 100,000 jobs a month, but 43% higher at 1,000, because fixed costs spread over fewer jobs. Details are on the cost ranking page.
What this doesn't tell you
Twelve workloads aren't your workload. Intervals cover sampling noise, not your repo. We didn't measure every provider (Depot, Namespace, WarpBuild and others aren't in v1), and prices change, so every price on the list prices page carries its source and the date it was read. The full method lists what's out of scope.
If you run one of these services and think a number is wrong, tell us and we'll fix it, or submit your own configuration for the next result set. The data is CC BY 4.0.
Top comments (0)