DEV Community

Santosh Pawar
Santosh Pawar

Posted on • Edited on

Comparing different options for GitHub self-hosted runners

We are in the phase of transitioning from Bitbucket to GitHub Enterprise.

Considering larger GitHub ecosystem, it also makes sense to move the CI/CD governance processes to GitHub instead of using a completely different solution. So we decided to migrate our CI/CD pipelines to GitHub Actions from current Jenkins/CloudBees based solution.
As part of this journey (which we are currently in), we had to work on various aspects of this migration.

  1. Migration of Bitbucket repositories to GitHub. This involves migrating source code, LFS, commit history, repository/branch permissions, webhooks, PR approvers etc.
  2. Designing the GitHub Actions based solution that maps from the existing pipelines defined in Jenkins/CloudBees. This involves considering existing shared libraries, utility/helper pipelines, webhooks, RBAC model etc.
  3. Build execution environment. In Jenkins, we use VMs as nodes to run the pipelines. In CloudBees, we use dynamic pods using EKS cluster. We need to identify the right build execution environment for GitHub Actions.

The below table talks more about #3.

GitHub-hosted Larger Runners GitHub-hosted Runners Containers on GitHub-hosted Runners Self-hosted Runners using VMs and servers Self-hosted runners using Kubernetes ARC (Actions Runner Controller) operator AWS CodeBuild Azure Container App job
Details VMs hosted by GitHub Large VMs hosted by GitHub, GPU and ARM support Your specified containers that run on VMs hosted by GitHub VMs or servers managed by you Uses Kubernetes cluster managed by you Uses AWS CodeBuild projects in your account Uses Azure Container App environment in your subscription
Supported Operating Systems Linux, Windows, macOS Linux, Windows, macOS Any OS the container has Linux, Windows, macOS Linux, Windows CodeBuild defaults - Amazon Linux, Ubuntu, Windows 2019. You can use your own image with any OS You can use your own image with any OS
Authentication Managed by GitHub Managed by GitHub Managed by GitHub You manage Kubernetes authentication GitHub App PAT GitHub App PAT
Preinstalled Software Most of the packages preinstalled Most of the packages preinstalled You install required software in Docker images You install required software You install required software in Docker images You install required software in buildspec.yml OR use a custom image with preinstalled software use a custom image with preinstalled software
Maintenance GitHub maintains the VM images GitHub maintains the VM images You maintain the Docker images You maintain VM images You maintain the Docker images You maintain the CodeBuild project and any custom Docker images You maintain the Container App environment and any custom Docker images
Security New VM that acts as a runner is created each time and deleted when job is finished, so less security risk. New VM that acts as a runner is created each time and deleted when job is finished, so less security risk. New VM is created each time for container that acts as a runner, and deleted when job is finished, so less security risk. You
maintain the security. Do not allow for public repos to avoid running untrusted workflows New pod that acts as a runner is created each time, so less security risk. However, you manage the security of Kubernetes cluster. New CodeBuild project execution (container) that acts as a runner created each time, so less security risk. However, you manage the security of CodeBuild project. New Container App job execution (container) that acts as a runner created each time, so less security risk. However, you manage the security of Container App
Scaling Autoscaling to support concurrent workflows Autoscaling to support concurrent workflows Autoscaling to support concurrent workflows You manage the scaling and make sure VMs/Servers have enough resources You configure EKS cluster HPA and autoscaling to support concurrent workflows CodeBuild autoscaling is used. Container App autoscaling is used
Pricing
    $
  • Free (public repos)
  • Limited Free (private repos)
  • Usage-based pricing
    $$
  • Free (public repos)
  • Limited Free (private repos)
  • Usage-based pricing
    $
  • Free (public repos)
  • Limited Free (private repos)
  • Usage-based pricing
 $$
  • Varies based on your infrastructure, but on-prem costs usually are more than using managed services
 $$$
  • Hourly pricing
    $
  • Usage-based pricing
    $
  • Usage-based pricing
Cost (1 hr on Linux) Linux 2-core x64-powerd Ubuntu $0.0008 * 60 = $0.48 Linux Advanced 2-core x64-powerd Ubuntu $0.0008 * 60 = $0.48
=========
Linux Advanced 64-core x64-powerd Ubuntu $0.256 * 60 = $15.36 		Linux 2-core x64-powerd Ubuntu $0.0008 * 60 = $0.48 EKS cluster 3 t3.medium worker nodes with 1TB EBS volumes $0.10 [cluster] + ($0.0416 * 3) [worker nodes] + $0.22 [EBS and snapshots] $0.45
Note: EKS charges will be continues, not just for the workflow executions		 CodeBuild project with Ubuntu image 3 GB memory, 2 vCPUs $0.30 Container App job with 2 vCPUs and 3GB memory with pay-as-you-go plan
$0.21

Top comments (0)