Comparing different options for GitHub self-hosted runners

We are in the phase of transitioning from Bitbucket to GitHub Enterprise.

Considering larger GitHub ecosystem, it also makes sense to move the CI/CD governance processes to GitHub instead of using a completely different solution. So we decided to migrate our CI/CD pipelines to GitHub Actions from current Jenkins and CloudBees based solution.
As part of this journey (which we are in currently), we had to work on various aspects of this migration.

1. Migration of BitBucket repositories to GitHub. This involves migrating source code, LFS, commit history, repository/branch permissions, webhooks etc.
2. Designing the GitHub Actions based solution that maps from the existing pipelines defined in Jenkins/CloudBees. This involves considering existing shared libraries, utility/helper pipelines, webhooks, RBAC model etc.
3. Build execution environment. In Jenkins, we use VMs as nodes to run the pipelines. In CloudBees, we use dynamic pods using EKS cluster. We need to identify the right build execution environment for GitHub Actions.

The below table talks more about #3.

	GitHub-hosted Larger Runners	GitHub-hosted Runners	Containers on GitHub-hosted Runners	Self-hosted Runners using VMs and servers	Self-hosted runners using Kubernetes ARC (Actions Runner Controller) operator	AWS CodeBuild
Details	VMs hosted by GitHub	Large VMs hosted by GitHub, GPU and ARM support	Your specified containers that run on VMs hosted by GitHub	VMs or servers managed by you	Uses Kubernetes cluster managed by you	Uses AWS CodeBuild projects in your account
Supported Operating Systems	Linux, Windows, macOS	Linux, Windows, macOS	Any OS the container has	Linux, Windows, macOS	Linux, Windows	Amazon Linux, Ubuntu, Windows 2019
Authentication	Managed by GitHub	Managed by GitHub	Managed by GitHub		GitHub App PAT
Preinstalled Software	Most of the packages preinstalled	Most of the packages preinstalled	You need to installed required software in Docker images	You need to installed required software	You need to installed required software in Docker images	You need to install required software in buildspec.yml OR use a custom image for CodeBuild project
Maintenance	GitHub maintain the VM images	GitHub maintain the VM images	You need to maintain the Docker images	You need to maintain VM images	You need to maintain the Docker images	You need to maintain the CodeBuild project and any custom Docker images
Security	New VM that acts as a runner is created each time and deleted when job is finished, so less security risk.	New VM that acts as a runner is created each time and deleted when job is finished, so less security risk.	New VM is created each time for container that acts as a runner, and deleted when job is finished, so less security risk.	You need to maintain the security. Do not allow for public repos to avoid running untrusted workflows	New pod that acts as a runner is created each time, so less security risk. However, you need to manage the security of Kubernetes cluster.	New CodeBuild project execution (container) that acts as a runner created each time, so less security risk. However, you need to manage the security of CodeBuild project.
Scaling	Autoscaling to support concurrent workflows	Autoscaling to support concurrent workflows	Autoscaling to support concurrent workflows	You need to manage the scaling and make sure VMs/Servers have enough resources	You need to configure EKS cluster HPA and autoscaling to support concurrent workflows	CodeBuild autoscaling is used.
Pricing	$ Free (public repos) Limited Free (private repos) Usage-based pricing	$$ Free (public repos) Limited Free (private repos) Usage-based pricing	$ Free (public repos) Limited Free (private repos) Usage-based pricing	$$ Varies based on your infrastructure, but on-prem costs usually are more than using managed services	$$$ Hourly pricing	$ Usage-based pricing
Cost (1 hr on Linux)	Linux 2-core x64-powerd Ubuntu $0.0008 * 60 = $0.48	Linux Advanced 2-core x64-powerd Ubuntu $0.0008 * 60 = $0.48 ========= Linux Advanced 64-core x64-powerd Ubuntu $0.256 * 60 = $15.36	Linux 2-core x64-powerd Ubuntu $0.0008 * 60 = $0.48		EKS cluster 3 t3.medium worker nodes with 1TB EBS volumes $0.10 [cluster] + ($0.0416 * 3) [worker nodes] + $0.22 [EBS and snapshots] $0.45 Note: EKS charges will be continues, not just for the workflow executions	CodeBuild project with Ubuntu image 3 GB memory, 2 vCPUs $0.30