Automating GitLab Runner Setup on AWS EC2 Using Terraform 🦊☁️

Introduction 👋

As DevOps engineers, our daily tasks often involve setting up and managing infrastructure to support CI/CD pipelines. Automating these processes can significantly enhance our efficiency and consistency. In this blog post, I'll walk you through provisioning and configuring a GitLab Runner on an AWS EC2 instance using Terraform. This setup will allow you to integrate GitLab CI/CD with AWS infrastructure seamlessly.

Prerequisites

Before we start, make sure you have the following:

• A GitLab account: https://gitlab.com/
• Create new group > new project on GitLab if needed, e.g: thedevopshub/hello-gitlab.git
• An active AWS account: https://signin.aws.amazon.com/signup?request_type=register
• Terraform installed on your machine
• AWS CLI installed and configured on your machine

You can refer to my previous Terraform post here to learn more about Terraform

Steps Overview

• Prepare the environment
• Configure Terraform
• Provision the GitLab Runner on AWS
• Verify GitLab CI/CD pipeline runs on new EC2 runner
• Clean up resources

NOTE: All the code for this blog post are available at: https://github.com/TheDevOpsHub/TerraformHub/tree/main/AWS/aws-ec2-gitlab-runner

Step 1: Prepare the Environment

Create an AWS Key Pair:

Follow the AWS documentation to create a key pair: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html

Generate a GitLab Runner Registration Token:

Navigate to your GitLab project or group, then go to Settings > CI/CD > Runners, and register a new runner to get the token. Register GitLab Runner

Clone the Repository:

Clone the repository containing the Terraform configuration or create a new directory for your Terraform files.

git clone [e-mail]:TheDevOpsHub/TerraformHub.git
# or creating your file manually: `mkdir -p TerraformHub/AWS/aws-ec2-gitlab-runner`
cd TerraformHub/AWS/aws-ec2-gitlab-runner

Step 2: Configure Terraform

Create a main.tf file with the following content to define the AWS resources:

resource "aws_security_group" "gitlab_runner" {
  name        = "gitlab-runner-sg"
  description = "Security group for GitLab Runner"

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_instance" "gitlab_runner" {
  # Check all available us-east-1 AMIs at: https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#AMICatalog
  ami             = "ami-03972092c42e8c0ca" # Amazon Linux 2 AMI
  instance_type   = var.instance_type
  key_name        = var.key_name
  security_groups = [aws_security_group.gitlab_runner.name]

  user_data = templatefile("install_runner.tpl", {
    gitlab_runner_registration_token = var.gitlab_runner_registration_token
  })

  tags = {
    Name = "AWS EC2 GitLab Runner"
  }
}

Create an install_runner.tpl file for the user data script:

#!/bin/bash
# Install necessary dependencies
# set -x enables a mode of the shell where all executed commands are printed to the terminal
set -x
echo "Hello from EC2 user data script"

yum update -y
yum install -y curl git

# Install GitLab Runner
curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
chmod +x /usr/local/bin/gitlab-runner


useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

# Start the GitLab Runner service
/usr/local/bin/gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
/usr/local/bin/gitlab-runner start

# Register the GitLab Runner
/usr/local/bin/gitlab-runner register --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "${gitlab_runner_registration_token}" \
  --executor "shell" \
  --description "AWS GitLab Runner" \
  --tag-list "aws,linux" \
  --run-untagged="true" \
  --locked="false"

systemctl status -l gitlab-runner.service

Create a variables.tf file to define variables:

variable "aws_region" {
  description = "The AWS region to create resources in."
  type        = string
  default     = "us-east-1"
}

variable "instance_type" {
  description = "The type of instance to create."
  type        = string
  default     = "t2.micro"
}

variable "key_name" {
  description = "The name of the key pair to use for the instance."
  type        = string
  default     = "ec2-ssh-key" # change to yours
}

variable "gitlab_runner_registration_token" {
  description = "The GitLab Runner registration token."
  type        = string
  sensitive   = true
}

Create a terraform.tfvars file:

aws_region = "us-east-1"
instance_type = "t2.micro"
key_name = "your-key-name"
gitlab_runner_registration_token = "your-registration-token"

Create an output.tf file to display outputs:

output "instance_id" {
  description = "The ID of the instance."
  value       = aws_instance.gitlab_runner.id
}

output "instance_public_ip" {
  description = "The public IP of the instance."
  value       = aws_instance.gitlab_runner.public_ip
}

Step 3: Provision the GitLab Runner on AWS

Initialize Terraform:

Navigate to your project directory and initialize Terraform.

terraform init

Plan the Configuration:

Create a plan to see the actions Terraform will take.

terraform plan -out "runner.tfplan"

Apply the Configuration:

Apply the Terraform plan to provision the infrastructure.

terraform apply "runner.tfplan"

Verify the Deployment:

After Terraform completes, verify that the GitLab Runner is running on the EC2 instance.

# Connect to your instance
ssh -i "your-key-name.pem" ec2-user@your-instance-public-ip

# Check GitLab Runner status
systemctl status gitlab-runner.service

On GitLab, navigate to your project > Setting > CICD > Expand the runner you would see the new active runner available.

Step 4: Verify GitLab CI/CD pipeline runs on new EC2 runner

Add the .gitlab-ci.yml file to the root directory of main branch:

# .gitlab-ci.yml
stages:
  - verify-runner

verify-runner:
  stage: verify-runner
  script:
    - echo "Hello, World!"
    - cat /etc/os-release
    - hostname -f
    - date
  tags:
    - aws

Commit the change and our pipeline will be run successfully on your newly registered EC2 runner:

Pipeline flow:

Logs:

Step 5: Clean Up Resources

Once you no longer need the runner, you can destroy the resources to avoid unnecessary costs:

terraform destroy

Conclusion

By following these steps, you can automate the setup of a GitLab Runner on AWS EC2 using Terraform. This approach not only saves time but also ensures consistency and repeatability in your CI/CD pipeline infrastructure. If you have any questions or suggestions, feel free to leave a comment below.

---

Want to learn more about DevOps? Visit https://github.com/TheDevOpsHub to explore various repositories from document/hands-on/practical guide

Thank you for reading and Happy DevOps-ing! 🚀

Comments

[Benoit COUETIL 💫]

Thank you for sharing ! Maybe you could improve it a little bit by limiting the uptime to working hours, and save 2/3 of the price of your runner. Not to mention the spot instance trick 😉.