DEV Community: Arunodhayam

EC2 instance as self-hosted GItHub runners

Arunodhayam — Tue, 11 Oct 2022 05:56:39 +0000

By default, GitHub offers hosted runners to run our workloads inside. For their Linux runners, GitHub offers runners with specs: 2-core CPU, 7 GB of RAM, 14 GB of SSD disk space.

Some of your CI/CD workloads may require more powerful hardware than GitHub-hosted runners could provide. In such case, you can configure an EC2 instance to act as your GitHub runner instead.

For example, you may run a c5.4xlarge instance type for some of your workloads, or even a r5.xlarge for workloads that process large data sets in-memory.

Let us begin:

#1. Create a GitHub personal access token

Go to your GitHub Settings
Click the Developer settings

Click Personal access token to create one

Create a new GitHub personal access token with the repo scope in the name of GH_PERSONAL_ACCESS_TOKEN. The action will use the token for self-hosted runners management in the GitHub account on the repository level.

#2. Create an AMI image

Create a new EC2 instance based on a Linux distribution of your choice
Connect to the instance ```sh

ssh -i user_name@IP_ADDRESS

- Install packages and configure the instance according to your workflow requirements
- After that, land on the EC2 console, select your instance, right-click on it to find `Image and templates` , hover to see `Create image`, click it

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/buiy3gqonksd0yabhsxg.png)
- Enter the `Image name`, `Image description`, and check `No reboot`, and at last , hit `Create`
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k00gnvipndhmh2avxaor.png)
-  The AMI should show up in the AMI console 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/83mf1vod1nav7k5h2rhv.png)
- You can safely terminate the instance as it's of no use any more, of course, you could just keep it running if you plan on making any changes to it, however

## #3. Create VPC with subnet and security group
- Create a new [VPC and Subnet](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-vpcs.html) or use an existing
- Create a new [Security group](https://docs.aws.amazon.com/cli/latest/userguide/cli-services-ec2-sg.html) for the runners in the VPC and only allow outbound traffic to port 443, for pulling jobs from GitHub. No port opening for inbound traffic is necessay.

## #4. Configure the environment secrets
- The following environment variables must be set for this action 

| Secrets | Description |
| ------- | ----------- |
| `GH_PERSONAL_ACCESS_TOKEN` | Add the GitHub pat token in secret |
| `AWS_REGION` | Add the region in secret |
| `AWS_ACCESS_KEY_ID` | Add the access key in secret |
| `AWS_SECRET_ACCESS_KEY` | Add the secret key in secret |
| `AMI` | Add the EC2 AMI ID in secret |
| `INSTANCE_TYPE` | Provide the instance type like t2.medium, t2.micro |
| `SUBNET` | Add the subnet ID in secret |
| `SECURITY_GROUP` | Add the security group ID in secret |

- Go to your project `repo settings` to add the above Secrets

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bpbvapzu5qh3qckwpfqr.png)

- Expand the `Secrets` drop-down and click `Actions`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y8041s4x1caywdahqwvo.png)

- In the _Actions secrets_ tab click `New repository secret` 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5fmybdpgemqq2vv6s1ci.png)

- Name your secrets and add their values to be consumed by the action

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3kx83orynk8n3mz16vwr.png)

- A complete listing of the required secrets should look something like

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gjd7ez1jzkxgwqrwmkul.png)


## #5. Configure the GitHub workflow

- Create a new GitHub Action with the below example workflow
- Please don't forget to set up a job for removing the EC2 instance at the end of the workflow execution.
   Otherwise, the EC2 instance won't be removed and continue to run even after the workflow execution is finished.

Now you're ready to go!

### Inputs

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Required                                   | Description                                                                                                                                                                                                                                                                                                                           |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mode`                                                                                                                                                                       | Always required.                           | Specify here which mode you want to use: <br> - `start` - to start a new runner; <br> - `stop` - to stop the previously created runner.                                                                                                                                                                                               |
| `GitHub-token`                                                                                                                                                               | Always required.                           | GitHub Personal Access Token with the `repo` scope assigned.                                                                                                                                                                                                                                                                          |
| `ec2-image-id`                                                                                                                                                               | Required if you use the `start` mode.      | EC2 Image Id (AMI). <br><br> The new runner will be launched from this image. <br><br> The action is compatible with Amazon Linux 2 images.                                                                                                                                                                                           |
| `ec2-instance-type`                                                                                                                                                          | Required if you use the `start` mode.      | EC2 Instance Type.                                                                                                                                                                                                                                                                                                                    |
| `subnet-id`                                                                                                                                                                  | Required if you use the `start` mode.      | VPC Subnet Id. <br><br> The subnet should belong to the same VPC as the specified security group.                                                                                                                                                                                                                                     |
| `security-group-id`                                                                                                                                                          | Required if you use the `start` mode.      | EC2 Security Group Id. <br><br> The security group should belong to the same VPC as the specified subnet. <br><br> Only the outbound traffic for port 443 should be allowed. No inbound traffic is required.                                                                                                                          |
| `label`                                                                                                                                                                      | Required if you use the `stop` mode.       | Name of the unique label assigned to the runner. <br><br> The label is provided by the output of the action in the `start` mode. <br><br> The label is used to remove the runner from GitHub when the runner is not needed anymore.                                                                                                   |
| `ec2-instance-id`                                                                                                                                                            | Required if you use the `stop` mode.       | EC2 Instance Id of the created runner. <br><br> The id is provided by the output of the action in the `start` mode. <br><br> The id is used to terminate the EC2 instance when the runner is not needed anymore.                                                                                                                      |
| `iam-role-name`                                                                                                                                                              | Optional. Used only with the `start` mode. | IAM role name to attach to the created EC2 runner. <br><br> This allows the runner to have permission to run additional actions within the AWS account, without having to manage additional GitHub secrets and AWS users. <br><br> Setting this requires additional AWS permissions for the role launching the instance (see above). |
| `aws-resource-tags`                                                                                                                                                          | Optional. Used only with the `start` mode. | Specifies tags to add to the EC2 instance and any attached storage. <br><br> This field is a stringified JSON array of tag objects, each containing a `Key` and `Value` field (see example below). <br><br> Setting this requires additional AWS permissions for the role launching the instance (see above).                         |
| `runner-home-dir`                                                                                                                                                              | Optional. Used only with the `start` mode. | Specifies a directory where pre-installed actions-runner software and scripts are located.<br><br> |

### Outputs

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Description                                                                                                                                                                                                                               |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `label`                                                                                                                                                                      | Name of the unique label assigned to the runner. <br><br> The label is used in two cases: <br> - to use as the input of `runs-on` property for the following jobs; <br> - to remove the runner from GitHub when it is not needed anymore. |
| `ec2-instance-id`                                                                                                                                                            | EC2 Instance Id of the created runner. <br><br> The id is used to terminate the EC2 instance when the runner is not needed anymore.                                                                                                       |

### Example workflow

`ci.yml`

```yml


name: ci
on: pull_request
jobs:
  start-runner:
    name: Start self-hosted EC2 runner
    runs-on: ubuntu-latest
    outputs:
      label: ${{ steps.start-ec2-runner.outputs.label }}
      ec2-instance-id: ${{ steps.start-ec2-runner.outputs.ec2-instance-id }}
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}
      - name: Start EC2 runner
        id: start-ec2-runner
        uses: machulav/ec2-github-runner@v2
        with:
          mode: start
          github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
          ec2-image-id: ${{ secrets.AMI }}
          ec2-instance-type: ${{ secrets.INSTANCE_TYPE }}
          subnet-id: ${{ secrets.SUBNET }}
          security-group-id: ${{ secrets.SECURITY_GROUP }}
          iam-role-name: my-role-name # optional, requires additional permissions
          aws-resource-tags: > # optional, requires additional permissions
            [
              {"Key": "Name", "Value": "ec2-github-runner"},
              {"Key": "GitHubRepository", "Value": "${{ github.repository }}"}
            ]
  do-the-job:
    name: Do the job on the runner
    needs: start-runner # required to start the main job when the runner is ready
    runs-on: ${{ needs.start-runner.outputs.label }} # run the job on the newly created runner
    steps:
      - name: Hello World
        run: echo 'Hello World!'
  stop-runner:
    name: Stop self-hosted EC2 runner
    needs:
      - start-runner # required to get output from the start-runner job
      - do-the-job # required to wait until the main job is done
    runs-on: ubuntu-latest
    if: ${{ always() }} # required to stop the runner even if the error happened in the previous jobs
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}
      - name: Stop EC2 runner
        uses: machulav/ec2-github-runner@v2
        with:
          mode: stop
          github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
          label: ${{ needs.start-runner.outputs.label }}
          ec2-instance-id: ${{ needs.start-runner.outputs.ec2-instance-id }}

After a successful first workflow execution, you should see this

Deploy NodeJS application on AWS ECS (EC2 Launch Type)

Arunodhayam — Thu, 06 Oct 2022 12:08:57 +0000

In this post, we will build the docker image for a nodejs application and deploy it onto AWS ECS (EC2 Launch type)

Prerequisites

#1: Create a simple Node app

Create a directory and navigate to it

mkdir node-app
cd node-app

Initialize packages

npm init --y

Install ExpressJS

npm install express

Enter this block of code into an index.js file

const express = require('express');
const app = express();
app.get('/', (req,res) => {
    res.send("This is ECS deployment")
});
app.listen(8080,() => {
    console.log("Server started")
});

Issue the below command to run the application locally

node index.js

You can now view the application at http://localhost:8080

#2: Dockerize Node app

Next, we create a Dockerfile in the project root to build an image out of

# Use a Node runtime as a parent image
FROM node: alpine
# Set the working directory to /app
WORKDIR /app
# Copy package.json and package-lock.json to the working directory
COPY package*.json ./
# Install any needed packages specified in package.json
RUN npm install
# Copying the rest of the code to the working directory
COPY . .
# Make port 8080 for the application port
EXPOSE 8080
# Run index.js for application expose for the container
CMD [ "node", "index.js"]

#3: Build and Push the docker image to AWS ECR

Navigate to ECR service in your AWS Console

Click on Create repository in top right corner to create the repo

Create either a private or public repo based on your need

Once the repository is created, click view push commands from inside to push the image

You can from the instructions, infer the commands to log in to AWS ECR, build the image, tag the build, and push the tagged image to AWS ECR.

Finally, your pushed image should look like this

#4: Create the ECS Cluster

Click on Clusters from the ECS console sidebar

Choose EC2 Linux + Networking as the cluster template

Input the instance and networking configuration

The cluster then is created and cloudformation by default takes care of launching the rest of the resources

#5: Create a Task Definition

Task Definitions in ECS sort of acts as a blueprint of how an application should be deployed

Click on Task Definitions right under Clusters from the ECS console sidebar and click on Create new Task Definition.

Select EC2 as the launch type

Assign the Name, Task role, and Network mode of the container

vCPU and Memory count is optional, as we have opted for the EC2 launch type. Click Add container once filled.

Now, time to configure the container: Container name, Image; copy the AWS ecr image URI, specify the Memory limit - 300MB and above, and set Port mapping to the given application port

Now your task definition is ready to be used in tandem with the Service definition

#6: Create Service

Navigate to the Cluster tab, scope in on the service tab, and click Create

Configure the service with launch type EC2, choose the Task definition we have created, and Service type as Replica.

Configure the VPC within which the ECS service should go. We are selecting None on Load balancing and Service discovery, as we aren't aiming for a larger scale.

Configure auto-scaling if need be

Do a quick review glance once all the configuration is done

Clicking the services tab will give you info on running containers and their configurations

Navigate to the Tasks tab and you'll find the task

Here you can view the details of the task

Click the drop-down next to the container name to reveal the External Link

And there we have it

The NodeJS application is successfully deployed on AWS ECS.