Unlocking CI/CD Power: My Journey with GitHub Self-Hosted Runners (DevOps SRE Challenge Day 7)

Hey everyone! 👋

Today marks Day 7 of the incredible Daily DevOps SRE Challenge - Season 2, and what a day it has been! Our mission: to set up GitHub Self-Hosted Runners and deploy a classic web-based Snake game. This challenge was a fantastic deep dive into customizing CI/CD pipelines, and I'm really excited to share my experience.

First off, a huge thank you to our brilliant mentor, Sagar Utekar. 🙏 His structured challenges and insightful guidance are truly making a difference in my DevOps journey. This specific task helped me bridge the gap between theoretical knowledge and practical, real-world application.

The Mission: Beyond Standard CI/CD

The core of today's challenge was to deploy a simple Flask-based Snake game. While GitHub's standard runners are great, the real learning came from using a self-hosted runner.

Why self-hosted, you ask? This is a question both new learners and experienced recruiters might have, and it highlights a key DevOps concept. For me, it was about:

• Customization: Full control over the build environment (OS, software versions, specific tools).
• Resource Optimization: Potentially more cost-effective for large, consistent workloads than paying for GitHub's hosted minutes.
• Security & Compliance: Meeting specific enterprise security policies by keeping builds on private infrastructure.
• Access to Private Networks: Allowing workflows to interact with resources behind a firewall that GitHub's public runners can't reach.

Understanding this "why" is crucial for any aspiring SRE or DevOps engineer. It shows you're thinking beyond just getting the job done, but doing it in the most effective and secure way possible.

Step-by-Step: My Self-Hosted Runner Deployment Journey

Here's how I tackled the challenge, detailing each phase for anyone reading, whether you're just starting out or reviewing my capabilities!

1. Setting Up the Host: My AWS EC2 Instance

The first order of business was provisioning the infrastructure for my self-hosted runner.

• Cloud Instance: I spun up an AWS EC2 instance (Ubuntu 22.04). This is a common and robust choice for hosting applications and services in the cloud.
• Network Access: Crucially, I configured the EC2's security group to allow inbound traffic on Port 22 (for SSH access) and Port 80 (for the web-based Snake game to be accessible).
• SSH Connection: Once the instance was running, I used SSH to connect and start configuring it.

2. Registering the GitHub Self-Hosted Runner

This is where the EC2 instance became part of my GitHub Actions ecosystem.

• I navigated to my GitHub repository's Settings > Actions > Runners.
• GitHub provided clear instructions and commands to download, extract, and configure the runner application on my EC2 instance.
• After running ./config.sh, I then started the runner with ./run.sh. This crucial step made my EC2 instance actively "listen" for workflow jobs from my repository. Seeing it show up as "Online" in GitHub was a satisfying moment!

3. Dockerizing the Snake Game (and My First Hurdle! 🚧)

The Snake game project uses Python Flask, and to ensure consistent deployment, Dockerization was key. This is where my first real challenge popped up, and it was a great learning experience!

• Missing Dockerfile: The project itself didn't come with a Dockerfile. This meant I had to create one from scratch. For a newbie, this is a perfect exercise in understanding how to containerize an application. For a recruiter, it shows initiative and the ability to adapt a project for container-based deployment.
  

  # My simple Dockerfile for the Snake game
  FROM python:3.8-slim
  WORKDIR /app
  COPY requirements.txt .
  RUN pip install --no-cache-dir -r requirements.txt
  COPY . .
  EXPOSE 80
  CMD ["python", "app.py"]
  

• Docker Daemon Permissions: After creating the Dockerfile, I ran into permission denied errors when trying to run docker build. This is a super common issue: my ubuntu user wasn't part of the docker group!

  • The Fix: I simply ran sudo usermod -aG docker $USER and then logged out/in (or newgrp docker). This granted my user the necessary permissions to interact with the Docker daemon without sudo. A small fix, but a vital lesson in Linux user management!

With the Dockerfile in place and permissions sorted, I could finally run:

docker build -t snake-game .
docker run -d -p 80:80 snake-game

4. Crafting the GitHub Actions Workflow: My Automation Pipeline

This was the core automation piece. I created a .github/workflows/deploy.yml file to orchestrate everything.

name: Deploy Snake Game to EC2

on:
  push:
    branches: [main] # Trigger on pushes to main branch
  workflow_dispatch: # Allows manual triggering

jobs:
  build-and-deploy:
    runs-on: self-hosted # <-- The magic! Tells GitHub to use my EC2 runner!

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Docker Hub Login
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }} # Using GitHub Secrets for security!

      - name: Build and push Docker image
        run: |
          docker build -t ${{ secrets.DOCKERHUB_USERNAME }}/snake-game:latest .
          docker push ${{ secrets.DOCKERHUB_USERNAME }}/snake-game:latest

      - name: Deploy application
        run: |
          docker stop snake-game || true # Stop existing container gracefully
          docker rm snake-game || true   # Remove it
          docker run -d --name snake-game -p 80:80 ${{ secrets.DOCKERHUB_USERNAME }}/snake-game:latest # Run new one!

Key takeaways from the workflow:

• runs-on: self-hosted: This single line is what directs the job to my custom EC2 instance. This is a powerful feature for integrating your own infrastructure into GitHub's CI/CD.
• GitHub Secrets: I used GitHub Secrets (DOCKERHUB_USERNAME, DOCKERHUB_TOKEN, EMAIL_RECIPIENT) to securely store sensitive information. This is a non-negotiable best practice for any production-ready pipeline.
• Robust Deployment Logic: The docker stop || true and docker rm || true commands ensure that deployments are idempotent – meaning they can be run multiple times without causing issues.
• Validation is Key: The curl loop to validate the application's health was crucial. My workflow initially failed here with curl exit code 56 (failure to receive data).
  • The Fix: This led me to debug the running container directly on the EC2 instance using docker ps -a and docker logs <container_id>. It turned out there was a small issue causing the Flask app to exit immediately after starting. Fixing the application code within the container resolved the issue. This iterative debugging process is very common in DevOps!
• Notifications: Integrating an email notification step ensures that the relevant team members are always aware of deployment outcomes.

The Final Result!

After overcoming these challenges, I successfully built an end-to-end CI/CD pipeline that:

1. Listens for code changes.
2. Utilizes a custom, self-hosted runner on AWS EC2.
3. Containerizes the application with Docker.
4. Deploys the containerized application.
5. Validates its health and sends a notification.

And of course, the proof is in the pudding:

A Screenshot of the GitHub Actions Workflow running on my self-hosted runner:

A Screenshot of the Snake Game running live in the browser on my EC2's Public IP:

A Screenshot of my AWS EC2 Instance details:

What's Next & Join the Community!

This challenge significantly boosted my confidence in managing custom CI/CD infrastructure. I'm already looking forward to exploring the bonus tasks like auto-scaling runners and implementing Nginx for load balancing!

A huge thank you again to Sagar for these invaluable lessons!

#getfitwithsagar #SRELife #DevOpsForAll #GitHubActions #SelfHostedRunner #Docker #AWS #CI/CD

---