DEV Community: Thierry Enongene

Infrastructure as Code Challenge - Terraform with GitLab CI/CD

Thierry Enongene — Thu, 31 Aug 2023 15:53:13 +0000

Full Project link on Github: https://github.com/users/tenongene/projects/14

Summary of Challenge and Plan

I recently came across an infrastructure as code challenge on Github called "DocNetwork's DevOps Challenge" and I decided to star it and give it a try. I am very passionate about furthering my skills and knowledge in all things cloud and devops, and this challenge seemed like a good one to test my skills.

A description of the challenge:

Challenge Link: https://github.com/docnetwork/Infra-as-code-challenge

PLANNING

-Design and code a Quote Generator single page Application with NodeJS and Express, that will fetch a
random quote from a free quote API, and also display a random inspirational image of nature also
fetched from a free photo collection API.

-Containerize the application with Docker, and pushing the image to the DockerHub registry.

-Write and code terraform manifest files for deployment on AWS via an AutoScaling Group of EC2
instances, configured with a load balancer.

-Build a CI/CD pipeline with GitLab, configuring a Webhook in DockerHub and writing a Lambda function
for updating the instances.

-Write and code terraform manifest files for deployment on AWS via an AutoScaling Group of Task
definitions in an ECS cluster, configured with a load balancer.

=================================

Initial Quote Generator application design with NodeJS and Express

The challenge said it had provided an already containerized application. That did not seem to be available as it's an old repository.
I decided to build my own from scratch using NodeJS, Express and some free APIs.

Wrote a single page quote generator application using NodeJs and the Express framework. The page is intended to generate random inspiring quotes from a free quote API called Zenquotes (https://zenquotes.io).
Along with the quote is displayed a random photograph of nature from a free photo collection API called Pexels (https://www.pexels.com).

A user will click on a "Generate" interactive button that will fetch a new picture along with a new quote with each call of the APIs.

Configured clicking of the "Generate" button to hit the "GET" route in express which will make sequential calls to the quote API, then the photo API, returning and resolving promises at each step to return the "picture", "quote" and "author" variables for injection into the page.

Masked the API key and Port Number using the dotenv module for security.

Made use of a random number function that generates a random number which I then attached as a variable to the API query string to generate a random quote/image.

Used the EJS templating engine to pass in the variables "picture", "quote" and "author" from the API that would be received from the APIs.
I used simple styling from materialize css.

Initial pageview from localhost:7272 during development.

==============================

Containerization of application with Docker

-I wrote the Dockerfile for building the application image. I used a slim node base image available in
docker hub.

-Copied the current directory to the image working directory which I named as "app".
Ran "npm install" so that all the application dependencies from the package.json file got included with
the image.

-Exposed port 7272 which is the application port as written in Express.

-The command was "node index.js" which will start the application in the container.

-Built the image by running "docker build -t tenongene/quotegen ." in the current directory.
Ran "docker push tenongene/quotegen" to push the image to docker hub.

Application image now in Docker Hub.

=========================

Writing Terraform manifests for deployment on AWS

Wrote manifest files to deploy application on AWS via an autoscaling group of EC2 instances.

Configured the VPC using the Terraform provider VPC module.

Entered security group rules for load balancer access, SSH and metrics export for possible observability.
Created 4 public subnets, 1 in each of 4 availability zones which will each host the application, for high availability.

Entered the autoscaling group launch templates as required by the module. As this is a simple exercise, did not go into much detail as to configuring launch configurations, scaling rules and instance lifecycles as may be required in production settings.
Specified the desired capacity to 4 with the intent on placing an instance in each subnet per availability zone.

Obtained an ami id number from the console to use in the manifest.

Registered the autoscaling group as targets to the load balancer target group by referencing the target group arn.

Entered a base64 encoded user data script which will bootstrap the EC2 instances on launch with instructions to install Docker onto the instance, login to the docker registry, run a docker container by pulling the image that was built and pushed to docker hub. Mapped the application port 7272 to the instance port.

Defined the target groups for subsequent registering of autoscaling group.
Associated the VPC by it's id.
Configure an http listener for the load balancer, with ingress at port 80 with a backend target port of 7272 for the instances/application via the autoscaling group.

Initially had access to application from both the instances and load balancer as I had configured the security group to allow access to port 7272 from everywhere ("0.0.0.0/0"). However I later realized this did not satisfy the requirement of access only via the load balancer. Thus created a second security group resource unique to the load balancer. Configured the default security group in the ASG to accepting incoming traffic into port 7272 only from the load balancer security group.

Wrote an output file for important infrastructure resources identification.

=============================

Infrastructure deployment results

Ran Terraform Init, plan, and apply and infrastructure was deployed.

i)- The 4 EC2 instances launched.

ii)- The security groups (default and load balancer) and associations/rules.

iii)-The deployed network load balancer

iv)- The target group, showing one instance per availability zone in a subnet.

v)- The deployed autoscaling group.

View from 4 different web browsers (Firefox, Opera, Edge, Chrome) when application is accessed via the load balancer.

Accessing each instance via it's public IP address at port 7272 does not reach the application, thus proof that application is only accessible via the load balancer.

===============================

Building a CI/CD Pipeline in GitLab, with a DockerHub Webhook configured for Lambda Function

I migrated the application code to GitLab as my preferred platform for continuous integration and continuous deployment.

I configured environment variables in the GitLab pipeline with credentials to access the DockerHub registry for automatically pushing any image updates.

Wrote the .gitlab-ci.yaml file definition for the pipeline that will propagate changes in the application to the image upon a push to the code repository.
I used a Docker in Docker runner for the pipeline.

Ran initial pipeline with a test change to the application code to verify successful build of application docker image.

I wrote a lambda function in AWS named "quotegen-update" that will update the application image in the EC2 instances using the NodeJS SDK for AWS. The function URL will be used to configure a Webhook in docker hub. The Webhook will send an HTTP POST request to the function URL with each push of a new image to docker hub.

The idea is for the function to send instructions to terminate the currently running instances, at which point the Autoscaling group will fire up new instances to replace the terminated ones. However, based on the embedded user data script, the new instances will have the latest application image pulled from docker hub when creating the containers.

The lambda function handler makes use of the DescribeInstancesCommand and TerminateInstancesCommand
methods of the EC2 Client Module.

The DescribeInstancesCommand was used as a method to filter out the instances by the resource Tags of
Name and Instance Running Sate. The instance ID was not ideal because each new instance will have a
new instance ID and a new public IP address when fired up by the autoscaling group, so it wouldn't be
possible to code which instances to stop.
The resource Tags are the most efficient method since that is common to all instances created.

The result of the filtered instances provided the running instances, which I transformed into an array to be used in the TerminateInstancesCommand method.

Created function with function URL: *https://aicu5rzgtfpqmwhaua4z7sijyi0snlsk.lambda-url.us-east-1.on.aws/
*

I configured a webhook for the application image repository in docker hub with the URL of the lambda function for which a POST request will be sent for image update.

===========================

Demonstration of application CI/CD with Results

With the complete pipeline now in place from the GitLab code repository through to the instances in AWS,
I made a change to the application page by adding a greeting in the html page.
I then committed the changes and pushed them to the GitLab code repository.

Push to the repository triggered the CI/CD pipeline which immediately began building the image.

Successful completion of the pipeline resulted in a push of the new image to DockerHub, which then triggered the Webhook to send a post request to the lambda function URL, and thus invoking the lambda function.

The lambda function executed by terminating the running EC2 instances.

The Autoscaling group then began initializing new instances to replace the terminated ones, but the new instances will initialize by pulling the newly updated image.

The updated application is now available when accessed via the load balancer as before.

============================

Writing Terraform manifests for deployment on AWS via an Autoscaling group associated ECS Cluster

To deploy the application via ECS, I had to modify the autoscaling group to be a capacity provider for a new ECS cluster and write new resources for the creation of a cluster, a service and a task definition to run the containers.

The load balancer and target groups had to be modified as well.

I decided to go with an application load balancer this time around, since I garnered several insights from the first phase of the project. I changed the application port in the application to serve at port 80, and added a health check route at "/health" in the application, and that is how the load balancer would report instances as healthy.

Created a launch template resource with the required EC2 configuration. The ami used thi time was an ECS optimized Amazon Linux ami.
Saved the cluster name to the environment variable and inserted into the ECS agent config file.
Created and IAM instance profile for the instances for authorization.

Created the cluster, capacity provider and attached the capacity provider to the cluster.

Wrote the service definition file, referencing the capacity provider strategy for autoscaling, the load balancer, the task definition (below) to use and I attached an IAM role that I created on the console to be used by the service load balancer.

The task definition for the container parameters with the container name, the image to pull form docker hub, the port mapping, and attached the execution role created on the console for authorization.
The task is to use the "bridge" network mode.

I created the lambda function for the CI/CD process, whose URL will be used as a Webhook in docker hub.

The idea is to use the "Update Service Command" of the ECS client Nodejs SDK. It has a method called "ForceNewDeployment", and when set to "true" and executes on an event, it simply instructs the service to create a new deployment. The service will then create new deployment of the same quantity, but in a ROLLING UPDATE strategy where one container at a time is terminated then updated. This prevents downtime for the service and creates a smoother transition to any image updates. The new containers will then have to pull the newest image in docker hub.

The function URL was:

=============================

Infrastructure deployment Results with ECS autoscaling

The deployed ECS Cluster

The service

Cluster infrastructure with containers in each availability zone

Load balancer and target group

The autoscaling group

Accessing application

The CI/CD process and making a change to the application.

Updated image in DockerHub

Cloudwatch logs showing function invocation by the webhook

Rolling update of the service in the cluster with new containers being spun up in turn, over about 15 minutes.

Updated application is now accessible with the newly updated image

Project challenges, insights and lessons learned

The with the initial phase of the project with EC2 instances, my user data script was not bootstrapping the EC2 instances with docker in order to pull images and then run the container:
Further research pointed out that there should be no indentation to the right of the user data script, in the terraform manifest file. When I took out the indentation, the EC2 instances then bootstrapped correctly with docker running.
I also learned, only after completing the 2nd phase of the project with ECS that I also had the option of using an Amazon Linux ECS optimized AMI, which already comes with docker pre-installed.

Indentations in user data script

Do not indent.

I initially could not reach the application although everything seemed to have been configured correctly.
My debugging method was to access the instances via SSH and check the docker logs. I noticed that docker could not load the express module in the application during the image pull step. That means I forgot to run "NPM install" to install the dependencies from the package.json during image the building step. I added this, and rebuilt the image and then the image pulled correctly.

During the ECS deployment phase of the project, at first, although things seemed to be configured correctly, the tasks could not start and none of the containers could be deployed for some reason. I couldn't find any issues and the EC2 instances where running and passing system checks.

Upon further research, I learned I could check the logs of the ECS agent at /var/log/ecs/ecs-agent.log to get a clue of what could be wrong. I checked the logs and saw that the ECS agent did not have the proper credentials and permissions to pull images. My understanding was that an instance profile was required to do so. So I created an instance profile and attached to the ASG definition and that got resolved.

After that resolution the new issue was an error that the containers could not be registered because the cluster was inactive, although I could clearly see on the console that the cluster was active.
Further research with an answer from stack overflow suggested that the ECS agent only creates a "default" cluster, unless the cluster is specifically added as an environment variable (ECS_CLUSTER) and added to the ECS agent's configuration. I added it via a user data script and everything got resolved.

Terraform - Kubernetes Deployment plus S3 backend with state locking in DynamoDB

Thierry Enongene — Sun, 19 Mar 2023 23:08:26 +0000

In this exercise, I will be deploying an application to a kubernetes cluster using terraform infrastructure-as-code, and also using an AWS S3 bucket as a remote backend with state-locking in an AWS DynamoDB table.

This exercise is part of the Terraform challenge administered by KodeKloud. All credits to kodekloud.com for the design and structure of the exercise.

Architecture

Source: KodeKloud.com

A kubernetes application that will deploy 4 replicas.

Task List

Source: KodeKloud.com

Infrastructure-As-Code (Terraform file)

terraform {
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.11.0"
    }
  }
}

provider "kubernetes" {
  # config_path    = "~/.kube/config"
  # config_context = "docker-desktop"
  config_path    = "/root/.kube/config"
  config_context = "kubernetes@kubernetes_admin"
}

##Service 

resource "kubernetes_service" "webapp-service" {
  metadata {
    name = "webapp-service"
  }
  spec {
    selector = {
      name = "webapp"
    }
    port {
      port = 8080
      node_port = 30080
    }
    type = "NodePort"
  }
}

## Deployment 

resource "kubernetes_deployment_v1" "frontend" {
  metadata {
    name = "frontend"
    labels = {
      name = "frontend"
    }
  }

  spec {
    replicas = 4

    selector {
      match_labels = {
        name = "webapp"
      }
    }

    template {
      metadata {
        labels = {
          name = "webapp"
        }
      }

      spec {
        container {
          image = "kodekloud/webapp-color:v1"
          name  = "simple-webapp"
          port {
            container_port = 8080
            protocol = "TCP"
          }
        }
      }
    }
  }
}

Images Source: KodeKloud.com

Displaying in browser - both local and remote deployments

Create an AWS s3 bucket for a remote backend and a DynamoDB table for state locking

Creating an s3 bucket in AWS for the remote state, and also enabling versioning in case rollback is needed for disaster recovery.

Creating a DynamoDB table for state locking. Partition key must be "LockID" (spelling and case sensitive)

Reconfigure for Terraform configuration to use S3 bucket as remote backend

Updated Configuration file:

terraform {
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.11.0"
    }

    aws = {
      source = "hashicorp/aws"
      version = "4.49.0"
    }
  }

  backend "s3" {
    bucket = "tenon-remote-backend"
    key = "terraform.tfstate"
    region = "us-east-1"
    dynamodb_table = "tenon_state_locking"
  }
}

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = "docker-desktop"
  # config_path    = "/root/.kube/config"
  # config_context = "kubernetes@kubernetes_admin"
}

provider "aws" {
  shared_config_files      = ["~/.aws/config"]
  shared_credentials_files = ["~/.aws/credentials"]
  profile                  = "iamadmin-general"
}



##Service 
resource "kubernetes_service" "webapp-service" {
  metadata {
    name = "webapp-service"
  }
  spec {
    selector = {
      name = "webapp"
    }
    port {
      port = 8080
      node_port = 30080
    }
    type = "NodePort"
  }
}

## Deployment 

resource "kubernetes_deployment_v1" "frontend" {
  metadata {
    name = "frontend"
    labels = {
      name = "frontend"
    }
  }

  spec {
    replicas = 4

    selector {
      match_labels = {
        name = "webapp"
      }
    }

    template {
      metadata {
        labels = {
          name = "webapp"
        }
      }

      spec {
        container {
          image = "kodekloud/webapp-color:v1"
          name  = "simple-webapp"
          port {
            container_port = 8080
            protocol = "TCP"
          }
        }
      }
    }
  }
}

The new backend requires reconfiguration with terraform init -reconfigure

State has been refreshed successfully and Terraform is now using the remote s3 bucket as the state file backend, with state locking via the DynamoDB table

Sample Node.js Application with PostgreSQL CRUD operations

Thierry Enongene — Thu, 16 Mar 2023 17:27:53 +0000

This article catalogs the development of a sample Node.js application with the express framework, that interacts with a containerized PostgreSql database via CRUD operations. The database is run in a docker container, and I made use of a sample dataset generated with Mockaroo. Th generated dataset is a simple list of users with name, gender and favorite color as properties. The application lists these users in table format, with a button to delete users. There is also a button to add users at the top of the page that redirects to a data entry form for a new user. The application is designed using materialize css for a quick UI design.

PostgreSQL Container and Database

I started by pulling and running the official postgres docker image to my system.

I named the docker container "postgres" and mapped the port 5432 (default postgres port) of the container to the same port on my system (the host). I passed in the default postgres password as admin in the command as well.

With the database application now running in the container on port 5432, I connected to the container using the psql CLI. The psql utility was previously downloaded on my system, but if needed, it can be downloaded via "apt-get" for debian linux systems.
A "postgres" (user) prompt means the connection into the container is successful.

Now I created a database called node_test and switch to it to get out of the default "postgres" database.

On mockaroo.com, I created the sample database table with the properties, and then downloaded the generated file in sql format.

The generated database table as seen in VS Code.

Application UI with EJS and Materialize CSS

For a quick UI, I used materialize css which is a flexible framework for quickly generating UI components and behavior. I used the EJS node package as a templating engine for rendering, with 2 views. One for displaying the table of users, and the other view for data entry of the new user.

Table View

New User entry view

Node.js Server application with Express, EJS and CRUD Operations.

I wrote the nodejs application with express, and also using the 'pg' package to interact with a postgres database. The code also made use of a random number generator to randomly assign a user ID for the database for any new users added. The postgres connection makes use of a "postgres client" with parameters that specify the postgres connection credentials.
For the CRUD operations, I made use of the 'get' and 'post' routes provided by the express framework. An issue of complexity that arose was for the 'delete' operation. Because the actual delete operation is done by the postgres client via the delete query, it was tricky to use use the actual 'delete' method by express. I researched further as to how that could be accomplished but was not successful, so settled on a work-around to treat the delete operation as a 'get' request to the postgres client which could be easily accomplished. I will update this if I eventually find out how to use a the delete method.

Clicking the red delete user button removes from the database, and re-renders the UI with the updated table. Demo of the top 2 users on the above table with 'delete user' clicked, and UI re-rendered (below image) with users removed from the table.

Clicking on the green 'add' button redirects to the '/adduser' route for entry, and then a new user is added to postgres on submit. The UI renders the new updated table. Demo for 2 new users added below.

Conclusion

This was a demonstration on how to run a node.js application that interacts with a containerized postgres database. It highlights the basics of a CRUD operation with a database, and demonstrates the speed, ease and convenience of running containerized applications on docker. On high level production systems, the application is also containerized and both containers can interact with each other in a container orchestration system like docker-compose or kubernetes. The Postgres database would also need to have its data persisted on the host my mapping the volume on the container to a volume on the host.