DEV Community: David WOGLO

From legacy to cloud serverless - Part 4

David WOGLO — Wed, 04 Sep 2024 11:46:09 +0000

Note: This article was originally published on Feb 5, 2024 here. It has been republished here to reach a broader audience.

Hello and welcome to this article in a journey of migrating a legacy-built app to the cloud. In this section, we will focus on three aspects: interfacing the application with Cloud Firestore, automating deployment, and exploring how Binary Authorization can reinforce supply chain security while aligning with security policies.

If you're joining us midway, I encourage you to take a look at the previous articles to get up to speed. Otherwise, let's dive in! 😊

integrating the app with Cloud Firestore

Our previous code interacted with MongoDB. With the migration to Google Cloud, we are transitioning away from MongoDB in favor of Firestore, which is Google Cloud's managed NoSQL document database built for automatic scaling, high performance, and ease of application development. To achieve this, we'll need to make modifications to our code, ensuring that our application seamlessly integrates and functions with Firestore.

We will replace the old MongoDB code with the following Firestore integration:

Old:

from pymongo import MongoClient
from bson.objectid import ObjectId
import mongomock
...
if os.environ.get('TESTING'):
    client = mongomock.MongoClient()
else:
    client = MongoClient(os.environ['MONGO_URI'])
db = client.flask_db
todos = db.todos

New:

from google.auth import compute_engine
from google.cloud import firestore
...
credentials = compute_engine.Credentials()
db = firestore.Client(credentials=credentials)
todos = db.collection('todos')

from google.auth import compute_engine: This line imports the compute_engine module from the google.auth library, which is used for authentication in Google Cloud environments
fromgoogle.cloudimport firestore: This line imports the firestore module from the google.cloud library, enabling interaction with Google Cloud Firestore.
The compute_engine.Credentials() call retrieves the default credentials provided by Google Cloud in its environment. These credentials are essential for authenticating with Firestore. In a local or non-Google Cloud service environment, you would need to generate a service account key before being able to authenticate with Firestore. However, in our case, since the code will be deployed on Cloud Run, authentication will be handled using the default service account of Cloud Run.
todos = db.collection('todos'). Here, we're defining a Firestore collection. Collections are used to organize documents in Firestore.

Data Insertion: When a POST request is made, the new todo item is added to the Firestore collection 'todos' using the add method. The data is stored as a dictionary.

@app.route('/', methods=['GET', 'POST'])
def index():
    if request.method == 'POST':
        content = request.form.get('content')
        degree = request.form.get('degree')
        todos.add({'content': content, 'degree': degree})

Old:

todos.insert_one({'content': content, 'degree': degree})

New:

todos.add({'content': content, 'degree': degree})

This modification reflects the adjustment needed in the code for Firestore, moving from the insert_one method in MongoDB to the add method in Firestore for adding documents.

Data Retrieval: In the new code, we utilize todos.stream() to obtain a stream of documents from the Firestore collection. In the old code, we used todos.find() to get a cursor to the documents in the MongoDB collection.

Old:

all_todos = todos.find()

New:

all_todos = [{'_id': doc.id, **doc.to_dict()} for doc in todos.stream()]

We now use todos.stream() to iterate over documents and convert them to a dictionary format for retrieval. The '_id' field represents the document ID in Firestore.

Data Deletion: In the new code, we employ todos.document(id).delete() to remove a document from the Firestore collection. In the old code, we used todos.delete_one({"_id": ObjectId(id)}) to delete a document from the MongoDB collection.

Old:

todos.delete_one({"_id": ObjectId(id)})

New:

todos.document(id).delete()

The todos.document(id).delete() method is used to delete a specific document by its ID in Firestore.

After all these updates, the new app.py should look like this:

import os
from flask import Flask, render_template, request, url_for, redirect
from google.auth import compute_engine
from google.cloud import firestore

app = Flask(__name__, template_folder='templates')

# Use the default credentials provided by the Cloud Run environment
credentials = compute_engine.Credentials()

# Use these credentials to authenticate with Firestore
db = firestore.Client(credentials=credentials)

todos = db.collection('todos')

@app.route('/', methods=['GET', 'POST'])
def index():
    if request.method == 'POST':
        content = request.form.get('content')
        degree = request.form.get('degree')
        todos.add({'content': content, 'degree': degree})
        return redirect(url_for('index'))

    all_todos = [{'_id': doc.id, **doc.to_dict()} for doc in todos.stream()]
    return render_template('index.html', todos=all_todos)

@app.route('/<id>/delete/', methods=['POST'])
def delete(id):
    todos.document(id).delete()
    return redirect(url_for('index'))

The adjustments include using Firestore methods for data insertion (todos.add()), retrieval (todos.stream()), and deletion (todos.document(id).delete()), along with integrating the appropriate syntax for Firestore operations.

Testing the new code

To ensure the correctness of the new 'app.py' code, we have to update also the testing approach. The tests aim to verify the functionality of critical components, such as data insertion and deletion, within the context of Firestore integration.

import unittest
from unittest.mock import patch, MagicMock
from app import app

class TestApp(unittest.TestCase):
    def setUp(self):
        self.app = app.test_client()

    @patch('app.todos.add')
    def test_index_post(self, mock_add):
        response = self.app.post('/', data={'content': 'Test Todo', 'degree': 'Test Degree'})

        mock_add.assert_called_once_with({'content': 'Test Todo', 'degree': 'Test Degree'})
        self.assertEqual(response.status_code, 302)

    @patch('app.todos.document')
    def test_delete(self, mock_document):
        mock_delete = MagicMock()
        mock_document.return_value.delete = mock_delete

        response = self.app.post('/123/delete/')

        mock_delete.assert_called_once()
        self.assertEqual(response.status_code, 302)

if __name__ == '__main__':
    unittest.main()

Let's delve into the primary components of this testing suite:

Data Insertion Test (test_index_post):

* This test simulates a POST request to the root endpoint ('/') of the application when a new todo item is added.

* The `@patch` decorator is utilized to mock the `todos.add` method, ensuring that actual Firestore interactions are bypassed during testing.

* The test asserts that the 'add' method is called with the expected data, and the response status code is as expected (302 for a successful redirect).

Data Deletion Test (test_delete):

* This test mimics a POST request to the endpoint for deleting a specific todo item ('//delete/').

* The `@patch` decorator is applied to mock the `todos.document` method, and a MagicMock is used to mock the 'delete' method of the Firestore document.

* The test verifies that the 'delete' method is called once and asserts the response status code after the deletion operation (302 for a successful redirect).

These tests ensure that data insertion and deletion operations interact seamlessly with Firestore. The use of mocking allows for isolated testing, focusing on specific components without the need for actual Firestore connections during the testing phase.

Now that the test has been added, you can re-run your Cloud Build pipeline to address any potential minor issues before proceeding with the deployment on Cloud Run.

Automating deployment (CD)

To automate the deployment on Cloud Run after building and pushing the image, add the following step to your Cloud Build configuration (cloudbuild.yaml):

# Step 8: Deploy the image to Cloud Run
- id: 'deploy-image'
  name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: 'gcloud'
  args:
  - 'run'
  - 'deploy'
  - '$_SERVICE_NAME'
  - '--image'
  - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA'
  - '--region'
  - '$_REGION'
  - '--platform'
  - 'managed'
  - '--allow-unauthenticated'
  waitFor: ['push-image']

This segment of the Cloud Build configuration file handles the deployment of the Docker image to Google Cloud Run. Here's a breakdown of each line's purpose:

id: 'deploy-image': Provides a unique identifier for this step within the Cloud Build configuration file.
name: 'gcr.io/google.com/cloudsdktool/cloud-sdk': Specifies the Docker image to be used for this step, which, in this case, is the Google Cloud SDK image.
entrypoint: 'gcloud': Sets the Docker entrypoint to 'gcloud,' the command-line interface for Google Cloud Platform.
args: A list of arguments passed to the 'gcloud' command.
- 'run' 'deploy' '$_SERVICE_NAME': Deploys a new revision of the Cloud Run service identified by $_SERVICE_NAME.
- '--image' '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA': Specifies the Docker image to deploy, located in the designated Google Cloud Artifact Registry repository.
- '--region' '$_REGION': Specifies the region where the Cloud Run service is deployed.
- '--platform' 'managed': Indicates that the Cloud Run service uses the fully managed version of Cloud Run.
- '--allow-unauthenticated': Permits unauthenticated requests to the Cloud Run service.
waitFor: ['push-image']: Directs Cloud Build to wait for the completion of the 'push-image' step before initiating this step.
Afterwards, don't forget to update the substitution section of your Cloud Build configuration to reflect the variables used in this new step.

Now, you can push your code to trigger the pipeline. If the pipeline runs successfully, you will obtain the access link for your application. Navigate to the Google Cloud Console, go to Cloud Run, and click on the name of your newly deployed service to retrieve the access URL for your application.

Binary Authorization

The Binary Authorization is a security control mechanism during the deployment of container images on Google Cloud platforms like Cloud Run, GKE, Anthos Service Mesh, and Anthos Clusters. Its primary function is to either authorize or control the deployment of images that have been attested as secure or trusted. This attestation involves subjecting the image to various processes such as testing, vulnerability scanning, and even manual signatures. Only after the image meets predefined conditions is it considered validated and allowed for deployment on the platforms.

Binary Authorization is responsible for defining and enforcing this policy. The controls or checks are executed using attestators, which can be custom-created, fixed, or generated by tools like Cloud Build (currently in preview).

For this project, I explored configuring Binary Authorization using the built-by-cloud-build attestor to deploy only images built by Cloud Build. With a well-crafted and robust Cloud Build configuration (incorporating various tests, vulnerability analyses, etc.), this approach can significantly save time compared to creating and using a custom attestor. However, as of the time of writing this article, Binary Authorization with cloud build attestor is in preview.

The main challenge with using the built-by-cloud-build attestor is that it is generated only once during the build with Cloud Build. This may not align well with continuous delivery (CD), especially for recurrent executions of the CI/CD pipeline. Ideally, with each new run of the pipeline, a new attestor should be generated to update the Binary Authorization policy. This becomes problematic, especially if the Binary Authorization policy is configured at the organization level, impacting all other deployments. From a personal perspective, to address this, it would be beneficial if Cloud Build generates the attestor once and uses it for subsequent pipeline executions. Currently, the custom attestor provides a workaround for this limitation. However, for simplicity, it would be ideal if Cloud Build handles this process seamlessly.

Follow this link for the setup of Binary Authorization."

This concludes the article. Thank you for reading. You can find the configurations and code for this project in the following Git repository.

From legacy to cloud serverless - Part 3

David WOGLO — Wed, 04 Sep 2024 11:38:11 +0000

Note: This article was originally published on
Dec 25, 2023 here. It has been republished here to reach a broader audience.

Welcome to the third part of this serie! In this segment, we dive into testings and pipeline configuration on Google Cloud, specifically focusing on continuous integration using Cloud Build, On-demand Vulnerability Scanner, and Artifact Registry. You can find the project repository here, or, if you prefer, you can bring your own project.

Let me walk you through the pipeline. With each push to the main branch, Cloud Build is triggered. First, it runs unit tests on the code. If the tests pass, it proceeds to build the image. After the image is built, Cloud Build invokes the image scanner to ensure it's free of vulnerabilities. If all is well, the image is sent and stored in the Artifact Registry, ready for deployment. But for this article, we'll focus solely on the CI part. Let's start with the tests.

Unittest

Here's the code we plan to test

import os
from flask import Flask
from pymongo import MongoClient
from flask import Flask, render_template, request, url_for, redirect
from bson.objectid import ObjectId
import mongomock



app = Flask(__name__, template_folder='templates')

if os.environ.get('TESTING'):
    client = mongomock.MongoClient()
else:
    client = MongoClient(os.environ['MONGO_URI'])


db = client.flask_db
todos = db.todos


@app.route('/', methods=('GET', 'POST'))
def index():
    if request.method=='POST':
        content = request.form['content']
        degree = request.form['degree']
        todos.insert_one({'content': content, 'degree': degree})
        return redirect(url_for('index'))

    all_todos = todos.find()
    return render_template('index.html', todos=all_todos)


@app.post('/<id>/delete/')
def delete(id):
    todos.delete_one({"_id": ObjectId(id)})
    return redirect(url_for('index'))

For an explanation of the code, refer to the first article in this series.

Now, let's move on to the testing phase

The test is written using Python's built-in unittest module, which provides a framework for writing and running tests.

Import necessary modules and create a mock MongoDB instance

The test begins by importing the necessary modules. unittest is the testing framework, patch and MagicMock from unittest.mock are used to replace parts of the system that you're testing with mock objects, and ObjectId from bson.objectid is used to create unique identifiers. The app and todos are imported from the app.py file. mongomock is used to create a mock MongoDB instance for testing, and flask is used to manipulate the request context during testing.
```
import unittest
from unittest.mock import patch, MagicMock
from bson.objectid import ObjectId
from app import app, todos
import mongomock
import flask

mock_db = mongomock.MongoClient().db
```
Define the test case

A test case is defined by creating a new class that inherits from unittest.TestCase. This class will contain methods that represent individual tests.
```
class TestApp(unittest.TestCase):
```
Set up the test environment

The setUp method is a special method that is run before each test. Here, it's used to create a test client instance of the Flask app and enable testing mode.
```
def setUp(self):
    self.app = app.test_client()
    self.app.testing = True
```
Write the test

The test_index_post method is the actual test. It tests the behavior of the app when a POST request is sent to the index route (/).
```
def test_index_post(self):
```
Mock the database operation

The patch function is used to replace the insert_one method of todos with a MagicMock. This allows the test to simulate the behavior of the database operation without actually interacting with a real database.
```
with patch('app.todos.insert_one', new_callable=MagicMock) as mock_insert_one:
```
Create a test request context

A test request context is created for the app using app.test_request_context. This allows the test to simulate a request to the app.
```
with app.test_request_context('/'):
```
Set the request method and form data

The request method is set to 'POST' and the request form data is set to a dictionary with 'content' and 'degree' keys.
```
flask.request.method = 'POST'
flask.request.form = {'content': 'Test Content', 'degree': 'Test Degree'}
```
Send a POST request to the app

A POST request is sent to the app using self.app.post. The form data is passed as the data argument.
```
result = self.app.post('/', data=flask.request.form)
```
Assert the expected results

The assertEqual method is used to check that the status code of the response is 302. The assert_called method is used to check that the insert_one method was called.
```
self.assertEqual(result.status_code, 302)
mock_insert_one.assert_called()
```

This test ensures that when a POST request is sent to the index route with the correct form data, the app responds with a 302 status code and inserts the data into the database.

Your test code should look something like the following:

import unittest
from unittest.mock import patch, MagicMock
from bson.objectid import ObjectId
from app import app, todos
import mongomock
import flask

## Create a mock MongoDB instance
mock_db = mongomock.MongoClient().db

class TestApp(unittest.TestCase):
    def setUp(self):
        # Create a test client instance
        self.app = app.test_client()
        # Enable testing mode. Exceptions are propagated rather than handled by the the app's error handlers
        self.app.testing = True 

    def test_index_post(self):
        # Patch the insert_one method of todos with a MagicMock
        with patch('app.todos.insert_one', new_callable=MagicMock) as mock_insert_one:
            # Create a test request context for the app
            with app.test_request_context('/'):
                # Set the request method to 'POST'
                flask.request.method = 'POST'
                # Set the request form data
                flask.request.form = {'content': 'Test Content', 'degree': 'Test Degree'}
                # Send a POST request to the app
                result = self.app.post('/', data=flask.request.form)
                # Assert that the status code of the response is 302
                self.assertEqual(result.status_code, 302)
                # Assert that the insert_one method was called
                mock_insert_one.assert_called()

Now, to execute the test, set the environment variable TESTING=true, Setting TESTING=True switches the application to use a mock MongoDB client for testing, instead of the real MongoDB database.

Now, if your test is successful, let's move on to configuring Cloud Build.

Cloud Build setup

Follow the guide to connect Cloud Build to your repository and this one for initial configurations.

Once that's done, let's move on to writing the Cloud Build configuration file, where we'll instruct it on how to execute the pipeline, the steps involved, dependencies, and so on.

Cloud Build config file

The Cloud Build Config file is written in YAML, a human-readable data serialization language.

Here are the main sections of our config file:

Substitutions: These are user-defined variables that can be replaced in the Cloud Build configuration file. They are defined under the substitutions key. In this case, _REGION, _REPOSITORY, _IMAGE, and _SEVERITY are defined.
```
substitutions:
  _REGION: us-central1
  _REPOSITORY: from-legacy-to-cloud
  _IMAGE: from-legacy-to-cloud
  _SEVERITY: '"CRITICAL|HIGH"'
```
Steps: These are the operations that Cloud Build will perform. Each step is a separate action and they are executed in the order they are defined.

* **Step 0: Install test dependencies**: This step uses a Python 3.10 Docker image to install the test dependencies listed in `docker/requirements-test.txt`. The `entrypoint` is set to `/bin/bash`, which means that the command that follows will be executed in a bash shell. The `args` key specifies the command to be executed, which in this case is a pip install command. The `-c` flag tells bash to read commands from the following string. The `|` character allows us to write multiple commands, which will be executed in order.

    ```yaml
    - name: 'python:3.10-slim'
      entrypoint: '/bin/bash'
      args:
        - '-c'
        - |
          pip install --user -r docker/requirements-test.txt
      id: 'install-test-dependencies'
    ```

* **Step 1: Run unit tests**: This step also uses a Python 3.10 Docker image to run the unit tests defined in [`test.py`](http://test.py). The `export TESTING=True` command sets an environment variable `TESTING` to `True`, which can be used to change the behavior of the application during testing. The `cd docker` command changes the current directory to `docker`, where the test file is located. The `python -m unittest` [`test.py`](http://test.py) command runs the unit tests in [`test.py`](http://test.py).

    ```yaml
    - name: 'python:3.10-slim'
      entrypoint: '/bin/bash'
      args:
        - '-c'
        - |
          export TESTING=True
          cd docker 
          python -m unittest test.py
      id: 'run-tests'
    ```

* **Step 2: Build the Docker image**: This step uses the `docker` Cloud Builder to build a Docker image from the Dockerfile located in the `docker/` directory. The image is tagged with the commit SHA. The `waitFor` key is used to specify that this step should wait for the `run-tests` step to complete before it starts. The `args` key specifies the command to be executed, which in this case is a docker build command. The `-t` flag is used to name and optionally tag the image in the 'name:tag' format.

    ```yaml
    - name: 'gcr.io/cloud-builders/docker'
      args: ['build', '-t', '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA', 'docker/']
      waitFor: ['run-tests']
      id: 'build-image'
    ```

Step 3: Inspect the Docker image and write the digest to a file: This step uses the docker Cloud Builder to inspect the Docker image and write the image digest to a file. The image digest is a unique identifier for the image. The docker image inspect command retrieves detailed information about the Docker image. The --format option is used to format the output using Go templates. The {{index .RepoTags 0}}@{{.Id}} template retrieves the first tag of the image and the image ID. The > operator redirects the output to a file. The && operator is used to execute the cat command only if the previous command succeeded.
```
- name: 'gcr.io/cloud-builders/docker'
  entrypoint: '/bin/bash'
  args:
    - '-c'
    - |
      docker image inspect $_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA --format '{{index .RepoTags 0}}@{{.Id}}' > /workspace/image-digest.txt &&
      cat /workspace/image-digest.txt
  id: 'inspect-image'
```
Step 4: Scan the Docker image for vulnerabilities: This step uses the cloud-sdk Cloud Builder to scan the Docker image for vulnerabilities. The scan ID is written to a file. The gcloud artifacts docker images scan command scans the Docker image for vulnerabilities. The --format='value(response.scan)' option is used to retrieve the scan ID from the response. The > operator redirects the output to a file.
```
- id: scan
  name: gcr.io/google.com/cloudsdktool/cloud-sdk
  entrypoint: /bin/bash
  args:
  - -c
  - |
    gcloud artifacts docker images scan $_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA \
    --format='value(response.scan)' > /workspace/scan_id.txt
```
Step 5: Check the severity of any vulnerabilities found: This step uses the cloud-sdk Cloud Builder to list the vulnerabilities found in the Docker image and check their severity. If any vulnerabilities with a severity matching _SEVERITY are found, the build fails. The gcloud artifacts docker images list-vulnerabilities command lists the vulnerabilities found in the Docker image. The --format='value(vulnerability.effectiveSeverity)' option is used to retrieve the severity of each vulnerability. The grep -Exq $_SEVERITY command checks if any of the severities match _SEVERITY. The echo command prints a message and the exit 1 command terminates the build if a match is found.
```
- id: severity check
  name: gcr.io/google.com/cloudsdktool/cloud-sdk
  entrypoint: /bin/bash
  args:
  - -c
  - |
    gcloud artifacts docker images list-vulnerabilities $(cat /workspace/scan_id.txt) \
    --format='value(vulnerability.effectiveSeverity)' | if grep -Exq $_SEVERITY; \
    then echo 'Failed vulnerability check' && exit 1; else exit 0; fi
```
Step 6: Push the Docker image to Google Cloud Artifact Registry: This step uses the docker Cloud Builder to push the Docker image to the Google Cloud Artifact Registry. The waitFor key is used to specify that this step should wait for the severity check step to complete before it starts. The docker push command pushes the Docker image to a repository.
```
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA']
  id: 'push-image'
  waitFor: ['severity check']
```
Images: This key specifies the Docker images that Cloud Build should build and push to the Google Cloud Artifact Registry. In this case, it's the Docker image built in Step 2.
```
images:
- '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA'
```

This cloudbuild.yaml file defines a complete CI/CD pipeline for our application. It installs test dependencies, runs unit tests, builds a Docker image, inspects the image, scans the image for vulnerabilities, checks the severity of any vulnerabilities found, and pushes the image to the Google Cloud Artifact Registry. This pipeline ensures that the application is tested, secure, and ready for deployment.

The complete config file should look like this:

substitutions:
  _REGION: us-central1
  _REPOSITORY: from-legacy-to-cloud
  _IMAGE: from-legacy-to-cloud
  _SEVERITY: '"CRITICAL|HIGH"'

steps:
# Step 0: Install test dependencies
- id: 'install-test-dependencies'
  name: 'python:3.10-slim'
  entrypoint: '/bin/bash'
  args:
    - '-c'
    - |
      pip install --user -r docker/requirements-test.txt

# Step 1: Run unit tests
- id: 'run-tests'
  name: 'python:3.10-slim'
  entrypoint: '/bin/bash'
  args:
    - '-c'
    - |
      export TESTING=True
      cd docker 
      python -m unittest test.py

# Step 2: Build the Docker image
- id: 'build-image'
  name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA', 'docker/']
  waitFor: ['run-tests']

# Step 3: Inspect the Docker image and write the digest to a file.
- id: 'inspect-image'
  name: 'gcr.io/cloud-builders/docker'
  entrypoint: '/bin/bash'
  args:
    - '-c'
    - |
      docker image inspect $_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA --format '{{index .RepoTags 0}}@{{.Id}}' > /workspace/image-digest.txt &&
      cat /workspace/image-digest.txt

# Step 4: Scan the Docker image for vulnerabilities
- id: scan
  name: gcr.io/google.com/cloudsdktool/cloud-sdk
  entrypoint: /bin/bash
  args:
  - -c
  - |
    gcloud artifacts docker images scan $_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA \
    --format='value(response.scan)' > /workspace/scan_id.txt

# Step 5: Check the severity of any vulnerabilities found
- id: severity check
  name: gcr.io/google.com/cloudsdktool/cloud-sdk
  entrypoint: /bin/bash
  args:
  - -c
  - |
    gcloud artifacts docker images list-vulnerabilities $(cat /workspace/scan_id.txt) \
    --format='value(vulnerability.effectiveSeverity)' | if grep -Exq $_SEVERITY; \
    then echo 'Failed vulnerability check' && exit 1; else exit 0; fi

# Step 6: Push the Docker image to Google Cloud Artifact Registry
- id: 'push-image'
  name: 'gcr.io/cloud-builders/docker'
  args: ['push', '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA']
  waitFor: ['severity check']

images:
- '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPOSITORY/$_IMAGE:$COMMIT_SHA'

View build results

Now, commit and push your changes. If the Cloud Build triggers are configured correctly, the build should be triggered. Connect to the Google Cloud Console, go to Cloud Build > History to view your builds.

If it fails, click on it to see the error messages and troubleshoot to resolve the issues. Once the build succeeds, you can access the Artifact Registry and see the stored image, ready for use.

What next ?

Well, that wraps up this article. In the next one, we'll delve into automating deployments—the CD part. After vulnerability scanning of container images, we'll be putting security policies in place through Binary Authorization, allowing only approved/trusted images to be deployed on Cloud Run. But before that, we'll migrate our Mongo database to Google Firestore. After that, we'll deploy our app on Cloud Run and connect it to Firestore to make it fully operational.

See you in the next article. Until then, I'm available on social media (I'm more active on LinkedIn) for any information or additional suggestions. Thanks for reading!

From legacy to cloud serverless - Part 2

David WOGLO — Wed, 04 Sep 2024 11:26:13 +0000

Note: This article was originally published on Nov 12, 2023 here. It has been republished here to reach a broader audience.

Hey, how's it been since the last article? If you haven't had a chance to check out the previous installment in the series, I invite you to discover it here. Perhaps you've already tackled something similar to what was described in the previous article, and this one seems to be a good resource to continue your project. Welcome aboard!

In this article, we'll be transforming Docker Compose services into Kubernetes objects and deploying them in a Kubernetes environment.

To follow along, you'll need some knowledge of Kubernetes, have completed the lab described in the previous article, or have done something similar, make sure you have a Kubernetes environment ready. As of now, I'm using Digital Ocean's Kubernetes Engine. I mention 'as of now' because if you've been here from the beginning, you're probably aware that our project's ultimate goal isn't just deploying on K8s. It's a journey of migrating a traditional app to a serverless cloud setup. The next step in this series will involve migrating to Google Cloud. Oh, did I forget to mention? I'm all about Google Cloud—I recently even snagged my Professional Cloud Architect certification. So, expect Google Cloud to pop up regularly in my discussions, and the rest of this series will be purely GCP-focused.

Enough chatter, let's dive into the real stuff!

Build the application image and push it to the Docker registry

If you haven't done so already, I invite you to clone our project's repo here. Navigate to the docker folder, where all the Docker-related elements of the project are stored. Explore the content a bit, and once you're ready, come back, and let's continue. If you don't have a Docker Hub account yet, I recommend creating one.

Now, in your terminal, log in with docker login using your Docker Hub account information. After that, build the image, tagging it with your username and the image name.

docker build -t <username>/<image-name> .

Finally, push the image to Docker Hub.

docker push <username>/<image-name>

Export MongoDB data

As part of our migration process, it's crucial to ensure we retain our data. To achieve this, let's export the data stored in the MongoDB container that we'll later use when deploying MongoDB on Kubernetes.

Export the existing MongoDB database from the Docker Compose setup:

Access the MongoDB database container shell.

docker exec -it <mongo_db_service> bash

Export all data from the MongoDB database.
```
mongodump <file name>
```
Exit the MongoDB database container shell.
```
exit
```
Copy the 'dump' folder from the MongoDB container to a specified destination.
```
docker cp <mongo_db_service>:/dump <destination>
```

Install MongoDB on Kubernetes

Now, while connected to the Kubernetes cluster, let's install MongoDB using Helm:

helm install mongo-helm oci://registry-1.docker.io/bitnamicharts/mongodb --set auth.rootUser=root,auth.rootPassword="defineYourRootPassword"

This command leverages Helm, a Kubernetes package manager, to install MongoDB from a chart hosted on Docker's registry.

The part --set auth.rootUser=root,auth.rootPassword="DefineYourPassword" specifies the username and password for the MongoDB root user.

Make sure to save the output of this command; we'll be using it to construct the database connection URI.

Verify that everything is installed correctly with the following commands:

kubectl get pods
kubectl get services

Restore data

It's time to restore the database:

Navigate to MongoDB Kubernetes pod

kubectl exec -it --namespace default mongodb_pod -- /bin/bash
mongosh
use admin
db.auth('root', 'password')

Create a non-root MongoDB user:

db.createUser({
  user: 'username',
  pwd: 'password',
  roles: [
    { role: 'readWriteAnyDatabase', db: 'admin' },
    { role: 'dbAdminAnyDatabase', db: 'admin' },
    { role: 'clusterAdmin', db: 'admin' }
  ]
})
exit

Restore the Docker Compose database dump to the new MongoDB pod:

Copy the database dump folder previously copied into the MongoDB pod:

kubectl cp <mongodb_dump_location_filename> <mongodb_pod>:/tmp/

Navigate to the MongoDB pod shell:

kubectl exec -it --namespace default mongodb_pod -- /bin/bash

Change the directory to the dump directory and list all MongoDB folders to verify the contents:

cd /tmp/dump
ls

Restore the app database:

mongorestore --uri="mongodb://username:password@localhost:27017/?authSource=admin" app_db -d app_db

Here's how it's formed:

mongodb://: This is the prefix to identify that we're connecting to a MongoDB instance.
<username>:<password>@: This part specifies the username and password to connect to the MongoDB instance. You would replace <username> and <password> with the actual username and password. In your case, the username is the one created earlier.
mongo-helm-mongodb.default.svc.cluster.local:27017: This is the host and port where the MongoDB server is running. mongo-helm-mongodb.default.svc.cluster.local is the DNS name for the MongoDB service in your Kubernetes cluster, and 27017 is the default port for MongoDB.

Exit the MongoDB pod shell:

exit

The new MongoDB is now ready for use.

Deploy and connect the application to the database

First, let's create the Kubernetes secret that will contain the connection string for the database. We're using the secret object because our connection string contains sensitive information. Kubernetes provides the secret object precisely for scenarios like this. If it were just configuration information or environment variables, a ConfigMap object would be more suitable.

apiVersion: v1
kind: Secret
metadata:
  name: app-secret
type: Opaque
data:
  mongo-uri: <base64-encoded-mongo-uri>

Create a YAML file and paste this content into it. Name the file as you see fit. Note that the mongodb-uri field under data should contain the base64-encoded MongoDB URI. Replace the placeholder with the actual base64-encoded connection string.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-to-cloud-deployment
  labels:
    app: legacy-to-cloud
spec:
  replicas: 3
  selector:
    matchLabels:
      app: legacy-to-cloud
  template:
    metadata:
      labels:
        app: legacy-to-cloud
    spec:
      containers:
      - name: legacy-to-cloud
        image: docker_username/image_name:tag
        ports:
        - containerPort: 5000
        env:
        - name: MONGO_URI
          valueFrom:
            secretKeyRef:
              name: mongodb-uri-secret
              key: mongodb-uri

---
apiVersion: v1
kind: Service
metadata:
  name: legacy-to-cloud-service
spec:
  type: LoadBalancer
  selector:
    app: legacy-to-cloud
  ports:
  - protocol: TCP
    port: 80
    targetPort: 5000

Create a second YAML file for your application's manifest and paste this content into it. The env section of the container in the Deployment references the MongoDB URI from the secret we created earlier. Ensure that the secret name and key match the values used in the secret manifest. Also, ensure that the selector in the Service matches the one in the Deployment. This is crucial for linking the pods to the service.

If everything looks good, let's proceed with deploying our application. You can use the following command to validate the syntax of your YAML file and perform a dry run:

kubectl apply -f filename.yaml --dry-run=client --validate=true

This command checks the syntax of your YAML file and prints out the resources that would be created or modified without actually applying the changes. If there are any syntax errors, this command will highlight them.

If everything is okay, create the resources with the following command:

kubectl apply -f filename1.yaml -f filename2.yaml

Replace filename1.yaml and filename2.yaml with the actual names of your YAML files.

Get the access IP address with the command:

kubectl get svc

Identify the service for your application and copy its external IP. Paste it into your browser to access the application.

Well, that wraps up this section on the migration to Kubernetes.

A little gift for the road?

Haha, did you know there's a tool to speed things up? Because here, we've created YAML manifests to deploy K8s resources. This deployment is a simple one, but imagine if it were a massive deployment with hundreds of Docker Compose services, unimaginable complexities, etc. Would we sit down and manually create manifests for all that complexity? Of course not :) Enter Kompose. Kompose is a conversion tool for Docker Compose to container orchestrators like Kubernetes. It takes a Docker Compose file and translates it into Kubernetes resources.

Kompose is a handy tool for those familiar with Docker Compose but aiming to deploy their application on Kubernetes. It automates the creation of Kubernetes deployments, services, and other resources based on the services defined in the Docker Compose file.

However, it's worth noting that not all Docker Compose features and options are supported by Kompose, so some manual tweaking of the generated Kubernetes resources might be necessary. Here's an excellent guide that addresses our use case well.

What next?

And that's a wrap for this article! In the next one, we're heading to the GOOGLE CLOUUUUUUUD :) and beginning to introduce DevOps tools and practices to automate and speed up our work. We're talking about stepping up the game. We'll be using Google Cloud DevOps tools—Cloud Build for CI/CD, Artifact Registry for container images, GKE for deployments. Plus, we'll dive into DevSecOps tools and practices, leveraging the security available within the Google Cloud ecosystem.

Thanks for reading, and see you soon in the next article in the series!"

From legacy to cloud serverless - Part 1

David WOGLO — Wed, 04 Sep 2024 11:17:51 +0000

Note: This article was originally published on Nov 4, 2023 here. It has been republished here to reach a broader audience.

Welcome to the first article in a series that will walk you through the process of migrating a legacy app from on-premises to the cloud, with a focus on modernization, serverless platforms, and integrated DevOps practices.

In this article, we will focus on containerizing your app. However, if you're building an app from scratch, that's perfectly fine (in fact, it's even better). For this example, I'm using this DigitalOcean guide to build a simple TODO app using Python (Flask) and MongoDB as the database. I've made some customizations to make it look better, but the main point is to build something that uses a NoSQL document-based database, as this will be required for the upcoming work.

You can clone the repository of the app here on GitHub if you haven't built your own.

Once you have your app built, let's get started!

Dockerfile

Here is the structure of the application directory that we will containerize, followed by the Dockerfile.

.
├── app.py
├── LICENSE
├── README.md
├── requirements.txt
├── static
│   └── style.css
└── templates
    └── index.html

The app.py file is the main application file that contains the Flask app code. The requirements.txt file contains the list of Python dependencies required by the application. The static/ directory contains static files such as CSS, JavaScript, and images. The templates/ directory contains the HTML templates used by the Flask app.

# Use a minimal base image
FROM python:3.9.7-slim-buster AS base

# Create a non-root user
RUN useradd -m -s /bin/bash flaskuser
USER flaskuser

# Set the working directory
WORKDIR /app

# Copy the requirements file and install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Add the directory containing the flask command to the PATH
ENV PATH="/home/flaskuser/.local/bin:${PATH}"

# Use a multi-stage build to minimize the size of the image
FROM base AS final

# Copy the app code
COPY app.py .
COPY templates templates/
COPY static static/

# Set environment variables
ENV FLASK_APP=app.py
ENV FLASK_ENV=production

# Expose the port
EXPOSE 5000

# Run the app
CMD ["flask", "run", "--host=0.0.0.0"]

Here's a walkthrough and breakdown of the Dockerfile:

The Dockerfile starts with a FROM instruction that specifies the base image to use. In this case, it's python:3.9.7-slim-buster, which is a minimal base image that includes Python 3.9.7 and some essential libraries.
The next instruction creates a non-root user named flaskuser using the RUN and useradd commands. This is a security best practice to avoid running the container as the root user.
The WORKDIR instruction sets the working directory to /app, which is where the application code will be copied.
The COPY instruction copies the requirements.txt file to the container's /app directory.
The RUN instruction installs the dependencies listed in requirements.txt using pip. The --no-cache-dir option is used to avoid caching the downloaded packages, which helps to keep the image size small.
The ENV instruction adds the directory containing the flask command to the PATH environment variable. This is necessary to run the flask command later.
The FROM instruction starts a new build stage using the base image defined earlier. This is a multi-stage build that helps to minimize the size of the final image.
The COPY instruction copies the application code (app.py), templates (templates/), and static files (static/) to the container's /app directory.
The ENV instruction sets the FLASK_APP and FLASK_ENV environment variables. FLASK_APP specifies the name of the main application file, and FLASK_ENV sets the environment to production.
The EXPOSE instruction exposes port 5000, which is the default port used by Flask.
The CMD instruction specifies the command to run when the container starts. In this case, it runs the flask run command with the --host=0.0.0.0 option to bind to all network interfaces.

With this Dockerfile, the application can be containerized and executed. However, it's important to note that our app requires a database to store the data created or generated while it's running. Of course, you could separately pull a MongoDB database image and run it independently. Then, make adjustments on both sides to establish communication between the two containers so that the app can successfully store data in the database. While this approach works, it may consume time and be a bit tedious. To streamline the process, we will instead move forward with Docker Compose. In Docker Compose, everything is declared in a YAML file, and by using the docker-compose up command, we can start and operate the different services seamlessly, saving time and effort.

Streamlining Database Integration with Docker Compose

Here is the basic Docker Compose YAML file that we will use to streamline the process.

version: '3.9'

services:
  db:
    image: mongo:4.4.14
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db

  web:
    build: .
    container_name: "myflaskapp"
    ports:
      - "5000:5000"
    environment:
      - MONGO_URI=mongodb://db:27017
    depends_on:
      - db

volumes:
  mongo-data:

This Docker Compose YAML file is configured to set up two services: a MongoDB database (db) and a web application (web). Here's a breakdown:

Version: Specifies the version of the Docker Compose file format being used (3.9 in this case).
Services:
- Database (db):
  - Uses the MongoDB version 4.4.14 image.
  - Maps the host port 27017 to the container port 27017.
  - Utilizes a volume named mongo-data to persistently store MongoDB data.
- Web Application (web):
  - Builds the Docker image from the current directory (.).
  - Sets the container name as "myflaskapp."
  - Maps the host port 5000 to the container port 5000.
  - Defines an environment variable MONGO_URI with the value mongodb://db:27017, establishing a connection to the MongoDB service.
  - Specifies a dependency on the db service, ensuring that the database is started before the web service.
Volumes:
- Defines a volume named mongo-data for persisting MongoDB data.

In summary, this Docker Compose file orchestrates the deployment of a MongoDB database and a Flask web application, ensuring they can communicate and function together seamlessly.

Now navigate to the directory with the Docker Compose file and run docker-compose up to start MongoDB and a Flask web app. Access the app at http://localhost:5000 to ensure everything works as expected.

To stop, use docker-compose down.

All good? Next up: migrating the workflow to Kubernetes in the next article.

Continuous Deployment to Kubernetes with ArgoCD

David WOGLO — Fri, 07 Jun 2024 17:43:01 +0000

Continuous deployment (CD) is the process of automatically deploying changes to production. It is a key part of the DevOps toolchain, and it can help organizations to improve their software delivery speed, reliability, and security.

ArgoCD is a Kubernetes-native CD tool that can help you to automate the deployment of your applications to Kubernetes. It is a declarative tool, which means that you can define the desired state of your applications in a Git repository. ArgoCD will then automatically synchronize the actual state of your applications with the desired state.

ArgoCD is a powerful tool that can help you to improve your CD process. It is easy to use, and it can be integrated with a wide range of other tools. If you are looking for a way to automate the deployment of your applications to Kubernetes, then ArgoCD is a great option.

In this blog post, we will explore the process of setting up continuous integration (CI) using GitHub Actions, and then we will delve into configuring ArgoCD to handle the continuous deployment (CD) aspect.

Why argoCD ?

For a brief overview of the benefits and reasons for using ArgoCD, I recommend checking out my LinkedIn post on the subject. In the post, I discuss the key advantages of leveraging ArgoCD and provide valuable insights into how it can enhance your deployment process. Click below to access the LinkedIn post and gain a quick understanding of why ArgoCD is a valuable tool for your software development and deployment needs

David W. on LinkedIn: #kubernetes #cicd #argocd #devops #continuousdeployment #gitops…

Why ArgoCD is My Preferred Tool for Continuous Deployment on Kubernetes ? 🚀 In a typical CI/CD flow using commons tools like Jenkins, Gitlab CI/CD, or GitHub…

linkedin.com

Requirements

Installed kubectl command-line tool.
Have a Kubernetes cluster and a kubeconfig file. The default location for the kubeconfig file is ~/.kube/config. If you don't have a Kubernetes cluster set up, you can follow this guide to quickly bootstrap Minikube.
A GitHub account.

Setting Up Continuous Integration (CI) Using GitHub Actions

For this activity, we will use a simple web application written in Python and utilizing Flask. The application has been specifically designed with cloud demonstrations and containers in mind.

To obtain the application code, you can fork this Github repository to your own Github account and then clone it to your local machine to start making changes and customizations as needed.

To create the workflows instruction for GitHub Actions, you'll need to create a YAML file following a specific structure. Start by creating a file named main.yml inside the .github/workflows directory of your repository. This file will serve as the configuration file for workflows. By following this standardized structure, you'll be able to define and customize the actions, triggers, and steps that make up your CI/CD pipeline.

Let's start the workflow configuration with the following structure:

name: ArgoCD demo Build

on:
  push:
    branches:
      - "main"
  pull_request:

In this configuration, we've named the workflow as "ArgoCD Demo Build". It will be triggered on both push events to the "main" branch and pull requests. The workflow will run on an "ubuntu-latest" virtual machine. This setup forms the foundation of the workflow.

jobs:
  test:
    name: 'Test'
    runs-on: ubuntu-latest
    steps:
      - 
        name: Checkout
        uses: actions/checkout@v2

      -
        name: Run tests
        run: make test

Above, we define a job called "Test" that will run on the latest Ubuntu environment (ubuntu-latest).

The "Checkout" step ensures that the repository's code is available by using the actions/checkout@v2 action.
The "Run tests" step executes the command make test to run the tests.

  build:  
    name: 'Build & Push to Docker Hub'
    runs-on: ubuntu-latest
    needs: test
    steps:
      - 
        name: Checkout
        uses: actions/checkout@v2

      -
        name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      -
        name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: ${{ secrets.DOCKERHUB_USERNAME }}/image-name:tag

The next job is "Build & Push to Docker Hub," which also runs on the ubuntu-latest environment.

The "Checkout" step ensures that the repository's code is available by using the actions/checkout@v2 action.
The "Login to Docker Hub" step authenticates with Docker Hub using the credentials that should be defined in the repository secrets in the GitHub repository settings.
The "Set up Docker Buildx" step uses the docker/setup-buildx-action@v2 action to set up Docker Buildx for building the Docker image.
Finally, the "Build and push" step uses the docker/build-push-action@v4 action to build the Docker image based on the specified Dockerfile and push it to Docker Hub. Make sure to modify the tags field to match your desired image name and version. And also add credentials to the repository secret before moving on.

Once everything is in place, you can initiate the workflow by pushing your changes to the repository. This action will automatically trigger the workflow to start. To monitor and gain insights into the workflow execution, navigate to the "Actions" tab in GitHub. Here, you'll be able to view the workflow status, check the progress of each step, and identify any errors encountered. If any issues arise, carefully review the error messages provided and make the necessary fixes before proceeding to the next part, which involves setting up the continuous deployment (CD) using ArgoCD.

Setting Up Continuous Deployment (CD) with ArgoCD

In this section, we will explore the process of setting up continuous deployment (CD) using ArgoCD. Building upon the foundation of continuous integration (CI) we established earlier with GitHub Actions, we will now focus on automating the deployment of our application to Kubernetes cluster.

You have the flexibility to utilize any Kubernetes (k8s) cluster at your disposal, whether it's a cloud-based cluster, a bare-metal setup, or even local environments such as Minikube or MicroK8s. ArgoCD is compatible with various Kubernetes configurations, allowing you to seamlessly integrate it into your preferred infrastructure. This versatility enables you to leverage your existing infrastructure or choose a setup that best suits your needs for continuous deployment (CD) with ArgoCD.

To proceed further, we will be utilizing Minikube for our setup. Minikube provides a convenient and lightweight way to run a single-node Kubernetes cluster locally.

Now, let's proceed with the installation of ArgoCD. We will walk through the steps to set up ArgoCD on your chosen Kubernetes cluster, in this case, Minikube.

Installing ArgoCD

To install ArgoCD on your Kubernetes cluster, execute the following commands:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

The first command creates a namespace called "argocd" where ArgoCD will be installed. The second command applies the ArgoCD installation manifest, which can be accessed from the official ArgoCD GitHub repository. By executing these commands, you will initiate the installation process and set up ArgoCD within your cluster.

Once the installation is completed, you can verify the installation status by running the following command:

kubectl get nodes -n argocd

This command will display the nodes in the "argocd" namespace, confirming that ArgoCD is successfully installed.

To access the ArgoCD web interface, you can use kubectl port-forwarding to connect to the API server. Execute the following command:

kubectl port-forward svc/argocd-server -n argocd 8080:443

This command will create a port-forwarding tunnel, allowing you to access the ArgoCD UI locally at https://localhost:8080. Simply open a web browser and navigate to the provided URL to access the ArgoCD interface.

To log in to the ArgoCD UI, you will need to retrieve the password from the argocd-initial-admin-secret secret. Follow these steps:

Retrieve the secret by executing the following command:

kubectl get secret argocd-initial-admin-secret -n argocd -o yaml

The output will include a field called data, which contains the base64-encoded password. Copy the value associated with the password key.
Decode the password using the echo and base64 commands. Replace encodedpassword in the command below with the copied value:

echo encodedpassword | base64 --decode

The decoded password will be displayed in the terminal. Copy the password string.
Return to the ArgoCD UI login page. Enter admin as the username and paste the decoded password into the password field.

Currently, ArgoCD is empty as we haven't configured any applications yet. Let's proceed with configuring ArgoCD to connect to a GitHub repository where our deployment files will be hosted.

It's important to note that in best practices, it is recommended to separate the application repository from the deployment repository. However, for the purpose of this activity, we will keep the deployment files alongside the application files. Please keep in mind that this is not a recommended practice for production-ready environments. In such scenarios, it is crucial to separate the two repositories to ensure a more organized and manageable deployment workflow.

Configuring ArgoCD

To configure ArgoCD to connect to your GitHub repository and deploy your application,

Create a YAML file, such as argocd-config.yaml, and add the following content:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-cd-demo
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://github.com/davWK/argoCD-demo.git
    targetRevision: HEAD
    path: deploy/kubernetes/
  destination: 
    server: https://kubernetes.default.svc
    namespace: demo-app-for-argo-cd

  syncPolicy:
    automated:
      selfHeal: true
      prune: true

Now, let's break down what each section of the YAML file does:

metadata: Specifies the metadata for the ArgoCD application, including its name and namespace.
spec.project: Specifies the project within ArgoCD where the application belongs. In this case, it is set to the default project.
source: Defines the source repository details:
- repoURL: Specifies the URL of the GitHub repository where your application's deployment files are hosted.
- targetRevision: Specifies the target revision of the repository to deploy. Here, it is set to HEAD, meaning the latest revision.
- path: Specifies the path within the repository where your application's Kubernetes deployment files are located. the path ArgoCD will track for any modification
destination: Specifies the destination details for the deployment:
- server: Specifies the URL of the Kubernetes API server. Here, it is set to https://kubernetes.default.svc. It can be an external cluster
- namespace: Specifies the target namespace in which the application will be deployed. In this case, it is set to demo-app-for-argo-cd.
syncPolicy: Defines the synchronization policy for the application:
- automated: Specifies that the synchronization should be automated, enabling self-healing and pruning capabilities.
  - selfHeal: Enables self-healing, ensuring the application stays in the desired state.
  - prune: Enables pruning, removing any resources that are no longer defined in the deployment files.

Save the file and apply the configuration by running the following command:

kubectl apply -n argocd -f argocd-config.yaml

By applying this configuration, ArgoCD will establish a connection to the specified GitHub repository, fetch the deployment files from the specified path, and deploy the application to the designated namespace within the Kubernetes cluster.

Once you apply the configuration using the command kubectl apply -n argocd -f argocd-config.yaml, you will no longer need to manually apply any changes to your Kubernetes files. ArgoCD takes over the responsibility of tracking and applying changes automatically.

After the initial deployment, ArgoCD continuously monitors the specified GitHub repository and the Kubernetes files within it. Whenever there is a change detected in the repository, ArgoCD will automatically apply those changes to your Kubernetes cluster. This ensures that your application remains up-to-date with the latest version defined in the repository.

With ArgoCD in place, you can focus on making changes to your application's deployment files in the repository, and ArgoCD will handle the synchronization and deployment to the Kubernetes cluster for you. This simplifies the deployment process and provides a seamless experience for maintaining the desired state of your applications.

Writing the python app deployment file for kubernetes

At this stage, the configuration will be created in ArgoCD, but no application pods or services will be available. This is because we have not yet defined the Kubernetes deployment manifest that contains the deployment information for our Python demo app. However, once this manifest is in place, ArgoCD will automatically apply it, resulting in the deployment of the application.

To proceed, you need to create the Kubernetes deployment manifest file that describes the desired state of your application, such as the container image, ports, and any other necessary configurations. Once you have the deployment manifest ready, commit and push it to your GitHub repository.

ArgoCD will then detect the changes in the repository and automatically apply the deployment manifest, triggering the creation of the corresponding pods and services. This automatic synchronization ensures that the deployed application aligns with the desired state defined in the deployment manifest.

To proceed with defining the Kubernetes deployment manifest for the Python demo app:

Inside the deploy/kubernetes directory, create a new deployment.yaml.
Open the deployment.yaml file and add the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-app-deployment
  labels:
    app: python-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: python-app
  template:
    metadata:
      labels:
        app: python-app
    spec:
      containers:
      - name: image-name
        image: imageurl
        ports:
        - containerPort: 5000
---
apiVersion: v1
kind: Service
metadata:
  name: python-app-service
spec:
  type: NodePort
  selector:
    app: python-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 5000
      nodePort: 30000

Save the file.

This deployment manifest defines a Kubernetes Deployment and Service for the Python app. It specifies the container image, ports, replicas, and other necessary configurations.

The Deployment creates three replicas of the Python app pods.
The Service exposes the app using a NodePort type, making it accessible on port 30000 of the cluster nodes.

Commit and push the deployment.yaml file to your GitHub repository. ArgoCD will automatically detect the changes and apply the deployment manifest, leading to the creation of the Python app deployment and service.

Once the synchronization is complete, you should see the app pods running and the service available for access.

To access the deployed Python app:

Run the following command to get the service information:

kubectl get svc -n <namespace for the Python app>

Replace <namespace for the Python app> with the actual namespace where your Python app is deployed. This command will provide you with the details of the service, including its name, type, cluster IP, and port.

Once you have the service information, run the following command to set up port forwarding:

kubectl port-forward -n <namespace for the Python app> svc/python-app-service 8083:<service port>

Replace <namespace for the Python app> with the actual namespace where your Python app is deployed, and <service port> with the port number specified in your service configuration (e.g., 80).

This command establishes a connection between your local machine and the Python app service running in the Kubernetes cluster. It forwards traffic from your local port 8083 to the specified service port.

Now, you can access the deployed Python app by opening a web browser and navigating to http://localhost:8083. This will direct your requests to the Python app service running in the Kubernetes cluster.

Conclusion

In conclusion, setting up Continuous Integration (CI) and Continuous Deployment (CD) processes are crucial for efficient software development and deployment. In this article, we explored the steps to configure CI using GitHub Actions and CD with ArgoCD. By integrating these tools into your workflow, you can automate the build, test, and deployment processes, leading to faster and more reliable software delivery.

To learn more about ArgoCD and its capabilities, you can refer to the official ArgoCD documentation available here. The documentation provides comprehensive information, including installation guides, usage examples, and advanced configurations.

For a practical demonstration and understanding of ArgoCD, you can watch the "ArgoCD tutorial" on YouTube by TechWorld with Nana.

To grasp the concept of GitHub Actions and its integration with CI/CD processes, you can watch the "GitHub Action Tutorial" video by TechWorld with Nana. This video explains the fundamentals and basic concepts of GitHub Actions.

Thanks for reading I hope you found the information helpful and informative. If you have any questions or comments, please feel free to reach out to me or leave a comment below.

AWS Cloud resume challenge

David WOGLO — Wed, 19 Apr 2023 02:13:39 +0000

In this article I describe how things I worked on the AWS CRC project With some slight comparison with Google Cloud based on my own experience. By the way, I also wrote an article about the Google Cloud version of this project, you can have a look at it here.

It is better if you are new to all this Cloud related stuff, to take the AWS Cloud Practitioner certification exam, as recommended in the CRC guide as a first step. If you are not familiar with cloud environments or come from another cloud provider, taking this exam will help you validate your knowledge of the cloud and your familiarity with the different services of AWS (in case you come from another cloud provider).

Personally, I had to quickly complete the AWS Cloud Practitioner Quest to refresh my knowledge of AWS as I am quite familiar with the cloud and regarding AWS, I have already worked with some services. The quest is free; you can try it here.

Well now it's good for the intro, let's get to the heart of the matter

Big picture of deployments

This project is about creating a website which is hosted on amazon S3, then on the site display the number of visitors which is calculated with an AWS Lambda Function and stored in DynamoDB table, then automate all process, Website publication and the resources deployment via a CI/CD pipeline.
My resume page is available https://aws.davidwoglo.me/

Website

The first steps of the project consist in setting up a website, using a completely different approach than the traditional one. In a traditional way, we set up a machine or VM on which we install a web server (Apache, Nginx, or whatever you choose) then we upload the html/CSS files etc. and we proceed to some configuration to make the site available.

There I'm spared this server preparation task (and the underlying configs). I just uploaded the website files to Amazon S3 bucket, the object storage service of AWS, the equivalent of Cloud Storage at Google Cloud, then I did some small configuration in two three clicks, just to let S3 know that I want to use this bucket to host a static website, and then we have a ready to use website. No need for a physical server or additional tools to install. This is the serverless approach, and this is what the rest of the project is based on, I didn't need to use any server or VM.

In order for my site to be accessible via a user friendly and secure HTTPS URL, it is necessary to manage the DNS and the ssl certificate configuration, so I used AWS Certificate Manager to obtain a certificate for my domain which ownership had to be verified by automatic mail, due to some problems related to my Custom Domain provider but it is recommended to use a CNAME record to do that. Then to route the DNS traffic I used Amazon Route 53, and the distribution of website content is sped up by Amazon CloudFront(CDN Service). All these configurations were done manually in a separate way and tied together at the end to make things work.

At this point let's make a small comparison with how Google Cloud handles it. At Google Cloud all this can be included in the creation of a Load Balancer where we will just have to activate the automatic management of SSL for HTTPS and CDN for content caching.

Counting website visitors

My web page include a visitor counter that displays how many people have accessed the site.To do this, I created an AWS Lambda function, a DynamoDB table and a REST API. On one side, I wrote a python code that is executed by Lambda, the python function is to get the current number of visitors stored in DYnamoDB and increment it by 1 each time a visitor access my page, on the other side, I added a javaScript code to the files of my site. The job of this script is to get the number of visitors present in the Dynamo table and display it on my page, the communication between the JS code and the database is done via a REST API that I set up using Amazon API Gateway.

This is the part that gave me headaches when I was doing the project on Google Cloud. I didn't use an API Gateway there (because honestly, I didn't know), so I used an open source Functions Framework for Python in which I used client library API to communicate with Cloud Firestore which is the equivalent of DynamoDB.

Automation (CI/CD, IaC, Source Control)

To accelerate and simplify the update of my deployments, whether it is the website (frontend) or the underlying resources (backend), I need to set up a CI/CD pipeline. A CI/CD pipeline is a series of steps that must be performed in order to deliver a new version of software.
Well for the website, it's ok, we can manage it as software, (it's composed of files as a software is), the question you could ask yourself is : What about the cloud resources in the background ? Since they are not files, right? This is where Infrastructure as Code (IaC) comes in. But before talking about it, let's see how the CI/CD for the front end was set up.
I created a source control repository on Github where I put the website files, then I wrote a workflow file that instructs Github Action on how to update my website every time I make a push.
Now let's talk about the Infrastructure as Code stuff, i.e. how to manage provision resources through machine-readable definition files, rather than use an interactive configuration as it is traditionally done.
Well I used Terraform to define the DynamoDB table, the API Gateway, the Lambda function configurations in a template and deploy them with Terraform CLI. You see, now that we can also manage our infrastructure as a software, we can integrate it in a CICD to accelerate the process of deployment and update of the infrastructure resources.
I proceeded in the same way as the website to set up the Backend pipeline, just that here the Github Actions workflow file is a bit more complex.

You can access my frontend repository here and the backend one here

Well, here is how things went during this project. Thanks for reading :)
Please check below for some useful resources.

Useful resources

Quickly deploy website on kubernetes locally

David WOGLO — Wed, 19 Apr 2023 02:09:36 +0000

Several reasons can justify the need to do local Kubernetes deployments, it can be and often is for learning reasons with a lack of resources or with limited resources, or for testing reasons where either we don't want to pay the bill for our use in the cloud, or it is not allowed to do tests on the production environment in case of on-prem deployment. as bootstrapping a Kubernetes cluster still requires considerable resources in terms of computing and time which may not be worth it when doing just simple tests.

So in this article, I show you an example of deploying a website on Kubernetes without worrying about your resources problems (in a light way).

Tools used in this lab :

Kubectl: It will be used to interact with the Kubernetes cluster. It is the Kubernetes command-line tool, which allows running commands against Kubernetes clusters. How to install kubctl
Docker: Docker Engine is an open-source containerization technology for building and containerizing applications. It will be used to containerize the website. How to install Docker
Minikube: Minikube is a local light Kubernetes, focusing on making it easy to learn and develop for Kubernetes. How to install Minikube

Once all these tools are installed, you are ready to start. Create a directory for the lab, then in this directory create another one where you will put the website files.

mkdir lab
cd lab
mkdir files

Once the website files are in place go back to the lab directory and let's start the containerization

Containerization

In the lab directory, create the Dockerfile.

Docker builds images automatically by reading the instructions from a Dockerfile -- a text file that contains all commands, in order, needed to build a given image. A Dockerfile adheres to a specific format and set of instructions which you can find at Dockerfile reference

For this lab, we will create a dockerfile as simply as possible. Use nano Dockerfile to create the file and paste below content in it then save it.

FROM nginx
COPY /files /usr/share/nginx/html

the image we will create here will use a Nginx image as a base image and will copy the website files from the /files directory to the /usr/share/nginx/html directory in the container. you can use another image if you want, like an apache image for example.

After that, we can start building the container image of our website.

Build the container with the docker build command. the dot . in the command bellow indicates the location of the dockerfile which is the current directory

docker build -t website .

Let's check if the image is available by running docker images command

We can see that our image is available, now let's try to run it and access our website

docker run -it -d -p 80:80 website

The “-d” option detaches the container from the current shell and runs in the background. as output we will have the container ID.

The -p 80:80 maps to container’s port 80 to local machine port 80.

Once this command is executed we can access our website from the browser via http://localhost:80

Congratulations, you just deployed a website using a container. now let's move on to the next phase of this lab by scaling up a bit and deploying our website using Kubernetes.

Deployment on Kubernetes

if not already done, start the cluster with the minikube start command.

Before you continue we need to make a small basic adjustment to minikube. By default the minikube node uses its own Docker repository that’s not connected to the Docker registry on the local machine, so without pulling, it doesn’t know where to get the image from. If we don't fix this, we'll observe an error ErrImageNeverPull. To fix this, let's use minikube docker-env the command that outputs environment variables needed to point the local Docker daemon to the minikube internal Docker registry.

As a recommendation on the output, To point the shell to minikube's docker-daemon, run: eval $(minikube -p minikube docker-env)

After that, the image has to be rebuilt again. Once this is done, we can move on to the actual deployment.

We can create a K8s deployment either via an imperative method (by writing directly the commands in the terminal) or via a declarative method where we declare in a yaml or json manifest file the desired state of our deployment and apply it. We will proceed with the declarative method here.

We can either start writing a manifest from scratch and deal with all the possible errors, or we can save ourselves from all that and use a really cool option of kubectl and quickly generate a file ready to use or on which we can work.

So to save us from all this hustle let's generate a yaml file with the dry-run option.

kubectl create deployment website --image=website --replicas=3 --port=80 --dry-run=client -o yaml > website.yaml

Then apply the manifest file

kubectl create -f website.yaml

Now let's see if the deployment has been successful.

kubectl get po

Oh oh ! It seems we have a problem.

Kubernetes tries to pull the image specified in the manifest, but this image is not in the minikube registry or a public Docker registry. So to fix this you have to prevent the image from being pulled from a public registry by setting the image pull policy to never. Edit the website.yaml file the in container section add imagePullPolicy: Never

then save the file then apply tby first deleting the erroneous deployment recheck again.

kubectl delete -f website.yaml
kubectl create -f website.yaml
kubectl get po

It should work now

Voila, it's okay now.

Next, expose the website deployment as a Kubernetes Service, specifying a port where it will be accessible with --type=NodePort and --port=80

kubectl expose deployment website --type=NodePort --port=80

Let's check whether the service is running.

kubectl get svc

Now let's retrieve a URL that is accessible outside of the container.

minikube service website --url

once the url has been obtained, it can be accessed via a browser.

Well, that's what ends this article, thank you and see you very soon

Deploy a highly available application in a scalable VPC architecture on AWS.

David WOGLO — Wed, 19 Apr 2023 02:04:09 +0000

In this article, we are going to deploy a VPC architecture in a scalable way. It is mainly about deploying two VPCs, one for a bastion host in a public subnet, and the second one for the main resources of the architecture, namely two private subnets that will be used to host the different instances of an autoscaling group that will be distributed across the two availability zones to ensure the high availability of the application. Since these instances will be in private subnets they will not be able to have public IPs so no internet access. So to allow them to have internet access, a NAT gateway will be set up in a public subnet, and the route tables will be updated to route the outgoing traffic by default to the NAT gateway to allow the resources in the private subnets to have internet access. To serve the requests coming from the internet to the application, a Network Load Balancer will be set up in front of the auto-scaling group. And finally to allow private communication between the two VPCs a transit gateway will be set up.

Preparation for the deployment

This application will be based on an ec2 instance with all the prerequisites and dependencies for the proper functioning of the application. So to do this we will create an ec2 instance on which to install and make all the necessary configurations for the web application to be ready to use. Of course, feel free to set up your web application, so everything it will need to be ready to use (web server, DB, etc.)

Once everything is ready, we will create a golden AMI based on the EC2 instance.

An Amazon Machine Image (AMI) is a supported and maintained image provided by AWS that provides the information required to launch an instance. You can create your own AMI, customize the instance (for example, install software on the instance), and then save this updated configuration as a custom AMI. Instances launched from this new custom AMI include the customizations that you made when you created the AMI

So to create your custom AMI:

stop the ec2 instance.
Click on action>images and templates> create an image**.
Once the necessary fields are filled in, click on create an image.
Wait for the image creation to finish
Deleting (terminating) the instance.

VPCs Deployment

Bastion Host VPC

To create the VPC:

Navigate to VPC dashboard
Click on the button create VPC
In the VPC settings, choose only VPC
Define IPv4 CIDR the address block and click on create VPC to validate the creation of the VPC.

As you may have seen on the architecture at the beginning, in this VPC there must be a public subnet in which the bastion host will reside. To set up a public subnet, several elements come into play, the main ones are the route tables and the internet gateway. For a subnet to be public, it must be associated with a route table that directs traffic to an internet gateway.

Create Internet Gateway

So let's create the internet gateway first.

In the VPC menu, click on internet gateway on the left,
Then click on the create internet gateway button, enter the gateway name and click on create internet gateway to validate.
Once the internet gateway is created you will see a banner that invites you to attach it to a VPC, click on it then attach it to the previously created VPC.

Create the public subnet

In the VPC menu click left on the subnet
Click the create subnet button
In the VPC section choose the VPC in which the subnet must be located
In the subnet parameters, define a name, then define the IPv4 CIDR block.
Click on create subnet to validate.

Create route tables and add routing rules

When creating a VPC, a route table is created by default, you can just add to it a new route depending on what you want to get, or create a new one. Since the route we want to add here will allow public access, I will not create a new one. But in case you want to create a new one,

Navigate to the VPC menu
Click on route tables on the left then the create route table button
Choose the VPC to which it should belong, give it a name then validate.
Once the route table is available, select it then in the routes section click on edit routes
Click on add route. at the destination level, type 0.0.0.0/0 for any destination then at the target level, choose the internet gateway we just created then save changes.

After the routes are updated, it is necessary to associate the table to a subnet, so in the section subnet associations, click on edit subnet associations, then select the subnet to which we want to associate the route table then save associations.

Production VPC

In the production VPC there are a public subnet and two private subnets. For the public subnet, proceed as done in the previous steps.

Create NAT Gateway

Now for the private subnet, we don't want the resources to be directly accessible on the internet, but the outbound traffic by default still points to the internet. This is possible by setting up a NAT gateway and then creating a route table that routes the default traffic to NAT for an outbound internet connection.

In the VPC menu, click on Nat gateway on the left
Click on the create NAT gateway button, give it a name
Choose the public subnet in which it must be located at the connection type level
Choose public to allocate it an Elastic public IP then validate by clicking on create NAT Gateway.

After you have to create a new route table that will direct the default traffic to NAT for outbound internet connection, to do this you have to proceed as before but here you will have to change the target to redirect it to the nat gateway instead of the internet gateway and do not forget to associate the private subnet.

Transit gateway

Using the transit gateway here will enable private communication between the Bastion Host VPC and the Production VPC, to create the transit gateway,

Still, in the VPC menu, click on the left on the transit gateway
Click on the create transit gateway button, give it a name
Validate by clicking on create transit gateway.

After its creation, you must add the attachments for the VPCs, so on the left just below the transit gateway is transit gateway attachments

Click on it then click on the create transit gateway attachment button,
Give it a name
Choose the required transit gateway ID, for the attachment type choose VPC, then at the VPC id level choose the required VPC then validate the creation of the attachment, and then it's good.

VPC traffics observability

Here we're going to create a CloudWatch Log Group with two Log Streams to store the VPC Flow Logs of both VPCs and enable Flow Logs for both VPCs and push the Flow Logs to Cloudwatch Log Groups and store the logs in the respective Log Stream for each VPC.

In the CloudWatch menu, click on "Logs" and then click on the "Create log group" button.
In the Create log group dialog, enter a name for the log group and click "Create".
Once the log group has been created, click on the log group and then click on the "Create log stream" button.
In the "Create log stream" dialog, enter a name for the log stream and click "Create".
Repeat this step to create a second log stream.

Next, you will need to enable VPC Flow Logs for the two VPCs. To do this,

Navigate to the VPC service in the AWS Management Console
Select one of the VPCs.
In the VPC menu, click on "Flow Logs" and then click on the "Create flow log" button.
In the "Create flow log" dialog, select the log group that you created previously as the destination for the flow logs.
Repeat these steps for the second VPC to enable VPC Flow Logs.

The application deployment

We will deploy the application by setting up an Auto Scaling group based on the Golden AMI we created above

Create launch template

To create an Auto Scaling group based on the golden AMI, you must use a launch template. To create a launch template

Go to the Amazon EC2 console and select "Launch Templates" from the navigation pane.
Click the "Create launch template" button.
On the "Create launch template" section, enter a name and choose the required VPC.
Under Auto Scaling guidance, select the check box to have Amazon EC2 provide guidance to help create a template to use with Amazon EC2 Auto Scaling
Under Launch template contents, fill out each required field and any optional fields as needed, select the Golden AMI created above, instance type, key pair, network settings, security group, and other instance details for the instances in the Auto Scaling group.
Click the "Create launch template" button to create your launch template.

Create Auto Scaling group

Once your launch template has been created, you can use it to create an Auto Scaling group.

But first, let's create the load balancer that will be in front of the auto-scaling group

Navigate to the Amazon EC2 dashboard.
In the left-hand menu, under "Load Balancing," select "Load Balancers."
Click the "Create Load Balancer" button and select "Network Load Balancer."
Follow the prompts to configure the NLB, including the name, listeners, and target groups.

To configure the NLB, you will need to specify the following details:

Name: This is the name of the NLB. It should be unique within your AWS account.
Listeners: These are the ports that the NLB will listen on. You can add multiple listeners with different ports and protocols. Here we use port 80
Target groups: These are groups of Amazon EC2 instances that the NLB will route traffic to. You will need to create at least one target group and add instances to it.

Once you have configured the NLB, navigate to the EC2 dashboard and select "Auto Scaling Groups" in the left-hand menu.

Click the "Create Auto Scaling group" button.
Follow the prompts to configure the Auto Scaling group, including the group name, network, and subnet.
In the "Load Balancer" section, select the NLB that you just created
Configure the other settings for the Auto Scaling group as desired, including the minimum and a maximum number of instances and the scaling policies.
Click the "Create Auto Scaling group" button to create the Auto Scaling group with the NLB.

Once the Autoscaling group is created and the NLB becomes active. You can then test the NLB by sending traffic to it and verifying that it is routing traffic to the target groups as expected.

Troubleshooting tips

in case of unreachability, the two main points that should be reviewed are the security groups and the route tables, make sure that the security group allows the type of traffic you want and the source. also, make sure to apply the right route table configurations. An excellent tool that can help debug in case of unreachability is the network reachability analyzer. it helps troubleshoot reachabilities by configuring paths and analyzing them for insightful explanations that can help solve problems.

The unreachability issue I had during my configurations was related to the transit gateway. The ec2 instances in the private subnets could not be accessed via ssh from the jump (bastion) host. To fix this I used the network reachability analyzer tool to get some information on to base my hypothesis, and this helped in part to solve the issue. the problem was due to the lack of a route within the VPC bastion and also at the production VPC level. I had to add a route that redirected traffic destined for the private subnet to the transit gateway by default, and also do the same at the private subnet level, ie direct by default the traffic destined for the bastion to the transit gateway.

Deploying and Securing an App on AWS EKS with Gitlab CI/CD and Checkov

David WOGLO — Wed, 19 Apr 2023 01:58:55 +0000

Introduction

Deploying an application on AWS EKS (Elastic Kubernetes Service) can be a powerful way to ensure scalability and reliability for your application. However, the process can be complex and time-consuming, especially when it comes to ensuring the security and compliance of your deployment. In this article, we'll show you how to simplify the process and ensure your deployment is secure with GitLab CI/CD and Checkov. GitLab CI/CD provides a powerful toolset for automating the deployment process and improving collaboration among team members, while Checkov is a Security as Code tool that can help you automatically scan your configuration files for potential security and compliance issues. By integrating these tools into your deployment pipeline, you can ensure your deployment is secure and compliant with industry best practices, all while saving time and effort.

Prerequisites

Before proceeding you will need the following :

Set up a GitLab project with runners to execute CI/CD jobs
A container registry (a docker hub repo is more than enough)
A running AWS EKS cluster
Some knowledge of Docker and Kubernetes

The different steps are as follows

Set up the application code and the Dockerfile
Define CI/CD GitLab variables
Set Kubernetes manifest files
Set up the CI/CD pipeline
Trigger the pipeline with a git push.

Directory structure


├── Dockerfile
├── .gitlab-ci.yml
├── .k8s
│   ├── deployment.yaml
│   └── services.yaml
└── src
    ├── app.py
    └── requirements.txt

The application files and the Kubernetes configurations are respectively in the src and .k8s directories. and the Dockerfile and the GitLab-ci script are at the root of the directory.

Set up the application code and the Dockerfile

Use whatever language or framework you want to create the application you want, the main thing is to have an application that you can containerize with a Dockerfile. Personally, I used a simple Python code that uses the Flask framework to create a web application that displays a "Hello, World!" message.

For the Dockefile, here is an example of what it could look like :

FROM python:3.9-slim-buster

WORKDIR /app

COPY src/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY src/app.py .

CMD ["python", "./app.py"]

I recommend you test your docker image locally before continuing.

Define CI/CD GitLab variables

To connect to AWS, Kubernetes, and Docker Hub from GitLab CI, you need to define variables in the GitLab CI/CD pipeline. You can define these variables in the GitLab project settings under CI/CD > Variables.

To connect to AWS

${AWS_ACCESS_KEY_ID}: This variable contains the access key ID for the AWS account used to deploy the application.
${AWS_SECRET_ACCESS_KEY}: This variable contains the secret access key for the AWS account used to deploy the application.
${AWS_DEFAULT_REGION}: This variable contains the AWS region where the application will be deployed.

The variables related to the Docker hub or container registry

${CI_REGISTRY_USER}: This variable contains the username used to authenticate with Container Registry, it can be docker hub, GitLab registry or whatever you want
${CI_REGISTRY_PASSWORD}: This variable contains the password used to authenticate with the Container Registry.
${CI_REGISTRY_IMAGE}: This variable contains the name of the Docker image in the Container Registry.
${CI_REGISTRY_IMAGE_VERSION}: This variable contains the version or tag of the Docker image in the Container Registry.

The configuration file to access Kubernetes

${KUBECONFIG}: This variable of type file contains the Kubernetes configuration file used to authenticate with the Kubernetes cluster. this file is available at this path ~/.kube/config so when adding it to Gitlab make sure you choose File as the type

Set Kubernetes manifest files

Now it's time to define manifest files for Kubernetes deployments. As you would have seen above these files are located in the .k8s folder.

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: my-app
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION}
        resources: {}
status: {}

deployment.yaml defines a Kubernetes Deployment for the application named "my-app". The Deployment creates a single replica of the application and specifies the container image to be used for the application using the variable ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION}. This variable references the Docker image built and pushed to a Docker registry.

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: my-app
  name: my-app
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 5000
  selector:
    app: my-app
status:
  loadBalancer: {}

services.yaml defines Kubernetes Service for the "my-app" application. The Service exposes the application on port 80 and routes traffic to the listening port of our hello world app which is port 5000 on the application container. The file also specifies the labels to be used to identify the application in Kubernetes.

Hint: To generate a quick template on which to base and write these configuration files and thus reduce the risk of error and save time, there is an option of the kubectl command that can be very useful.

The --dry-run=client -o yaml option in the kubectl command generates a YAML representation of the Kubernetes resource that would be created or modified without actually creating or modifying the resource. Here is an example of how to generate our yaml file

#deployment
kubectl create deployment <app_name> \
     --image=<the_docker_image> \
     --dry-run=client -o yaml > deployment.yaml

#service
kubectl expose deployment <app_name> \
     --port=80 --target-port=5000 \
     --dry-run=client -o yaml > service.yaml

This should generate the two yaml files that you will adjust to fit your use. You can also use the --dry-run option with the kubectl command to validate a Kubernetes YAML file without actually applying it to a cluster.

To use the --dry-run option to validate a Kubernetes YAML file, you can run the following command:

kubectl apply --dry-run=client -f <yaml_file_path>

Set up the CI/CD pipeline

Now we can start setting up the GitLab script for our pipeline. The script in our case here will have 5 steps.

The script consists of the following stages:

docker build: Builds the Docker image and tags it with the registry information.
docker push: Pushes the Docker image to the registry.
test: Runs the Checkov tool to validate the Kubernetes deployment and service files.
deploy services: Deploy the Kubernetes services to EKS using the kubectl command.
deploy the app: Deploys the Kubernetes application to EKS using the kubectl command.

Let's go a little further into the test phase, our test here is much more security oriented. we have integrated Checkov into the pipeline. Checkov is an open-source tool used for static code analysis of infrastructure-as-code (IAC) files. In this case, it will be used to perform security and compliance checks on the Kubernetes YAML files in the .k8s directory.

By running Checkov on the .k8s/deployments.yaml and .k8s/services.yaml files, the GitLab CI/CD pipeline can ensure that the Kubernetes resources being deployed meet the security and compliance requirements defined in the policies and rules being enforced by Checkov.

stages:
  - docker build
  - docker push
  - test
  - deploy services
  - deploy app


Build docker image:
  image: docker:stable
  stage: docker build
  before_script:
    - docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD}
  script:
    - docker build -t the-app .
    - docker tag the-app:latest ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION} 

Push to registry:
  image: docker:stable
  stage: docker push
  before_script:
    - docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD}
  script:
    - docker push ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION}

Test:
  image: bridgecrew/checkov:latest
  stage: test
  script:
    - checkov -d .k8s/deployments.yaml
    - checkov -d .k8s/services.yaml
  allow_failure: true



Deploy services on EKS:
  image: ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION}
  stage: deploy services
  before_script:
    - export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
    - export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
    - export AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
  script:
    - cd .k8s
    - kubectl --kubeconfig ${KUBECONFIG} apply -f services.yaml
  rules:
    - changes:
      - .k8s/services.yaml


Deploy app on EKS:
  image: ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION}
  stage: deploy app
  before_script:
    - export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
    - export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
    - export AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
  script:
    - cd .k8s
    - kubectl --kubeconfig ${KUBECONFIG} apply -f deployment.yaml
    - kubectl --kubeconfig ${KUBECONFIG} rollout status deployments

Docker build stage: Builds a Docker image for the application specified by the Dockerfile using the official docker:stable image. Before building the image, it logs in to the Docker registry using the username and password provided as GitLab CI environment variables ${CI_REGISTRY_USER} and ${CI_REGISTRY_PASSWORD}. After building the image, it tags it with ${CI_REGISTRY_USER}/${CI_REGISTRY_IMAGE}:${CI_REGISTRY_IMAGE_VERSION}.
Docker push stage: Pushes the Docker image created in the previous stage to the GitLab CI registry using the docker push command.
Test stage: Runs checkov tool for Kubernetes manifest files deployment.yaml and services.yaml. in this case, this stage is allowed to fail and does not prevent the pipeline from continuing.
Deploy services on EKS stage: Deploys the Kubernetes services specified in the services.yaml file to the Amazon Elastic Kubernetes Service (EKS) cluster. Before deploying, it sets the AWS credentials and region environment variables. It only runs the deployment if the services.yaml file has been modified since the last pipeline run.
Deploy app on EKS stage: Deploys the Kubernetes deployment specified in the deployment.yaml file to the EKS cluster. Before deploying, it sets the AWS credentials and region environment variables. After deploying, it checks the rollout status of the deployment.

To make sure that this script is valid, you can use a lint tool to validate the script, it will help to quickly fix the errors and save time. For my part, I used glab CLI tool which with the command glab ci lint allows validating the script to make sure that everything is correct.

Trigger the pipeline with a git push.

To trigger this GitLab CI pipeline, you need to commit and push the code changes to the GitLab repository that contains this GitLab CI script.

Once you have pushed the changes to the repository, GitLab CI automatically detects the changes and starts running the pipeline. The pipeline can also be triggered manually by clicking the "CI/CD" tab in the GitLab repository and clicking the "Run Pipeline" button.

Note that to run the pipeline successfully, you need to ensure that you have configured the necessary environment variables on GitLab CI, such as CI_REGISTRY_USER, CI_REGISTRY_PASSWORD, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, and KUBECONFIG. These variables are used to log in to the GitLab registry, authenticate with AWS, and connect to the Kubernetes cluster.

You can find the files I used here on my GitHub or GitLab

I remain open to any contribution and suggestion to improve my work. Do not hesitate to let me know your contribution or suggestion by opening an issue.

Thanks for reading :)

My Cloud Resume Challenge

David WOGLO — Wed, 17 Aug 2022 21:55:00 +0000

How did it all start?

It all started after I passed my certification. I thought to myself, this is it, you've achieved your first short term goal of getting into the Cloud world, something you've been studying tirelessly for in recent months. But I must admit I thought that once certified the offers would fall left and right. Haha !!! it's not the case and it wasn't going to happen, we can't make you an offer like that just because you are certified inh, Certified people, we can find all over the place lately. Certification at best is just the key to open the door of Cloud house 😀, it just offers the opportunity to be consulted, to have a little bit of visibility or I would say consideration. But to get a job (like settling down in the Cloud house 😀), to get offers as I thought, you have to justify your know-how and your knowledge in the Cloud by practical, relevant and convincing experiences. And I can see from afar this question that we juniors often ask ourselves "How can we get experience if we are not given the opportunity to join a company where we can develop these experiences? 🤔" Honestly (at least for the IT world) we can do without it, we can justify our skills, our experiences, our knowledge by projects, challenges, that make us go through the pitfalls, the mistakes, the solutions, the obstacles, the dark days, the light that make us build and help us to polish a profile well adapted and that constitutes a solution to the needs and problems of companies.
It is with this in mind that after watching a video of Forrest Brazeal, I came across the Cloud Resume Challenge.

What is The Cloud Resume Challenge ?

The Cloud Resume Challenge is a hands-on project designed to help bridge the gap from Cloud certification to Cloud job. It incorporates many of the skills that real Cloud and DevOps engineers use in their daily work. The Cloud Resume Challenge is a multiple-step (It's roughly 16 steps) resume project, from the creation of a website to the implementation of a CI/CD pipeline, which helps build and demonstrate skills fundamental to pursuing a career as a Cloud Engineer.

Certification

The first step of the challenge is to get a Cloud certification of beginner or associate level. I don't think it's an obligation, but from my personal point of view I strongly recommend it, especially when you are a beginner or if you have a non-technical background. it will allow you to acquire and validate the fundamentals that you need to pursue a career in the cloud. For me, I had to pass the associate level certification of Google Cloud, the Google Cloud Associate Cloud engineer thanks to the Google Africa Developer ScholarshipGADS 2021 program

The architecture

Website

Setup

To go fast and get to the heart of things, I didn't have to write HTML files from scratch, I rather took a free template that I modified and adapted to my needs with the little and old knowledge I have in HTML and CSS. Once the website files are ready, it must be hosted, right? Since the challenge is based on a serverless spirit, the site is not hosted on a server but on the Object storage service of Google Cloud. By uploading website contents as files to Cloud Storage, we can host static websites on buckets. See how to do it.

Make the site accessible via a domain name and secure it

Usually to make a site accessible via a domain name, you must map the IP address of the server hosting the site to a domain name. But in our case there is no server in play that has an IP address, what can I do? Fortunately we have the managed HTTP Load Balancer on Google Cloud which can allow us to set up a LoadBalancer with a public IP address and point this address to a Bucket that hosts a static site. This solves the problem of IP address, we now have an IP address to access our website, we can now link a domain name to this IP address and access our website as usual.
But with only that, the site is only accessible in HTTP, for more security the site must use HTTPS, again thanks to Google Cloud's LB we can set this up, without too much trouble, Google Cloud's HTTPS LB has an option for automatic management of SSL certificates that can be activated during the implementation of the LB.

But in a corporate environment it is better that the certificates are managed and controlled by you
For the domain name I used Namecheap, however you can use Cloud DNS from Google Cloud to do this

Website visitors count

This part of the challenge consists in having the number of visitors who have visited the site.
For this I wrote a JavaScript code in addition to the website files on one side, and on the other side a function in python which is hosted on Google Cloud Function. The purpose of the javascript is to call the execution of the python function which in turn is executed, calculates and stores the result in the Document NoSQL database, Firestore and returns the total number of visitors in Json format to the JS code which is responsible for displaying it on the site, This operation is done each time there is a visit to the page. The python function in this case serves as an API to avoid having the JS code communicate directly with the database

Test

To ensure that the python code works as it should and returns the expected result, it was necessary to write python a test using the unittest module provided by the Python standard library which help write and run tests for Python code. Since the JS code needs to get the data concerning the number of visitors in json format, the python test here verifies if the data returned by the function are indeed in json format

Well I think we can take a coffee break ☕

So far everything that has been done has been manually clicking around in the Google Cloud console. In the following stages, the methods and techniques used by DevOps Engineers will intervene, in particular Infrastructure as Code, GitOps and the CI/CD (I admit it's not that hard in this challenge but it's the same operating mode)

Infrastructure as Code

Here I discovered a rather brilliant thing. Before telling you, know that it is planned here to define resources in a Terraform template and deploy them using the Terraform CLI. But instead of getting into another hassle of rewriting everything I've done so far in Terraform, testing it, destroying it, aligning it to what exists and all that, I discovered a Google Cloud tool that I can use to generate Terraform code for resources in a project, folder, or organization.
The gcloud beta resource-config bulk-export --resource-format=Terraform command exports resources currently configured in the project, folder, or organization and prints them to the screen in HCL code format.
(Of course that's what I did)
So what's left for me to do is manage these Terraform templates with a source control system, so I can integrate it into the pipeline later. aSo what's left for me to do is manage those templates with Github, so I can integrate it into the pipeline later. and when I need to make changes to my Cloud resources , I just have to adjust a few lines and that's it.

But I point out, before using this tool, it is necessary to already have Terraform bases. Here's a great guide to getting started

CI/CD

To setup the CI/CD pipeline or I would rather say the pipelines, since there are two, one for the frontend and another for the backend. I created a Github repository for the frontend (i.e. the website files), and thanks to Cloud Build I automate the update of the website which is triggered each time I make a push. On the other side for the Backend (i.e. the infrastructure resources that are defined in a Terraform template), I also created a Github repo for this purpose. repo (changes to .tf files) my Cloud resources are automatically updated, whether it's deletion, change or addition, all of this is done automatically without any intervention on my part.

In the end ...

Well, that's an overview of how I managed to complete this challenge, which by the way is very rewarding, allowed me to learn and discover really useful things, to identify my weaknesses and how to strengthen them and above all has allowed me to have relevant experience in the cloud. and I don't intend to stop there 😉
Of course some of the steps were not fun, but thanks to the help of some of my SWE connections, in particular Vincent
Bakpatina, Corneille ALLOGBALO, Abdel-Khafid ATOKOU and Samtou Assekouda I was able to overcome.

Here are some useful resources 👇🏾

Here is my Github repo for the frontend, concerning the backend, since it contains some sensitive information I have not made it public

Host a static website on Google Cloud Storage
Functions Framework for Python
Automated static website publishing with Cloud Build
Invalidate cached content
Managing infrastructure as code with Terraform, Cloud Build, and GitOps
Export your Google Cloud resources into Terraform format

Quickly automate resources deployment on Google Cloud using an IaC and CI/CD Platform

David WOGLO — Tue, 09 Aug 2022 08:27:00 +0000

In this article I will show you in a simple way, how to set up a CI/CD pipeline that automatically deploys your google cloud infrastructure resources using Terraform, Cloud Build and Github.

Objectives

Automatically deploy resources to Google Cloud from Terraform code hosted in the source control repository.

Requirements

To be able to realize all the steps of this article, you will need a functional google cloud account (You can use the free trial ), a Github account, and some basic knowledge in Google Cloud and Terraform.

Granting necessary permissions to Cloud Build

To be able to perform the necessary deployments on the infrastructure, Cloud Build will need proper permissions. In this lab I will go faster by giving the service account the project editor role. Get the Cloud Build service account and give it the necessary permissions so that it can make required changes to the resources.

Of course, in a production environment it is necessary to comply with the principle of least privilege.

To do so, run the following command in the cloud shell
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:theCloudBuidServiceAccount --role roles/editor

To get Cloud Build service account, click on Cloud Build then settings

And there you will find the email address of the service account

Setup the Github repo and connect Cloud Build to it

Login to Github and create a new repo , then upload Terraform files or edit new ones directly on Github. [Click here](https://Github.com/davWK/ci-cd-terraform-cloudbuild_basics to fork my example infrastructure files repository, or if you are comfortable with Terraform and want to deploy a custom infrastructure write ones from scratch. After that go to Cloud Build to set up automated deployment with a build trigger, you will use Cloud Build and its build triggers to deploy your ressources automatically every time you push a new git commit to the source repository.

Go to Cloud Build
And on the left select trigger
click on create trigger
Give it a name, and for the event choose push to the branch
For the source , select** repository** and click connect new repository
Here it is possible to link a Github repo to Cloud Build by mirroring a Github repository to Cloud Source Repositories or by using Google Cloud Build Github app. We will use the application in this case
see how to configure the application . After configuring the app,
Back to create trigger page, and click on repository and choose the newly created repository
In branch set it to ^master$ or ^main$
For the configuration type choose Cloud Build configuration file (yaml or json)
and in your Github repo create a cloudbuid.yaml with the content below.

steps:
- id: 'tf init'
  name: 'hashicorp/terraform:1.0.0'
  entrypoint: 'sh'
  args: 
  - '-c'
  - |
      terraform init

- id: 'tf apply'
  name: 'hashicorp/terraform:1.0.0'
  entrypoint: 'sh'
  args: 
  - '-c'
  - |
      terraform apply -auto-approve

Back yo trigger page, in the location, by selecting repository, put the path to the yaml file or choose inline (in this case you would not need to create the yaml file in the repo but rather paste the yaml content directly into code editor)
Leave the other values as default and click on create

Voila :) the deployment of your resources should start automatically if you make a push of the yaml file created previously, if not you can run it manually for the first time, for the next times as soon as you update your Terraform configuration the update of your resources should be done automatically