DEV Community: Valon Januzaj

Hosting Django Static Files in AWS using S3 and CloudFront: A Comprehensive Guide

Valon Januzaj — Tue, 17 Oct 2023 07:48:50 +0000

In today’s digital landscape, having a reliable and scalable infrastructure for hosting static files is crucial for web applications. Django, a popular Python web framework, offers seamless integration with Amazon Web Services (AWS) to accomplish this task. In this guide, we will explore how to host Django static files in AWS using CloudFront, a powerful content delivery network (CDN) service, to ensure high availability and fast content delivery.

The article is a step-by-step guide on how to achieve the goal, but nevertheless, I assume that the reader has basic knowledge of Django and AWS.

Understanding Django Static Files

Before diving into the technical aspects, it’s important to understand Django static files. These files include CSS stylesheets, JavaScript scripts, images, and other assets that are served directly by the web server without any processing. Managing static files efficiently enhances the overall performance and user experience of your Django application.

Normally, when you deploy a service into production using Django, especially when you put the settings variable DEBUG=False Django excepts you to provide a way to handle these static files like a cloud service or CDN. What happens in this flow it’s that whenever the service wants to access those files, they’re fetched from the CDN or a cloud service instead of living in the same codebase as the web service itself.

For this article, we are going to use AWS Services called S3 and CloudFront.

Amazon S3 is a highly scalable and durable cloud storage service provided by Amazon Web Services (AWS). It allows you to store and retrieve large amounts of data, such as images, videos, documents, and backups, in a secure and reliable manner. S3 provides an object-based storage model, where each object is stored in a bucket and accessed using a unique key.
Amazon CloudFront is a content delivery network (CDN) also provided by Amazon Web Services. It helps deliver content, such as web pages, images, videos, and other static or dynamic files, to users with low latency and high transfer speeds. CloudFront caches the content at edge locations, which are distributed globally, closer to the users, reducing the distance and network latency. This improves the performance of delivering content to users across different regions.

Setting up IAM User

Login with your root user and then continue to create a new user that you will use during this exercise. You’ll need to create a user that has access to
S3 and CloudFront:

Navigate to IAM > Users > Add New User
Enter the username you want for your user and in the next step, select Attach policies directly tab.
Attach these two policies: CloudFrontFullAccess and AmazonS3FullAccess

Or if you want to skip this part (for simplicity), just use the root user (but this is not recommended in any case for the sake of security)

Setup S3 and CloudFront

First, we need the S3 bucket on which we are going to upload our static files. Navigate to the S3 console and click > Create Bucket

Enter a name for the bucket and select the region
Keep ACL-s disabled
Tick Block all public access and leave everything else as default

You’re good to go!

Now let’s set up CloudFront for the newly created Bucket.
Since we kept everything private in our bucket, we need to let CloudFront access some of the folders that exist in our bucket.

Navigate to the CloudFront console and click Create a CloudFront distribution. After that, you might need to fill up some information:

Origin Domain: Choose the S3 Bucket that you created
Origin Access: Choose Legacy access identities and then click Create new OAI
On the bucket policy select: Yes, update the bucket policy
Allowed HTTP methods: GET, HEAD, OPTIONS
Web Application Firewall (WAF): Do not enable security protections
Response headers policy SimpleCORS

That’s it, keep everything else as default. Now a new CloudFront distribution will be created and will return back a Distribution domain name which in my case looks something like this:

https://d29u7vv9xp7q8y.cloudfront.net

Great, now let’s make the necessary changes in our Django application.

Setting up our Django application — Uploading static files

In order to upload our static files to the AWS S3 bucket, we need to turn off DEBUG mode and also update some settings and configurations in our Django application. To start off, first, install django-storageslibrary which is a library to handle storages better in Django.

$ pip install django-storages

With that done, create a new Python module named storage_backends.py and add the following code:

from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage


class StaticStorage(S3Boto3Storage):
    location = 'static'
    custom_domain = settings.CLOUDFRONT_DOMAIN

This will tell that you want to upload static files to a folder named static and the custom domain to handle and redirect the static files would be the CLOUDFRONT_DOMAIN which is set up in settings.py

NOTE: to not hard-code the environment variables as below and use a mechanism to manage the environment variables. A simple one would be os.getenv().

Modify the settings and add the following environment variables:

DEBUG = False
AWS_ACCESS_KEY_ID = 'YOUR_AWS_ACCESS_KEY_ID'
AWS_SECRET_ACCESS_KEY = 'YOUR_AWS_SECRET_ACCESS_KEY'
AWS_STORAGE_BUCKET_NAME = 'NAME_OF_BUCKET'
AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
AWS_S3_REGION_NAME = "YOUR_LOCATION_IN_AWS"
AWS_S3_SIGNATURE_VERSION = "s3v4"
AWS_QUERYSTRING_EXPIRE = 604800
CLOUDFRONT_DOMAIN = 'YOUR_CLOUD_FRONT.cloudfront.net'

STATIC_LOCATION = "static"
STATIC_URL = f'{CLOUDFRONT_DOMAIN}/static/'
# Add your path in the STATICFILES_STORAGE
STATICFILES_STORAGE = 'django_static.storage_backends.StaticStorage'

This should be enough. Now before you start your Django application, open a new console and collect the static files:

python manage.py collectstatic

You have requested to collect static files at the destination
location as specified in your settings.

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes

125 static files copied.

Now navigate to your AWS account go to your bucket and see if the files are copied to a folder named static , if they’re, everything worked as expected.

Testing CloudFront Integration:

Now it’s time to test whether CloudFront is serving your static files correctly. Start your Django development server and access your application. Inspect the network requests using your browser’s developer tools to verify that the static files are being fetched from the CloudFront URL you specified.

python manage.py runserver    
Performing system checks...

System check identified no issues (0 silenced).
July 30, 2023 - 16:58:13
Django version 4.2.3, using settings 'django_static.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Now visit localhost:8000/admin or 127.0.0.1/admin and open developer tools, for example in Google Chrome: CTRL+SHIFT+I and go to the network tab. Click one of the .css files and navigate to Headers to see the requested URL and see if it’s pointing to the CloudFront URL that you defined in settings.py.

Enabling CloudFront Cache Invalidation:

To ensure that your users receive the latest versions of your static files, configure CloudFront cache invalidation. There are two common approaches to achieving this. You can manually invalidate the CloudFront cache whenever you update your static files by using the AWS Management Console or AWS CLI. Alternatively, you can implement cache invalidation techniques in your Django application, such as versioning static files or appending a query string parameter.

Wrapping up

In this article, we have set up a Python application with Django & Deploy and serve the static files from CloudFront.

You can find the full source code of the article on the GitHub repository, with the instructions.

www.github.com/vjanz/django-static-files-s3-cloudfront

If you found it helpful, please don’t forget to clap & share it on your social network or with your friends.

If you want to support my work, you can buy me a coffee by clicking the image below 😄

If you have any questions, feel free to reach out to me.

Connect with me on LinkedIn, GitHub

Deploy a dockerized FastAPI application to AWS

Valon Januzaj — Fri, 03 Feb 2023 09:29:59 +0000

You’ve created your FastAPI application and now you want to make it public by deploying it? — No worries got that covered.
In this article, I am going to explain step-by-step from creating a simple application with FastAPI, dockerizing it, and deploying to AWS EC2.

What is FastAPI?

From the official docs:

FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints.

Setting up the project:

$ mkdir fastapi-demo
$ cd fastapi-demo

After creating the directory for your project, dive in, and create a virtual environment. You can skip this step but it’s always good to have your dependencies isolated from the outside world. I want to stay simple here so I am using the virtualenv tool to create python virtual environments.

$ virtualenv <name_of_environment>
$ source venv/Scripts/activate

Now install FastAPI and uvicorn:

pip install fastapi uvicorn

Now let’s set the project structure:

$mkdir src && cd $_ 
$ touch __init__.py main.py

Now in the main.py let’s just create an instance of the FastAPI application, add a route and test it out:

from fastapi import FastAPI

app = FastAPI*()


*@app.get("/")*
*def root(*)*:
    return *{*"Hello": "World"*}*

So now we test if the application is working by running the application with uvicorn. Command has this format:
uvicorn .:

uvicorn src.main:app -- reload # for live-reloading

Visit http://localhost:8000/ to see the application in action 🔥, or http://localhost:8000/docs to see the API documentation, which is generated automatically? — Isn’t it awesome?🙌

I am not going to dive deeper into the project structure, how can we organize things better, etc.. as the goal here is just to get started with FastAPI and learn how to dockerize and deploy it to AWS, and for the other tips and tutorials about FastAPI I highly recommend to refer to the official docs as it has really well-organized documentation, and it dives deep in all the concepts of the frameworks.

We set up a simple application with FastAPI, and now you want to deploy it so you can access it or share it, or whatever is your intention. There are several approaches, like deploying manually by grabbing the source code and running manually as we did locally, but we want to be smarter here, so we’re going to dockerize the application and run it on AWS EC2 instances. Let’s go!

Dockerizing the Application

So we basically created the application and tested it locally, all good. We could skip the steps of creating virtual environments and running the code locally, by defining the requirements and just adding the same logic, but it’s always a good practice to run the application locally and test it before going further.

Note: I am assuming that you already have docker installed in your machine if you don’t get it from here.

Why would we use docker?

Get rid of the popular saying “it worked on my machine”
Isolate the dependencies
Run in an isolated environment which will serve only to run your application and its dependencies
Easier to deploy, etc.

In the root directory of our project create a Dockerfile

$ touch Dockerfile

A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image.
Before we write our configuration for the Dockerfile, let’s generate our application requirements:

$ pip freeze > requirements.txt

Open the Dockerfile with your favorite editor, and write the following:

FROM python:3.8.1-slim # Image from dockerhub

ENV *PYTHONUNBUFFERED *1 
EXPOSE 8000 # Expose the port 8000 in which our application runs
WORKDIR /app # Make /app as a working directory in the container

# Copy requirements from host, to docker container in /app 
COPY ./requirements.txt .

# Copy everything from ./src directory to /app in the container
COPY ./src . 

RUN pip install -r requirements.txt # Install the dependencies

# Run the application in the port 8000
CMD *[*"uvicorn", "--host", "0.0.0.0", "--port", "8000", "src.main:app"*]*

Great, now let’s build our image by running:

$ docker build -t fastapi-demo .

As we are in the same directory we don’t need to specify the path to the Dockerfile as the name of the file is following the conventions. Now, wait for a couple of minutes as you will download the image from the remote repository from the docker hub, and then build your project on top of it.

To make sure that nothing failed run $ docker images and you should see the name of the image in the list of the images in your machine. Great, now let’s run it:

$ docker run -dp <host_port:docker_port> <name_of_image>

# -d - Detached mode, runs in the background
# -p - to map the port on where do you want to access the #application in my case localhost:8000/
We have exposed port 8000 in our Dockerfile so we're good to go.

So putting it all together:

$ docker run -dp 8000:8000 fastapi-demo

To make sure that the image is running ( after the image runs it becomes a container ) check with docker ps and you should see a container with our image running in the port 8000. Great now visit localhost:8000 and you should see the application running just as we did before locally. Done...

Deploying the application to AWS

So we have created our FastAPI application and we have dockerized into an isolated environment that runs everything. That’s great for local, but we want to expose the application to the public, so let’s deploy to AWS which is the most popular cloud provider.

Everything that we’ll be using in this article is supposed to be without any additional charge, so you can use AWS free tier to follow along, but in general, do a clean-up after you have played around with it.

Note: I am assuming that you already have an AWS Account and you can use free-tier services. I am going to explain everything step-by-step so let’s continue.

Create EC2 Instance
For the sake of demonstration, I will do everything from my AWS root account, but it’s not recommended, so make sure that you always create a separate user, and give the necessary permissions to accomplish the tasks.

Navigate to EC2 service, and click Launch Instance

Select the Amazon Linux 2 AMI (HVM) as it’s free tier eligible. Stay with the default settings and just click Review and Launch . After that it will ask you to generate a new key which you will use when you need to ssh into the instance, so create a new one, name whatever you want, and then download and keep them in your machine as you will need them right away. Click Launch instance and your instance will be ready in a bit.

After the instance has been created you might want to ssh into it, and install docker, so let’s do it.

In your EC2 Dashboard, after the instance state has changed to running, right-click over it, and choose Connect . That is enough to give you information on how to ssh into your instance. Leave that open, and open a new bash terminal.
Navigate to the downloaded .pem key, and do the following:

$ chmod 400 fastapi-deploy-demo.pem # Substitute with your key

From your EC2 Dashboard, grab the latest command which tells you how to ssh to the instance and paste that into your terminal, (make sure you’re in the same directory as the .pem file is ):

# Same directory where .pem key is located

ssh -i "fastapi-deploy-demo.pem" ec2-user@<your-details-here>.compute-1.amazonaws.com

And you should be inside your instance now, so let’s install docker. Just follow along with the commands and you should be fine:

# Update the installed packages and package cache on your instance.

sudo yum update -y

# Install the most recent Docker Community Edition package.sudo 

amazon-linux-extras install docker

# Start the Docker service.

sudo service docker start

# Add the ec2-user to the docker group so you can execute Docker #commands without using sudo.

sudo usermod -a -G docker ec2-user

Now reboot your EC2 instance, because in most cases it won’t take the configured permissions, so do it from the EC2 Dashboard and follow the steps above to connect again.

Now write docker info to see if you can run it without sudo, and if you can you’re good to go.

Create a repository, and push the image to it
As we are deploying in AWS I prefer that we keep everything here, so instead of Dockerhub, I am going to use Amazon ECR.

Amazon Elastic Container Registry (ECR) is a fully-managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images

Navigate to Amazon Container Services (ECR) and create a new repository. Name it whatever you want (fastapi-deploy-demo).

Now it’s time to push or local image that we built earlier to this remote repository. You first need to authenticate with your AWS credentials from your AWS CLI (Local), so if you don’t have it go ahead and install it from here:

I am assuming that you’re already authenticated using your credentials

Now go to your ECR, and select your created repository, and click VIEW PUSH COMMANDS . Authenticate your docker client with your registry using the first command:

After that build the image again using the repository name, tag and push the image to the repository:

# Navigate to the project directory first and type the following:

docker build -t fastapi-deploy-demo .

# Tag the image
docker tag fastapi-deploy-demo:latest <YOUR ID>.dkr.ecr.us-east-1.amazonaws.com/fastapi-deploy-demo:latest

# PUSH
docker push <YOUR ID>.dkr.ecr.us-east-1.amazonaws.com/fastapi-deploy-demo:latest

Done...
Now you’ve got everything set up, we need to ssh to the instance and pull the image, then run it with docker.

Just as we did locally, now in our instance we have AWS CLI shipped we just need to authenticate by running

$ aws configure

# This will ask you for:
AWS Access Key ID: []
AWS Secret Access Key: []
Default region name: []
Default output format: []

# Make sure you use the same credentials as you used when you authenticated localy with AWS CLI

And then authenticate with the docker client, see docs here
After you have been authenticated you can pull the image from ECR, so go ahead and navigate to the ECR once again, click on the repository and grab the Image URI.

Get back to your EC2 instance and write the type the following command:

docker pull <IMAGE_URI> # that you grab from ecr repository

After you’ve pulled the image, go ahead and run as we did locally.

Grab the name of the image first by running docker images and then write:
docker run -dp 80:8000 .. Now we have mapped port 80 in our instance with port 8000 in docker.
Great, verify that the container is running with the command: docker ps
is so, we have one last thing to configure, the traffic into our instance... 🚥

Go to your instance in the EC2 dashboard, click on it, and then go to the Security tab, and after that click on your security group(mine is sg-0881f2399bc659e6b), and then click Edit inbound rules .

Click add a new rule, and select all traffic, which is not good for security but just for sake of demonstration.

Finally, navigate to your EC2 Instance, grab the public IPV4of the instance and test that in your browser:

We have successfully deployed our FastAPI application in your AWS EC2 Instance using Docker 🚀 🔥

Play around with it, and when you’re done with it, remove the instance and the repository as you don’t want to get any additional charge from AWS.

Wrapping up

In this article, I wanted to show you step-by-step how to deploy your application to dockerize and deploy your application in AWS.
As I did my research out there, this is a nice and straightforward approach to deploying your application, but you can use other services like ECS if you are intentions are to run applications in scale, which is a fully managed container orchestration service, so you can easily scale applications by only having images of your application as we did.

If you would like to support my work, you can buy me a coffee by clicking the link 😄:

BuyMeCoffe

Feel free to reach out to me if you have any questions.
Connect with me on 👉 LinkedIn, Github

From local development to Kubernetes — Cluster, Helm, HTTPS, CI/CD, GitOps, Kustomize, ArgoCD — Part[2]

Valon Januzaj — Fri, 03 Feb 2023 00:51:21 +0000

From local development to Kubernetes — Cluster, Helm, HTTPS, CI/CD, GitOps, Kustomize, ArgoCD — Part[2]

This is the second part of the series: From local development to Kubernetes — Cluster, Helm, HTTPS, CI/CD, GitOps, Kustomize, ArgoCD. You can find the [PART 1] here

This part includes:

Introduction to GitOps — ArgoCD installation
Using Kustomize to write Kubernetes manifest
Secret management in Kubernetes with SealedSecrets
Add a basic Continuous integration pipeline with GitHub actions
Creating and running our services in ArgoCD

Intro to GitOps — Continuously integrating and deploying applications with ArgoCD, Kustomize, and GitHub Actions

If we go to the GitHub repo you can find all the manifests that we worked on so far: Deployment, Service, ClusterIssuer, Ingress, and Secret. This is okay at some point… but look at that secret.yaml file, which is very unlikely to be there, especially when it’s encoded with base64 when we know that is so easy to decrypt. Aside from that, it’s so hard to separate environments: production, staging, development, etc. There can also be lots of inconsistencies, where I can have some manifest locally and apply it, and another version of that manifest exists in the repo, so it’s very hard to reproduce the same environment if we delete everything.

Ideally, we would love to have a reproducible environment where everyone that works on infra knows the state of an application and can easily make changes to the manifest while everyone sees the changes, so let’s aim for this!

Intro to GitOps — What is GitOps

GitOps is a way to manage infrastructure and applications using Git as a single source of truth. It’s a method to use Git as a centralized source of truth for declarative infrastructure and applications. The idea is to use Git to store the desired state of the infrastructure and applications, and then use automation tools to ensure that the actual state matches the desired state. This approach helps to ensure that the infrastructure and applications are always in a known, good state, and it makes it easy to roll back changes if something goes wrong.

GitOps solves several problems in software development and deployment, including:

Version control: By using Git as the central source of truth, GitOps allows teams to easily track changes and roll back to previous versions if necessary.
Collaboration: GitOps allows multiple people to work on the same codebase and infrastructure, making it easier to collaborate and share knowledge.
Automation: GitOps uses automation tools to ensure that the desired state of the infrastructure and applications is always in sync with the actual state, reducing human error and increasing efficiency.
Auditability: With GitOps, every change is tracked and auditable, making it easier to understand how and why changes were made.
Speed: GitOps allows for faster deployment and rollback, as well as faster iteration and experimentation, as teams can quickly and easily test new features and changes.
Scalability: GitOps allows teams to scale their infrastructure and applications easily and efficiently, with the ability to easily add and remove resources as needed.

Gitops is a methodology that uses Git as the single source of truth for infrastructure and application deployments. It is not a specific tool, but rather a way of organizing and managing the deployment process. ArgoCD is a tool that implements the GitOps methodology by automating the deployment and management of applications and infrastructure in a Git-based workflow.

What is ArgoCD — Installing ArgoCD in the cluster — Implementing GitOps methodology

ArgoCD is an open-source GitOps tool that automates the deployment of applications to Kubernetes clusters. It uses Git as the source of truth and continuously monitors the state of the cluster to ensure that it is in sync with the Git repository. ArgoCD also provides a web-based UI that makes it easy to view and manage deployments.

To install ArgoCD we might use the same methodology as before, installing from the helm chart or directly applying the manifests that ArgoCD has published.

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

After a couple of minutes, ArgoCD should be installed and we can port-forward the service and access the ArgoCD UI locally, so let’s do that:

$ kubectl port-forward --namespace argocd svc/argocd-server 3000:443

# Get password - Use this password when loging in from the UI
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

Now visit localhost:3000 and log in:

username: admin
password: the password that got output when you executed the command above

Normally when working in GitOps, there is a known pattern where you have another Git repository where you keep all the manifest/infrastructure related resources in a declarative way. For this one I am going to create a new repository named kubernetes-demo-gitops.

➜  ~ gh repo create kubernetes-demo-gitops --private
✓ Created repository vjanz/kubernetes-demo-gitops on GitHub

I will create an Argo AppProject so we can separate the application and not leave them in a default project. Think of an AppProject as a namespace in Kubernetes, just as a layer to isolate the resources.

I will push argo-project.yaml to my new repo in projects/fastapi-app.yaml so the repo is not empty. Now my GitOps repo will look like this:

.
└── projects       
    └── fastapi-app.yaml

Now let’s connect the GitHub repo that we just created on ArgoCD. On ArgoCD UI navigate to Settings > Repositories > Connect Repo

We need to generate an SSH key and add it to the repository (the public key) and we need to add the private key in ArgoCD:

➜  .ssh ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/pc/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/pc/.ssh/id_rsa
Your public key has been saved in /home/pc/.ssh/id_rsa.pub

Now we need to do two things:

Add the public key that we generated to GitHub deploy keys
Add the private key on ArgoCD

Copy the public key that ends with the extension .pub and create it in the repository on GitHub:

Now navigate in ArgoCD and add the private key of the same key that you generated.

cat ~/.ssh/id_rsa

Now the repo is connected and we can continue to write the manifests. Argo CD supports a wide variety of templates, including:

Kubernetes manifests in YAML or JSON format
Helm charts
Kustomize bases and overlays (We are using this one)
JSONnet and Jsonnet templates
Ksonnet and ksonnet-lib
Templates in the Open Policy Agent (OPA) Rego language
Argo CD’s own JSONnet library (argocd-lib)

Install Kustomize from here

What is Kustomize — Convert manifests in Kustomization way — Start using SealedSecrets to manage secrets

Kustomize is a tool used to customize Kubernetes manifests, which are files that define the desired state of a Kubernetes cluster. It allows you to modify and extend existing manifests, or create new ones, without having to write everything from scratch.

For example, you may have a base manifest that defines the deployment of a certain application, and you want to use that same manifest in multiple environments, but with some slight variations. With Kustomize, you can create a separate “overlay” for each environment, that specifies the specific changes you want to make to the base manifest, and then apply those overlays to the base manifest to generate a final, customized version that you can use to deploy the application.

Now we’re going to take the plain manifests that we wrote at the beginning: deployment, ingress, service, etc. and convert them into the format of Kustomize.

In our GitOps repo create a new directory named apps. This would be the directory where we would list all the apps that we want to manage with ArgoCD. Remember, your Git repo according to GitOps can (and should) manage all the projects and infrastructure for your projects or organization. Copy the deployment and the service that we created in previous parts to apps/fastapi-service/base without making any modification for now. The repository structure should be like this.

├── apps
│   └── fastapi-service
│       ├── base
│       │   ├── deployment.yaml
│       │   ├── kustomization.yaml
│       │   └── service.yaml
│       └── overlays
│           ├── development
│           └── production
├── argocd
└── projects
    └── fastapi-app.yaml

On the base, I added resources that will be part of any environment, but in overlays > development and production I will add stuff that is related only to those environments. Simply said, add everything that is common for all the environments on base and anything that is specific to the environment to the overlays/

Let’s build the base first. In the base directory, I am going to paste the deployment and service as they’re and then we’ll do some modifications. You can also see that I’ve included a new file named kustomization.yaml where I define which resources I want to include.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

Normally when you create a new programming feature and push it to the GitHub repository we want to deploy that change. What happens is we create a new docker image and we push it to the registry. After that, we update the Kubernetes deployment to use the new image. So image has to be changed dynamically for example vjanz/kubernetes-demo:v1 can be one version and another one can be vjanz/kuberentes-demo:v2-my-feature

This is where the beauty of Kustomize comes in, as we can put some placeholders in our manifest and then update them as we want. Edit base/deployment.yaml and update:

containers:
  - name: kubernetes-demo
    image: valonjanuzaj/kubernetes-demo:latest

to:

containers:
  - name: kubernetes-demo
    image: example-image

I updated the image name to my-image which doesn’t make so much sense now, because that is not even a valid image name, but now we can use Kustomize to update the my-image to something else in an indirect way.

# change directory to base
$ cd apps/fastapi-service/base
$ kustomize edit set image my-image=valonjanuzaj/kubernetes-demo:v2

This command won’t change deployment.yaml directly, but it will update the kustomization.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml

# It will add the following part 
images:
- name: my-image
  newName: valonjanuzaj/kubernetes-demo
  newTag: something

Next time we use Kustomize to build a specific environment, the tool will look at kustomization.yaml file and it will update the values accordingly, let’s check it as we build the base environment manifests with kustomization:

# Assume we are in the base directory
$ kustomize build apps/fastapi-service/base

If you see the output and look carefully, you can see that the image is not my-image but it’s whatever we edited before:

....  
 - envFrom:
        - secretRef:
            name: demo-secrets
        image: valonjanuzaj/kubernetes-demo:something # Updated by Kustomize
        name: kubernetes-demo
        ports:
        - containerPort: 8000

Awesome isn’t it? — Now let’s imagine a scenario where CI (Continuous Integration) pipeline builds a new docker image and tags it with some random hash for example valonjanuzaj/reponame:0447995 then we can easily update the manifests with kustomize and use the newly generated image (we’ll do exactly this when we implement the CI/CD pipeline in the beginning).

Now remove the images key from kustomization.yaml that is on the base directory, because we will start to create our development environment in our GitOps repo. Onoverlays/development we create a new file named kustomization.yaml which holds a reference to the base file plus you can add any additional resources:

resources:
  # references everything in base directory, as we want to include them
  - ../../base 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kubernetes-demo-dev # we put the namespace here

Now we have imported deployment and service which are common for all the environments, but we see that we added a namespace to separate resources and we will add specific resources that only make sense for a specific environment. Let’s start by adding ingress for development.

Now we need to create an A record for this subdomain, but I will not go through how to do it as It’s explained above. Let’s add this in kustomization.yaml which is in the development directory.

resources:
  - ../../base
  # added ingress to kustomization on /overlays/development as
  # I want to use this ingress only for development
  - ingress.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kubernetes-demo-dev

In the first part when we deployed the application in the traditional way, we say that secret management becomes hard as it was easy to decrypt a base64 value and it wasn’t safe at all to push the secrets in the repository. In GitOps methodology the secrets should also be part of the repository as we want to build a system that is easily reproducible. So let’s find a way to make the secrets safe even when they’re in the repository.

Secrets management in GitOps — SealedSecrets

SealedSecrets is a Kubernetes-native solution for managing secrets using GitOps. It allows you to encrypt sensitive information like passwords, API keys, and certificates and store them in your Git repository while keeping them securely encrypted at rest and in transit.

To start working with SealedSecrets we need two things:

Install SealedSecrets in the cluster
Install kubeseal locally (CLI tool) to encrypt the secrets

To install SealedSecrets in the cluster we can install it again through a helm chart:

➜  ~ helm search repo sealed                  
NAME                          CHART VERSION APP VERSION DESCRIPTION                                       
my-repo/sealed-secrets        1.2.1         0.19.3      Sealed Secrets are "one-way" encrypted K8s Secr...
sealed-secrets/sealed-secrets 2.7.3         v0.19.4     Helm chart for the sealed-secrets controller.

# Installation
$ helm install sealed-secrets my-repo/sealed-secrets --namespace kube-system

The command will install a controller in the cluster in kube-system namespace and it will also create a certificate that will be used to encrypt the secrets. This is great because even though we commit the secrets in the repo, the secrets are encrypted with a certificate that exists only in our cluster, so they cannot be decrypted with a random certificate.

To install the kubeseal tool locally, look for the instructions here. After you install the kubeseal, we can easily create secrets that we can push to the repository. Let’s grab the secret for Postgres that we created earlier and let’s convert it to a SealedSecret.

Our old secret.yaml looks like this:

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: demo-secrets
type: Opaque
data:
  POSTGRES_USER: cG9zdGdyZXM=
  POSTGRES_PASSWORD: TDVYT3lacTViUg==
  POSTGRES_PORT: NTQzMg==
  POSTGRES_DB: a3ViZXJuZXRlcy1kZW1v
  POSTGRES_SERVER: cG9zdGdyZXMtcG9zdGdyZXNxbC5wb3N0Z3Jlcw==

Now let’s create a version of this but with kubeseal tool, which uses the certificate that exists in our cluster to encrypt the data:

kubeseal \
  --controller-name=sealed-secrets \ # name of the controller
  --controller-namespace=kube-system \ # namespace where controller is 
  --scope cluster-wide \ # To allow decryption from all 
  --format yaml < secret.yaml > sealed-secret.yaml # Output file

Now the generated sealed-secret.yaml can be added to the GitOps repo as it’s encrypted using the certificate that is inside the cluster, and the file would look something like this in my case:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  annotations:
    sealedsecrets.bitnami.com/cluster-wide: "true"
  creationTimestamp: null
  name: demo-secrets
spec:
  encryptedData:
    POSTGRES_DB: AgBLjgBM7CIlaEnxTWVedkmhd5HgO+Ep9HUNfwGNLe4K7tFS540xoVvvwV9g7UZJM547dcn3F5thfKal4ilah4UixQ1Y5w9ZG38jf4zo1AwiXaV+1YdvXjC7NLRAQhh3Ya8bwJT7QOJRS0vGioJRWkB9BY5JUHlgTJHvVcuMdwoD0vR34M3Z5XswrMr+uBBLyasrDSKtrhhIOxGTsMHtYWzfWm2UiJRp/s6hnsZG4N5IAFDB8HYcMCWGkTtZ3DGsX6XD30JrK6txpGmRb4PjYIFiKtFgp3uKWHS4XN2rgiK0VdpvgdZbgLVclX24NK+o/P+75cwHyVP6aGY9DIWFmovj3afaEfVYcnO91EC0l0V716HE6q3lKB204hpsZ3ioTPV+9MSzW6YixafX2t+J2wQiUd8q996v5lNWomRPdyjw0P2lXzJUbjkQjHeMK2UBu7xLz/ODb7QDhQpOnKGm/Wz1Tj/brd6vpDWVdntY8+9KDW1n3e6E4po8P2P09ihP4OtkFbD2jKC/56FUvV5y3wlP5XJxn9jqoZJBcq4PGS3cpngjUSimOfc9WpvG3wLhpXSFLlVXrWxoPeklBcX++sy4blFQ34JHXiD+48qk5GY7ceW45PcJOQv6REUUo2OCdWt9KUqrWiC6WgM1STJ7sScWNHL0ito9N3eNm+T8nwaI/jNwSLNCQNHkIUId8t86w9NqclSUsda1xQuGUqj/JU8=
    POSTGRES_PASSWORD: AgCBdPMRPxax/teLQw9fzLiYnXcDOZY+6Ly+eqem2qIePzg65guoTkqnMaQnCi3veUIi4RlimiMBoLxhMe6sHBX2LrEESfbtf2nRtZrs69QNG1lvOKMylXNpNHYISEKh6D3GWApcdYM/phXr0QbMZY6+CP0dMAn3tPTXBj1HZ3MJgwZYMKnKdAQbY49FprHwO0N28te96IvqagdlEIWKkYXBtazHG7lAIJDKfleHDyWa1FLjbtrjb+oXbx3eBd/scKagYdZc/I7EkelbMKNuzGgMRnKjaN/fez1dvwnzPWqRKgiAMQP05jfO15bjOWGqlwU2UFd1RQuB1gzrJViDo3tWI7vYXpegIWbBPes1jCC3y5hybxprGoWMkiMRXmj+anVbLRl1ZH+SRcZldCUOTzhUFI9J/vc2rb6kOj+aetR0eJKrhZ6/SkR5Sa9kHzakUDROmdC9cIzSVfZ3RA1aBSs56JCX7gLvDndPGpT/BFfMMt41DiA6O7TtM3CEM/qB+YDs9XFJVPsDlHkdMziv0bAR5jRNQTa5xCTSMt6VU/ef0+415pv1iJqau85TK5hSptSq/3Fn6ARhTtcw1RpvY3USd8PDVHMbQkdLW5SnEAFp37WUrjjqi7VcrGcVNGwQZbAzzyg4ns3EQ3p3TU4uXzbcTHeLHfzA/NKDRAzqMV1d3PWPlMuDfVqfQaXBPv8LKeFMWNt75URrUov4
    POSTGRES_PORT: AgCLN36HvCVsCc+7MT8IwP9bpcTJcoqh80USdqsUmhwEFYhzAo8Kux3/gwxnghDDvEya9WCQSAEbuAD6hX4Yo7T+sbSD5+oxDRZAU+YFPdYjAJs0tOhMAM2AwAmj1cJLoGFRUqqCFI2uFRExB1nJkr1e0QOgingnnLWPPvUIP0v/Gj3Bh/+FC925LppZcjJxuJ9xyRYuj1bqLoqAmw9YtXPYOArlAYn1t1+xDseSxvAYo7UU1+QCx82zBZVyXnEpyPaGjKsqIE9O4MaV4g62W7VbBNtRbK7lCinggjFzQLv/T8s0IVgmqGMtou4oPamtlZN8OThUZF2W5B+PBBBHsKXIiOAoWVCF27x3mEC7OLlpRwwVpic4y9nDHkLLg2V0Wpcnu8m41voyjQywT8fDP2ogl3sDHeUpouG2UduumWz4PZpDyNBriJ9cZUwa+de00mLftA170scDBqqw3hTkmvnbwoy/+L6mYjJn1/yl1lUUMd4ezYm1Ki7dwRzrXfvy/zIpHHQt2T1Av2JHpCEhlW8DoBPAP5C0nobUUSRxNqvBgeN81GRORohsfKjs76wwSJyyOXuU4Y9eLD0JDZuJ9aei6T/jAd8nubcef7jw2pwIuAQ5RrCGot8mHWQwXjd52/XWqyVUzcksqnMcsuK7u93/SKcKZb9tr2wMzLw75PjYRXYrTblyH5DAmOAhNVjLzculWGDZ
    POSTGRES_SERVER: AgBTTiwCjdNsETS/9aoJzvSVtPsUWfY5ZSnHmsDxQxgaPR1TWbZx1iNuZjOIw0XZm6T3OBnGoVKq09kQdMS0DOvOtZ+XNoP7S+88Ee82lyymMZyCBDlMcAUyHxR6Xa/RpqE1IFtZB5m/aubN23A9vevZkH73cwWTwl/CTVmsb9x0dY6W3NExOG5FQ7HaOsTrnyirDZSyLRYGnYNCeqzY1OFPiPQLcyYJoFwDfATQ7e7x0O3S6vhnj/KeUCxunsMpSsIavjdo/t8DgFtkUhNaWfCr3LWB4WdL2uIeCs8gebyzO7xaxR+/XKCHSrH9WeLHkQknwwfVdWFidGMtUXLvUvOM26EsrmIcAnioD6rxpRtIszWuDYNAl7qdk+s4WsXJFWuiUzALNioWuwUGDlICb6ViWGdlbTXI2W8PYQFHiuCTByGk93hc46T+jdsiM+gxzik5FdhFMAnsqZLzkvfqJfeBT5Sr/+AGfjke/SH5ses/KB+61NtCRiBwaL10S73KwKmzk6wC/zBv1sEICWJhf08Z+VU2q76HcJJXu9Ll66uvo/YWViNPR1W7Rt881QGzof1/MEf3Rc1xy0Ni65Z87mQEMs68wzjLb2eLpPk5x3AAPgjGVQw1CVgnutoOlwZevwayCP/5kNIE5Bzhm4pgx41sjeBItqZxvkXqgNpBIfcxKPs6rECgLRws1tRNId0xLhkkvfma5ckpC0M4UlagplpnfuriNzOnZv3MZ+q2
    POSTGRES_USER: AgCTJA1LSrYSdLn/3IrGyxjbI3LXIu5sN+Swt84RsLDgOvqAuqJ9aL8YHotWwumUBpxxdm3MdCUoTcWqnTgUSst9hRvgrO72YQr9ej2YBe5CBbPXbO6Y7PQNbm0rz73AKo7HAI5GOP77Kd/o2ovos9f1dWLayI7+6HDvl4FjBCHTQ8B+e2kJnBHSB8/P6PdGAOks5qMBK8hCMu9gUjpxygGgZDAiW+ITInbzKABh+6AbMgXSl7WRGZVgwGZ1Mlezh13rDOmMhy/68j2s5HaOlgnvGmJYqyS3k5NegO7nhTwvlqzQCI/zuca8e+PpoedIa6XG3c6Y91psrIwZn2e7KT2z5sXUSRq/IX+PnH+Qx+SJIJP/0UL2cuVal+DXyr1jA2lZSizFBW5ZpLteDVRMFcjwVj0gaz5EUkpeGJPRJa/yQVi/KnB/KHRx8VxxE6k2fDY3NKb9hsx6E0Mjwsc9fdHC5W18pbwc/QGiN7bO9WSCusoibletolLS9eS7YBEORG+4LiYyhzI9KAzp3a9FZu15R4CZW6QNvxjo7sOAMkH71U8JNpz1h77bpKelEgPpVPtS9WCFQ8acGsJn+kVpxV28TdOPX4CKBdiuv/pSKvfyZ54nxZToJZuQ8hwjjUf2LXcqYIqvIR7Oe5wO7JFobKIXHUVcPndhY5SPGjShH9+HhaTvAL1h6DkkeYsPMtoJ47vLgMDDgtnqdA==
  template:
    metadata:
      annotations:
        sealedsecrets.bitnami.com/cluster-wide: "true"
      creationTimestamp: null
      name: demo-secrets
    type: Opaque

Now I will be adding this file to the GitOps repo on overlays/development with name sealed-secret.yaml and I will update the kustomization file to include this resource my kustomization file will look like this:

resources:
  - ../../base
  - ingress.yaml
  - sealed-secret.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kubernetes-demo-dev

Let’s explore other features of kustomize by doing something that you may need. On the base deployment, we have 2 replicas of that pod, but what if we want to scale up the number of replicas but only for specific environments (let’s say development)? We can create a patch and we can apply that one only for development. So let’s create a new file named replica-count in development, as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-demo
spec:
  replicas: 3

and in kustomization, modify as:

resources:
  ...
# you add this key
patchesStrategicMerge:
- replica-count.yaml
...
namespace: kubernetes-demo-dev

Now if we check with kustomize:

$ kustomize build apps/fastapi-service/overlays/dev

we will see that the number of replicas for the development environment has changed to 3. This is the power of kustomize when it comes to separating the environments in style where you can extend what you want, but also override anything you want.

At this point, we have completed all the necessary resources to replicate a development environment using ArgoCD and Kustomize. Now we won’t apply these manifests through kubectl, or in any manual way because we want to keep everything consistent and reliable so instead, we are going to let ArgoCD apply these manifests and create them accordingly. Now we need to create a Continuous Integration pipeline for our sample app that builds the docker image, pushes it to the registry, and updates the GitOps repo with the name image name in order for ArgoCD to get the changes and deploy the new version of the application.

Continuous Pipeline with GitHub Actions — Continuous Delivery with ArgoCD

In the GitOps world, the CD part is normally handled by a tool that implements GitOps, like ArgoCD or Flux. We are using ArgoCD, so basically, ArgoCD is connected with the repo that holds the manifests, which I am referring to as the GitOps repo and when there is a change, ArgoCD synchronizes the change automatically in the cluster.

Let’s go to our codebase and create a CI pipeline. GitHub excepts the workflows to exist under .github/workflows so let’s create the directories and the respective files and also a branch for development:

➜  kubernetes-demo-app git:(main) ✗ mkdir -p .github/workflows           
➜  kubernetes-demo-app git:(main) ✗ touch .github/workflows/workflow.yaml
➜  kubernetes-demo-app git:(main) ✗ git add .
➜  kubernetes-demo-app git:(main) ✗ git commit -m "Add workflow files"                                     
 create mode 100644 .github/workflows/workflow.yaml
➜  kubernetes-demo-app git:(main) git push 
# I created a new branch as I want to associate development with the
# overlay that I created on GitOps repo
➜  kubernetes-demo-app git:(main) git checkout -b development       
Switched to a new branch 'development'
➜  kubernetes-demo-app git:(development)We are going to use GitHub actions to build our pipeline. GitHub Actions is a powerful tool for automating software development workflows. It allows you to trigger actions based on events in your GitHub repository, such as commits, pull requests, and releases. In this case, we will use GitHub Actions to trigger a deployment to our Kubernetes cluster every time a change is pushed to the master branch.

Our CI pipeline has two goals:

Build the docker image and push it to the registry
Update the repo that we use to store manifests with the latest image that is pushed

Let’s explain the workflow a bit. There are three jobs:
build: This job runs on an ubuntu-latest environment and checks out the code from the repository. It then sets up Python 3.9 and installs any dependencies specified in the requirements.txt file.
build-and-push: This job also runs on an ubuntu-latest environment and sets up QEMU and Docker Buildx. It then logs in to Docker Hub using the username and token stored as secrets, and builds and pushes a Docker image to Docker Hub with the tag valonjanuzaj/kubernetes-demo:$github.sha, where github.sha is the commit SHA.
update-manifest: This job checks out a separate repository named vjanz/kubernetes-demo-gitops, which contains the Kubernetes manifests and updates the development manifests in the $K8S_YAML_DIR/overlays/development directory with the new image version, which is built and pushed in the build-and-push job. The changes are then committed and pushed back to the repository.

Basically what happens is, each time we create a Pull request or we push in the development branch, a new image will be built and pushed to the repository. Aside from this, the pipeline will checkout in our GitOps repo to update the image with the latest one based on the environment that the Pull Request or push has happened.

There are some secrets that are being used in the workflow. To set up GitHub secrets, see the instructions here, and to set up GitHub personal token (PAT) see the instructions here. The secrets corresponding to Dockerhub login information and GitHub personal token to be able to access the other repository.

The updated manifests in the GitOps repository are then pulled by ArgoCD, which ensures that the deployed application in the cluster is in sync with the desired state defined in the GitOps repository. This way ArgoCD ensures that the application version deployed in the cluster is always up-to-date and aligned with the version in the GitOps repository.

That’s everything, we have set up our CI pipeline which will build, push the image and then update the repository which holds the manifests. After the update is done, ArgoCD will see the changes and it will update the cluster accordingly.

Check if the pipeline is working as excepted

Now let’s check if the pipeline that we build is working as excepted. Normally all the jobs should pass, the image should be built, and there should be a commit in the GitOps repository with the new image tag.

Let’s make a change in one of the routes and push the code!

@app.get("/health")
def health():
    return {"status": "App is running!!"}

$ git add .
$ git commit -m "Update /health endpoint"
$ git push

If everything is set up correctly (keep an eye on GitHub secrets) the workflow should be complete without any errors. If we check the GitOps repo we should see an update on overlays/development since we pushed on the development branch. We have configured that when a change is made in the development branch we deploy to a development environment on Kubernetes and we can configure it so that when we push to the main we update the production environment (this is based on your preferences).

Perfect, so now let’s just add our application to be monitored byArgoCD as this is the only step missing in the picture.

Setting up the application on ArgoCD — Automating the whole workflow!

Now that the continuous pipeline is set up and the application is continuously being integrated, all we need to do is to set up the ArgoCD application, which will listen for changes in a specific directory, and if there is any change it will automatically deploy. In our case, we have added our manifest at: apps/fastapi-service/overlays/development.

There are at least two ways to add an application in ArgoCD:

Configuring all the options from the ArgoCD UI.
Adding the application in a declarative way (I recommend this one)

Creating an Argo application in a declarative way is generally considered to be better than creating it from the UI for a few reasons:

Reproducibility: Declarative manifests allow you to version control your application configurations, making it easy to roll back to a previous version if something goes wrong.
Automation: Declarative manifests can be easily automated, allowing for repeatable and consistent deployments.
Audibility: Declarative manifests provide a clear and concise representation of the desired state of the application, making it easier to understand and audit the configuration of the application.
Portability: Declarative manifests can be easily ported across different environments, allowing for simpler migration and disaster recovery.
Easier to scale: Declarative manifests can be easily scaled up or down with minimal changes, making it easier to manage the application as it grows.

Below find the Argo application for our application in the declarative form:

Let’s explain what those configurations mean:

name — the name of the application that is installed on ArgoCD (can be anything)
project — name of the project we want this application to be associated with
repoURL — the repo that we have linked with our ArgoCD (where we keep the manifest, GitOps repo)
path — Where is the application manifest located
destination, server — With ArgoCD you can manage multiple clusters, so in this case I am telling ArgoCD to install on the same cluster where ArgoCD is installed
namespace — Which namespace do we want the application to be deployed at
syncPolicy, automated — means that any change will be automatically synchronized without any manual interventions

the other configuration seems quite self-explanatory, so I will not go on for each of them.

Let’s create the application:

$ kubectl apply -f fastapi-service-development.yaml

Now the structure on the repo looks like this:

├── apps
│   ├── argocd
│   │   └── fastapi-service-development.yaml
│   └── fastapi-service
│       ├── base
│       │   ├── deployment.yaml
│       │   ├── kustomization.yaml
│       │   └── service.yaml
│       └── overlays
│           ├── development
│           │   ├── ingress.yaml
│           │   ├── kustomization.yaml
│           │   ├── replica-count.yaml
│           │   └── sealed-secret.yaml
│           └── production
└── projects
    └── fastapi-app.yaml

This is the application that we are deploying for a development environment. Note that if you want to separate another one for production, you should create another manifest for it too, giving its respective configurations.

Now if we head back to the ArgoCD UI, we can see that the application should be up and running!

Awesome isn’t it? — Now note that you cannot make any changes from outside the cluster, all the changes should happen from the Git Repository. For example, if we delete a pod from outside the cluster with kubectl or any other tool, ArgoCD will look at the state of the cluster and the state defined in the GitOps repository, and where there is a change between states, ArgoCD will automatically choose the one that is on GitOps repository and this is really good as you have only one source of truth when it comes to writing and managing these files.

Now let’s see if the changes are applicable if we change something, like changing the number of replicas. I am going to update this file to make the deployment have only one pod (1 replica).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-demo
spec:
  replicas: 1 # this was 3

Now let’s just push the changes to the GitOps repository:

$ git add .
$ git commit -m "Change replica count"
$ git push

and we can see that two pods will get immediately killed by ArgoCD as the state on the GitOps has changed compared to the one on the cluster. Keep in mind that these two should always be in sync!

Since we tried that, let’s try to change the number of replicas with kubectl and see what happens:

$ kubectl get deployment -n kubernetes-demo-dev
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-demo   1/1     1            1           30m

$ kubectl scale --replicas=3 deployment/kubernetes-demo -n kubernetes-demo-dev
deployment.apps/kubernetes-demo scaled

At first, you may think that you changed the number of replicas and everything is going to be fine but as soon as ArgoCD detects the change, will rollback the change to the state that is defined on the repository which was (1), and it will kill both newly created pods.

Wrapping up

So we have completed the full workflow, from creating the application and running it locally to Kubernetes using the best practices out there. I hope you find it helpful and I am sure that you learned a lot from this article as I dedicated a lot of time to writing it, using my knowledge regarding the topic and the problems related to it.

If you want to support my work, you can buy me a coffee by clicking the image below 😄

BuyMeCoffe

If you have any questions, feel free to reach out to me.

Connect with me on LinkedIn, GitHub

Links and resources

Part 1, Part 2

Github repo for this part:
https://github.com/vjanz/kubernetes-demo-app

From local development to Kubernetes — Cluster, Helm, HTTPS, CI/CD, GitOps, Kustomize, ArgoCD — Part[1]

Valon Januzaj — Fri, 03 Feb 2023 00:48:53 +0000

From local development to Kubernetes — Cluster, Helm, HTTPS, CI/CD, GitOps, Kustomize, ArgoCD — Part[1]

Nowadays you may hear a lot about Kubernetes… You open the browser and search for it and all you get is some resources that are really advanced and you can’t even understand what is happening or basically, you don’t see where is your problem fitting into the frame. At least, this was my experience when I started digging around this topic.

Now I wrote this article to share my knowledge and to make your life easier when it comes to using solving problems with Kubernetes using the best practices.

I will go in an ordered way from explaining what Kubernetes is, how to prepare local development, deployment, etc. each step-by-step and explaining in detail what each component is and why you need it in the bigger picture. Are you ready? — Let's go!

Since the article is going to cover a lot of concepts, it’s separated into two parts to make it easier for you to read and follow along. Please find below the link for the second part of this article. This article assumes that the reader has basic knowledge of programming and Kubernetes.

Tutorial steps

The tutorial series progresses with increased complexity, where:

Part 1(This one):

What is Kubernetes and its most important components
Build a basic Python application using FastAPI
Dockerize and run it locally with docker-compose
Setting up a Kubernetes Cluster in the cloud
Deploy our app to Kubernetes in the traditional way
Deploying a PostgreSQL database in our cluster using Helm
Configure domain, Ingress Controller, HTTPS

Part 2 includes (click to go to Part 2):

Introduction to GitOps — ArgoCD installation
Using Kustomize to write Kubernetes manifest
Secret management in Kubernetes with SealedSecrets
Add a basic Continuous integration pipeline with GitHub actions
Creating and running our services in ArgoCD

If you know Kubernetes architecture, you can skip the following part (What is Kubernetes), even though a refresh is highly recommended

What is Kubernetes

Kubernetes is a system for managing and orchestrating containerized applications. In simple terms, it allows you to define how your applications should run, and then it manages the running of those applications for you. It can run on a variety of different platforms, including on your own infrastructure or in the cloud. Kubernetes is especially useful when you have multiple applications or microservices that need to run together, as it can help you to automate the deployment and scaling of those applications, and it makes it easier to manage the entire system.

API Server — The Kubernetes API server is the central way to interact with a Kubernetes cluster. It exposes the Kubernetes API, which is a set of RESTful APIs that you can use to create, update, delete, and list the objects in a cluster

Scheduler — The Kubernetes scheduler is a component of the Kubernetes system that is responsible for assigning pods (the smallest deployable units in Kubernetes) to nodes in the cluster. When a pod is created, the scheduler selects the node where the pod should run based on the resource requirements of the pod, the resource availability on each node, and other constraints that may be specified by the user.

Controller Manager — The Kubernetes controller manager is a daemon that runs on the Kubernetes master node and is responsible for the management and execution of various controllers

etcd — is a distributed key-value store that is used by Kubernetes to store data that needs to be shared across all of the nodes in the cluster. This data includes information about the state of the cluster, such as the configuration of the various applications and services that are running on the cluster.

Pod — a group of one or more containers that are deployed together on a host. Pods are the smallest deployable units in Kubernetes and provide a higher level of abstraction than individual containers.

Node — may be a virtual machine or a physical machine, depending on the cluster. Each node has a Kubernetes agent (also called a “kubelet”) that communicates with the control plane components and runs containers on the node. Docker is also installed on each node as it’s used to manage the containers that run inside a pod.

Now that we know a bit about the architecture of a Kubernetes cluster, let’s dive into code and build a simple Python web application that defines some API endpoints, connects to a Postgres database, has some environment variables and also has Dockerfile and docker-compose to simulate the application locally.

Building a basic Python application locally — Run with Docker & docker-compose

To save time here, I have prepared a basic Python application developed on FastAPI and if you want to follow along you can get the code from GitHub Repository here.

├── app                   # Base directory
│   ├── crud.py           # Crud Logic 
│   ├── database.py       # Database engine
│   ├── main.py           # API endpoints and FastAPI initialization
│   ├── models.py         # Database models
│   └── schemas.py        # Schemas used to exchange data in API
├── docker-compose.yaml   # docker-compose with web and postgres
├── Dockerfile            # Dockerfile for FastAPI application
└── requirements.txt      # Requirements for project

I will focus on the points that are important and that you should keep in mind while developing an application for Kubernetes or other platforms that are related to containerized applications like:

Developing a service in containerized methodology
Giving environment variables in a dynamic way
Running a service locally with docker-compose

I am stopping at this point as this as its the beginning of the whole process as Kubernetes itself is based on containers (running containerized applications).

The sample app in the repo is a basic FastAPI application that I extracted from official docs, but I added a PostgreSQL database to make the whole tutorial a bit more challenging. The application is very simple, it exposes some endpoints to create Users and Items.

To start the application locally, simply copy .env_example to .env and run the application with Docker

$ cp .env_example .env
$ docker-compose up -d --build

The application should be up and running, and you should be able to see the API definitions at http://localhost:8000/api/docs

Let’s make sure the service is working as excepted by making some API requests:

# Create a user

curl -X 'POST' \
  'http://localhost:8000/users/' \
  -H 'Content-Type: application/json' \
  -d '{
  "email": "test@demo.com",
  "password": "somepassword"
}'
------------------------------------------------------------
{"email":"test@demo.com","id":1,"is_active":true,"items":[]}                                        ➜  ~

# Create a item for the user with id 1

curl -X 'POST' \
  'http://0.0.0.0:8000/users/1/items/' \
  -H 'Content-Type: application/json' \
  -d '{
  "title": "Item1",
  "description": "Some Description"
}'

Excepted response:
{"title":"Item1","description":"Some Description","id":1,"owner_id":1}                                  ➜  ~

So as excepted, the users and the respective items are being created and stored in PostgreSQL which is running in Docker. I added a PostgreSQL on purpose to also explain how you connect a service to a database running on Kubernetes.

Setting up Kubernetes Cluster

To setup a Kubernetes cluster you can decide if you want to create locally with minikube or kind, or create it in a cloud provider like AWS, GCP, DigitalOcean, etc. in their respective platforms for Kubernetes like EKS, GKE, DOKS.

For this demo, I want to be as realistic as possible, as I want to also set up:
Domain, HTTPS, Reverse Proxy, etc. so I am going to use a cloud-based cluster in DigitalOcean (similar steps are applicable in other cloud providers) or if you follow along using Minikube or Kind.

Steps to set up a Kubernetes cluster may be different in other cloud providers, but the concept are almost the same, so follow along:

Here are the general steps for setting up a Kubernetes cluster on DigitalOcean:

Sign up for a DigitalOcean account here
Navigate to Kubernetes
Click “Create a Kubernetes Cluster”
Fill up the necessary information, and click “Create Cluster”

We set up a Kubernetes cluster, which is located in Germany, with 2 nodes, which we named my-cluster while the node pool name is my-node-pool . It’s always recommended to have at least two nodes, as Kubernetes concepts are regarding scaling the services between the nodes, so if one goes down, the pods can be created in the other nodes, so in order to keep your services up and running is highly recommended to have at least two nodes.

After creating the Kubernetes Cluster, you need to configure tools to connect to the cluster. The CLI tool to interact with the Kubernetes cluster is kubectl which you can easily download from here.

kubectl requires an active context in order to know in which cluster to make the request. After the cluster has been provisioned, you can download the kubeconfig, and then you can make a request against the cluster.

For now, navigate to the directory in which you have downloaded the config and export a variable named KUBECONFIG . Remember, if you configure this way, you always need to export this variable when you change the terminal.

$ export KUBECONFIG=name_of_downloaded_file.yaml 

# on windows you can use setx

To verify that we are successfully authenticated to cluster, run a simple command:

$ kubectl get nodes

------------------------------------------------------
NAME                 STATUS   ROLES    AGE     VERSION
my-node-pool-mhag8   Ready    <none>   7m4s    v1.25.4
my-node-pool-mhagu   Ready    <none>   6m54s   v1.25.4

Deploying the application to Kubernetes — Traditional Way

To deploy our services to Kubernetes, we need to write the manifests in a declarative way, speaking of which we need to create:

Please read each one from the official docs on kubernetes.io

Before we do so, let’s push our docker image to a registry (You can skip this step and use my image instead as I am leaving it as public, but feel free to create your own registry if you want to customize for your needs)

I am using the docker hub registry on which I have created a public repository. (How to create a Docker Hub Repository). After creating the repository, I need to rename and tag the image that I built with docker-compose to the name of the repo that I created on Dockerhub

$ docker login # login to your dockerhub account
$ docker tag fastapi-app_app valonjanuzaj/kubernetes-demo:latest
$ docker push valonjanuzaj/kubernetes-demo:latest

Now that we have the application image on the docker hub, we can start creating the manifest for our application. Let’s start by creating a deployment and service. To simplify, I am using the same file and I am separating the manifest with --- which is valid YAML syntax.

To deploy the application in Kubernetes you need to execute:

$ kubectl apply -f deployment-service.yaml

The application will get deployed, but it won’t work as it tries to access the database which we still don’t have. Remember, in the local application, we defined some environment variables regarding the Postgres database and here we still didn’t add any environment variable for the pod.

Deploying a PostgreSQL Database

There are several ways how you can deploy a database in the cluster (you can also use a remote database hosted in the cloud, but we want to make this practice more challenging so we can touch more concepts regarding Kubernetes). You can manually write all the manifests (deployment, service, secrets, volumes, etc) or you can use Helm to deploy the database and many other services. I highly recommend using helm charts (verified ones) to deploy this kind of service instead of writing these by yourself.

Helm is a package manager for Kubernetes. It is used to automate the installation, upgrade, and management of Kubernetes applications.A Helm package is called a “chart”. Charts are made up of a collection of files that describe a related set of Kubernetes resources. For example, a chart might contain a deployment, a service, and a cluster role.

After installing Helm locally, you need to find the chart that you need to install. I usually go with Bitnami charts. To install any chart with helm, we first need to add the repo from which we want to pull the charts, and then we can install after finding the chart that we want to.

$ helm repo add my-repo https://charts.bitnami.com/bitnami
$ helm install postgres my-repo/postgresql --namespace postgres --create-namespace

After you install the chart, all the instructions to connect to the database will be shown. Basically, you would need to port-forward the service of PostgreSQL and connect to it locally to create a new database or just manipulate it.

The command above will install PostgreSQL in the cluster, it will attach the necessary resources and will make it ready to be used by applications. What is even more interesting is that when the chart gets installed it will assign some persistent volumes for data based on the cloud provider that you’re using, in this case, do-block-storage, so even if we delete this chart, we can still re-wire a database to the volume without losing the data.

Let's connect to the database and create a database named kubernetes-demo for our FastAPI service.

Connect the application to the database

Now that the database is created and running in our cluster, we can set up the environment variables to connect our FastAPI service to this database. To do that, we need to provide some environment variables, just like we did locally on the application with docker-compose, but now to the container that is running on the pod that we deployed, and those are:

POSTGRES_USER
POSTGRES_PASSWORD
POSTGRES_DB
POSTGRES_SERVER
POSTGRES_PORT

We’ll these are considered secrets and you need to be careful about the way you store them. Currently, we are going to create a secret and then we’re going to make some changes in deployment-service.yaml to reference the secrets that we have created because currently, we haven’t set any environment variable for the container.

Creating the secret: To create the secret, you need to encode the values with base64 which can be easily done on the terminal:

$ echo -n "value_to_encrypt" | base64
# Example
$ echo -n "SomeSecret" | base64
Output: U29tZVNlY3JldA==

Similarly, you encode all the values and then add them to the secret.yaml:

I’ve encoded real values of the Postgres database that we have created. Now create the secret with kubectl:

$ kubectl apply -f secret.yaml

Be very careful when you add the POSTGRES_SERVER; Since the postgres is deployed on a separate namespace named “postgres” and the app is running on “kubernetes-demo” namespace, you need to add the server like: ., which in our case would be something like: postgres-postgresql.postgres

These kinds of manifests are not safe to be pushed on the repository, since base64 can be easily decoded to plain text. We are leaving it like this for now as this is the purpose of the article, to show you the traditional way first then we’ll move to advanced ways of managing these kinds of resources.

Now let’s update deployment-service.yaml to reference the created secrets:

    spec:
      containers:
        - name: kubernetes-demo
         ...
         #######################################
         #Add the following part to the yaml file
          envFrom:
            - secretRef:
                name: demo-secrets

Apply the manifest again with:

kubectl apply -f deployment-service.yaml

And the application should be up and running. To access the app now we need to expose something, but for now, let’s just port forward and access it locally to see if everything is working fine:

$ kubectl port-forward --namespace kubernetes-demo svc/kubernetes-demo 8000:80

# The application should be up and running at http://localhost:8000/api/docs

We port-forwarded the service of the application named kubernetes-demo which has port 80 to local port 8000. Now we can access the application at http://localhost:8000/api/docs

Setting up the domain—Installing Ingress Controller — Securing the traffic with HTTPS

Now let’s expose our application for the whole world to see it. To do that we need a domain (you can use the load-balancer IP if you don’t have a domain and don’t want to set one, so bear with me). If you want to get one domain, go check out Namecheap as you can find cheap domains to play around with or for your personal purposes.

I have already one domain purchased get.tech, so I am going to use that one. To manage the domain from DigitalOcean (and any cloud provider), you need to add the domain to it and then point the nameservers to that provider from your domain registrar. To add the domain in DigitalOcean, navigate to Networking > Domains and add your custom domain.

After you add the domain you need to point the nameservers to DigitalOcean (see here how), which are:

ns1.digitalocean.com.
ns2.digitalocean.com.
ns3.digitalocean.com.

After a bit, your domain should be pointing to the DO nameservers and you can manage your records from the DigitalOcean dashboard.

Setting up Kubernetes ingress controller

Note that the same steps are applicable to other cloud providers

An ingress controller acts as a reverse proxy and load balancer. It implements a Kubernetes Ingress. The ingress controller adds a layer of abstraction to traffic routing, accepting traffic from outside the Kubernetes platform and load balancing it to Pods running inside the platform through internal services. The whole flow is as follows:

A user makes a request to the URL that we have exposed. The load balancer accepts the request and based on the mappings that are defined on the ingress forwards the request to a specific service. The service then forwards the request to one of the pods that are available to handle that request.

Now that we understand a bit about what is ingress controller, we need to install one in our cluster.

You may do these steps manually but it would take a lot of work to implement one of the ingresses that are already out there, so I highly recommend installing one from the versions that are out there using Helm. To do that we can install it again from the Bitnami repo that we added before when installing the PostgreSQL, so let’s go ahead and install the nginx-controller implementation:

➜  ~ helm search repo nginx           
NAME                             CHART VERSION APP VERSION DESCRIPTION                                       
my-repo/nginx                    13.2.21       1.23.3      NGINX Open Source is a web server that can be a...
my-repo/nginx-ingress-controller 9.3.24        1.6.0       NGINX Ingress Controller is an Ingress controll...
my-repo/nginx-intel              2.1.13        0.4.9       NGINX Open Source for Intel is a lightweight se...


$ helm install nginx-controller my-repo/nginx-ingress-controller --namespace nginx-controller --create-namespace

The controller will be installed at nginx-controller namespace in a bit. What happens when we install the controller in a managed Kubernetes service through the helm chart is that a LoadBalancer will automatically get created for you, and you can use that created LoadBalancer to send the traffic to your cluster (see picture above, we are implementing the same idea). If we navigate to Networking > Load Balancers we see that the LoadBalancer is created and is ready to be used. Similarly, a load balancer would be created in other cloud providers such as AWS, GCP, etc.

Now that the domain is created, and the LoadBalancer is ready, we can go ahead and create a new subdomain to use for our application. To do so, we navigate again to:

Networking > Domain > Choose our domain and we create an “A domain”

An A record, also known as an “address record,” is a type of DNS record that maps a hostname to a specific IPv4 address.

Just be careful to redirect to the LoadBalancer IP that we have just created so then that Ingress can take control of the request and redirect as it should. Now we are ready, so let’s create an Ingress and map the subdomain to our FastAPI service.

Creating a Kubernetes Ingress

Create a new file named ingress.yaml and paste the following content.

It’s pretty clear right? We are mapping the host demo.*.tech to the service named kubernetes-demo (which exists in the same namespace), in the port 80, when path / is triggered. The ingressClassName should be the name of the ingress controller that we created.

Create the ingress with kubectl:

kubectl apply -f ingress.yaml

To check if the ingress is created and assigned to the right service, we execute the command belowto get more data about the Kubernetes resource:

kubectl describe ingress -n kubernetes-demo kubernetes-demo-ingress
-------------------------------------------------------------------------

Name:             kubernetes-demo-ingress
Labels:           <none>
Namespace:        kubernetes-demo
Address:          164.92.182.82
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                      Path  Backends
  ----                      ----  --------
  demo.vjanztutorials.tech  
                            /   kubernetes-demo:80 (10.244.0.123:8000,10.244.0.251:8000)

We can see that ingress is created successfully and it’s redirecting the traffic to the kubernetes-demo service, which is redirecting to the two pods that we have created for that Deployment. Perfect, now let’s see if we can access the application from the exposed host:

curl -X 'GET' \
  'http://demo.vjanztutorials.tech/health' \
  -H 'accept: application/json'
--------------------------------------------
{
  "status": "Running!"
}

Or just visit the service URL at: yourdomain.com/health in my case demo.vjanztutorials.tech/health

Everything working as excepted, but there is only one thing we can do better and that is securing our traffic with HTTPS, so why not, let’s do that! If you have doubts about what HTTPS is and why we need it please take a refresher here, and then let's move on.

Securing the traffic with HTTPS

HTTPS is a way to make sure that when you visit a website, the information you share with that website is private and can’t be seen by anyone else. It’s like a secret code between your computer and the website that makes sure that only you and the website can read what’s being sent.

To do so, we need an SSL/TLS certificate for our application. To retrieve one we need to communicate with a certificate authority (CA) to verify the authenticity of the domain and then issue a certificate that can be used to secure the communication between the application and the user.

We will install Cert-Manager to manage our SSL/TLS certificates in an automated way. Cert-manager is a tool that helps manage and automate the process of obtaining and renewing SSL/TLS certificates for your websites and applications. It works by communicating with certificate authorities (CA) to request and renew certificates and then automatically updating your web server or application to use the new certificate. This eliminates the need for manual intervention and ensures that your certificates are always up-to-date and secure.

# search
➜  ~ helm search repo cert-manager
NAME                 CHART VERSION APP VERSION DESCRIPTION                                       
my-repo/cert-manager 0.8.10        1.10.1      cert-manager is a Kubernetes add-on to automate...

# install
helm install cert-manager my-repo/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true

Now that we have cert-manager installed we need to install an Issuer in our cluster, which can be ClusterIssuer or Issuer.
The differences between them are that the Issuer is a namespaced resource and it is not possible to issue certificates from an Issuer in a different namespace while ClusterIssuer is almost identical to the Issuer resource, however, is non-namespaced so it can be used to issue Certificates across all namespaces.

I am going to use ClusterIssuer and Let’s Encrypt as CA

Substitute the email with your email and that should be it. Now create it:

$ kubectl apply -f cluster-issuer.yaml

To issue a TLS certificate for our domain, we’ll annotate ingress.yaml with the ClusterIssuer that we created, so let’s modify it as:

Apply the changes with kubectl:

$ kubectl apply -f ingress.yaml

And check if the certificate is created:

$ kubectl get certificate -n kubernetes-demo
--------------------------------------------
NAME                  READY   SECRET                AGE
kubernetes-demo-tls   True    kubernetes-demo-tls   1m


$ kubectl describe certificate kubernetes-demo-tls -n kubernetes-demo
---------------------------------------------------------------------
Normal  Issuing    2m   cert-manager-certificates-issuing          The certificate has been successfully issued

We can also see that a secret has been created in the same namespace named kubernetes-demo-tls which contains the tls.crt and tls.key which is used to encrypt the traffic between pods, services, and other resources within the cluster.

Perfect, now let’s visit our application through the browser to see if the connection is being through HTTPS.

Perfect, our application is now running on HTTPS, which provides a secure and encrypted connection for users. This means that any information shared on our website, such as login credentials or personal information, is protected from potential hacking or data breaches. Additionally, HTTPS also improves website performance and is a ranking factor for search engines.

Everything is working smoothly, but let’s start to think on:

How we are going to work in a team on this project
How we are going to continuously integrate and deploy our app
How we are going to build a deployment strategy that is secure and reliable, etc.

Summary

In this chapter, we learned what is Kubernetes and its components, creating and running a simple application locally, installing PostgreSQL in the cluster using Helm, and connecting our running application to it. After that, we exposed the application to the world with Ingress and secured the connections with HTTPS.

In the next part, I am going to discuss some of the downsides of the way that we organized the project and manifests and what we could do better using the most modern tools that are out there regarding Kubernetes and deployment strategies to make it developer friendly, reliable, and secure so stay tuned!

If you have any questions or any problem following along, feel free to reach out to me.

Connect with me on LinkedIn, GitHub

Links and resources

Part 1, Part 2

Github repo for this part:
https://github.com/vjanz/kubernetes-demo-app

Implement API Caching with Redis, Flask, and Docker [Step-By-Step]

Valon Januzaj — Wed, 09 Jun 2021 07:14:03 +0000

You want your API to be faster, more consistent, and to reduce the requests to the server? — That’s where caching comes into play. In this article, I will show you how to implement API Caching with Redis on Flask. I am taking Flask as an example here, but the concept about Caching are the same regardless of the technologies.

What’s caching?

Before we move into the practical part about implementing Caching with Redis and Flask, let’s first know what’s caching as a definition and learn it as a concept so you know then what would the use cases be.

Caching is the ability to store copies of frequently accessed data in several places along the request-response path. When a consumer requests a resource representation, the request goes through a cache or a series of caches (local cache, proxy cache, or reverse proxy) toward the service hosting the resource. If any of the caches along the request path has a fresh copy of the requested representation, it uses that copy to satisfy the request. If none of the caches can satisfy the request, the request travels to the service (or origin server as it is formally known). This is well defined with two terminologies, which are cache miss and cache hit.

Cache hit — A cache hit is a state in which data requested for processing by a component or application is found in the cache memory. It is a faster means of delivering data to the processor, as the cache already contains the requested data.

Cache miss — Cache miss is a state where the data requested for processing by a component or application is not found in the cache memory. It causes execution delays by requiring the program or application to fetch the data from other cache levels or the main memory.

As mentioned above, there are several ways to implement caching. That can be on the client-side through Web Caching, on the server-side through Data Caching (Relational Databases, Redis, etc), Application Caching through plugins that get installed on the application (ex: plugins on WordPress). For this tutorial we’re going to use Redis, to save the responses from the API, and then use those responses instead of making the requests to the server to fetch the data.

Flask and Redis — Implementation

Prerequisites:

Docker & Docker-compose
Flask
Python 3.*+

We are going to use docker to isolate our services, and then docker-compose to orchestrate the services together (putting them on the same network, communication between them, environment variables, etc). If you don’t know about Docker, I suggest you refer to the official docs here.

Project setup:

Create python virtualenv and install Flask, redis, flask-caching and requests:

$ python -m venv venv
$ source venv/Scripts/activate
$ (venv) pip install Flask redis flask_caching requests

Our application will look something like this:

/root
├── app.py                 - Application entrypoint
├── config.py              - Config file for Flask
├── docker-compose.yml     - Docker compose for app and redis
├── Dockerfile             - Dockerfile for Flask API
├── .env                   - Environment variables

So lets go ahead and create files that are necessary for this setup:

$ touch Dockerfile docker-compose.yml .env
$ pip freeze > requirements.txt
$ touch config.py app.py

What we are going to implement?

We are just going to make a simple endpoint which fetches the university data from Hipolabs universities API, and based on the country that we sent as a query parameter, we get a list with the universities for the specified country.

Let’s go ahead and in app.py create an instance of Flask, and use that to create an endpoint that fetches universities data.

So basically, based on the query parameter country it makes the request to the external API and gets back the data in JSON format . Let’s go ahead and try it:

$ export FLASK_APP=app.py      # To tell where your flask app lives
$ export FLASK_ENV=development # Set debug mode on
$ flask run

I will be using Postman to make the request because I also want to see the time that my request takes to process.

Okay, now we see that we have the results, and everything is working fine as excepted. With the red color, you can see the time that it took to get the data from that endpoint. We can try and make the same request several times, and the performance won’t change. That is because we’re always making a new request to the server. Our goal is to minimize this, and as explained at the beginning to make fewer requests to the server. So let’s go ahead and do that.

Add redis and dockerize the application

We saw that it worked fine locally, but now we want to implement caching, and for that, we’re going to need Redis. There are several approaches you can take here as:

Installing Redis (Officially compatible in Linux, not in Windows, see here)
Host a Redis instance and use that one (ex: Redis instance on Heroku)
Start the Redis instance with Docker (We are doing this)

We are going to dockerize the application and add Redis as a service so we can easily communicate from our application. Let’s go ahead and write the Dockerfile for the Flask application:

We don’t have command here to run the image, as I will use docker-compose to run the containers. Let’s configure docker-compose to run our application and Redis:

So we simply add two services, which are our application and Redis. For the application, we expose the port 5000 in and out, and for Redis, we expose 6379. Now let’s start the services with docker-compose.

$ docker-compose up -d --build

Our services should be up and running, and if we go again and make the same request as we did above when we were running the application without Docker, we will have the same output. To check if the services are running enter the following command:

$ docker ps

Now let’s configure our application to connect with Redis instance, and also to implement caching in our endpoint. We can go straight and set the variables directly in the code, but here I am trying to show you some good practices while developing with Flask and Docker. In the docker-compose from the above gist, we can see that for the environment variables I refer to the .env file, and then I use config.py to map these variables to the Flask application. For the flask-caching library to work, we need to set some environment variables, which are for Redis connection and caching type. You can read more about the configuration from the documentation of the library, based on the caching type that you want to implement.

# .e
CACHE_TYPE=redis
CACHE_REDIS_HOST=redis
CACHE_REDIS_PORT=6379
CACHE_REDIS_DB=0
CACHE_REDIS_URL=redis://redis:6379/0
CACHE_DEFAULT_TIMEOUT=500

In the .env we set some variables like caching type, host, db, etc. Since we have these variables mounted from docker-compose inside our container, now we can get those variables using the os module. Let’s get those variables in config.py and we’ll use them later to map the values to our Flask application.

From the configuration side of things, we’re good. Now let’s initialize the cache on top of Flask and integrate that with our application.

We have added a new decorator which is @cache.cached then we specify a timeout which is the time that this response will be cached in Redis memory. So basically after the first request, we will have this response stored for 30 seconds, after that there’ll be a fresh request that will update the memory again. The second parameter is query_string=True which in this case makes sense because we want to store the responses based on the query string that we store instead of the static path.

query_string — Default False. When True, the cache key used will be the result of hashing the ordered query string parameters. This avoids creating different caches for the same query just because the parameters were passed in a different order.

And we’re done, let’s build the containers again and test this out in action with caching in place.

docker-compose up -d --build

Now let’s go to Postman again, and do the same request on the universities endpoint.

For the first time we’ll have approximately the same time as we did when we weren’t using caching, but if we do the same request again, we’ll have significant improvements and all that thanks to Redis. So what we’re doing is, we are saving the response to an in-memory database, and then while the data are still stored there they’ll be returned from there, instead of making the request from the server.

Dive deeper? — Let us see that in action, by using a GUI tool, to query our Redis store. I am using TablePlus, for the sake of the visualization, but you can also use Redis CLI to query the data. To connect to our redis instance we will specify host as localhost and then for the port we enter 6379 just as we exposed in docker-compose .

After that, we can see the data that are being stored in our Redis instance. When there’s a response saved, you can see a db0 and if we look for more, we’ll see our cached response including [key; value; type; ttl].

We can clearly see that the response that is cached is /universities?* and that is available for the time that appears on the ttl. This section was a bit outside of the scope, but it’s good to know what is happening in the background.

So with that, we have implemented API caching with Redis and Flask. For more option please refer to the documentation of the flask-caching library which is a wrapper to implement caching around different clients.

Conclusions

So we implemented API Caching using Redis. This is a simple example, but it includes lots of details regarding this topic. Caching is really important when you write applications, as it helps a lot on the performance and when possible, you should implement it, but make sure that you’re targeting the right use case.

You can find the full source code of the article on the GitHub repository, with the instructions.
vjanz/flask-cache-redis

If you found it helpful, please don’t forget to clap & share in your social network or with your friends.

If you have any question, feel free to reach out to me.

Connect with me on: LinkedIn, GitHub

References:

https://www.cloudflare.com/learning/cdn/what-is-caching/
https://redislabs.com
https://docs.docker.com/
https://flask-caching.readthedocs.io
http://universities.hipolabs.com/

Asynchronous tasks in Python with Celery + RabbitMQ + Redis

Valon Januzaj — Sat, 24 Apr 2021 21:06:44 +0000

Asynchronous tasks in Python with Celery + RabbitMQ + Redis

In this article, we are going to use Celery, RabbitMQ, and Redis to build a distributed Task queue.
But what is a distributed task queue, and why would you build one?

A distributed task queue allows you offload work to another process, to be handled asynchronously (once you push the work onto the queue, you don’t wait) and in parallel (you can use other cores to process the work).

So it basically it gives you the ability to execute tasks in the background while the application continues to resolve other tasks.

Use Cases of Task Queues

The most basic and understandable example would be sending emails after the user is registered. In this case, you don’t know how much time is it going to get to send the email to the user, it can be 1ms but it can be more, or sometimes even not sent at all, because, in these case scenarios, you are not responsible or simply said you’re not aware of the task is going to be successfully done, because it’s another provider who is going to do that for you.
So now that you got a simple idea of how you can benefit from the task queues, identifying such tasks is as simple as checking to see if they belong to one of the following categories:

Third-party tasks — The web app must serve users quickly without waiting for other actions to complete while the page loads, e.g., sending an email or notification or propagating updates to internal tools (such as gathering data for A/B testing or system logging).
Long-running jobs — Jobs that are expensive in resources, where users need to wait while they compute their results, e.g., complex workflow execution (DAG workflows), graph generation, Map-Reduce like tasks, and serving of media content (video, audio).
Periodic tasks — Jobs that you will schedule to run at a specific time or after an interval, e.g., monthly report generation or a web scraper that runs twice a day.

Setting up the dependencies for Celery

Celery requires a message transport to send and receive messages. Some candidates that you can use as a message broker are:

For this tutorial we are going to use RabbitMQ, you can use any other message broker that you want (ex. Redis).

It’s also good to mention for what are we going to use Redis now since for the message transporter we are using RabbitMQ.
When tasks are sent to the broker, and then executed by the celery worker, we want to save the state, and also to see which tasks have been executed before. For that, you’re going to need some kind of data-store and for this one, we are going to use Redis.

For the result stores we also have many candidates:

AMQP, Redis
Memcached,
SQLAlchemy, Django ORM
Apache Cassandra, Elasticsearch, Riak, etc

To set up these services we are going to use docker as it’s easy to set up, it’s isolated environment and you can easily reproduce the same environment when you have a configuration (Dockerfile or docker-compose).

Project setup

Let’s start a new python project from scratch. First let’s create a new directory, create all the files necessary for the project, and then initialize the virtual environment.

$ mkdir celery-python && cd $_
$ touch __init__.py
$ touch tasks.py
$ touch docker-compose.yaml
$ touch requirements.txt

# create & activate the virtualenv

$ python -m venv env
$ source env/Scripts/activate

Now let’s install the project requirements. For this project, we are just going to need celery and Redis.

pip install celery redis

Now it’s time to configure docker-compose to run RabbitMQ and Redis. In the docker-compose.yaml paste the following YAML configuration.

Here we simply start up two services, by defining the image key to point to the image in dockerhub , mapping the ports host:docker and adding environment variables. To see what types of environment variables you can use with your image, you can simply go to the corresponding image in dockerhub, and see the documentation. For example you can check how to use RabbitMQ image here

Now, let’s initialize the celery app to use RabbitMQ as a message transporter and Redis as a result store.
In the tasks.py, let’s go ahead and paste the following code:

I tried to keep the code as minimal as possible, so you can understand the purpose of this tutorial.
As you can see, we have defined the URLs for RabbitMQ and Redis, and then we simply initialize the celery app using those configurations. The first parameter tasks is the name of the current module.

Then we have decorated the function say_hello with @app.task which tells that the function is marked as a task, and then can later be called using .delay() which we will see in a bit.

Normally we would have a module celery_app.py to only initialize the celery application instance, and then a separate moduletasks.py in which we would define the tasks that we want to run by celery.

Build and run services with docker

Now we only need to run the services (RabbitMQ and Redis) with docker. To run the images inside a container we simply run:

$ docker-compose up -d

This will take a while if you don’t have these images pulled locally. Then to verify that the containers are up and running we write:

$ docker ps

And you should see two services running, and additional information for each one, if not check the logs for any possible error.
Now let’s start the celery worker, and then let’s try to run some tasks with python interactive shell.

# Starting the Celery worker

$ celery -A tasks worker -l info --pool=solo

This will run celery worker, and if you see the logs it should tell that it has successfully connected with the broker.

Now let’s run a task.

# Running celery tasks

$ python
---------------------------------
Type "help", "copyright", "credits" or "license" for more information.
>>> from tasks import say_hello
>>> say_hello.delay("Valon")
<AsyncResult: 55ad96a9-f7ea-44f4-9a47-e15b90d6d8a2>

We can see that we called the function using .delay() and then passing the name argument. This method is actually a star-argument shortcut to another method called apply_async(). Then we see that we get <AsyncResult back which is the task that was passed to the broker, and after that will get consumed and finished in background by celery.

If you look at your worker now, you will see in the logs that the worker received a task and then after 5 seconds will tell you that the task finished successfully.

Now let’s run the same task but let’s put the results store in the game now. In the python shell let’s store the result in a variable, and then lets its properties.

If we didn’t have the backend configured at the celery (Redis), we couldn’t access these properties or functions, because by default it wouldn’t store any state, but since we have it, we can see and get the pieces of information about our tasks. If you wanna dig deeper you can access your Redis database with a tool like table plus or you can set Flowerto monitor Redis and RabbitMQ.

As you can see on the image above, all the tasks are stored in redis.

Wrapping up

In this article we have set up a python application with Celery, RabbitMQ and Redis from scratch. The purpose of the article was to show you what is task queue, what can we benefit from it, and how to implement.
The examples of the task are just for demonstration, but you can use the same configuration as I did on this one, adding tasks in the tasks module and the configuration in celery_app.py. See the docs here

I highly encourage you to use celery in your application as it quite useful when you have things that take longer time, you need to schedule tasks, etc.

If you read the article and found it useful don’t forget to clap.
If you have any question, feel free to reach out to me.
Connect with me on 👉 LinkedIn, Github