DEV Community: Kipchirchir Langat Emmanuel

Django automated deployments to Digital ocean using GitHub actions

Kipchirchir Langat Emmanuel — Fri, 13 Jan 2023 18:16:34 +0000

In this article, you will learn how to build an automated GitHub actions pipeline that deploys a Django application to a digital ocean droplet.

Prerequisites:

A GitHub account.
A docker hub account.
A digital ocean account.

1 Base application.

For this guide, we shall be using a base project that can be cloned here.

Navigate to your desired folder/directory.

cd Desktop

Clone the project

git clone https://github.com/manulangat1/github-actions-do-article.git

Create and activate a virtual environment (use whatever tool you are comfortable with to achieve this.)

python3 -m virtualenv venv 
source venv/bin/activate
#install the required dependancies 
pip install -r requirements.txt

Run the Django development server to confirm that all is well.

python3 manage.py runserver

2. Setting up the GitHub actions file.

On the root of the project create a folder .github, and inside it create another folder named workflows and a file inside the workflows folder named integrations.yaml. The structure should be as illustrated in the image below.

Paste the following code into the intergrations.yaml file

name: Continuous Integration and Delivery # workflow name

on: # this is the entry point to the events, we specificy when the actions should be run
  push:
    branches: [develop] #specificy which branch should the workflow be triggered
  pull_request:
    branches: [develop] #specificy which branch should the workflow be triggered

jobs:
  testing-docker-compose-auto-deploy-digital-ocean: # name of the job 
    runs-on: ubuntu-latest # specify that the app will run on the latest version of ubuntu
    steps: # steps that should be followed while building and deploying the image.
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v2
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      - uses: actions/checkout@v2

3. Dockerize the application.

Create a new file named Dockerfile at the root of the project and paste the following lines of code.


# FROM python:3.9.6-alpine 
FROM python:3.10.1-slim-buster 

# WORKDIR 
ENV APP_HOME=/app
RUN mkdir $APP_HOME
RUN mkdir $APP_HOME/staticfiles
WORKDIR $APP_HOME

LABEL maintainer='manulangat1'

LABEL description="This is an application that shows how to create an automatic ci pipepline that deploys the django app to a digital ocean droplet" 

ENV PYTHONDONTWRITEBYTECODE=1

ENV PYTHONUNBUFFERED=1  

ENV DEBUG=False

ENV ENVIRONMENT=staging

RUN apt-get update \
    && apt-get install -y build-essential \
    && apt-get install -y libpq-dev \
    && apt-get install -y gettext \
    && apt-get -y install netcat gcc postgresql \
    && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
    && rm -rf /var/lib/apt/lists/*

RUN pip3 install --upgrade pip 

COPY requirements.txt $APP_HOME/requirements.txt 

COPY . $APP_HOME/
RUN pip3 install -r $APP_HOME/requirements.txt 

COPY /entrypoint /entrypoint
RUN sed -i 's/\r$//g' /entrypoint
RUN chmod +x /entrypoint

ENTRYPOINT ["/entrypoint"]

Once done, create a new file docker-compose.yaml and paste the following lines of code.

version: '3.8'

services:
  web:
    build: 
      context: .
      dockerfile: Dockerfile
    command: gunicorn do-guide.wsgi:application --bind 0.0.0.0:8000
    volumes:
      - static_volume:/home/app/web/staticfiles

    image: manulangat/auto-deploy-do-github-actions:latest
    ports:
      - "8000:8000"
    networks:
      - bms_staging

volumes:
  static_volume:

networks:
  bms_staging:

With this in place, you can now add the steps to build the docker images in the GitHub actions workflow file

      - uses: actions/checkout@v2
      - name: Build the stack
        run: docker-compose -f docker-compose.yaml up -d --build

4 Deploy to Dockerhub.

We want the latest image to be deployed to docker hub, Login to your docker hub account and create a new repository.

Grab the image name eg manulangat/auto-deploy-do-github-actions:tagname and head over to the workflow file.

In the workflow file, add a step that logins it to docker hub using your username and password, add these as repository secrets on GitHub

Once done with that, head over to the workflow file and add a step that login into Dockerhub.

- name: Login to DockerHub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

The workflow file should at this point resemble the one below

name: Continuous Integration and Delivery

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  testing-docker-compose:
    runs-on: ubuntu-latest
    steps:
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - uses: actions/checkout@v2
      - name: Build the stack
        run: docker-compose -f docker-compose.yaml up -d --build

      - name: Get docker logs
        run: docker ps

      - name: Login to DockerHub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

At this point, the step that pushed the built docker image can now be added after the login to Dockerhub step.

- name: Push to dockerhub
        run: |
          docker-compose push # this pushes the image to the repo that we have defined in the docker-compose.yaml file.

And a step that stops the containers

- name: stop containers
        run: docker-compose -f docker-compose.yaml down --volumes

5. Auto-deploy to digital ocean droplet.

Head over to the digital ocean page and log in, upon logging in, create a Docker droplet, for this tutorial, a shared CPU of 2Gb and 50Gb SSD disk is sufficient. Give the droplet whatever name you see fit and hit the create droplet button.

To configure firewalls and enable port 8000 and port 7000, which our Django app will be using communicate with the outside world, go to the networking tab, then the firewall option. This will redirect you to the page below, fill name, add a custom TCP inbound rule to port 8000 and 7000, attach it to the droplet created above and hit create firewall button.

With the droplet and firewall now up and running, you can now add a step that automatically deploys the django app to the digital ocean droplets, but before that you need to add a few secrets on github repo.

Ssh into your digital ocean droplet

ssh root@YOUR_DO_IP

Run the following command to generate a new ssh key pair that the pipeline will use to connect to the droplet.

ssh-keygen

To get the value of the generated private key navigate to the ssh folder by

cd ~/.ssh

To get the private key, run the following command

cat id_rsa

make sure that you copy the whole of the contents and add it to your repository secrets as DO_PRIVATE_KEY.

With that in place, copy the public key

cat id_rsa.pub

and add it to the authorised_keys file

nano authorised_keys

paste the contents and save and close the file.

Go back to the root of the project and create a new folder django-ci-example, create a .env file if need be and a 'docker-compose.yaml' file and paste into it the contents of the file into.

nano docker-compose.yaml

And paste in the following lines of code

version: '3.8'

services:
  web:
    build: 
      context: .
      dockerfile: Dockerfile
    command: gunicorn do-guide.wsgi:application --bind 0.0.0.0:8000
    volumes:
      - static_volume:/home/app/web/staticfiles

    image: manulangat/auto-deploy-do-github-actions:latest
    ports:
      - "8000:8000"
    networks:
      - django-ci-example

volumes:
  static_volume:

networks:
  django-ci-example:

Head over to your workflow file and add the following step

- name: Executing remote command and deployment to digital ocean for dev enviroment
        uses: appleboy/ssh-action@master
        with:
          host: "YOUR_DO_IP"
          USERNAME: "root"
          PORT: 22
          KEY: ${{ secrets.DO_PRIVATE_KEY}}
          script: |
            cd django-ci-example/
            docker system prune -af
            docker compose -f docker-compose.staging.yaml down --volumes
            echo "${{secrets.DOCKER_PASSWORD}}" | docker login -u ${{secrets.DOCKER_USERNAME}} --password-stdin
            docker system prune -af
            docker compose -f docker-compose.staging.yaml pull
            docker compose -f docker-compose.staging.yaml up --build --remove-orphans -d --force-recreate
            # docker-compose -f docker-compose.staging.yaml up --build -d

At this point, your actions file should be similar to the one below.

name: Continuous Integration and Delivery

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  testing-docker-compose:
    runs-on: ubuntu-latest
    steps:
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - uses: actions/checkout@v2
      - name: Build the stack
        run: docker-compose -f docker-compose.yaml up -d --build

      - name: Get docker logs
        run: docker ps

      - name: Login to DockerHub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Push to dockerhub
        run: |
          docker-compose push

      - name: stop containers
        run: docker-compose -f docker-compose.yaml down --volumes

      - name: Executing remote command and deployment to digital ocean for dev enviroment
        uses: appleboy/ssh-action@master
        with:
          host: "YOUR_DO_IP"
          USERNAME: "root"
          PORT: 22
          KEY: ${{ secrets.DO_PRIVATE_KEY}}
          script: |
            cd django-ci-example/
            docker system prune -af
            docker compose -f docker-compose.staging.yaml down --volumes
            echo "${{secrets.DOCKER_PASSWORD}}" | docker login -u ${{secrets.DOCKER_USERNAME}} --password-stdin
            docker system prune -af
            docker compose -f docker-compose.staging.yaml pull
            docker compose -f docker-compose.staging.yaml up --build --remove-orphans -d --force-recreate
            # docker-compose -f docker-compose.staging.yaml up --build -d

To test this out, make commit and push your works and the actions should be triggered. Once the actions is done, navigate to the web page ie YOUR_DO_IP:7000 and you should be welcomed with a page as shown below.

The code for this project can be found at this GitHub repo .

Kindly follow me to get instant notifications whenever I post articles and guides on Devops and Django.

Happy hacking.

Deploying Django using Kubernetes - a gentle intro.

Kipchirchir Langat Emmanuel — Sun, 18 Dec 2022 09:47:42 +0000

1. What is Kubernetes?

Kubernetes is an open-source container orchestration tool developed by Google. Kubernetes offers the following features:

High availability
Scalability or high performance.
Disaster recovery (backups and restore)

To get an in-depth read of what Kubernetes is and the problem it solves, read the official guide

To install Kubernetes in your preferred operating system, follow the guides on the

official documentation

2. A brief intro to services will be covered in this article.

In this article, you will learn about the following Kubernetes components:

2.1 Pods.

A pod is the smallest unit of Kubernetes. It provides an abstraction over a container and helps you to interact only with the Kubernetes layer and is usually meant to run one container per pod this is because the pods are scaled up and down as a single unit regardless of a container need thus resulting in expensive cloud costs.

Each pod gets allocated an IP address upon creation and they use the IP address to communicate. Pods are ephemeral and can die easily thus the need for services

2.2 Services and Ingress.

A service is a static IP address that can be attached to the pods and is not connected to the lifecycle of a Pod thus when a pod dies and is recreated. There are various types of services namely:

ClusterIP service - is the default type of service and is used to expose a service on an Ip internal to the cluster. Access is only permitted to objects with the cluster.
HeadlessIp Service
Nodeport Service
LoadBalancer Service

Ingress -

2.3 ConfigMaps and Secrets.

ConfigMap contains the external configuration of your application both are local volumes but are rather own components.

Secret - used to store credentials and it is stored in base64 format. used for individual key-value pairs or for mounting files.

2.4 Deployments.

Is an abstraction over the pod's blueprint for creating pods. In practice, you will be creating deployments and not pods since it provides an avenue to scale up and down.

2.5 Volumes.

It is useful when persisting data in k8s for pod restart and does not depend on the pod lifecycle Should be available on all nodes. Highly available that survives even if cluster crashes

My SQL - data gets added, updated but once you restart a pod the data goes as k8s and does not

We will briefly cover the 3 classes of Kubernetes volumes

Persistent Volume

a cluster resource
created via a YAML file.
Takes storage from physical storage eg local storage, NFS server etc
decide what type and manage them eg backups etc
They are not namespaced. local vs remote volume types: local volume violates
being tied to 1 specific node.
surviving cluster crashes

For DB persistence, always use remote storage.

Persistent Volume Claim.

Admin configures storages (create and configure).
Admin creates PV from these storage backends.
App claim volumes and also created using YAML files. must exist in the same ns as the PVC
do not care where your storage is,
Easier for developers.

Pod -> PVC -> CLaim

Storage class. provisions persistent volumes dynamically whenever PVC claims it. created via a YAML class claimed by PVC - add a storage class Name

2.6 Namespaces.

Namespaces can be thought of as a virtual cluster inside a cluster. It is used to isolate resources/ clusters together.

It is important to note that you cannot access most of the resources from another namespace eg if you have a config map in project A ns, you cannot use that configmap in project B namespace but instead, you have to create the same in the other project. However, a service can be referenced in another project. To get a list of all components that can be namespace run

kubectl api-resources --name-spaced=true

Volumes are accessible throughout the cluster To get a list of all non-namespaces components run

kubectl api-resources --name-spaced=false

This brings up the question, why use namespaces?

Group resources into namespaces eg DB ns, monitoring ns, Nginx ingress ns
Many teams are using the same application - one team can easily overwrite another team's work. Each team can work independently
Resource sharing - staging and development environments.
When you use blue-green deployment for your application.
Access and Resource limits on Namespaces, 2 teams - give the team access to their own ns and define resource quotas.

By default, K8s has the following namespaces defined.

Kubernetes-dashboard - This ns is shipped only with minikube and you will not have this in a normal cluster.
kube-system - Not meant for your use. Components deployed here include the system processes,
Kube-public - Has a config map that contains cluster information and which is accessible without auth. Type kubectl cluster info on your terminal and you should see info about your minikube installation.
kube-node-list - Holds info about the heartbeats of every node and determines the availability of nodes.
Default namespace - Used to create a new namespace at the beginning.

To get a list of all the namespaces run

kubectl get namespace

You can use a CLI command as shown below or via a namespace configuration file.

Via a CLI(command line ) command

 kubectl create namespace my-namespace

And you can now assign the created namespace to a Kubernetes namespace as:

apiVersion: v1 
kind: ConfigMap 
metadata:
  name: my-configmap
  name: my-namspace

To switch over your namespaces, install kubens as detailed in this GitHub

repository

2.6 Sample YAML configuration file.

In this part, you will learn the basic syntax of a Kubernetes YAML file and how it is usually structured.

The main parts of a YAML file include:

Metadata. - Contains the metadata of the file eg name and others such as labels.
Specification - specific to the kind you are creating.
Status- automatically generated and added by K8s. This is the basis of the self-healing nature of k8s eg you have specified you need 2 replicas and if one dies, it will look at the status for the desired state and spin up a new pod.

Attached below is a simple k8s YAML file of kind Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: name-of-deployment
  labels:
    app: name #should be the same with labels in the template
spec:
  replicas: 2
  selector:
    matchLabels:
      app: name
  template: # blue print of a pod
    metadata:
      labels:
        app: name
    spec:
      containers:
        - name: container-name
          image: image-name-ref
          imagePullPolicy: Always

3. Django app that will be used (Optional).

For the demonstration, the following application will be used

ecommerce monolith. Navigate to your desired folder ie

cd Desktop

Clone the application by using the following command.

git clone https://github.com/manulangat1/ecomerce-monolith.git

Once the cloning is done navigate into the cloned directory and change into a new branch.

git switch -f "feat-h8s-test-your-name"

N/B. This step is optional and you can use any of your code repositories to follow along with this tutorial.

4. Deployments and services set up.

4.1 Deployment file

On the root of your project, create a folder named k8s and navigate inside the newly created folder

mkidr k8s

Navigate into the created k8s folder

cd k8s

Create a file named django-deployment.yaml

touch django-deployment.yaml

Once created, add the following lines of code.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-be-deployment
  labels:
    app: django-be #should be same with labels in template
spec:
  replicas: 1
  selector:
    matchLabels:
      app: django-be
  template: # blue print of a pod
    metadata:
      labels:
        app: django-be
    spec:
      containers:
        - name: django-be
          image: manulangat/django-ecomerce-monolith
          imagePullPolicy: Always
          command: ["gunicorn"]
          args: ["--bind=localhost:8000","monolithEcommerce.wsgi:application"]
          ports:
            - containerPort: 8000

Run the following command at the project root to apply the changes we have added.

kubectl apply -f django-deployment.yaml

To get the pod details of the app and to get the progress, run the following command.

kubectl get pod | grep django

It should have a similar output as the one shown below, kindly note that the container creation process may take some bit of time when the command is run initially.

To get the logs of the deployment created initially run

kubectl logs -f deployment/django-deployment

and the output should be similar to the image shown below if you are using the base project shared above.

4.2 Secrets set up.

Now that our deployment service is up and running, we may need to pass some environment variables such as the Django-secret key that should be hidden from the public, this is better done through the use of secrets and thus the need for the secrets file. Note that values stored in Secretes should be base64 encoded

While on the project root, create a new file inside the k8s folder by

cd k8s 
touch django-secrets.yaml

Once that is done, add the following lines of YAML code into the newly created django-secrets.yaml file. To get a base64 encoded value of your secret run the following command

echo -n "YOUR_SECRET_KEY" | base64


apiVersion: v1
kind: Secret
metadata:
  name: django-secrets
type: Opaque
data:
  DJANGO_SITE_SECRET_KEY: WU9VUl9TRUNSRVRfS0VZ

To apply and create the secret, run

kubectl apply -f k8s/django-secrets.yaml

To confirm whether the secret was created successfully, use the below command

kubectl get secrets | grep django

and the output should be similar to the image attached below

The next step is now accessing the secrets in our django deployment file. Head over to the k8s/django-deployment.yaml file and below the ports option add the following configurations to load the secret and access the value for the key DJANGO_SECRET_SITE_KEY.

env:
            - name: DJANGO_SITE_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: django-secrets # name given to the secret in django-secrets.yaml file
                  key: DJANGO_SITE_SECRET_KEY

At this stage, your django-deployment.yaml the file should resemble this

apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-be-deployment
  labels:
    app: django-be #should be same with labels in template
spec:
  replicas: 1
  selector:
    matchLabels:
      app: django-be
  template: # blue print of a pod
    metadata:
      labels:
        app: django-be
    spec:
      containers:
        - name: django-be
          image: <REF_OF_YOUR_IMAGE>
          imagePullPolicy: Always
          command: ["gunicorn"]
          args: ["--bind=localhost:8000", "monolithEcommerce.wsgi:application"] # replace monolithEcommerce with the name of your
          ports:
            - containerPort: 8000
          env:
            - name: DJANGO_SITE_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: django-secrets # name given to the secret in django-secrets.yaml file
                  key: DJANGO_SITE_SECRET_KEY

At this stage, we have gone through Kubernetes deployments and secrets and tested it out via kubectl commands all is well, in the next step, we shall learn how to expose our application to the outside world through Kubernetes services.

4.3 Services- Exposing our django app to the outside world.

Create a new file in the k8s folder ie

touch k8s/django-service.yaml

And add the following pieces of yaml code inside it


apiVersion: v1
kind: Service
metadata:
  name: django-service
spec:
  selector:
    app: django-be # name of app as defined in the 
  type: LoadBalancer # specifies the type of service created. 
  ports:
    - protocol: TCP
      port: 1338
      targetPort: 8000
      nodePort: 30001

Run the commands below to apply our changes and to confirm whether everything works as expected. The output of the second command should be similar to the image attached below

The service can now be exposed via minikube tunnel command. Head to your terminal and paste in the following :

minikube tunnel NAME_OF_YOUR_SERVICE

It will prompt you to input your password and once done, it will show the logs similar to the one in the picture attached below.

To access the application, get the IP address stated above and navigate to it ie

192.168.49.2:30000/

In this article, we have gone through the basics of Kubernetes and how to use components such as Deployments, services and secrets in our Django app. In the next article in this series, you will learn about using volumes, exposing our app using ingress and deploying it to a digital ocean-managed Kubernetes service.

Happy hacking!!

Create and use custom Django signals by building a blog application

Kipchirchir Langat Emmanuel — Wed, 19 Oct 2022 19:11:16 +0000

In this article, you will learn how to create and use custom Django signals. Then use them to log all actions undertaken by a user on the application

1. What are Django signals?

According to the Django documentation, a signal is In a nutshell, signals allow certain senders to notify a set of receivers that some action has taken place. Theyre especially useful when many pieces of code may be interested in the same events.

There are various built-in signals that you can read comprehensively here. However, for the scope of this project, you shall learn about creating custom signals.

2. Set up a new Django project.

2.1 Navigate into your desired folder and create a virtual environment.

Navigate to your desired folder, e.g Downloads and create and activate a virtual environment.

cd Desktop

create a folder that will house the Django project

mkdir audit-logs-app && cd audit-logs-app

create and activate a virtual environment

python3 -m virtualenv venv

Activate the virtual environment

source venv/bin/activate # if you are using a Linux distro 
.\venv\Scripts\activate # if running on windows

2.2 Install Django and create project

You can now install Django

pip install django

Create a new Django project

django-admin startproject djangoauditlog

Navigate into the django project. This is where we will create our apps and write the code.

cd djangoauditlog

3. Create django apps (authentication and posts)

3.1 Create an authentication app and install drf and django-rest-knox

django-admin startapp authentication

Head over to your settings.py file and add the authentication app to the list of INSTALLED_APPS as shown below.

Now that is done install django rest framework and django-rest-knox by

pip install djangorestframework django-rest-knox

After a successful installation head over to your settings.py file and add the rest_framework and knox to the list of INSTALLED_APPS as shown below.

Still on the settings.py file, add the following lines of code which tells rest_framework the authentication classes that is should use.

REST_FRAMEWORK = {
    "DEFAULT_AUTHENTICATION_CLASSES": (
        "knox.auth.TokenAuthentication",
        "rest_framework.authentication.TokenAuthentication",
        "rest_framework.authentication.SessionAuthentication",
        "rest_framework.authentication.BasicAuthentication",
    ),
    # 'DEFAULT_FILTER_BACKENDS': ('django_filters.rest_framework.DjangoFilterBackend',)
}

3.2 Create a custom User model.

We will create a custom user model using the AbstractUser abstract class. Head to the apps/authentication/models.py and add the following code.

from django.db import models
from django.contrib.auth.models import AbstractUser
from django.utils import translation

#local imports 
#from apps.common.models import TimeStampedModel
# Create your models here.

class User(AbstractUser):
    bio = models.TextField(verbose_name='user bio', null=True, blank=True)

    def __str__ (self) -> str:
        return self.email

3.3 Create your auth serializer files.

Head over to the apps/authentication/serializers.py file and add the following lines of code.

from rest_framework import serializers 

#django imports 
from django.contrib.auth import get_user_model, authenticate
#local imports
from apps.authentication.models import Profile

# instantiate our user model 
User = get_user_model()

class UserSerializer(serializers.ModelSerializer):

    class Meta:
        model = User
        fields = ( 
            "id",
            "username",
            "email",
            "first_name",
            "last_name"
        )

class ProfileSerializer(serializers.ModelSerializer):
    username = serializers.CharField(source="user.username")
    class Meta:
        model = Profile 
        fields = ( 
            "username",
            "city",
            "state",
            "country"
        )

class RegisterSerializer(serializers.ModelSerializer):
    class Meta:
        model = User
        fields = ('id', 'username', 'email', 'password')
        extra_kwargs = {'password': {'write_only': True}}

    def create(self, validated_data):
        user = User.objects.create_user(validated_data['username'], validated_data['email'], validated_data['password'])
        return user

class LoginSerializer(serializers.Serializer):
    username = serializers.CharField()
    password = serializers.CharField()

    def validate(self, data):
        user = authenticate(**data)
        if user and user.is_active:
            return user
        raise serializers.ValidationError('Incorrect Credentials Passed.')

3.4 Create auth API endpoints.

Head over to the apps/authentication/views.py file and add the following lines of code.

from django.shortcuts import render
from rest_framework import views, generics, permissions, status
from rest_framework.response import Response

#local imports
from apps.authentication.serializers import UserSerializer, RegisterSerializer, LoginSerializer

from apps.authentication.exceptions import ProfileNotFound
# third party imports
from knox.models import AuthToken

# Create your views here.

class RegisterAPI(generics.GenericAPIView):
    serializer_class = RegisterSerializer
    def post(self, request):
        serializer = self.get_serializer(data=request.data)
        serializer.is_valid(raise_exception=True)
        user = serializer.save()
        print(user)
        user.save()
        token = AuthToken.objects.create(user)
        return Response({
            "users": UserSerializer(user, context=self.get_serializer_context()).data,
            "token": token[1]
        }, status=status.HTTP_200_OK)

class LoginAPI(generics.GenericAPIView):
    serializer_class = LoginSerializer

    def post(self,request):
        serializer = self.get_serializer(data=request.data)
        serializer.is_valid(raise_exception=True)
        user = serializer.validated_data
        token = AuthToken.objects.create(user)
        return Response({
            "users": UserSerializer(user, context=self.get_serializer_context()).data,
            "token": token[1]
        }, status=status.HTTP_200_OK)

class UserAPI(generics.GenericAPIView):
    serializer_class = UserSerializer
    permission_classes = ( 
        permissions.IsAuthenticated,
    )
    def get(self,request):
        user = self.request.user 
        serializer = UserSerializer(user)
        return Response(serializer.data, status=status.HTTP_200_OK)

3.5 Create the authentication URLs.

Head over to the apps/authentication folder and create a urls.py file if it did not exists. After successful creation, paste in the following lines of code.

from django.urls import path 
from . import views
urlpatterns = [ 
    path('auth/register/', views.RegisterAPI.as_view(), name="registration_api"),
    path('auth/login/', views.LoginAPI.as_view(), name="login_api"),
    path('auth/me/', views.UserAPI.as_view(), name="get_logged_in_user_api"),
]

3.6 Testing out the authentication APIs

Head over to postman or the tool you use and test out the APIs one by one and confirm that all are working fine. Feel free to tweak the code as you wish to achieve your desired functionality.

4. Create the blog application and the custom signal.

4.1 Create the blog application using django-admin command.

django-admin startapp blog

4.2 Create the blog models.

Head over to your blog/models.py file and paste in the following lines of code.


from django.db import models
from django.contrib.auth import get_user_model

import uuid

# Create your models here.

User = get_user_model()

class TimeStampedUUIDModel(models.Model):
    pkid = models.BigAutoField(primary_key=True, editable=False)
    id = models.UUIDField(default=uuid.uuid4, editable=False, unique=True)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    class Meta:
        abstract = True

class Tag(TimeStampedUUIDModel):
    title = models.CharField(max_length=199)

    def __str__ (self) -> str:
        return self.title

class Post(TimeStampedUUIDModel):
    user = models.ForeignKey(User, related_name="posts",
                             on_delete=models.CASCADE)
    title = models.CharField(max_length=199)
    content = models.TextField()
    tags = models.ManyToManyField(Tag)
    isPublished = models.BooleanField(default=False)
    image = models.CharField(max_length=255, null=True)
    slug = models.CharField(max_length=255, null=True)

    def __str__ (self) -> str:
        return self.title

You can now run your migrations by heading over to your terminal and using the commands.

python3 manage.py makemigrations
python3 manage.py migrate

4.3 Create the posts serializer.

At this point, you will handle the serializer classes that will help in converting the queryset objects to a JSON readable format.

Head over to blogs/serializers.py and paste in the following lines of code.

from rest_framework import serializers
from .models import Post, Tag

class TagSerializer(serializers.ModelSerializer):
    class Meta:
        model = Tag
        fields = (
            "id",
            "title",
        )

class PostSerializer(serializers.ModelSerializer):
    class Meta:
        model = Post
        fields = (
            "id",
            "title",
            "content",
        )

class PostDetailSerializer(serializers.ModelSerializer):
    tags = serializers.SerializerMethodField()

    class Meta:
        model = Post
        fields = ("id", "title", "content", "tags")

    def get_tags(self, obj):
        return TagSerializer(obj.tags.all(), many=True).data

4.4 Blog application API endpoints. Create the views.

You will now create API endpoints, using django rest framework that will be used to communicate with the models (CRUD operations). Head over to the blogs/views.py file and paste in the following lines of code.


from rest_framework import permissions, status, generics
from rest_framework.views import APIView
from rest_framework.response import Response

import requests
# Create your views here.

from .models import Post, Tag
from .serializers import TagSerializer, PostDetailSerializer, PostSerializer

class TagAPI(generics.ListCreateAPIView):
    serializer_class = TagSerializer
    queryset = Tag.objects.all().order_by("-created_at")
    permission_classes = [permissions.IsAuthenticated]

class PostAPI(generics.ListCreateAPIView):
    serializer_class = PostSerializer
    queryset = Post.objects.all().order_by("-created_at")
    permission_classes = (permissions.IsAuthenticated,)

    def perform_create(self, serializer):
        data = self.request.data
        new_post = serializer.save(
            user=self.request.user, title=data["title"], content=data["content"]
        )
        all_posts = Post.objects.all().order_by("-created_at")
        serializer = PostSerializer(all_posts, many=True)
        return Response(serializer.data, status=status.HTTP_201_CREATED)

class PostDetailAPI(APIView):
    def get(self, request, id):
        post = get_post(id)
        serializer = PostDetailSerializer(post)
        return Response(serializer.data, status=status.HTTP_200_OK)

With the views now in place, go ahead and create a new file blogs/urls.py and add the following lines of code.

from django.urls import path
from . import views

urlpatterns = [
    path("v1/posts/", views.PostAPI.as_view(), name="get_a_list_of_all_blogs"),
    path("v1/tags/", views.TagAPI.as_view(), name="get_all_tags"),
    path("v1/posts/<str:id>/", views.PostDetailAPI.as_view(), name="get_a_post_detail"),
]

4.5 Test out the blogs API endpoints.

Head over to postman / insomia based on your preferences and test out the blog api's one by one. If you face any issue , feel free to leave a comment or get in touch for more clarifications.

5. Implementing the audit logs functionality.

At this stage, you have both authentication and the blog features up and running, you can create, view, edit and delete posts, however, you cannot, as an admin / super user know who accessed what part of the system and the action that was undertaken. This drives us to implement the audit trail functionality.

5.1 Create the AuditTrail model.

On your blogs/models.py, add the audit trail model

class AuditTrail(TimeStampedUUIDModel):
    login_IP = models.GenericIPAddressField(null=True, blank=True)
    action_datetime = models.DateTimeField(auto_now=True)
    user = models.ForeignKey(User, on_delete=models.CASCADE) # keeps track of the user 
    changed_object = models.CharField(max_length=40)
    event_category = models.CharField(max_length=40)
    user_agent_info = models.CharField(max_length=255)
    is_deleted = models.BooleanField(default=False)
    action = models.CharField(max_length=40)
    change_summary = models.CharField(max_length=199)

Run your migrations by using the commands to ensure we can start using the

python3 manage.py makemigrations
python3 manage.py migrate

5.2 Custom Signal implementation.

With the audit trail model in place, you can now go ahead and create the custom signal implementation that will be used to create the logs.

Go and create a new file blogs/signals.py file and add in the following piece of code.


import django.dispatch
from django.dispatch import receiver

from .models import AuditTrail
import logging
import datetime

audit_trail_signal = django.dispatch.Signal(providing_args=['user', 'request', 'model', 'event_category', 'method', 'summary']) # creates a custom signal and specifies the args required. 

logger = logging.getLogger( __name__ )

#helper func that gets the client ip
def get_client_ip(request):
    x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')
    if x_forwarded_for:
        ip = x_forwarded_for.split(',')[0]
    else:
        ip = request.META.get('REMOTE_ADDR')
    return ip

@receiver(audit_trail_signal)
def log_audit_trail(sender, user, request, model, event_category, method,summary,**kwargs):
    try:

        user_agent_info = request.META.get('HTTP_USER_AGENT', '<unknown>')[:255],
        print(summary)
        auditTrail = AuditTrail.objects.create( 
            user=user,
            user_agent_info=user_agent_info,
            changed_object = model ,
            event_category = event_category ,
            login_IP = get_client_ip(request),
            is_deleted = False,
            action=method,
            change_summary=summary
        )
        logger.info(f"Audit trail created {auditTrail.id} for user {auditTrail.username} and object {auditTrail.changed_object}")
    except Exception as e:

        logger.error("log_user_logged_in request: %s, error: %s" % (request, e))

5.3 Log user actions using the custom signal.

With the setup of the audit trail model and the custom signal done, it is now time to test out whether everything works as expected (finger's crossed). Head over to the views.py and in the PostAPI view

class PostAPI(generics.ListCreateAPIView):
    serializer_class = PostSerializer
    queryset = Post.objects.all().order_by("-created_at")
    permission_classes = (permissions.IsAuthenticated,)

    def perform_create(self, serializer):
        data = self.request.data
        new_post = serializer.save(
            user=self.request.user, title=data["title"], content=data["content"]
        )
        all_posts = Post.objects.all().order_by("-created_at")
        serializer = PostSerializer(all_posts, many=True)
        audit_trail_signal.send(sender=request.user. __class__ , request=request, 
        user=request.user, model="Blog",event_category="Blog", method="CREATE")
        return Response(serializer.data, status=status.HTTP_201_CREATED)

5.4 View to expose logs.

The logs set up is almost done, however as a dev, you may want to access the logs via an API. This is done by first creating a serializer class, the corresponding views and URLs. Go to the blog/serializers.py and paste in the following piece of code. Do not forget to import the Audit trail model.

class AuditTrailSerializer(serializers.ModelSerializer):
    class Meta:
        model = AuditTrail
        fields = ( 
            "id",
            "action_datetime",
            "user",
            "changed_object",
            "event_category",
            "change_summary",
        )

The addition of the AuditTrailSerializer sets the stage for the addition of the AuditTrailView and the corresponding URLs

# don't forget to import the audit trail model and serializer class 

class AuditTrailView(generics.ListAPIView):
    authentication_classes = ( permissions.IsAuthenticated,)
    queryset = AuditTrail.objects.all().order_by('-created')
    serializer_class = AuditTrailSerializer
    permission_classes = ( permissions.IsAuthenticated,)

and add the URL in the blog/urls.py

#add the following line to your urls.py file 
path('audit-trail/', views.AuditTrailView.as_view(), name="audit_trail"),

Congratulations, you have built a blog application using Django, Django rest framework and learnt about signals and learnt how to create and use a custom signal.

Happy hacking.

k

Kipchirchir Langat Emmanuel — Fri, 07 Oct 2022 16:56:40 +0000

Intergrate sonarqube in Jenkins. A gentle intro to Jenkins DevSecOps.

Kipchirchir Langat Emmanuel — Thu, 06 Oct 2022 21:02:11 +0000

In this article, you will learn how to set up Jenkins in a digital ocean droplet, configure it, Install sonarqube, configure and link it to our Jenkins pipeline and how to send emails in Jenkins. We will also finally deploy the application to an aws ec2 instance.

This article assumes a basic understanding of docker.

1. Set up, configure and install Jenkins on a digital ocean droplet.

1.1 Head over to Digital Ocean and create an account if you do not already have one, otherwise log in.

1.2 Create a digital ocean droplet with the following specs.

An image - Ubuntu: the Lts tag
Plan - 4GB 2 AMD CPUs
Datacenter region - (preferably one close to your location)
Authentication method - (ssh keys) and copy your public key to Digital Ocean. follow this link to create ssh keys.

1.3 Configure the firewall rules to open port 22 (for ssh connections ) and port 8080 (for our Jenkins server).

Ensure ports 22 and 8080 are open to receive requests.

1.4 Install docker in the droplet.

On your machine, ssh into the digital ocean droplet by

ssh YOUR_DROPLET_IP@root

Update the digital ocean droplet by

apt update

Upgrade the digital ocean dependencies by:

apt upgrade -y

Install docker runtime engine by:

apt install docker.io -y

1.5 Run Jenkins as a docker container

Run the following command whilst being ssh'ed into the digital ocean droplet.

docker run -p 8080:8080 -p 50000:50000 -d \ 
-v jenkins_home:/var/jenkins_home \
-v /var/run/docker.sock:/var/run/docker.sock \
-v $(which docker):/usr/bin/docker jenkins/jenkins:lts

Access the Jenkins server on the web using the server's public IP as shown below

YOUR_DO_DROPLET_IP:8080

After a successful installation of Jenkins in the Docker container, you should see a screen similar to the one below that will prompt you for an administrator password

#use this command to get the initialAdminPassword
cat /var/lib/docker/volumes/jenkins_home/_data/secrets/initialAdminPassword

copy the password and login into Jenkins. We can now create our first Jenkins user.

2. Install and configure sonarqube in the digital ocean droplet.

2.1 SSH into the digital ocean droplet by

ssh YOUR_DO_IP@root

2.2 Run sonarqube as a docker container

docker run -d --name sonarqube -p 9000:9000 -p 9092:9092 sonarqube

N/B ensure that port 9000 is configured on your firewall and can receive requests

You will be redirected to a page where you will reset the default password

You will now be redirected to the sonarqube dashboard

You have successfully installed sonarqube and Jenkins on the digital ocean droplet

2.3 Configure Sonarqube in Jenkins.

In your Jenkins dashboard, navigate to the manage Jenkins -> Manage Jenkins and install the Sonarqube scanner

After a successful installation, head over to the manage Jenkins - configure systems and head over to the Sonarqube Servers part and configure it as shown in the pic below.

Kindly ensure that you have created a token in Sonarqube by going to the My Account -> Security part and have added the token in Jenkins as a credential.

3. Create a multi-branch pipeline and configure it for Django.

Head over to your Jenkins Dashboard and click on the New Icon tab. You will be redirected to this page, input the job name and select the multi-branch job option.

On successful creation, head over to the configure section -> branch sources and set the source as GitHub. Now add in your GitHub credentials previous added and the remote branch.

You can now head over to your project and add the following code in the JenkinsFile.

#! env/bin/groovy 
def version 
// def newVersion
pipeline {
    agent any 
    stages { 
        stage("Init the application") {
            steps { 
                script {
                    echo "Hello there, ${BRANCH_NAME}"
                }
            }
        }

        stage("Create a virtual enviroment") { 
            steps{
                script{
                    echo "Now creating a virtualenv"
                    sh "python3 -m virtualenv venv"
                    sh "chmod +x venv/"
                    sh "ls"
                    sh ". venv/bin/activate"
                }
            }
        }
        stage("Read the current version of the application") { 
            steps { 
                script { 
                    echo" build number ${BUILD_NUMBER}"
                    def matcher = readFile(" __init__.py") =~ "version = (.+)"
                    echo "${matcher[0][1]}"
                    version = matcher[0][1]
                }
            }
        }
        stage("Test stage") { 
            steps { 
                script{
                    echo "now what"
                    // sh "pytest"
                }
            }
        }
        stage("Build") { 
            steps { 
                script{
                    echo "now what"

                    withCredentials([usernamePassword(credentialsId:'dockerhub-id', usernameVariable:"USER" , passwordVariable:"PASS")]) { 
                        // sh " echo ${PASS} | docker login -u $USER --stdin-"
                        sh ''' 
                    docker system prune -a -f 
                    docker-compose -f docker-compose.yaml down
                    docker-compose -f docker-compose.yaml up --build -d
                    echo $PASS | docker login -u $USER --password-stdin
                    docker push manulangat/django-ecomerce-monolith
                    '''

                    }

                }
            }
        }

        stage('Commit the version and tag'){
            steps {
                script {

                    echo" build number ${BUILD_NUMBER}"
                    def matchers = readFile(" __init__.py") =~ "version = (.+)"
                    echo "${matchers[0][1]}"

                    def vers = matchers[0][1]
                    echo "${vers}"
                    matchers = null
                    // def newVersion = matchers[0][1]
                    // echo "${newVersion}"
                    // sh "docker-compose -f docker-compose.yaml down"
// now
                    withCredentials([usernamePassword(credentialsId:"github-creds-hook", usernameVariable:'USER', passwordVariable:'PASS')]) { 
                        sh '''
                        chmod +x venv/
                        . venv/bin/activate
                        pip install -r requirements.txt

                        git config --global user.name jenkins
                        git config --global user.email jenkins@jobs.com
                        git status
                        git config --list
                        git remote set-url origin https://$USER:$PASS@github.com/manulangat1/ecomerce-monolith.git
                        git push --tags -f
                        git tag | xargs git tag -d
                        chmod +x README.md 
                        chmod +x __init__.py 
                        chmod +x setup.py 
                        chmod +x CHANGELOG.md
                        bump2version minor --allow-dirty
                        git describe --always
                        git push --tags -f
                        git tag | xargs git tag -d

                        '''
                        echo "Done pushing to github"
                    }
                }
            }
        }
    }

}

You can now connect the sonarqube server to the Jenkins server by going to the Jenkins dashboard and configure systems and head over to the Sonarqube installations part and fill in the required data.

You can now add the stage to run sast analysis using sonarqube

stage('SonarQube Analysis') {
                steps {
                    script {
                        def scannerHome = tool 'SonarScanner';
                            withSonarQubeEnv() {
                            sh "${scannerHome}/bin/sonar-scanner"
                            }
                    }
                }
            }

At this stage, your JenkinsFile should resemble the one below.

#! env/bin/groovy 
def version 
// def newVersion
pipeline {
    agent any 
    stages { 
        stage("Init the application") {
            steps { 
                script {
                    echo "Hello there, ${BRANCH_NAME}"
                }
            }
        }
        stage('SonarQube Analysis') {
                steps {
                    script {
                        def scannerHome = tool 'SonarScanner';
                            withSonarQubeEnv() {
                            sh "${scannerHome}/bin/sonar-scanner"
                            }
                    }
                }
            }
        stage("Create a virtual enviroment") { 
            steps{
                script{
                    echo "Now creating a virtualenv"
                    sh "python3 -m virtualenv venv"
                    sh "chmod +x venv/"
                    sh "ls"
                    sh ". venv/bin/activate"
                }
            }
        }
        stage("Read the current version of the application") { 
            steps { 
                script { 
                    echo" build number ${BUILD_NUMBER}"
                    def matcher = readFile(" __init__.py") =~ "version = (.+)"
                    echo "${matcher[0][1]}"
                    version = matcher[0][1]
                }
            }
        }
        stage("Test stage") { 
            steps { 
                script{
                    echo "now what"
                    // sh "pytest"
                }
            }
        }
        stage("Build") { 
            steps { 
                script{
                    echo "now what"

                    withCredentials([usernamePassword(credentialsId:'dockerhub-id', usernameVariable:"USER" , passwordVariable:"PASS")]) { 
                        // sh " echo ${PASS} | docker login -u $USER --stdin-"
                        sh ''' 
                    docker system prune -a -f 
                    docker-compose -f docker-compose.yaml down
                    docker-compose -f docker-compose.yaml up --build -d
                    echo $PASS | docker login -u $USER --password-stdin
                    docker push manulangat/django-ecomerce-monolith
                    '''

                    }

                }
            }
        }

        stage('Commit the version and tag'){
            steps {
                script {

                    echo" build number ${BUILD_NUMBER}"
                    def matchers = readFile(" __init__.py") =~ "version = (.+)"
                    echo "${matchers[0][1]}"

                    def vers = matchers[0][1]
                    echo "${vers}"
                    matchers = null
                    // def newVersion = matchers[0][1]
                    // echo "${newVersion}"
                    // sh "docker-compose -f docker-compose.yaml down"
// now
                    withCredentials([usernamePassword(credentialsId:"github-creds-hook", usernameVariable:'USER', passwordVariable:'PASS')]) { 
                        sh '''
                        chmod +x venv/
                        . venv/bin/activate
                        pip install -r requirements.txt

                        git config --global user.name jenkins
                        git config --global user.email jenkins@jobs.com
                        git status
                        git config --list
                        git remote set-url origin https://$USER:$PASS@github.com/manulangat1/ecomerce-monolith.git
                        git push --tags -f
                        git tag | xargs git tag -d
                        chmod +x README.md 
                        chmod +x __init__.py 
                        chmod +x setup.py 
                        chmod +x CHANGELOG.md
                        bump2version minor --allow-dirty
                        git describe --always
                        git push --tags -f
                        git tag | xargs git tag -d

                        '''
                        echo "Done pushing to github"
                    }
                }
            }
        }
    }

}

Now push your code to GitHub and head over to your Jenkins Dashboard and navigate to the multi-branch job created earlier and click on the Build now button to start a new build. Your pipeline should run as expected

Navigate to your sonarqube server and on the dashboard, you should see your latest build as shown in the image below. You can go through it to understand how your code compares against several metrics.

4. Intergrate Email sending functionality.

Email integrations into the pipeline will help in sending alerts when specific actions occur. This can be achieved by:

4.1 Install the email extension library

Head over to your Jenkins dashboard -> manage plugins section and install the Email Extension plugin

4.2 Configuring email extension Library

After successful installation, go to the configure system and scroll down to the extended email part and set up your SMTP servers.

Once that is done, head to your JenkinsFile and add the following block after all the stages.

    post{ 
    always { 
        sh '''
        docker system prune -a -f 
        docker-compose -f docker-compose.yaml down
        rm -rf venv
        '''
        emailext body: "$JOB_NAME - Build # $BUILD_NUMBER - :Check console output at $BUILD_URL to view the results.", recipientProviders: [[$class: 'DevelopersRecipientProvider'], [$class: 'RequesterRecipientProvider']], subject: 'Test'
    }
    failure { 
        echo "oops sth failed"
    }
}

This step shuts down our docker containers and sends an email about the build. Commit and push your code to GitHub and head over to Jenkins to trigger a build. You should receive an email about the build details.

In this article, we have gone through setting up Jenkins, Sonarqube and email integration and how they work together in a pipeline. In the next article, we shall look at how to set up slack notifications in the pipeline and how to integrate dast (dynamic application software testing) in the pipeline.

Happy hacking

Deploy a Django image to Jenkins using a multi-branch pipeline and conditionals.

Kipchirchir Langat Emmanuel — Thu, 08 Sep 2022 17:35:54 +0000

This is the fifth article in the series Deploy a Django app to AWS EC2 instance using Jenkins. A gentle guide to Jenkins..

In this article, you will build and configure a multi-branch Jenkins pipeline that will deploy a Django image to Dockerhub. You are required to have gone through the other articles in the series beforehand.

Now, you might wonder, what exactly is a multi-branch pipeline? The Multibranch Pipeline project type enables you to implement different Jenkinsfiles for different branches of the same project. In a Multibranch Pipeline project, Jenkins automatically discovers, manages and executes Pipelines for branches which contain a Jenkinsfile in source control.

This eliminates the need for manual Pipeline creation and management.

1. Create a multi-branch pipeline on Jenkins.

Head over to your Jenkins dashboard and click on the new item tab. Continue with the creation by giving the job a name and selecting the multi-branch pipeline option.

Upon successful creation, click on the branch sources tab and on the drop-down, select either Git or GitHub depending on the version control in use.

Now onto configuring the sources after selection. Attach the GitHub credentials you had earlier configured, and add your repo name in the project URL input box. On the discover branches dropdown, select the Filter by name (with regular expression) option. In the input, add the following regex

.* // dynamically discovers and loads all the branches (master, feature, hot fix etc)

Click on save and it should scan your GitHub repo branches if any Jenkins file exists and build the job.

You have now successfully created and configured your multi-branch pipeline.

2. Use conditionals in Jenkins.

Let's mimic a production workflow and you want to push the Django image to Dockerhub for the staging environment only. The below steps will guide you on how to do it.

Create and switch into a staging branch by using the commands.

 git switch -c "feat-staging"

Confirm that you are in the staging branch by

git branch

All good now, now shift the focus to the JenkinsFile, suppose you want to only push our docker image to Dockerhub when in the staging branch, how can that be achieved? Enter the use of the when block.

stage('build') { 
            when { 
                expression { 
                    BRANCH_NAME == 'staging' //this expression make sure that this step is only run when the branch name is staging
                }

            }
            steps{
                script { 
                    gv.buildApp()
                }
            }
        }

You have now successfully created, and configured a multi-branch pipeline in Jenkins and learnt how to use conditionals in Jenkins. In the next article, you will learn how to install, configure and use sonarqube in Jenkins.

Happy hacking!

Jenkins pipeline to build and push a Django image to dockerhub and GitHub webhook integration.

Kipchirchir Langat Emmanuel — Wed, 31 Aug 2022 11:41:25 +0000

This is the fourth article in the series Deploy a Django app to AWS EC2 instance using Jenkins. A gentle guide to Jenkins.

In this article, you will piece together all the fundamentals that you have learned and build a full Jenkins pipeline that will build and deploy a Django image to Dockerhub. You are required to have gone through the previous articles beforehand

1. Add the init stage and load the `script.groovy` file.

In your JenkinsFile, add the init stage that loads our groovy script.

def gv // we define this here to make it globally available. 
pipeline { 

    agent any
    stages { 

        stage('Init') { 

            steps { 
                script { 
                    gv = load "script.groovy"
                }
            }
        }
    }
}

Note that you should have a script.groovy file at the root of your project. If not , kindly create it by:

touch script.groovy

Add the following code inside the script.groovy file

def buildApp() { 
    echo "Building app"
}
return this // returns all the funcs inside the file.

2. Add a test stage to your JenkinsFile (optional).

You will now add a test stage to your JenkinsFile. This is however optional and you may add it if you want to run some tests.

You will first add a testApp function in your script.groovy file.

def testApp() { 
    echo "Testing our application" 
    // you can now run your test command here, e.g if using pytest you run 
    sh "pytest"
}

def buildApp() { 
    echo "Building app"
}

return this

You can now use the testApp func in the test stage in your JenkinsFile.

stage ('test') { 
            steps { 
                script {
                    gv.testApp()
                }
            }
        }

Disclaimer: This step is optional and can be skipped unless you have tests that you want to run.

3. Add a build stage to your pipeline.

You can now add a build stage to your JenkinsFile, as usual, you first start with adding the buildApp function in the script.groovy file.

You will make use of the withPassword functionality that you had covered before and the dockerhub credentials we had earlier created.

def buildApp() { 
    echo "Building app with build number #${env.BUILD_NUMBER}" // env variables are available and can be accesed in groovy files.
    withPassword([credentialsId:'dockerhub-id', passwordVariable: 'PASS', usernameVariable:'USER']){
        sh "docker build . -t YOUR_REPO_NAME:YOUR_TAG" // replace YOU_REPO_NAME and YOUR_TAG with the respective repo name and tag.
        sh "echo $PASS | docker login -u $USER --password-stdin"
        sh " docker push YOUR_REPO_NAME:YOUR_TAG" // replace YOU_REPO_NAME and YOUR_TAG with the respective repo name and tag.
    }
}

You can now use this in the build step

stage('build') { 
            steps{
                script { 
                    gv.buildApp()
                }
            }
        }

4. Add a deploy step to Jenkins pipeline.

You first add a deployApp to the script.groovy file.

def deployApp() { 
    echo "We will handle this in the next article!"
}

And use it in our JenkinsFile.


stage("deploy"){ 
            steps{
                gv.deployApp()
            }
        }

At the moment, your script.groovy should be similar to

// script.groovy
def testApp() { 
    echo "Testing our application" 
    // you can now run your test command here, e.g if using pytest you run 
    sh "pytest"
}

def buildApp() { 
    echo "Building app with build number #${env.BUILD_NUMBER}" // env variables are available and can be accesed in groovy files.
    withPassword([credentialsId:'dockerhub-id', passwordVariable: 'PASS', usernameVariable:'USER']){
        sh "docker build . -t YOUR_REPO_NAME:YOUR_TAG" // replace YOU_REPO_NAME and YOUR_TAG with the respective repo name and tag.
        sh "echo $PASS | docker login -u $USER --password-stdin"
        sh " docker push YOUR_REPO_NAME:YOUR_TAG" // replace YOU_REPO_NAME and YOUR_TAG with the respective repo name and tag.
    }
}
def deployApp() { 
    echo "We will handle this in the next article!"
}
return this

And the JenkinsFile

def gv // define here for it to be globally available
pipeline { 

    agent any
    parameters { 
        choice(name: 'Version', choices:['1.1.0', '1.2.0', '1.3.0'], description:'')
    }
    stages { 

        stage('Init') { 

            steps { 
                script { 
                    gv = load "script.groovy"
                }
            }
        }
        stage ('test') { 
            steps { 
                script {
                    gv.testApp()
                }
            }
        }
        stage('build') { 
            steps{
                script { 
                    gv.buildApp()
                }
            }
        }

        stage("deploy"){ 
            steps{
                gv.deployApp()
            }
        }
    }
}

You can now commit and push your changes to the version control of your choice then head over to your Jenkins dashboard and build. It should build successfully without any error. If you need any clarifications, kindly leave a comment and I will reply as fast as I can.

5. Integrate with GitHub.

Up to this point, you have been manually building our jobs, however, in large development teams, this is highly inefficient. You will now look at how to integrate GitHub with our jenkins server so that the jobs can run automatically.

Head over to your GitHub dashboard and click on the settings tab and navigate to the webhooks section.

In the Payload URL field, paste your Jenkins environment URL. At the end of this URL add /github-webhook/. In the Content type select: application/json and leave the Secret field empty

On the page Which events would you like to trigger this webhook? choose Let me select individual events. Then, check Pull Requests and Pushes. At the end of this option, make sure that the Active option is checked and click on Add webhook.

You are done with configurations on GitHub, now move to your Jenkins dashboard, select the job that you had created initially and select the Build triggers tab after clicking on configure.

Check on the GitHub hook trigger for GITScm polling option.

You have now successfully integrated GitHub with Jenkins. Head over to your code editor, make a change, push it to GitHub and you should see a build automatically runs.

In this article, you have come up with a full Jenkins pipeline that pushes a Django image to dockerhub and you have integrated GitHub with Jenkins for automated builds. In the next article, you will learn how to use Multi-branch pipelines , conditionals and how to set up email notifications (finally)

Happy hacking.

An introduction to Jenkins environment variables, parameters and use of groovy scripts.

Kipchirchir Langat Emmanuel — Tue, 30 Aug 2022 16:19:57 +0000

This is the third article in the series Deploy a Django app to AWS EC2 instance using Jenkins. A gentle guide to Jenkins.

In this article, we will go through how to use variables and parameters in Jenkins and how to use groovy scripts. We will be using the same project as before. You are required to have gone through the previous articles beforehand.

1. How to configure and use variables in a JenkinsFile.

While writing a JenkinsFile pipeline script, you may need to inject and use some dynamic values thus avoiding the hassle of hardcoding values in the pipeline hence the need for environment variables.

Jenkins environment variable is a global variable, exposed via env variable and used in Jenkins pipelines and anywhere in your Jenkinsfile file /Pipeline. Any value stored as the environment variable in the env variable is of string type.

Jenkins already has a list of pre-defined environment variables that you can use in your pipeline. A comprehensive list of the variables can be found by accessing the below URL on your browser.

${JENKINS_IP}/env-vars.html/
// Substitute JENKINS_IP with the IP address of the digital ocean droplet hosting our Jenkins server.

You should expect to see a page with all the Jenkins env variables.

Now that you can access a list of the environment variables, let us now access them in our JenkinsFile.

To access the environment variables in our JenkinsFile, we use the env object. The reference to Jenkins pipeline environment variables is made by surrounding it by${} in the following way: ${env.ENV_VARIABLE}

Let us now add an init stage in the JenkisFile where we can see this in use.

stage('Init') { 
            steps { 
                echo "Hello, this is build number ${env.BUILD_NUMBER}"
            }
        }

You can also access the environment variables without the env object

stage('Init') { 
            steps { 
                echo "Hello, this is build number ${BUILD_NUMBER}"
            }
        }

However, for pre-defined variables, it is common practice to access them with the env object.

Local environment variables in Jenkins

You can also define your own environment variables in a JenkinsFile Script. This is done by adding your variable to an environment block.

pipeline {
   agent any
   environment {
       DISABLE_AUTH = 'true' //can be used in whole pipeline
   }
   stages {
       stage(Build) {
           steps {
               echo env.DISABLE_AUTH
           }
       }
   }
}

You can also define local environment variables for a specific stage

pipeline {
   agent any
   environment {
       DISABLE_AUTH = 'true'
   }
   stages {
       stage(Build) {
           environment {
                   ENABLE_AUTH = false //can be used in this stage only
              }
           steps {
               echo env.DISABLE_AUTH
               echo env.ENABLE_AUTH
           }
       }
   }
}

You can also access credentials created via the GUI in the environment block. However, that is out of the scope of this article.

2. Parameters in your Jenkins.

You can also provide parameters in your Jenkins script. The parameters can be of the following types.

String parameters.
Boolean parameters.
Multi-line string parameters.
Choice parameters.

For the purpose of this article, we are going to see a use case which involves the use of choice parameters.

pipeline { 

    agent any
    parameters { 
        choice(name: 'Version', choices:['1.1.0', '1.2.0', '1.3.0'], description:'')
    }
    stages { 

        stage('Init') { 

            steps { 

                echo "Hello, this is build number ${BUILD_NUMBER}"
            }
        }
        stage("deploy"){ 
            steps{
                echo "Deploying Version ${VERSION}"
            }
        }
    }
}

You can now head over to the Jenkins dashboard and you will see a build with parameters option and which upon selecting the version a build will commence.

Disclaimer: This is simple into to parameters in Jenkins, a more in-depth article on the same is in the works.

Refactor the build step into a groovy script.

You will now start decomposing and refactoring the build step in our JenkisFile by moving the docker build commands into a groovy script. You can read more about the groovy scripts and its use-case in Jenkins in the documentation

Create a file named script.groovy at the root of the project and add the following code to it.

def buildApp() { 
    echo "Building app"
}
return this

To use this script in your JenkinsFile, add a script block in the init stage and load the file as follows.

        stage('Init') { 

            steps { 
                script { 
                    gv = load "script.groovy"
                }
            }
        }

To make this globally available we can define it at the top of our file.

def gv 
pipeline { 

    agent any
    parameters { 
        choice(name: 'Version', choices:['1.1.0', '1.2.0', '1.3.0'], description:'')
    }
    stages { 

        stage('Init') { 

            steps { 
                script { 
                    gv = load "script.groovy"
                }
            }
        }
        stage("deploy"){ 
            steps{
                echo "Deploying Version ${VERSION}"
            }
        }
    }
}

You are now able to access the buildApp function defined in the script.groovy file by:

stage('build') { 
            steps{
                scripts { 
                    gv.buildApp()
                }
            }
        }

Commit your code to version control and run a new build and it should pass.

We have introduced a new concept, the use of groovy scripts and this will be fully discussed in the next article.

At this stage, you should be fairly comfortable with using environment variables and parameters in Jenkins and should have a basic knowledge of the use case of groovy scripts and its importance. In the next article, we will now write a Jenkins file that builds, run tests and pushes our docker image to dockerhub by building on the foundations we have gone through today ( env variables, parameters, user inputs and use of groovy scripts.). We shall also set up email notifications to notify us of the build status (success/failure).

Happy hacking.

Dockerizing a Nest js application and deploying to Dockerhub (without docker-compose)

Kipchirchir Langat Emmanuel — Sun, 28 Aug 2022 14:58:53 +0000

Hi Peeps, Today we will look at how to dockerize a node application and deploy it to docker hub. We will be using this project for demo purposes.

1. Clone application

Navigate to your preferred folder and clone the application

cd Desktop 
git clone https://github.com/manulangat1/nest-docker-jenkins.git

2. Create your docker file.

Once you have cloned your project, on the root of the project create a file Dockerfile.

A Dockerfile is a text document that contains all the commands that a user would call on the command line for an image to be built.

FROM node:13-alphine # this is the base image in which our image will be based on. 
WORKDIR /usr/src/app # this sets the working directory for our image 
COPY package*.json . # this command copies the package.json and package-lock.json 
files from our host container to the docker container. 
EXPOSE 3000 # this exposes our port on the docker .
RUN npm install # this installs all our dependencies 
COPY . . # this copies all the other files from our host to our docker container
CMD ["npm" , "start:dev"] # this is the entry point of our application

3. Build our docker image.

We are now going to build our image using docker

docker build -t nest-docker:nd-1.0 .

The -t commands tells docker to tag our image with the name nest-docker and the . context tells it to look for the Dockerfile at the root of the project.

4. Run the docker image

We will now run our docker image

docker run -p 3000:3000 --name nest-service nest-docker:nd-1.0

-p binds our port to the port we had earlier exposed and --name gives a name to the current container that we are running.

5. Deploying to docker hub.

Head over to Dockerhub and create an account if you do not have one. Once you log in, create a repository and we are set to deploy our docker image to docker hub.

Run the following command to log in to dockerhub.

docker login

To push our docker image to docker hub, run the following command.

docker push nest-docker:nd-1.0

You have now built and deployed a dockerized nest application to dockerhub.

Follow me for more articles on docker, Jenkins and DevOps in general.

Set up Jenkins pipeline jobs to deploy a Django image to dockerhub

Kipchirchir Langat Emmanuel — Fri, 26 Aug 2022 14:50:27 +0000

This is the second tutorial in the Deploy a Django app to AWS EC2 instance using a Jenkins server series.

In this article, we will go through how to give our Jenkins permission to run docker commands and set up a basic pipeline job that deploys our Django docker build to dockerhub.

1. Give access to the jenkins user to run docker commands.

ssh into our droplet by using the commands

ssh root@YOUR-IP

Get the jenkins container id using the commands.

 docker ps

You should see an output similar to the one below.

Once we have the container id, we can now get into the container by using the commands.

docker exec -it CONTAINER_ID bash

run docker and you should be able to see a list of all docker commands.

Let's now try to run a Redis image by the commands

docker run redis

Wait, what happened? We should be hit with the following error

This error means that jenkins user does not have sufficient permissions to run certain commands.

Exit the session and get into the container as a root user by first running docker ps to get the container id, we will now get into the container as a root user by:

docker exec -u 0 -it CONTAINER_ID bash

We can now give our Jenkins user the permissions by:

chmod 666 /var/run/docker.sock

To verify that all went well, run ls -l var/run/docker.sock and you should expect an output like the one in the picture below.

To verify that all is well, get into the jenkins container as the jenkins user and run docker run redis. The error that we previously encountered should be non-existent and the command should run successfully.

2. Create a pipeline to deploy a Django docker image to dockerhub.

Head over to your jenkins page and create a new pipeline job

After successful creation, we need to connect it to our repository by clicking on the pipeline tab and scrolling down to the pipeline section.

Select the pipeline script from SCM option on the dropdown and add in the repo URL.

Make sure to add your github/ gitlab credentials as a username with a password option and the branch in which your JenkinsFile exists.

In your Django project, make sure that the JenkinsFile exists, we can now proceed to add the Jenkins configurations.

pipeline { 
    agent any 
    stages { 
        stage('Build') { 
            steps { 
                sh 'echo "Hello World"'
            }
        }
        stage('Done'){
            steps {
                echo " Wow! I'm done!"
            }
        }
        stage('Deploy staging ') { 
            steps { 
                echo 'I am deploying to a staging server
            }
        }

        stage ( 'Deploy prod') { 
            steps { 
                echo 'I am deploying to prod server

            }
        }
    }}

You can now push your code to your version control and head over to your Jenkins server web page, get into the job we created and click on the build now button to start a build.

You should be able to see your build status with the build number, you can click on the build number and it will redirect you to the build.

To check on the console output info, you can click on the console output.

We can go ahead and configure a stage that will build and deploy our Django image to dockerhub. Make sure that you have installed the credentials and the credentials binding plugins for this next step. Add your docker hub credentials with an id for you to be able to access it in your JenkinsFile.

pipeline { 
    agent any 
    stages { 
        stage('Build docker image') { 
            steps { 
                sh 'echo "Hello World"'
                script{ 
                //    
                withCredentials([ 
                    usernamePassword(credentials:'docker-hub-credentials', usernameVariable:USER, passwordVariable:PASSWORD)
                ]) { 
                    sh "docker build . -t YOUR_IMAGE_NAME"
                    sh "echo $PASSORD | docker login -u $USER --password-stdin"
                    sh "docker push YOUR_IMAGE_NAME"
                }
                }
            }
        }
    }
}

We have introduced a new concept, the withCredentials tool that we will discuss comprehensively in the next article.

At this stage, you have now set up Jenkins on a digital ocean droplet and used it to deploy a Django image to Dockerhub. In the next article, we shall cover how to use groovy scripts, and environment variables and set up a GitHub trigger that will run our builds automatically.

Cheers.

Set up Jenkins server on a digital ocean droplet.

Kipchirchir Langat Emmanuel — Thu, 25 Aug 2022 12:33:03 +0000

This article will go through the steps needed to set up a Jenkins instance on a digital ocean droplet

What is Jenkins?

Jenkins is an open-source automation server that helps development teams to automate repetitive tasks in CI/CD

Let's now head to set up our Jenkins server.

1. Register for a digital ocean account here

2. Create a droplet with the following specs.

An image - Ubuntu: the Lts tag
Plan - 4GB 2 AMD CPUs
Datacenter region - (preferably one close to your location)
Authentication method - (ssh keys) and copy your public key to Digital Ocean. follow this link to create ssh keys.

3. Configure firewall rules to open ports 22 and 8080 that our Jenkin server will use.

Click on the newly created droplet and navigate to the networking tab.

Ensure that ports 22 and 8080 are open to receive requests.

4. Install docker and run Jenkin as a docker container.

4.1 ssh into the droplet by using the command

         ```


            ssh root@YOUR_IP


#### 4.2. Update the server by using the command.

apt update




#### 4.3. Install recommended upgrades.

apt upgrade -y




#### 4.4. Install docker runtime engine.

apt install docker.io -y




## 5. Run Jenkins as a docker container using the following command

docker run -p 8080:8080 -p 50000:50000 -d \
-v jenkins_home:/var/jenkins_home \
-v /var/run/docker.sock:/var/run/docker.sock \
-v $(which docker):/usr/bin/docker jenkins/jenkins:lts




### 6. Access the Jenkins server on the web using the server's public IP as shown below

134.xxx.xx.xxx:8080




After a successful installation of Jenkins in the Docker container, you should see a screen similar to the one below that will prompt you for an administrator password

use this command to get the initialAdminPassword

cat /var/lib/docker/volumes/jenkins_home/_data/secrets/initialAdminPassword




copy the password and login into Jenkins. We can now create our first Jenkins user.

![Screenshot from 2022-08-25 15-07-43.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1661429412236/lz0Kc6M0H.png)

We have now successfully installed Jenkins.

In our next article, we shall go through how to deploy a Django application to an ec2 instance using the Jenkins server we have set up.

Dockerizing a nest app and deploying to Dockerhub (with Docker compose)

Kipchirchir Langat Emmanuel — Sun, 03 Jul 2022 20:42:08 +0000

Hello peeps,

In this article, we will go through how to deploy a nest app to dockerhub. The prerequisites needed are a dockerhub account and having docker and docker compose locally installed. We will be using this project as our starting point

1. Clone the application.

Navigate into your desired folder and clone the app using the following commands

cd Desktop 
git clone https://github.com/manulangat1/nest-bookmark-api

Once done, navigate into the cloned project

2. Adding a docker file.

A docker file is a text document that contains all the commands that will be used to build a docker image. At the root of your project, create a new file Dockerfile

touch Dockerfile

We can now start adding our commands.

FROM node:13-alphine # this is the base image that our docker image will be based on
WORKDIR /usr/src/app # this sets the working directory on our docker container 
COPY package*.json . #this command copies the package.json and package-lock.json from our host to the docker container 
RUN npm install # this installs all the packages defined in our package.json file. 
EXPOSE 3000 # exposes the port on our container 
COPY . . #copies everything from our host to our docker container
CMD ["npm" , "start:dev"] # this is the entry point of our application

3. Setting up our docker-compose file.

At the root of the project, create a new file called docker-compose.yml

touch docker-compose.yml

We can now proceed to add our docker-compose commands. Note that we will not spin up and database for this part, a more detailed post on docker-compose is in the works.

services:'3' # this tells the version of docker-compose that we are using. 
  dev-api: #this is the name of our container 
      build: . This tells docker-compose to look for the Dockerfile at the root (path where Dockerfile lives)
      environment:
      ports: # this binds the port on our local machine to our docker container
          - "3000:3000"  
      volumes: # creating a volume enables us to have a way of persisting our data. More about this soon. (we are using a named volume here)
          - db-data:/var/lib/postgresql/data
      restart: always # tells the behaviour of our application once the docker container fails

4. Fire up our docker container.

Moment of truth, we are going to fire up our container using the command:

 docker-compose up dev-api -d

-d tells the container to run in a detached mode.

To confirm whether our container is running, use the command below:

docker ps

and your output should be in the form of :

To check the logs of the container run

docker logs $container_id

To enter into our docker container use the following command.

docker exec -it $container_id /bin/sh

. 5 Deploy to dockerhub.

To deploy this image to docker hub, we have to run the following commands.

docker build -t node-nest:ndj-1.0 
docker login 
docker push node-nest:ndj-1.0

We have successfully dockerized and deployed our application to Dockerhub.

Follow for more articles on Docker, Jenkins, AWS and DevOps in general.

DEV Community: Kipchirchir Langat Emmanuel

Django automated deployments to Digital ocean using GitHub actions

1 Base application.

For this guide, we shall be using a base project that can be cloned here.

2. Setting up the GitHub actions file.

3. Dockerize the application.

4 Deploy to Dockerhub.

5. Auto-deploy to digital ocean droplet.

Deploying Django using Kubernetes - a gentle intro.

1. What is Kubernetes?

2. A brief intro to services will be covered in this article.

2.1 Pods.

2.2 Services and Ingress.

2.3 ConfigMaps and Secrets.

2.4 Deployments.

2.5 Volumes.

2.6 Namespaces.

2.6 Sample YAML configuration file.

3. Django app that will be used (Optional).

For the demonstration, the following application will be used

ecommerce monolith. Navigate to your desired folder ie

4. Deployments and services set up.

4.1 Deployment file

4.2 Secrets set up.

4.3 Services- Exposing our django app to the outside world.

Create and use custom Django signals by building a blog application

1. What are Django signals?

2. Set up a new Django project.

2.1 Navigate into your desired folder and create a virtual environment.

2.2 Install Django and create project

3. Create django apps (authentication and posts)

3.1 Create an authentication app and install drf and django-rest-knox

3.2 Create a custom User model.

3.3 Create your auth serializer files.

3.4 Create auth API endpoints.

3.5 Create the authentication URLs.

3.6 Testing out the authentication APIs

4. Create the blog application and the custom signal.

4.1 Create the blog application using django-admin command.

4.2 Create the blog models.

4.3 Create the posts serializer.

4.4 Blog application API endpoints. Create the views.

4.5 Test out the blogs API endpoints.

5. Implementing the audit logs functionality.

5.1 Create the AuditTrail model.

5.2 Custom Signal implementation.

5.3 Log user actions using the custom signal.

5.4 View to expose logs.

k

Intergrate sonarqube in Jenkins. A gentle intro to Jenkins DevSecOps.

1. Set up, configure and install Jenkins on a digital ocean droplet.

1.1 Head over to Digital Ocean and create an account if you do not already have one, otherwise log in.

1.2 Create a digital ocean droplet with the following specs.

1.3 Configure the firewall rules to open port 22 (for ssh connections ) and port 8080 (for our Jenkins server).

1.4 Install docker in the droplet.

1.5 Run Jenkins as a docker container

2. Install and configure sonarqube in the digital ocean droplet.

2.1 SSH into the digital ocean droplet by

2.2 Run sonarqube as a docker container

2.3 Configure Sonarqube in Jenkins.

3. Create a multi-branch pipeline and configure it for Django.

4. Intergrate Email sending functionality.

4.1 Install the email extension library

4.2 Configuring email extension Library

Deploy a Django image to Jenkins using a multi-branch pipeline and conditionals.

1. Create a multi-branch pipeline on Jenkins.

2. Use conditionals in Jenkins.

Jenkins pipeline to build and push a Django image to dockerhub and GitHub webhook integration.

1. Add the init stage and load the script.groovy file.

2. Add a test stage to your JenkinsFile (optional).

3. Add a build stage to your pipeline.

4. Add a deploy step to Jenkins pipeline.

5. Integrate with GitHub.

An introduction to Jenkins environment variables, parameters and use of groovy scripts.

1. How to configure and use variables in a JenkinsFile.

2. Parameters in your Jenkins.

Dockerizing a Nest js application and deploying to Dockerhub (without docker-compose)

1. Clone application

2. Create your docker file.

3. Build our docker image.

1. Add the init stage and load the `script.groovy` file.