DEV Community: Mohamed M El-Kalioby

Leopards with different file types

Mohamed M El-Kalioby — Thu, 30 Jan 2025 13:26:26 +0000

Introduction

This article covers how to use leopards with different file types.

CSV

DictReader from csv module can be used to read csv files as dictionaries as shown below.

import csv
from leopards import Q

data = csv.DictReader(open("data.csv"))
res = Q(data, {"age__gt": 15})

TSV

DictReader from csv module can be used to read tsv files as dictionaries as shown below.

import csv
from leopards import Q

data = csv.DictReader(open("data.csv"), delimiter="\t")
res = Q(data, {"age__gt": 15})

JSON

json.load can be used to read json files as dictionaries as shown below.


import json
from leopards import Q

data = json.load(open("data.json"))
res = Q(data, {"age__gt": 15})

XLS

xlrd library can be used to read xls files as dictionaries as shown below.

import xlrd
from leopards import Q

wb = xlrd.open_workbook("data.xls")
sh = wb.sheets()[0]
keys = sh.row_values(0)
data =[]
for n in range(1, sh.nrows):
    data.append({key: sh.row_values(n)[n2] for n2, key in enumerate(keys)})
res = Q(data, {"age__gt": 15})

ClickHouse

clickhouse_connect library can be used to read data from ClickHouse as dictionaries as shown below.

import clickhouse_connect
client = clickhouse_connect.get_client( host='localhost',  username='default',password='' )
rows = client.execute("SELECT * FROM TABLE")
data = rows.named_results()
res = Q(data, {"age__gt": 15})

MySQL

mysql-client library can be used to read data from MySQL as dictionaries as shown below.

import MySQLdb
from MySQLdb.cursors import DictCursor
from leopards import Q

db=MySQLdb.connect(user='root',password='PASS', database="db", cursorclass=DictCursor)
cursor = db.cursor()
cursor.execute("SELECT * FROM TABLE")
data = cursor.fetchall()
res = Q(data, {"age__gt": 15})

Passkeys as Q&A

Mohamed M El-Kalioby — Thu, 22 Jun 2023 11:12:40 +0000

Passkeys is the one of the hottest topics in the industry. It is supported by Apple, Google and Microsoft to kill passwords.

What are Passkeys?

Passkeys are an extension to the 3 years old W3C standard of Web Authentication API (WebAuthn). In WebAuthn the user is identified by a Public/Private key pair. The private key is on the authenticator and the public key is one the server. In time of authentication, the server sends a challenge to the user, the user verifies his presence to the authenticator who will sign the challenge for the server using the private key. The server will decrypt the encrypted challenge received by the public key stored on the server and if they get the original challenge sent. Then the user is present and can be authenticated.

What are the components of WebAuthn?

WebAuthn depends on three components

Server: The backend of the service that wants to register the passkey or to authenticate by the passkey.
Browser: Receives the request from the server and asks the user to select an authenticator to use either to generate new keypair or authenticate to the service and send the response back to the server.
Authenticator: The device that generates the RSA keypair and provides the public key back to the browser in case of registration or signed challenging in case of authentication.

What are the authenticators?

Authenticator is any device that can generate Public/Private Keypair. They are normally divided in 2 groups.

Platform Authenticators: These are the devices that reside inside your device. TouchID/FaceID on Apple iPhone, iPad or Macs, Windows Hello in any MS Windows installation, Android SafetyNet. In the method the authenticator verifies the user presence is different on each platform e.g on Windows, it can be Face recognition, Fingerprint, PIN or password, in other words it is anything supported by Windows Hello. On Android it can be PIN, pattern, Face recognition or Fingerprint.
Roaming Authenticators: These are the ones which are external, and can be used on several devices, they can communicate over USB, Bluetooth and/or NFC. The most popular brand is Yubico and they have Bio series that verifies the user by a fingerprint or PIN number if the fingerprint isn't recognized 3 times consequently.

How are Passkeys different from WebAuthn?

WebAuthn credentials used to primary reside on the device that generated it. e.g. if the key is generated on your Mac, you can't use it on your phone. This added a burden to developers to adopt WebAuthn because it means there shall be a fallback method to authenticate the user if he doesn't use his primary device.

In Passkeys, it is different, the key you generate on an Apple device, will be synced to iCloud keychain encrypted by the passcode, so you can use on any Apple device that your keychain is activated. In Andorid, the private keys are stored encrypted on Google Cloud.

OK, so the key will be synced to the cloud of my device's cloud provider, what if I use iPhone and need to login to the service on Windows PC?

Passkey has thought about this case and what will happen that the browser will ask you use the passkey from another device, and it gives you a QR code, you scan the code by your Phone/tablet and they will communicate over Bluetooth. The phone will confirm your presence and send the results back to the browser to send to the server to log you in.

Why are Passkeys phishing resistant?

Passkeys are phishing-resistant, because the authenticators validate the domain who is requesting to authenticate against the one who registered the keypair. For example if your key is registered for google.com, g00gle.com can't ask you to authenticate. In fact the authenticator will report that no key is available.

How to use Passkeys in my current application?

There are 2 options

Integrate a WebAuthn or Passkey library in your app and start managing the user keys with different login scenarios.
Delegate this to a Passkey provider like JustPass.me. Their service is OpenID Connect (OIDC) certified as well as FIDO2 certified. You use their service with any language and/or framework using OpenID client, they have Firebase Extension to support Mobile Apps either iOS and/or Andorid. Finally, You can use their service at zero cost up to 1,000 users.

Work on CSV files with Leopards

Mohamed M El-Kalioby — Sun, 16 Apr 2023 11:41:44 +0000

Filtering a CSV file is one of the first steps when trying to interpolate data for data science. This article will show how to use Leopards to filter a CSV file.

We will use this csv file to do our filtrations.

Install leopards

pip install leopards

Open csv file

We will use the build-in Python module csv to load CSV

import csv
rows = csv.DictReader(open('people-100.csv'), delimiter=",")

Example: Count How many of each gender is in the set

Let's see how many males/females are in the dataset

from  leopards import Count
res = Count(rows,['Sex'])
print(res)

output:

dict_values([{'Sex': 'Male', 'count': 47}, {'Sex': 'Female', 'count': 53}])

Lets run the same code again.

res = Count(rows,['Sex'])
print(res)

output:

dict_values([])

Explanation

Why did it return result in the first run but not the second run?
DictReader is an iterator so with the first run, we were on the first item, so leopards was able to iterate over it but with the second run, the iterator was already consumed so it is like an empty list. so to solve the issue either load the rows in memory like below rows=[r for r in rows] or recreate the iterator by rows = csv.DictReader(open('people-100.csv'), delimiter=",")

Example: Find people whose email is '@example.com'

from  leopards import Q
rows = csv.DictReader(open('people-100.csv'), delimiter=",")
res = Q(rows,Email__icontains='example.com')
for item in res:
    print(item)

output:

{'Index': '2', 'User Id': 'f90cD3E76f1A9b9', 'First Name': 'Phillip', 'Last Name': 'Summers', 'Sex': 'Female', 'Email': 'bethany14@example.com', 'Phone': '214.112.6044x4913', 'Date of birth': '1910-03-24', 'Job Title': 'Phytotherapist'}
{'Index': '3', 'User Id': 'DbeAb8CcdfeFC2c', 'First Name': 'Kristine', 'Last Name': 'Travis', 'Sex': 'Male', 'Email': 'bthompson@example.com', 'Phone': '277.609.7938', 'Date of birth': '1992-07-02', 'Job Title': 'Homeopath'}
{'Index': '4', 'User Id': 'A31Bee3c201ef58', 'First Name': 'Yesenia', 'Last Name': 'Martinez', 'Sex': 'Male', 'Email': 'kaitlinkaiser@example.com', 'Phone': '584.094.6111', 'Date of birth': '2017-08-03', 'Job Title': 'Market researcher'}

23 more records

Example: from the people with Email '@example.com' Return how many are 'Male'

from  leopards import Q
rows = csv.DictReader(open('people-100.csv'), delimiter=",")
res = Q(rows,Email__icontains='example.com',Sex='Male')
print(len(list(res)))

output:

The above code is eqivalent to this

from  leopards import Q
rows = csv.DictReader(open('people-100.csv'), delimiter=",")
kwargs={"Email__icontains":'example.com' ,"Sex":'Male'}
res = Q(rows,**kwargs)
print(len(list(res)))

Example: from the people with email '@example.com' Return how many are 'Male' and not born in 1992.

from  leopards import Q
rows = csv.DictReader(open('people-100.csv'), delimiter=",")
kwargs={"Email__icontains":'example.com' ,"Sex":'Male', "Date of birth__nstartswith":"1992"}
res = Q(rows,**kwargs)
print(len(list(res)))

output:

Note: Index '3' is removed.

This can be written as follows using NOT:

from  leopards import Q
rows = csv.DictReader(open('people-100.csv'), delimiter=",")
kwargs={"Email__icontains":'example.com' ,"Sex":'Male', "NOT": {"Date of birth__startswith":"1992"}}
res = Q(rows,**kwargs)
print(len(list(res)))

Example: Find the Males who emails are '@example.com' OR '@example.net'

from  leopards import Q
rows = csv.DictReader(open('people-100.csv'), delimiter=",")
kwargs = kwargs = {"Sex": "Male", "OR":[{"Email__icontains":'example.com'},{"Email__icontains":"example.net"}]}
res = Q(rows,**kwargs)
print(list(res)[:3])

output:

[{'Index': '1', 'User Id': '88F7B33d2bcf9f5', 'First Name': 'Shelby', 'Last Name': 'Terrell', 'Sex': 'Male', 'Email': 'elijah57@example.net', 'Phone': '001-084-906-7849x73518', 'Date of birth': '1945-10-26', 'Job Title': 'Games developer'}, 
{'Index': '3', 'User Id': 'DbeAb8CcdfeFC2c', 'First Name': 'Kristine', 'Last Name': 'Travis', 'Sex': 'Male', 'Email': 'bthompson@example.com', 'Phone': '277.609.7938', 'Date of birth': '1992-07-02', 'Job Title': 'Homeopath'},
{'Index': '4', 'User Id': 'A31Bee3c201ef58', 'First Name': 'Yesenia', 'Last Name': 'Martinez', 'Sex': 'Male', 'Email': 'kaitlinkaiser@example.com', 'Phone': '584.094.6111', 'Date of birth': '2017-08-03', 'Job Title': 'Market researcher'}]

Conculsion

This tutorial showed how easy it is to filter CSV and do some aggregations quickly using Leopards.

If you have any question, you can ask me.

Filter🐍 Python Iterables by Complex Queries

Mohamed M El-Kalioby — Mon, 14 Nov 2022 04:27:37 +0000

Python filter built-in function allows filtering Python iterables by function that shall return True/False. The problem with it ,that you can easily pass a query to that function to filter the iterable.

Example of Python filter

def filter_age(item):
    return item["age"] > 19
l = [{"name":"John","age":16}, {"name":"Mike","age":19},{"name":"Sarah","age":21}]
print(list(filter(filter_age,l)))

output

[{'name': 'Sarah', 'age': 21}]

What if you want to pass the age as parameter to filter_age function. This will need a partial which isn't complex but isn't comprehensive enough.

Leopards

Leopards is a Python library that provides an easy and fast way to filter members of a list or a tuple. The member can be a dict or object. The library supports AND, OR and NOT implementations.

Example

from leopards import Q
l = [{"name":"Karen","age":"16"}, {"name":"Mike","age":"19"},{"name":"Sarah","age":"21"}]
filtered= Q(l,{'name__icontains':"k", "age__gt":16})
print(list(filtered))

output

[{'name': 'Mike', 'age': '19'}]

For more examples and how to install/ use the library. Please check the github repo.

mkalioby / leopards

Query your python lists

Leopards

Leopards is a way to query list of dictionaries or objects as if you are filtering in DBMS You can get dicts/objects that are matched by OR, AND or NOT or all of them As you can see in the comparison they are much faster than Pandas.

Installation

pip install leopards

Usage

from leopards import Q
l = [{"name":"John","age":"16"}, {"name":"Mike","age":"19"},{"name":"Sarah","age":"21"}]
filtered= Q(l,{'name__contains':"k", "age__lt":20})
print(list(filtered))

output

[{'name': 'Mike', 'age': '19'}]

The above filtration can be written as

from leopards import Q
l = [{"name": "John", "age": "16"}, {"name": "Mike", "age": "19"}, {"name": "Sarah"

…

View on GitHub

Happy Coding.

Passkeys on Django

Mohamed M El-Kalioby — Sat, 29 Oct 2022 14:02:24 +0000

Passkeys is the state of the art technology standardized by FIDO Alliance to allow users to use generated Web Authication API credentials on a device to be used on another device.

For Demo, please refer to this webinar

The technology is now supported on Apple Ecosystem by the iOS 16 and iPadOS 16.1 and Mac OS X Ventura. Safari on these platforms can get credentials from Android or iOS based device. Android is currently in beta for passkey generation and Chromium based browsers (on PC and Laptop) allows picking up credentials from Android and iPhone/iPadOS.

Web Authencation API (and passkeys) is the only technology which is phishing resistant even during live Man-In-The-Middle attack, that is due to verification of the domain request to authenticate user on the authenticator to make sure it is the one registered credential before.

django-passkeys is a Django app that extends Django ModelBackend to provide a passkey authentication.

The application is a slim-down version of django-mfa2 which has been 🌟 more 150 times and downloaded round 135k. As this application did only one thing, it is much easier to integrate within your current application.

The steps needed to install and use the is in the README on GitHub repo.

mkalioby / django-passkeys

Django Authentication Backend For Passkeys

django-passkeys

An extension to Django ModelBackend backend to support passkeys.

Passkeys is an extension to Web Authentication API that will allow the user to login to a service using another device.

This app is a slim-down version of django-mfa2

Passkeys are now supported on

Apple Ecosystem (iPhone 16.0+, iPadOS 16.1, Mac OS X Ventura)
Chromium based browsers (on PC and Laptop) allows picking up credentials from Android and iPhone/iPadOS.
Android Credentials creation for ResidentKeys is currently in live now.

On May 3, 2023, Google allowed the use of Passkeys for the users to login, killing the password for enrolled users.

Installation

pip install django-passkeys

Currently, it support Django 2.0+, Python 3.7+

Usage

in your settings.py add the application to your installed apps
```
INSTALLED_APPS=(
'......'
'passkeys'
'......')
```
Collect Static Files python manage.py collectstatic
Run migrate python manage.py migrate
Add the following settings to your file
```
 AUTHENTICATION_BACKENDS
```
…

View on GitHub

Have fun :)

Django Sheet Upload

Mohamed M El-Kalioby — Mon, 17 Oct 2022 23:57:39 +0000

Most of the projects we worked on need a feature to upload data in batch, we always ended up writing another code to handle the Excel sheet and where each column shall go to each model. In the latest project we decided to handle it differently and why wrote a django app that handles this like ModelForm handles models and surprisingly, it worked.

Features

Generate a Sheet from a Django model or a combination of models (through Combined Sheet),
Add the validation rules for Choices, Foreign Keys, Integers automatically in Excel,
Showing the required field in red by default
Sheet can be validated automatically and check with is_valid()
Allow overriding the behavior of model in the class,
Allow alot of functions to override the class behavior (row_preprocess,save,pre_load,post_process),
Automatic Loading of the data in the sheet to the model.

Install

Install the package

pip install django-batch-sheet

Add it to INSTALLED_APPS

INSTALLED_APPS=[
...
'batch_sheet'
...
]

Example

Let's assume that we have the following models with these rules

Patient can have multiple tests but saved once as a Patient Object.

class Test(models.Model):
    name = models.CharField(max_length=50)

    def __str__(self):
        return self.name


class Gender (models.Model):
    name = models.CharField(max_length=50)

    def __str__(self):
        return self.name

class Patient(models.Model):
    name = models.CharField(max_length=50,verbose_name="Patient Name")
    MRN = models.CharField(max_length=50)
    gender= models.ForeignKey(Gender,on_delete=models.PROTECT,null=True, verbose_name="Gender")
    date_of_birth = models.DateField(null=True,blank=True,verbose_name="Birth Date")
    date_admitted = models.DateField(auto_now_add=True,auto_created=True)
    lastUpdate = models.DateTimeField(auto_now=True)


class RequestedTest(models.Model):
    patient = models.ForeignKey(Patient,on_delete=models.CASCADE)
    test = models.ForeignKey(Test,on_delete=models.PROTECT,verbose_name="Test")
    sample_date = models.DateField(auto_now_add=True,auto_created=True)
    sample_type = models.CharField(max_length=50, verbose_name="Sample Type",choices=(('Blood','Blood'),('DNA','DNA')))
    lastUpdate = models.DateTimeField(auto_now_add=True)

We want to combine both Models in on sheet, so we put the patient information with the requested test in one row.

So For Patient, we will implement the class PatientSheet as follows

class PatientSheet(Sheet):
    def save(self, obj:Patient, row_objs:dict):
        """Lets handle the save manually to check if the MRN exists"""
        patient = Patient.objects.filter(MRN = obj.MRN)
        if patient.exists():
            p = patient[0]
        else:
            p = obj
            p.save()
        return p
    class Meta:
        exclude=('id','date_admitted','lastUpdate')
        Model = Patient
        obj_name = "patient"
        title_header = True

For the Meta attributes, please go to the Meta Attributes sections

For RequestedTest Sheet, it goes like this

class RequestedTestSheet(Sheet):
    def save(self,obj,row_objs):
        patient = row_objs.get('patient')
        if patient is None:
            return None
        else:
            obj.patient = patient
            obj.save()
            return obj

    class Meta:
        exclude =('id','sample_date','lastUpdate')
        validation_exclude = ('patient',)
        title_header = True
        Model = RequestedTest

Now Lets combine them in one sheet

class TestSheet(CombinedSheet):
    patient = PatientSheet()
    test = RequestedTestSheet()

Now Let's Generate The sheet

$ python manage.py generate_sheet --xls App.xls --sheet test_app.sheets.TestSheet

This will the file to App.xls in the folder, now, lets open App.xls

We got the dropdown automatically, based on the value in the database.

Meta Options

    Class Meta:
        rows_count = 10              # Number of rows to apply validations on
        columns = ()                 # Columns to add from the Model to the sheet
        exclude = ()                 # Columns to exclude
        Model = None                 # Model to scan
        raw_cols = []                # Foreign Key Field that shouldn't be set as dropdown
        title_header = False         # Make the titles as "First Name"
        validation_exclude=[]        # Don't validate these field, important in case of objects relationship  
        object_name = None           # The name of the object saved, important in case of Combined Sheet.

Running Private Docker Registry for Kubernetes

Mohamed M El-Kalioby — Thu, 20 Jan 2022 16:56:16 +0000

As you work with Kubernetes, you will need a private docker registry to upload your images and then deploy them on Kubernetes, so this will summarize the steps

Note: You need a domain and a certificate from Lets Encrypt, the example domain is reg.example.com

The files are on github on the repo below

mkalioby / docker-reg-k8s

Run Docker registry for K8s

Install Apache2
```
sudo apt install apache2
```
Create a persistent storage /data on all nodes (based on the storageClass you perfer, the deployment use hostPath
Create passwd file on /data
```
 htpaswd -Bc /data/passwd username
```
Enter the password twice
Create a directory on /data/registry
```
mkdir /data/registry
```

Apply the deployment

kubectl apply -f registry_deployment.yaml

You can the file below

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: registry
name: registry
namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
       app: registry
  template:
     metadata:
       labels:
        app: registry
     spec:
       containers:
        - env:
          - name: REGISTRY_AUTH
            value: htpasswd
          - name: REGISTRY_AUTH_HTPASSWD_REALM
            value: Registry
          - name: REGISTRY_AUTH_HTPASSWD_PATH
            value: /auth/passwd
          image: registry:2
          name: registry
          ports:
            - containerPort: 5000
              protocol: TCP
          volumeMounts:
             - mountPath: /data/
               name: registry-data
             - mountPath: /auth/passwd
                name: passwd
       restartPolicy: Always
       volumes:
         - hostPath:
             path: /data/registry/
             type: Directory
           name: registry-data
         - hostPath:
            path: /data/passwd
            type: File
           name: passwd

Expose the service with NodePort
```
kubectl expose deployment registry --type NodePort
```
Make sure that CLUSTERIP/v2 works
Proxy from apache to the Nodeport
Enable Header mod
```
 a2enmod header
```
Generate a certificate from Lets encrypt, Make redirect always

Add the following to ssl config on your domain

Header add X-Forwarded-Proto "https"
RequestHeader add X-Forwarded-Proto "https"

Now login to docker registry
```
docker login https://reg.example.com
```
Enter the username and password.

Now build your image and push to the private repo

docker build -t reg.example.com/test:v1.0 . 
docker push reg.example.com/test:v1.0

Add your docker credentials to Kubernetes

kubectl create secret generic regcred --from-file=.dockerconfigjson=$HOME/.docker/config.json --type=kubernetes.io/dockerconfigjson

Create a deployment with the image from the private repo

spec:
  containers:
  - name: test
    image: reg.example.com/test:v1.0
    ports:
    - containerPort: 80
  imagePullSecrets:
  - name: regcred

Django on K8s (Part VI: Advanced Topics)

Mohamed M El-Kalioby — Thu, 13 Jan 2022 07:32:20 +0000

This part will be a collection of advanced topics that we need to make use of Kubernetes and it is the last part of this tutorial which I hope simplified the technology and gave you a good start, so let's start

For the sake of simplicity, we will use the SQLite version (master branch) of the application, but the MySQL branch will work as well.

Readiness and Liveness checks

They are similar in terms of configuration, but different when each gets invoked, that is why most guides state them together.

Rediness Check: When true the container is ready to accept traffic, can be useful if the container needs some time to prepare the environment and start receiving traffic.
Liveness Check: It is a check that Kubernetes will run periodically to make sure that the container is alive and kicking, in case it fails, the container will be considered dead and it will be terminated and a new container will be started.

Note: We said 'containers' and not 'pods' are the checks are per container.

Types of the Checks

The checks can be one of the following

command: that returns 0 if succeed, error otherwise
HttpGet: which is a success if the code is between 200 and 399.
tcpSocket: if Kubernetes can open the connection to the TCP port then it is a success else a failure.

Configure Probes:

As stated in the documentation, there are several configurations for the probes

initialDelaySeconds: the number of seconds before the check is initiated (after container starts), default is 0
periodSeconds: The interval between the checks, default 10 seconds
failureThreadhold: Try this number of times before considering the probe failed (which means restarting the container), default to 3

Example

Note: In our case (web-app), both readiness and liveness checks shall be 'httpGET', but the sake of the tutorial, we will configure the rediness check as a command and liveness to be httpGET

in the web-app container section, add the following YAML which is available in django-example_checks.yaml




     containers:
      - name: web-app
        image: django-example:v1.0
        ports:
        - containerPort: 80
        readinessProbe:                 #<-- Start here
          periodSeconds: 5
          initialDelaySeconds: 5
          failureThreshold: 100
          exec:
            command:
            - cat
            - /tmp/ready
        livenessProbe:
           httpGet:
             path: /
             port: 80
           initialDelaySeconds: 2
           periodSeconds: 3              #<-- End here
        volumeMounts:
            - name: sqlite-volume
              mountPath: /app/db.sqlite3

So the container are ready, if '/tmp/ready' exists, the container is alive it can response to the HTTP request on port 80.

Note: For the sake of the demo, we set the failureThreshold very high that should NOT be the case on a real environment.

Testing it



django-on-k8s$ kubectl apply -f django-example_checks.yaml

Wait for few seconds and then run



kubectl get po -l app=django-example

You will see that the po is not ready, so to know why it is ready run describe as shown



kubectl describe po -l app=django-example

and check at the end. You will see that the pod is unheathly as rediness probe is failing as shown

To solve the issue, connect to the pod to touch the final, this is describe in the next section 'Executing Commands in Pods'

Executing Commands in Pods

Sometimes, we need to run commands in the containers like we need to run a management command, here if it we demo, how to connect to the pod we created and touch the '/tmp/ready' so the the container becomes ready.



kubectl exec django-example-5c79c94df6-9vkr7 -c web-app -it -- /bin/bash

Notes
1. 'django-example-5c79c94df6-9vkr7' is the pod name
2. -c takes a container name, useful if there are multiple containers in the pod
3. What is after '--' is the command to pass to container, so here we started a bash session
4. '-it' is a flag to state that this is an interactive session.

Then once you get the terminal to the container, enter the following command



touch /tmp/ready

Then press CTRL+D to exit, and after check the status of the pod and you will find it ready as shown below.

Rolling Updates

Writing a complete bug-free application doesn't exist so we need to make an update to our web application now.

Rather greeting the user by the username, we will greet him by the first name.

Note: You can move to update branch if you dont want to make the change manually.

To do the update open 'django_app/accounts/templates/home.html' and Go to line 18 and change it to



  <h1>Welcome {{ request.user.first_name }}!</h1>

Next, you need to build a new docker image, to save time, build it against 'minikube docker'



django-on-k8s$ eval $(minikube docker-env)
django-on-k8s$ docker build -t django-example:v1.1 .

Then edit the deployment on Kubernetes by running



kubectl edit deployments.apps django-example

In the file, search for the image and change it to the new image and also, remove the readiness check as shown below

Now run



kubectl get po

See the old pod are terminating and a new one is starting.

Lets try the new application

Oops!, we made an issue, we didn't handle the case when there is no first name, like with the account created by 'create superuser', so first let's rollback the production to the old image, till we fix the issue.

Rollback

When we have an issue like this, don't freak-out, just ask Kubernetes to rollback like this.

First, let's get the history



kubectl rollout history  deployment django-example

This will show the change history as shown below

Note: I have multiple changes as I didn't remove the readiness check in the 2nd change.

So we want to go to 1 so we run



kubectl rollout  undo  deployment django-example --to-revision=1

Why is it ready, because of the ready state, so let's touch the file as we did above.

Let's refresh the website.

Wrap-Up

So in the last part of this tutorial, we showed readiness and liveness probes, executing commands in the container and finally how to roll and rollback an update.

There are some interesting topics we didn't cover, like
ConfigMaps and secrets, which allow to store information in Kubernetes (like the mysql password) instead on saving them on file, also Ingress and Ingress Controllers which may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting.

Finally, I hope this tutorial was useful and you learnt something.

Thank you.

Django on K8s (Part V: Running with MySQL)

Mohamed M El-Kalioby — Wed, 12 Jan 2022 07:52:16 +0000

Welcome back, in the last part, we discussed how to persist the storage between container restarts (assuming we have only one node), but we can't use SQLite on any environment, and we need a robust DBMS like MySQL to run our web-app. so In this part we will see how to use multiple containers per pod strategy to run Django with MySQL.

Warning: There is a better technique for production, where we have a MySQL service (which has replicas across several pods) and the master node is exposed to the other pods and Django apps can find it using ConfigMaps, but this is out-of-scope for this tutorial.

MySQL Container

The MySQL image on docker hub needs the databases folders '/var/lib/mysql' to be an external volume which makes sense as we need to persist the databases if the container crashes. also the MySQL image needs an env variable MYSQL_ROOT_PASSWORD to set the root password.

To configure the image, lets do the following steps

Create a storage folder on minikube
Start a MySQL container with the MYSQL_ROOT_PASSWORD environmental variable on the minikube docker
Create a database called 'django-example' which will the data for the application
Create an image of the created container,
Move the settings file to MySQL instead of SQLite.
Update the requirements file
Create new docker image of the new code.
Adjust the deployment.yaml to
1. Add MySQL container based on our image to the containers,
2. Add a command to the web container to migrate the database as the container starts

1. Create a storage folder on minikube

As minikube is our hosting we need to create a storage folder.



minikube ssh -- mkdir -p  /storage/mysql/django-example

2. Start a MySQL

Move your environment to use Docker inside minikube



$ eval $(minikube docker-env)

Start the container (with an external storage)



docker run -d -e MYSQL_ROOT_PASSWORD=pass1234 --name example-mysql  -v /storage/mysql/django-example:/var/lib/mysql mysql

The -v mount a folder from the host to the container.

After downloading all the image layers, you will get the container ID like the image below, next start a terminal and connect to mysql using the root and the password we set, as shown in the image below.

3. Create a database called 'django-example' which will the data for the application

Connect to MySQL & Create Database



$ docker exec -it example-mysql /bin/bash

mysql -u root -ppass1234



mysql> create database django_example;

Exit twice by (pressing CTRL+D) to get back to your terminal.

4. Create an image of the created container

Create a new database image using docker commit, and check the images list as shown in the image below



$ docker commit example-mysql django-example-db:v1.0
$ docker images

Verification Step

Start a new container from our image and let's check that the database is there.



docker run -d --name test-db -v /storage/mysql/django-example/:/var/lib/mysql django-example-db:v1.0
docker exec -it test-db /bin/bash
mysql -u root -ppass1234

on the mysql terminal, write



show databases

you shall find the database there as shown in the image below

Important: Clean what you done so we can continue, if you won't do so MySQL in the other containers won't start.



$ docker stop test-db 
test-db
$ docker rm test-db 
test-db

5. Move the settings file to MySQL instead of SQLite.

Note: All the steps descibed below are available in the MySQL branch in the repo

Note:In Kubernetes, when multiple containers are in the same pod, they share the same IP and they communicate using the loopback interface (localhost) so we gonna to configure Django to connect to MySQL as follows:



DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'django_example',
        'USER': 'root',
        'PASSWORD': 'pass1234',
        'HOST': '127.0.0.1'
    }
}

6. Update the 'requirements.txt' file to include 'mysqlclient' library



mysqlclient==2.1.0

7. Update Dockerfile to run the migrate command first.



FROM python:3.8-slim-bullseye

RUN apt update && apt install -y gcc libmariadb-dev-compat #<---New

RUN pip install gunicorn

WORKDIR /app

COPY django_app/requirements.txt .

RUN pip install -r /app/requirements.txt

COPY django_app/ /app/

CMD sleep 5                       #<---New
CMD python manage.py migrate      #<---New

EXPOSE 80/tcp

ENTRYPOINT gunicorn -w 4 k8s.wsgi -b 0.0.0.0:80

Run the docker build command



docker build -t django-example-mysql:v1.0 .

Adjust the deployment.yaml run the sql container with web container, as shown in django-example_mysql.yaml



apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-example-mysql
  labels:
    app: django-example-mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: django-example-mysql
  template:
    metadata:
      labels:
        app: django-example-mysql
    spec:
      volumes:                   # <-- Add MySQL Storage
        - name: mysql-volume
          hostPath:
            path: /storage/mysql/django-example/
            type: Directory
      containers:
      - name: web-app
        image: django-example-mysql:v1.0 #<--Use New Image
        command: ['/bin/bash']           #<-- Add Command to run at start-up
        args: ['-c','python manage.py migrate; gunicorn -w 4 k8s.wsgi -b 0.0.0.0:80']
        ports:
        - containerPort: 80
      - name: mysql                   #<-- Add Second Container
        image: django-example-db:v1.0
        volumeMounts:
          - name: mysql-volume
            mountPath: /var/lib/mysql/

Finally, Apply the configuration to Kubernetes



django-on-k8s$ kubectl apply -f django-example_mysql.yaml

Expose the service



$  kubectl expose deployment django-example-mysql

Start the minikube tunnel (in a new terminal window)



$ minikube tunnel

Get the IP



$ kubectl get svc

Now open a new browser window, and go to 'http://IP/' and try to login by 'admin','admin', you get 'Invalid Username or Password' as shown below.

I think, it is clear why the username and password are invalid, because the database doesn't have any users and it is just migrated by web container, so next we connect to the pod and run createsuperuser command to create the first user that is done as follows

Get the pod name through



$ kubectl get po -l  app=django-example-mysql

Note: This filters the pods by the app name
Then connect to the pod by



$ kubectl exec -it django-example-mysql-58459c6f4-q8zg8 -c web-app -- /bin/bash

The image below shows the steps

Now try to login again to the application.

You shall be able to login successfully, as shown below.

Verification

So, we logged-in successfully, but we need to make sure that the database will hold against containers failure so we can are going to delete the deployment and create it again and try to refresh and we shall be still login and we can logout and login again

To simulate pod failure, we will delete the pod automatically, which will force Kubernetes to start a new pod to match the deployment's requirements as shown in the image below



$ kubectl delete po/django-example-mysql-58459c6f4-q8zg8

Go to the browser window which is showing the application and refresh, you will find everything working and you logout and login again.

Wrap-up

In this part, we discussed how to start a MySQL container with our web-app container and make them talk together and now we have most of things to go production. Next part will show some advanced topics like rediness & liveness checks, rolling updates and rolling back and secrets, so stay tuned.

Django on K8s (Part IV: Adding Storage)

Mohamed M El-Kalioby — Tue, 11 Jan 2022 09:58:34 +0000

Welcome back,

So, at the last part, we deployed our application on Kubernetes and we were able to reach the web-app through the service.

There is a problem with containers that their storage is lost when the container dies, but users (admins included) assumes that the database changes in the application will life forever whatever happens.

Note: Storage is a complex topic and it depends on where are you running, so there is a StorageClass for AWS, GCE, or Azure, also other technologies like NFS and CEPH are supported natively in Kubernetes,so you shall work with your infrastructure team to know how the storage will be provisioned and/or claimed. In this tutorial, we will show the simplest method which is 'hostPath' (which isn't reliable on a real Kubernetes Cluster which has multiple hosts)

So In this part, we will see how to make the database file db.sqlite on a persistent storage so the app can handle being restarted.

This will include

Create the directory and copy initial db to minikube node,
Attaching a the file from the hostPath to the container in the deployment.

1. Create the directory and copy initial db to minikube node

As we will use hostPath and the host is the minikube, we need to create the folder and initial db so we can run.

django-on-k8s$ ssh -i $(minikube ssh-key)  docker@$(minikube ip) "sudo mkdir -p /storage/sqlite/example/; sudo chmod a+w /storage/sqlite/example/"
django-on-k8s$ scp -i $(minikube ssh-key) django_app/db.sqlite3 docker@$(minikube ip):/storage/sqlite/example/db.sqlite3

2. Attaching storage to new Container

Now see the new section django-example_db_storage.yaml

...............
spec:
      volumes:                # <-------
        - name: sqlite-volume
          hostPath:
            path: /storage/sqlite/example/db.sqlite3
            type: File
      containers:
      - name: web-app
        image: django-example:v1.0
        ports:
        - containerPort: 80
        volumeMounts:     # <------
            - name: sqlite-volume
              mountPath: /app/db.sqlite3

Now apply the deployment

django-on-k8s$ kubectl apply -f django-example_db_storage.yaml

Expose the service

kubectl expose deployment django-example

Next, Get IP of the service and connect to it,
then go to django-admin ('/admin/') on the container (Refer to Part III), so you and add new user, once added, Try to login with it, once succeed, kill the deployment, and start a new one with the following 2 commands

kubectl delete deploy/django-example
django-on-k8s$ kubectl apply -f django-example_db_storage.yaml

Finally, you can either login by the new user, or login with admin and check if the created user still exists.

Wrap Up

In this part, we showed how to persist storage between container results, but we just barely scratched the surface,and your next step can be reading further about Storage Classes.

Finally, In real applications, no one uses SQLite to host the database, we use a DBMS like MySQL or Postgres, so in the next part, we will show how to use multi-container per pod of Kubernetes to add a DBMS container within the pod, The explanation will be based on MySQL.

Django on K8s (Part III: Running on Kubernetes)

Mohamed M El-Kalioby — Mon, 10 Jan 2022 07:23:01 +0000

Welcome back,

In the last part, we were able to containerize our Django Application and run it on Docker and make sure it is working fine. In this part we will show how to load this image on Kubernetes and then add a deployment, and see how we can connect to it from our PC and from the network. so let's go.

Adding image to Kubernetes

Simulating a real environment, MiniKube has its own Docker registry, so we need to upload the image to that registry so we can use it to start containers, there are 2 ways to do that.

Download the image from local registry using docker save command on the disk and load it to the minikube's docker registry by docker load command

or
Build the image again on the minikube's registry using docker build command as shown in part II, which will be faster and easier as we won't save/load the full image everytime.

Note to interact with the docker registry on minikube, you have to change your ENV variables, which is easily done by running the following command
eval $(minikube docker-env) so to build the image against minikube registry run the following commands
Part

~/django-on-k8s$ eval $(minikube docker-env)
~/django-on-k8s$ docker build -t django-example:v1.0 .

Finally, check the image is there by running

 docker images

which shall show something like the image below

So now we ready to deploy our app to Kubernetes, so lets see how.

Deploy on Kubernetes

The smallest object in Kubernetes is 'Pod' which is a collection of containers running together and the one or more pods (of the same containers images) are replica-sets, which forms a deployment, so we will create a deployment with one replicas count as 1 and we will use our image as the container image.

We will use django-example_initial.yaml to do that

apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-example
  labels:
    app: django-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: django-example
  template:
    metadata:
      labels:
        app: django-example
    spec:
      containers:
      - name: web-app
        image: django-example:v1.0
        ports:
        - containerPort: 80

So we created a deployment django-example and it has 1 replicaset and it has one container web-app which is based on our django-example image and it exports one port which is 80.

So lets apply the YAML using

kubectl apply -f django-example_initial.yaml

You shall get the output like the one below

Then lets check the status of the deployment

 kubectl get deploy

and you shall get that the deployment is ready as in the image below.

Connect to the application

Now the deployment is up, we need to navigate to the web-app and that needs 2 things.

Expose the deployment on the network
Get the IP
Open a tunnel to the minikube network

1. Expose the deployment

We need to tell Kubernetes to expose the deployment so we interact with it from our network and since the port is already known from yaml file, it can be simply through the following command

kubectl expose deployment django-example

2. Get the Pod IP

After we exposed the deployment, let find the IP, this can be done through

kubectl get services

and IP is

Now, open a new browser window and Go to the http://IP/ and you shouldn't be able to connect to the Pod, that because there is no route for the minikube network, so you need to open a tunnel as show in the next section.

3. Opening a tunnel

As the minikube's containers are running under the docker container with their network CIDR, so we need to change our routes so we can reach them, that can be done by the following command

minikube tunnel

Note: You need the a sudoer account and run this command in a new terminal as it will keep running as shown below

so go a the browser window, you opened before and refresh it, and you shall be see the login page of the application and you shall be able to login as shown below.

Bonus: Exposing the service to the network

This won't work on MiniKube as Kubernetes is working on your PC under a container, but in real production environment, Kubernetes will be running on a server which is accessible on the network so if you want to expose the service to the network, you can use 'NodePort' as follows

kubectl expose deployment django-example --type NodePort

then get the port from the service list using

kubectl get service

The output will be as below

On Minikube, you get the Kubernetes IP by running

$ minikube ip
192.168.49.2

Then open a new browser window and connect to

http://192.168.49.2:31394

You will be able to reach the web-app as shown below.

Note: This does NOT require a tunnel to work which simulates the real environment.

Clean what we created

kubectl delete service django-example
kubectl delete deploy django-example

Wrap Up

So now, we have our web-app working in Kubernetes and we can reach it, but if we start adding users to the web-app and the container fails, the databases will be lost, which isn't acceptable for a web-app, the next part will show how to persist the database file between container restarts, so stay tuned.

Django on K8s (Part II: Create an app image)

Mohamed M El-Kalioby — Sun, 09 Jan 2022 08:08:18 +0000

Welcome back to the series, so now we installed Docker and minikube and we have our Django application ready which is on github. First, we will test the application on the PC using the dev server and then we will start creating an image on docker using docker file.

Test the application

We need to clone GiHub repo and set create the virtualenv.

Clone the repo

git clone https://github.com/mkalioby/django-on-k8s.git
enter the cloned folder

cd django-on-k8s
create a virtualenv

virtualenv env
start the virtualenv
source env/bin/activate
install the requirements.txt
pip install -r requiremnets.txt
run the application
cd django_app; python manage.py runserver
Go to http://127.0.0.1:8000 and login with admin admin
You shall get the same as the image below.

So now the application is working fine, let's building the image

Building the image

There are 2 ways to build an image,

start a container from a base image like Ubuntu or Python3.8 and start writing commands to make the container suitable for our requirements e.g install apt packages and/or pip packages and then create an image.
Automating the process with Dockerfile

The second method looks hard at first but once you learnt it, it will be easy to package the new code in the image.

So let start
Add the following to Dockerfile

FROM python:3.8-slim-bullseye

RUN pip install gunicorn

WORKDIR /app

COPY docker_app/requirements.txt .

RUN pip install -r /app/requirements.txt

COPY django_app/ /app/

EXPOSE 80/tcp

ENTRYPOINT gunicorn -w 4 -b 0.0.0.0:80 k8s.wsgi

I think the file is clear,we started from python:3.8, installed gunicorn (as it is NOT part of requirements.txt', set the working directory to '/app', Copied the requirements file to /app, installed the env, and then copied django_app to /app and told Docker to export port 80 and finally we ran the gunicorn command to start the server.

Note: we copied the requirements.txt separately to avoid reinstalling the pip packages if there is no updates to the requirements file e.g no packages updated.

But there are some files to skip copy as pyc, __pycache__ and the current environment so add the following to .dockerignore

*/*.zip
*/*.sh
*/*.pyc
*/__pycache__*

*.zip
*.sh
*.pyc
__pycache__
env/

Build the image

docker build -t django-example:v1.0 .

Note: the '.' at the end of the command.

You shall get something like the image below.

Test the image

Let's make sure that the image is good,
sudo docker run django-example:v1.0

then open another tab and get the container IP.

Run ps -ls to get the running containers. sudo docker ps -ls

Run inspect to get the container information based on id sudo docker inspect 890200dd1d12

Now go to http://172.17.0.2/ and you shall get the login page and login by admin, admin

This concludes the second part of the series, the next part is running this under Kubernetes, so stay tuned.