DEV Community: Mohamed M El-Kalioby

Installing gcc9 on CentOS 7

Mohamed M El-Kalioby — Wed, 26 Aug 2026 11:18:00 +0000

If you like to upgrade GCC to version 9 so you can a recent Python version like 3.14. You need to the following commands

Move to valut repo

sed -i -e 's/mirror.centos.org/vault.centos.org/g' /etc/yum.repos.d/CentOS-*.repo
sed -i -e 's/^#.*baseurl=http/baseurl=http/g' /etc/yum.repos.d/CentOS-*.repo
sed -i -e 's/^mirrorlist=http/#mirrorlist=http/g' /etc/yum.repos.d/CentOS-*.repo
sed -i s/mirror.centos.org/vault.centos.org/g /etc/yum.repos.d/CentOS-*.repo
sed -i s/^#.*baseurl=http/baseurl=http/g /etc/yum.repos.d/CentOS-*.repo
sed -i s/^mirrorlist=http/#mirrorlist=http/g /etc/yum.repos.d/CentOS-*.repo
sed -i s/mirror.centos.org/vault.centos.org/g /etc/yum.repos.d/CentOS-SCLo-*.repo 2>/dev/null || true
sed -i s/^#.*baseurl=http/baseurl=http/g /etc/yum.repos.d/CentOS-SCLo-*.repo 2>/dev/null || true
sed -i s/^mirrorlist=http/#mirrorlist=http/g /etc/yum.repos.d/CentOS-SCLo-*.repo 2>/dev/null || true

Update Yum Caches
```
yum clean all
yum makecache
```

Install devtoolset-9

yum install -y centos-release-scl
yum install -y scl-utils devtoolset-9-gcc devtoolset-9-gcc-c++

Enable toolset
```
scl enable devtoolset-9 bash
```
check the gcc version
```
gcc --version
```

Install Python 3.10+ on Old Linux

Mohamed M El-Kalioby — Tue, 25 Aug 2026 15:57:31 +0000

So, you have an old OS like CentOS 7 or Ubuntu 16.04, and you want to install a recent Python version like Python 3.11. The problem is that the openssl library in these operating systems is old, and you need a newer version of openssl to be able to use pip.

This post will discuss how to compile and install openssl on Ubuntu 16.04 and CentOS7 to install Python 3.11

Install requirements

CentOS 7

yum install -y make gcc perl-core pcre-devel wget zlib-devel gcc openssl-devel bzip2-devel libffi-devel bzip2-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel

Not: If build Python3.13+, you need to update gcc to be able to use --enable-optimizations by enabling SCL and Install GCC 9. Check this article to know how to do so.

Ubuntu 16.04

apt install -y make gcc perl-core pcre-devel wget zlib-dev gcc openssl-dev bzip2-dev libffi-dev

Download/Build openssl

Go to openssl website and download a supported version.
Extract the downloaded archive

tar xzf openssl-Ver.tar.gz
cd openssl-Ver

Configure the build

./Configure linux-x86_64 --prefix=/opt/openssl --openssldir=/opt/openssl shared

Build and Install

make install 
ldconfig /opt/openssl/lib64/

Test Build

/opt/openssl/bin/openssl version

Add the built components to the environment

export CPPFLAGS="-I/opt/openssl/include"
export LDFLAGS="-L/opt/openssl/lib64"
export PKG_CONFIG_PATH="/opt/openssl/lib/pkgconfig"
unset PYTHONHOME
unset PYTHONPATH

Download and Compile Python

Download the source from https://www.python.org/downloads/source/
Extract the source
run ./configure --enable-optimizations --enable-shared --with-openssl=/opt/openssl | grep -i ssl

Note Make sure that openssl is from /opt/ssl as screenshot below

Run make

make 
make altinstall

and you are done.
Notes:

libffi is crucial, so don't forget it.
--enable-shared is required if you will compile any library with C Extension.
pip will be installed by default.
If you get /usr/local/bin/python3.7: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory, then do the following
export LD_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib or
sudo ldconfig /usr/local/lib

Ubuntu Freezing Touchpad After Login on Matebook X

Mohamed M El-Kalioby — Wed, 19 Aug 2026 17:40:11 +0000

Problem

On my Huawei MateBook X Pro running Ubuntu 26.04 LTS, the built-in touchpad would behave strangely immediately after logging in.

The touchpad would:

Work normally for a few seconds.
Freeze for several seconds.
Start working again.
Occasionally repeat the behavior during the initial startup period.
Eventually work normally.

The problem occurred after Ubuntu started and was not a permanent touchpad failure.

I initially suspected USB autosuspend or general touchpad power management, but the problem turned out to be related to the Goodix touchpad firmware probing performed by fwupd.

Hardware and Software

My system:

Laptop:     Huawei MateBook X Pro
OS:         Ubuntu 26.04 LTS
Kernel:     7.0.0-30-generic
Touchpad:   GXTP7863
Device ID:  27C6:01E0
Interface:  I2C HID

The touchpad appears in libinput as:

Device:                  GXTP7863:00 27C6:01E0 Touchpad
Kernel:                  /dev/input/event4
Id:                       i2c:27c6:01e0
Capabilities:             pointer gesture

Diagnosing the Problem

First, install the libinput command-line tools if necessary:

sudo apt update
sudo apt install libinput-tools

Then:

libinput list-devices

The important device was:

GXTP7863:00 27C6:01E0 Touchpad

I then checked the kernel messages:

sudo dmesg | grep -Ei 'i2c|hid|touchpad|elan|synaptics|goodix'

The important errors were:

i2c_designware i2c_designware.0: controller timed out
i2c_designware i2c_designware.0: timeout in disabling adapter
i2c_hid_acpi i2c-GXTP7863:00: failed to get a report from device: -110

At first this looked like an I2C HID or kernel problem.

However, checking the complete boot log revealed something more interesting.

The Important Discovery

The following message appeared at almost exactly the same time as the touchpad freeze:

fwupd: failed to add device ... using on goodixtp:
failed to setup: gtx8 read version failed:
failed to read cfg version:
failed get report:
ioctl error: Connection timed out [110]

This was followed by:

i2c_designware.0: controller timed out

The touchpad is a Goodix device, and fwupd was attempting to communicate with it using its goodixtp plugin.

This interaction caused the I2C communication with the touchpad to time out temporarily.

Solution

The solution was to disable the goodixtp plugin in fwupd.

Edit the fwupd configuration:

sudo nano /etc/fwupd/fwupd.conf

Add:

[fwupd]
DisabledPlugins=goodixtp

If the [fwupd] section already exists, don't create a second section. Just add:

DisabledPlugins=goodixtp

Then restart fwupd:

sudo systemctl restart fwupd

Finally, reboot:

sudo reboot

After rebooting, the touchpad should work normally without the initial freezing.

Verify the Fix

After reboot, check the boot log:

journalctl -b -k | grep -Ei 'i2c|hid|GXTP|touchpad|timeout'

You should no longer see the goodixtp firmware probing causing the I2C timeout.

You can also check whether fwupd is still trying to use the plugin:

journalctl -b | grep -i goodixtp

The important thing is that the touchpad no longer freezes during startup.

Why This Happens

The MateBook X Pro uses a Goodix touchpad connected through the system's I2C controller.

The Linux input stack sees it as:

GXTP7863:00 27C6:01E0

The normal communication path is approximately:

Touchpad
   ↓
Goodix GXTP7863
   ↓
I2C HID
   ↓
Intel DesignWare I2C controller
   ↓
Linux input subsystem
   ↓
libinput
   ↓
MATE desktop

During startup, fwupd can also attempt to communicate with the Goodix touchpad through its goodixtp plugin:

fwupd
   ↓
goodixtp
   ↓
Goodix GXTP7863
   ↓
I2C

On this MateBook configuration, that firmware probe can cause the I2C controller to time out:

i2c_designware.0: controller timed out

which results in:

i2c_hid_acpi:
failed to get a report from device: -110

The result is a temporarily unresponsive touchpad.

Disabling the goodixtp plugin prevents fwupd from performing that problematic probe.

Quick Fix After Reinstalling Ubuntu

For future reference, the essential fix is:

sudo nano /etc/fwupd/fwupd.conf

Add:

[fwupd]
DisabledPlugins=goodixtp

Then:

sudo systemctl restart fwupd
sudo reboot

Useful Diagnostic Commands

If the problem returns after a future Ubuntu installation, first identify the touchpad:

libinput list-devices

Look for:

GXTP7863:00 27C6:01E0

Then check:

journalctl -b | grep -Ei 'i2c|hid|GXTP|goodix|fwupd|timeout'

The key symptoms are:

GXTP7863:00 27C6:01E0

combined with:

i2c_designware.0: controller timed out

and:

fwupd ... goodixtp ...

If those appear together, disabling goodixtp is the first thing to try.

Final Configuration

The important configuration to remember is:

File:

/etc/fwupd/fwupd.conf

Setting:

[fwupd]
DisabledPlugins=goodixtp

This fixed the MateBook X Pro touchpad startup freezing issue for me on Ubuntu 26.04 LTS.

Leopards with different file types

Mohamed M El-Kalioby — Thu, 30 Jan 2025 13:26:26 +0000

Introduction

This article covers how to use leopards with different file types.

CSV

DictReader from csv module can be used to read csv files as dictionaries as shown below.

import csv
from leopards import Q

data = csv.DictReader(open("data.csv"))
res = Q(data, {"age__gt": 15})

TSV

DictReader from csv module can be used to read tsv files as dictionaries as shown below.

import csv
from leopards import Q

data = csv.DictReader(open("data.csv"), delimiter="\t")
res = Q(data, {"age__gt": 15})

JSON

json.load can be used to read json files as dictionaries as shown below.


import json
from leopards import Q

data = json.load(open("data.json"))
res = Q(data, {"age__gt": 15})

XLS

xlrd library can be used to read xls files as dictionaries as shown below.

import xlrd
from leopards import Q

wb = xlrd.open_workbook("data.xls")
sh = wb.sheets()[0]
keys = sh.row_values(0)
data =[]
for n in range(1, sh.nrows):
    data.append({key: sh.row_values(n)[n2] for n2, key in enumerate(keys)})
res = Q(data, {"age__gt": 15})

ClickHouse

clickhouse_connect library can be used to read data from ClickHouse as dictionaries as shown below.

import clickhouse_connect
client = clickhouse_connect.get_client( host='localhost',  username='default',password='' )
rows = client.execute("SELECT * FROM TABLE")
data = rows.named_results()
res = Q(data, {"age__gt": 15})

MySQL

mysql-client library can be used to read data from MySQL as dictionaries as shown below.

import MySQLdb
from MySQLdb.cursors import DictCursor
from leopards import Q

db=MySQLdb.connect(user='root',password='PASS', database="db", cursorclass=DictCursor)
cursor = db.cursor()
cursor.execute("SELECT * FROM TABLE")
data = cursor.fetchall()
res = Q(data, {"age__gt": 15})

Passkeys as Q&A

Mohamed M El-Kalioby — Thu, 22 Jun 2023 11:12:40 +0000

Passkeys is the one of the hottest topics in the industry. It is supported by Apple, Google and Microsoft to kill passwords.

What are Passkeys?

Passkeys are an extension to the 3 years old W3C standard of Web Authentication API (WebAuthn). In WebAuthn the user is identified by a Public/Private key pair. The private key is on the authenticator and the public key is one the server. In time of authentication, the server sends a challenge to the user, the user verifies his presence to the authenticator who will sign the challenge for the server using the private key. The server will decrypt the encrypted challenge received by the public key stored on the server and if they get the original challenge sent. Then the user is present and can be authenticated.

What are the components of WebAuthn?

WebAuthn depends on three components

Server: The backend of the service that wants to register the passkey or to authenticate by the passkey.
Browser: Receives the request from the server and asks the user to select an authenticator to use either to generate new keypair or authenticate to the service and send the response back to the server.
Authenticator: The device that generates the RSA keypair and provides the public key back to the browser in case of registration or signed challenging in case of authentication.

What are the authenticators?

Authenticator is any device that can generate Public/Private Keypair. They are normally divided in 2 groups.

Platform Authenticators: These are the devices that reside inside your device. TouchID/FaceID on Apple iPhone, iPad or Macs, Windows Hello in any MS Windows installation, Android SafetyNet. In the method the authenticator verifies the user presence is different on each platform e.g on Windows, it can be Face recognition, Fingerprint, PIN or password, in other words it is anything supported by Windows Hello. On Android it can be PIN, pattern, Face recognition or Fingerprint.
Roaming Authenticators: These are the ones which are external, and can be used on several devices, they can communicate over USB, Bluetooth and/or NFC. The most popular brand is Yubico and they have Bio series that verifies the user by a fingerprint or PIN number if the fingerprint isn't recognized 3 times consequently.

How are Passkeys different from WebAuthn?

WebAuthn credentials used to primary reside on the device that generated it. e.g. if the key is generated on your Mac, you can't use it on your phone. This added a burden to developers to adopt WebAuthn because it means there shall be a fallback method to authenticate the user if he doesn't use his primary device.

In Passkeys, it is different, the key you generate on an Apple device, will be synced to iCloud keychain encrypted by the passcode, so you can use on any Apple device that your keychain is activated. In Andorid, the private keys are stored encrypted on Google Cloud.

OK, so the key will be synced to the cloud of my device's cloud provider, what if I use iPhone and need to login to the service on Windows PC?

Passkey has thought about this case and what will happen that the browser will ask you use the passkey from another device, and it gives you a QR code, you scan the code by your Phone/tablet and they will communicate over Bluetooth. The phone will confirm your presence and send the results back to the browser to send to the server to log you in.

Why are Passkeys phishing resistant?

Passkeys are phishing-resistant, because the authenticators validate the domain who is requesting to authenticate against the one who registered the keypair. For example if your key is registered for google.com, g00gle.com can't ask you to authenticate. In fact the authenticator will report that no key is available.

How to use Passkeys in my current application?

There are 2 options

Integrate a WebAuthn or Passkey library in your app and start managing the user keys with different login scenarios.
Delegate this to a Passkey provider like JustPass.me. Their service is OpenID Connect (OIDC) certified as well as FIDO2 certified. You use their service with any language and/or framework using OpenID client, they have Firebase Extension to support Mobile Apps either iOS and/or Andorid. Finally, You can use their service at zero cost up to 1,000 users.

Work on CSV files with Leopards

Mohamed M El-Kalioby — Sun, 16 Apr 2023 11:41:44 +0000

Filtering a CSV file is one of the first steps when trying to interpolate data for data science. This article will show how to use Leopards to filter a CSV file.

We will use this csv file to do our filtrations.

Install leopards

pip install leopards

Open csv file

We will use the build-in Python module csv to load CSV

import csv
rows = csv.DictReader(open('people-100.csv'), delimiter=",")

Example: Count How many of each gender is in the set

Let's see how many males/females are in the dataset

from  leopards import Count
res = Count(rows,['Sex'])
print(res)

output:

dict_values([{'Sex': 'Male', 'count': 47}, {'Sex': 'Female', 'count': 53}])

Lets run the same code again.

res = Count(rows,['Sex'])
print(res)

output:

dict_values([])

Explanation

Why did it return result in the first run but not the second run?
DictReader is an iterator so with the first run, we were on the first item, so leopards was able to iterate over it but with the second run, the iterator was already consumed so it is like an empty list. so to solve the issue either load the rows in memory like below rows=[r for r in rows] or recreate the iterator by rows = csv.DictReader(open('people-100.csv'), delimiter=",")

Example: Find people whose email is '@example.com'

from  leopards import Q
rows = csv.DictReader(open('people-100.csv'), delimiter=",")
res = Q(rows,Email__icontains='example.com')
for item in res:
    print(item)

output:

{'Index': '2', 'User Id': 'f90cD3E76f1A9b9', 'First Name': 'Phillip', 'Last Name': 'Summers', 'Sex': 'Female', 'Email': 'bethany14@example.com', 'Phone': '214.112.6044x4913', 'Date of birth': '1910-03-24', 'Job Title': 'Phytotherapist'}
{'Index': '3', 'User Id': 'DbeAb8CcdfeFC2c', 'First Name': 'Kristine', 'Last Name': 'Travis', 'Sex': 'Male', 'Email': 'bthompson@example.com', 'Phone': '277.609.7938', 'Date of birth': '1992-07-02', 'Job Title': 'Homeopath'}
{'Index': '4', 'User Id': 'A31Bee3c201ef58', 'First Name': 'Yesenia', 'Last Name': 'Martinez', 'Sex': 'Male', 'Email': 'kaitlinkaiser@example.com', 'Phone': '584.094.6111', 'Date of birth': '2017-08-03', 'Job Title': 'Market researcher'}

23 more records

Example: from the people with Email '@example.com' Return how many are 'Male'

from  leopards import Q
rows = csv.DictReader(open('people-100.csv'), delimiter=",")
res = Q(rows,Email__icontains='example.com',Sex='Male')
print(len(list(res)))

output:

The above code is eqivalent to this

from  leopards import Q
rows = csv.DictReader(open('people-100.csv'), delimiter=",")
kwargs={"Email__icontains":'example.com' ,"Sex":'Male'}
res = Q(rows,**kwargs)
print(len(list(res)))

Example: from the people with email '@example.com' Return how many are 'Male' and not born in 1992.

from  leopards import Q
rows = csv.DictReader(open('people-100.csv'), delimiter=",")
kwargs={"Email__icontains":'example.com' ,"Sex":'Male', "Date of birth__nstartswith":"1992"}
res = Q(rows,**kwargs)
print(len(list(res)))

output:

Note: Index '3' is removed.

This can be written as follows using NOT:

from  leopards import Q
rows = csv.DictReader(open('people-100.csv'), delimiter=",")
kwargs={"Email__icontains":'example.com' ,"Sex":'Male', "NOT": {"Date of birth__startswith":"1992"}}
res = Q(rows,**kwargs)
print(len(list(res)))

Example: Find the Males who emails are '@example.com' OR '@example.net'

from  leopards import Q
rows = csv.DictReader(open('people-100.csv'), delimiter=",")
kwargs = kwargs = {"Sex": "Male", "OR":[{"Email__icontains":'example.com'},{"Email__icontains":"example.net"}]}
res = Q(rows,**kwargs)
print(list(res)[:3])

output:

[{'Index': '1', 'User Id': '88F7B33d2bcf9f5', 'First Name': 'Shelby', 'Last Name': 'Terrell', 'Sex': 'Male', 'Email': 'elijah57@example.net', 'Phone': '001-084-906-7849x73518', 'Date of birth': '1945-10-26', 'Job Title': 'Games developer'}, 
{'Index': '3', 'User Id': 'DbeAb8CcdfeFC2c', 'First Name': 'Kristine', 'Last Name': 'Travis', 'Sex': 'Male', 'Email': 'bthompson@example.com', 'Phone': '277.609.7938', 'Date of birth': '1992-07-02', 'Job Title': 'Homeopath'},
{'Index': '4', 'User Id': 'A31Bee3c201ef58', 'First Name': 'Yesenia', 'Last Name': 'Martinez', 'Sex': 'Male', 'Email': 'kaitlinkaiser@example.com', 'Phone': '584.094.6111', 'Date of birth': '2017-08-03', 'Job Title': 'Market researcher'}]

Conculsion

This tutorial showed how easy it is to filter CSV and do some aggregations quickly using Leopards.

If you have any question, you can ask me.

Filter🐍 Python Iterables by Complex Queries

Mohamed M El-Kalioby — Mon, 14 Nov 2022 04:27:37 +0000

Python filter built-in function allows filtering Python iterables by function that shall return True/False. The problem with it ,that you can easily pass a query to that function to filter the iterable.

Example of Python filter

def filter_age(item):
    return item["age"] > 19
l = [{"name":"John","age":16}, {"name":"Mike","age":19},{"name":"Sarah","age":21}]
print(list(filter(filter_age,l)))

output

[{'name': 'Sarah', 'age': 21}]

What if you want to pass the age as parameter to filter_age function. This will need a partial which isn't complex but isn't comprehensive enough.

Leopards

Leopards is a Python library that provides an easy and fast way to filter members of a list or a tuple. The member can be a dict or object. The library supports AND, OR and NOT implementations.

Example

from leopards import Q
l = [{"name":"Karen","age":"16"}, {"name":"Mike","age":"19"},{"name":"Sarah","age":"21"}]
filtered= Q(l,{'name__icontains':"k", "age__gt":16})
print(list(filtered))

output

[{'name': 'Mike', 'age': '19'}]

For more examples and how to install/ use the library. Please check the github repo.

mkalioby / leopards

Query your python lists

Leopards

Leopards is a way to query list of dictionaries or objects as if you are filtering in DBMS You can get dicts/objects that are matched by OR, AND or NOT or all of them As you can see in the comparison they are much faster than Pandas.

Installation

pip install leopards

Usage

from leopards import Q
l = [{"name":"John","age":"16"}, {"name":"Mike","age":"19"},{"name":"Sarah","age":"21"}]
filtered= Q(l,{'name__contains':"k", "age__lt":20})
print(list(filtered))

output

[{'name': 'Mike', 'age': '19'}]

The above filtration can be written as

from leopards import Q
l = [{"name": "John", "age": "16"}, {"name": "Mike", "age": "19"}, {"name": "Sarah"

…

View on GitHub

Happy Coding.

Passkeys on Django

Mohamed M El-Kalioby — Sat, 29 Oct 2022 14:02:24 +0000

Passkeys is the state of the art technology standardized by FIDO Alliance to allow users to use generated Web Authication API credentials on a device to be used on another device.

For Demo, please refer to this webinar

The technology is now supported on Apple Ecosystem by the iOS 16 and iPadOS 16.1 and Mac OS X Ventura. Safari on these platforms can get credentials from Android or iOS based device. Android is currently in beta for passkey generation and Chromium based browsers (on PC and Laptop) allows picking up credentials from Android and iPhone/iPadOS.

Web Authencation API (and passkeys) is the only technology which is phishing resistant even during live Man-In-The-Middle attack, that is due to verification of the domain request to authenticate user on the authenticator to make sure it is the one registered credential before.

django-passkeys is a Django app that extends Django ModelBackend to provide a passkey authentication.

The application is a slim-down version of django-mfa2 which has been 🌟 more 150 times and downloaded round 135k. As this application did only one thing, it is much easier to integrate within your current application.

The steps needed to install and use the is in the README on GitHub repo.

mkalioby / django-passkeys

Django Authentication Backend For Passkeys

django-passkeys

An extension to Django ModelBackend backend to support passkeys.

Passkeys is an extension to Web Authentication API that will allow the user to login to a service using another device.

This app is a slim-down version of django-mfa2

Passkeys are now supported on

Apple Ecosystem (iPhone 16.0+, iPadOS 16.1, Mac OS X Ventura)
Chromium based browsers (on PC and Laptop) allows picking up credentials from Android and iPhone/iPadOS.
Android Credentials creation for ResidentKeys is currently in live now.

On May 3, 2023, Google allowed the use of Passkeys for the users to login, killing the password for enrolled users.

Installation

pip install django-passkeys

Currently, it support Django 2.0+, Python 3.7+

Usage

in your settings.py add the application to your installed apps
```
INSTALLED_APPS=(
'......'
'passkeys'
'......')
```
Collect Static Files python manage.py collectstatic
Run migrate python manage.py migrate
Add the following settings to your file
```
 AUTHENTICATION_BACKENDS
```
…

View on GitHub

Have fun :)

Django Sheet Upload

Mohamed M El-Kalioby — Mon, 17 Oct 2022 23:57:39 +0000

Most of the projects we worked on need a feature to upload data in batch, we always ended up writing another code to handle the Excel sheet and where each column shall go to each model. In the latest project we decided to handle it differently and why wrote a django app that handles this like ModelForm handles models and surprisingly, it worked.

Features

Generate a Sheet from a Django model or a combination of models (through Combined Sheet),
Add the validation rules for Choices, Foreign Keys, Integers automatically in Excel,
Showing the required field in red by default
Sheet can be validated automatically and check with is_valid()
Allow overriding the behavior of model in the class,
Allow alot of functions to override the class behavior (row_preprocess,save,pre_load,post_process),
Automatic Loading of the data in the sheet to the model.

Install

Install the package

pip install django-batch-sheet

Add it to INSTALLED_APPS

INSTALLED_APPS=[
...
'batch_sheet'
...
]

Example

Let's assume that we have the following models with these rules

Patient can have multiple tests but saved once as a Patient Object.

class Test(models.Model):
    name = models.CharField(max_length=50)

    def __str__(self):
        return self.name


class Gender (models.Model):
    name = models.CharField(max_length=50)

    def __str__(self):
        return self.name

class Patient(models.Model):
    name = models.CharField(max_length=50,verbose_name="Patient Name")
    MRN = models.CharField(max_length=50)
    gender= models.ForeignKey(Gender,on_delete=models.PROTECT,null=True, verbose_name="Gender")
    date_of_birth = models.DateField(null=True,blank=True,verbose_name="Birth Date")
    date_admitted = models.DateField(auto_now_add=True,auto_created=True)
    lastUpdate = models.DateTimeField(auto_now=True)


class RequestedTest(models.Model):
    patient = models.ForeignKey(Patient,on_delete=models.CASCADE)
    test = models.ForeignKey(Test,on_delete=models.PROTECT,verbose_name="Test")
    sample_date = models.DateField(auto_now_add=True,auto_created=True)
    sample_type = models.CharField(max_length=50, verbose_name="Sample Type",choices=(('Blood','Blood'),('DNA','DNA')))
    lastUpdate = models.DateTimeField(auto_now_add=True)

We want to combine both Models in on sheet, so we put the patient information with the requested test in one row.

So For Patient, we will implement the class PatientSheet as follows

class PatientSheet(Sheet):
    def save(self, obj:Patient, row_objs:dict):
        """Lets handle the save manually to check if the MRN exists"""
        patient = Patient.objects.filter(MRN = obj.MRN)
        if patient.exists():
            p = patient[0]
        else:
            p = obj
            p.save()
        return p
    class Meta:
        exclude=('id','date_admitted','lastUpdate')
        Model = Patient
        obj_name = "patient"
        title_header = True

For the Meta attributes, please go to the Meta Attributes sections

For RequestedTest Sheet, it goes like this

class RequestedTestSheet(Sheet):
    def save(self,obj,row_objs):
        patient = row_objs.get('patient')
        if patient is None:
            return None
        else:
            obj.patient = patient
            obj.save()
            return obj

    class Meta:
        exclude =('id','sample_date','lastUpdate')
        validation_exclude = ('patient',)
        title_header = True
        Model = RequestedTest

Now Lets combine them in one sheet

class TestSheet(CombinedSheet):
    patient = PatientSheet()
    test = RequestedTestSheet()

Now Let's Generate The sheet

$ python manage.py generate_sheet --xls App.xls --sheet test_app.sheets.TestSheet

This will the file to App.xls in the folder, now, lets open App.xls

We got the dropdown automatically, based on the value in the database.

Meta Options

    Class Meta:
        rows_count = 10              # Number of rows to apply validations on
        columns = ()                 # Columns to add from the Model to the sheet
        exclude = ()                 # Columns to exclude
        Model = None                 # Model to scan
        raw_cols = []                # Foreign Key Field that shouldn't be set as dropdown
        title_header = False         # Make the titles as "First Name"
        validation_exclude=[]        # Don't validate these field, important in case of objects relationship  
        object_name = None           # The name of the object saved, important in case of Combined Sheet.

Running Private Docker Registry for Kubernetes

Mohamed M El-Kalioby — Thu, 20 Jan 2022 16:56:16 +0000

As you work with Kubernetes, you will need a private docker registry to upload your images and then deploy them on Kubernetes, so this will summarize the steps

Note: You need a domain and a certificate from Lets Encrypt, the example domain is reg.example.com

The files are on github on the repo below

mkalioby / docker-reg-k8s

Run Docker registry for K8s

Install Apache2
```
sudo apt install apache2
```
Create a persistent storage /data on all nodes (based on the storageClass you perfer, the deployment use hostPath
Create passwd file on /data
```
 htpaswd -Bc /data/passwd username
```
Enter the password twice
Create a directory on /data/registry
```
mkdir /data/registry
```

Apply the deployment

kubectl apply -f registry_deployment.yaml

You can the file below

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: registry
name: registry
namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
       app: registry
  template:
     metadata:
       labels:
        app: registry
     spec:
       containers:
        - env:
          - name: REGISTRY_AUTH
            value: htpasswd
          - name: REGISTRY_AUTH_HTPASSWD_REALM
            value: Registry
          - name: REGISTRY_AUTH_HTPASSWD_PATH
            value: /auth/passwd
          image: registry:2
          name: registry
          ports:
            - containerPort: 5000
              protocol: TCP
          volumeMounts:
             - mountPath: /data/
               name: registry-data
             - mountPath: /auth/passwd
                name: passwd
       restartPolicy: Always
       volumes:
         - hostPath:
             path: /data/registry/
             type: Directory
           name: registry-data
         - hostPath:
            path: /data/passwd
            type: File
           name: passwd

Expose the service with NodePort
```
kubectl expose deployment registry --type NodePort
```
Make sure that CLUSTERIP/v2 works
Proxy from apache to the Nodeport
Enable Header mod
```
 a2enmod header
```
Generate a certificate from Lets encrypt, Make redirect always

Add the following to ssl config on your domain

Header add X-Forwarded-Proto "https"
RequestHeader add X-Forwarded-Proto "https"

Now login to docker registry
```
docker login https://reg.example.com
```
Enter the username and password.

Now build your image and push to the private repo

docker build -t reg.example.com/test:v1.0 . 
docker push reg.example.com/test:v1.0

Add your docker credentials to Kubernetes

kubectl create secret generic regcred --from-file=.dockerconfigjson=$HOME/.docker/config.json --type=kubernetes.io/dockerconfigjson

Create a deployment with the image from the private repo

spec:
  containers:
  - name: test
    image: reg.example.com/test:v1.0
    ports:
    - containerPort: 80
  imagePullSecrets:
  - name: regcred

Django on K8s (Part VI: Advanced Topics)

Mohamed M El-Kalioby — Thu, 13 Jan 2022 07:32:20 +0000

This part will be a collection of advanced topics that we need to make use of Kubernetes and it is the last part of this tutorial which I hope simplified the technology and gave you a good start, so let's start

For the sake of simplicity, we will use the SQLite version (master branch) of the application, but the MySQL branch will work as well.

Readiness and Liveness checks

They are similar in terms of configuration, but different when each gets invoked, that is why most guides state them together.

Rediness Check: When true the container is ready to accept traffic, can be useful if the container needs some time to prepare the environment and start receiving traffic.
Liveness Check: It is a check that Kubernetes will run periodically to make sure that the container is alive and kicking, in case it fails, the container will be considered dead and it will be terminated and a new container will be started.

Note: We said 'containers' and not 'pods' are the checks are per container.

Types of the Checks

The checks can be one of the following

command: that returns 0 if succeed, error otherwise
HttpGet: which is a success if the code is between 200 and 399.
tcpSocket: if Kubernetes can open the connection to the TCP port then it is a success else a failure.

Configure Probes:

As stated in the documentation, there are several configurations for the probes

initialDelaySeconds: the number of seconds before the check is initiated (after container starts), default is 0
periodSeconds: The interval between the checks, default 10 seconds
failureThreadhold: Try this number of times before considering the probe failed (which means restarting the container), default to 3

Example

Note: In our case (web-app), both readiness and liveness checks shall be 'httpGET', but the sake of the tutorial, we will configure the rediness check as a command and liveness to be httpGET

in the web-app container section, add the following YAML which is available in django-example_checks.yaml




     containers:
      - name: web-app
        image: django-example:v1.0
        ports:
        - containerPort: 80
        readinessProbe:                 #<-- Start here
          periodSeconds: 5
          initialDelaySeconds: 5
          failureThreshold: 100
          exec:
            command:
            - cat
            - /tmp/ready
        livenessProbe:
           httpGet:
             path: /
             port: 80
           initialDelaySeconds: 2
           periodSeconds: 3              #<-- End here
        volumeMounts:
            - name: sqlite-volume
              mountPath: /app/db.sqlite3

So the container are ready, if '/tmp/ready' exists, the container is alive it can response to the HTTP request on port 80.

Note: For the sake of the demo, we set the failureThreshold very high that should NOT be the case on a real environment.

Testing it



django-on-k8s$ kubectl apply -f django-example_checks.yaml

Wait for few seconds and then run



kubectl get po -l app=django-example

You will see that the po is not ready, so to know why it is ready run describe as shown



kubectl describe po -l app=django-example

and check at the end. You will see that the pod is unheathly as rediness probe is failing as shown

To solve the issue, connect to the pod to touch the final, this is describe in the next section 'Executing Commands in Pods'

Executing Commands in Pods

Sometimes, we need to run commands in the containers like we need to run a management command, here if it we demo, how to connect to the pod we created and touch the '/tmp/ready' so the the container becomes ready.



kubectl exec django-example-5c79c94df6-9vkr7 -c web-app -it -- /bin/bash

Notes
1. 'django-example-5c79c94df6-9vkr7' is the pod name
2. -c takes a container name, useful if there are multiple containers in the pod
3. What is after '--' is the command to pass to container, so here we started a bash session
4. '-it' is a flag to state that this is an interactive session.

Then once you get the terminal to the container, enter the following command



touch /tmp/ready

Then press CTRL+D to exit, and after check the status of the pod and you will find it ready as shown below.

Rolling Updates

Writing a complete bug-free application doesn't exist so we need to make an update to our web application now.

Rather greeting the user by the username, we will greet him by the first name.

Note: You can move to update branch if you dont want to make the change manually.

To do the update open 'django_app/accounts/templates/home.html' and Go to line 18 and change it to



  <h1>Welcome {{ request.user.first_name }}!</h1>

Next, you need to build a new docker image, to save time, build it against 'minikube docker'



django-on-k8s$ eval $(minikube docker-env)
django-on-k8s$ docker build -t django-example:v1.1 .

Then edit the deployment on Kubernetes by running



kubectl edit deployments.apps django-example

In the file, search for the image and change it to the new image and also, remove the readiness check as shown below

Now run



kubectl get po

See the old pod are terminating and a new one is starting.

Lets try the new application

Oops!, we made an issue, we didn't handle the case when there is no first name, like with the account created by 'create superuser', so first let's rollback the production to the old image, till we fix the issue.

Rollback

When we have an issue like this, don't freak-out, just ask Kubernetes to rollback like this.

First, let's get the history



kubectl rollout history  deployment django-example

This will show the change history as shown below

Note: I have multiple changes as I didn't remove the readiness check in the 2nd change.

So we want to go to 1 so we run



kubectl rollout  undo  deployment django-example --to-revision=1

Why is it ready, because of the ready state, so let's touch the file as we did above.

Let's refresh the website.

Wrap-Up

So in the last part of this tutorial, we showed readiness and liveness probes, executing commands in the container and finally how to roll and rollback an update.

There are some interesting topics we didn't cover, like
ConfigMaps and secrets, which allow to store information in Kubernetes (like the mysql password) instead on saving them on file, also Ingress and Ingress Controllers which may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting.

Finally, I hope this tutorial was useful and you learnt something.

Thank you.

Django on K8s (Part V: Running with MySQL)

Mohamed M El-Kalioby — Wed, 12 Jan 2022 07:52:16 +0000

Welcome back, in the last part, we discussed how to persist the storage between container restarts (assuming we have only one node), but we can't use SQLite on any environment, and we need a robust DBMS like MySQL to run our web-app. so In this part we will see how to use multiple containers per pod strategy to run Django with MySQL.

Warning: There is a better technique for production, where we have a MySQL service (which has replicas across several pods) and the master node is exposed to the other pods and Django apps can find it using ConfigMaps, but this is out-of-scope for this tutorial.

MySQL Container

The MySQL image on docker hub needs the databases folders '/var/lib/mysql' to be an external volume which makes sense as we need to persist the databases if the container crashes. also the MySQL image needs an env variable MYSQL_ROOT_PASSWORD to set the root password.

To configure the image, lets do the following steps

Create a storage folder on minikube
Start a MySQL container with the MYSQL_ROOT_PASSWORD environmental variable on the minikube docker
Create a database called 'django-example' which will the data for the application
Create an image of the created container,
Move the settings file to MySQL instead of SQLite.
Update the requirements file
Create new docker image of the new code.
Adjust the deployment.yaml to
1. Add MySQL container based on our image to the containers,
2. Add a command to the web container to migrate the database as the container starts

1. Create a storage folder on minikube

As minikube is our hosting we need to create a storage folder.



minikube ssh -- mkdir -p  /storage/mysql/django-example

2. Start a MySQL

Move your environment to use Docker inside minikube



$ eval $(minikube docker-env)

Start the container (with an external storage)



docker run -d -e MYSQL_ROOT_PASSWORD=pass1234 --name example-mysql  -v /storage/mysql/django-example:/var/lib/mysql mysql

The -v mount a folder from the host to the container.

After downloading all the image layers, you will get the container ID like the image below, next start a terminal and connect to mysql using the root and the password we set, as shown in the image below.

3. Create a database called 'django-example' which will the data for the application

Connect to MySQL & Create Database



$ docker exec -it example-mysql /bin/bash

mysql -u root -ppass1234



mysql> create database django_example;

Exit twice by (pressing CTRL+D) to get back to your terminal.

4. Create an image of the created container

Create a new database image using docker commit, and check the images list as shown in the image below



$ docker commit example-mysql django-example-db:v1.0
$ docker images

Verification Step

Start a new container from our image and let's check that the database is there.



docker run -d --name test-db -v /storage/mysql/django-example/:/var/lib/mysql django-example-db:v1.0
docker exec -it test-db /bin/bash
mysql -u root -ppass1234

on the mysql terminal, write



show databases

you shall find the database there as shown in the image below

Important: Clean what you done so we can continue, if you won't do so MySQL in the other containers won't start.



$ docker stop test-db 
test-db
$ docker rm test-db 
test-db

5. Move the settings file to MySQL instead of SQLite.

Note: All the steps descibed below are available in the MySQL branch in the repo

Note:In Kubernetes, when multiple containers are in the same pod, they share the same IP and they communicate using the loopback interface (localhost) so we gonna to configure Django to connect to MySQL as follows:



DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'django_example',
        'USER': 'root',
        'PASSWORD': 'pass1234',
        'HOST': '127.0.0.1'
    }
}

6. Update the 'requirements.txt' file to include 'mysqlclient' library



mysqlclient==2.1.0

7. Update Dockerfile to run the migrate command first.



FROM python:3.8-slim-bullseye

RUN apt update && apt install -y gcc libmariadb-dev-compat #<---New

RUN pip install gunicorn

WORKDIR /app

COPY django_app/requirements.txt .

RUN pip install -r /app/requirements.txt

COPY django_app/ /app/

CMD sleep 5                       #<---New
CMD python manage.py migrate      #<---New

EXPOSE 80/tcp

ENTRYPOINT gunicorn -w 4 k8s.wsgi -b 0.0.0.0:80

Run the docker build command



docker build -t django-example-mysql:v1.0 .

Adjust the deployment.yaml run the sql container with web container, as shown in django-example_mysql.yaml



apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-example-mysql
  labels:
    app: django-example-mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: django-example-mysql
  template:
    metadata:
      labels:
        app: django-example-mysql
    spec:
      volumes:                   # <-- Add MySQL Storage
        - name: mysql-volume
          hostPath:
            path: /storage/mysql/django-example/
            type: Directory
      containers:
      - name: web-app
        image: django-example-mysql:v1.0 #<--Use New Image
        command: ['/bin/bash']           #<-- Add Command to run at start-up
        args: ['-c','python manage.py migrate; gunicorn -w 4 k8s.wsgi -b 0.0.0.0:80']
        ports:
        - containerPort: 80
      - name: mysql                   #<-- Add Second Container
        image: django-example-db:v1.0
        volumeMounts:
          - name: mysql-volume
            mountPath: /var/lib/mysql/

Finally, Apply the configuration to Kubernetes



django-on-k8s$ kubectl apply -f django-example_mysql.yaml

Expose the service



$  kubectl expose deployment django-example-mysql

Start the minikube tunnel (in a new terminal window)



$ minikube tunnel

Get the IP



$ kubectl get svc

Now open a new browser window, and go to 'http://IP/' and try to login by 'admin','admin', you get 'Invalid Username or Password' as shown below.

I think, it is clear why the username and password are invalid, because the database doesn't have any users and it is just migrated by web container, so next we connect to the pod and run createsuperuser command to create the first user that is done as follows

Get the pod name through



$ kubectl get po -l  app=django-example-mysql

Note: This filters the pods by the app name
Then connect to the pod by



$ kubectl exec -it django-example-mysql-58459c6f4-q8zg8 -c web-app -- /bin/bash

The image below shows the steps

Now try to login again to the application.

You shall be able to login successfully, as shown below.

Verification

So, we logged-in successfully, but we need to make sure that the database will hold against containers failure so we can are going to delete the deployment and create it again and try to refresh and we shall be still login and we can logout and login again

To simulate pod failure, we will delete the pod automatically, which will force Kubernetes to start a new pod to match the deployment's requirements as shown in the image below



$ kubectl delete po/django-example-mysql-58459c6f4-q8zg8

Go to the browser window which is showing the application and refresh, you will find everything working and you logout and login again.

Wrap-up

In this part, we discussed how to start a MySQL container with our web-app container and make them talk together and now we have most of things to go production. Next part will show some advanced topics like rediness & liveness checks, rolling updates and rolling back and secrets, so stay tuned.