DEV Community: Hasanul Islam

uwsgi error: bind(): No such file or directory [core/socket.c line 230]

Hasanul Islam — Fri, 01 Oct 2021 16:18:28 +0000

Problem:

Sometimes, sites running on uwsgi and nginx cannot find socket file due to suddent restart of server. The most common error from /var/log/nginx/error.log in this case is:

bind(): No such file or directory [core/socket.c line 230]

Content of Nginx config file:

Support, content of /etc/nginx/sites-enabled/demo_app.ini is:

server {
    listen 80;
    server_name www.example.com;
    return 301 https://example.com$request_uri;
}

server {

    listen 80 default_server;
    server_name example.com;

    location = /favicon.ico { access_log off; log_not_found off; }

    location /static/ {
        alias /home/ubuntu/app/demo_app/static/;
            autoindex off;

            location ~* \.(eot|ttf|woff|woff2)$ {
            add_header 'Access-Control-Allow-Origin' '*';
        }
    }

    location /media/ {
        alias /home/ubuntu/app/demo_app/media/;
        autoindex off;
    }

    location / {
        add_header 'Access-Control-Allow-Origin' '*';

        include         uwsgi_params;
        uwsgi_pass      unix:/var/run/uwsgi/demo_app.sock;
    }
}

Content of uwsgi config file:

And, content of etc/uwsgi/sites/demo_app.ini file is:

project = demo_app
base    = /home/ubuntu/app

chdir   = %(base)/%(project)
home    = %(base)/venv
module  = conf.wsgi:application
env     = DJANGO_SETTINGS_MODULE=conf.settings

master  = true
processes = 5

socket = /var/run/uwsgi/%(project).sock
chmod-socket = 777
uid     = www-data
gid     = www-data

harakiri = 60
vacuum  = true
buffer-size = 32768

logger = file:/tmp/uwsgi.log
log-maxsize = 200000

Solution:

After a reboot socket file inside /var/run/uwsgi/ gets lost, because no folder exists with name uwsgi. That's why this folder is needed to be created.

mkdir /var/run/uwsgi

and

sudo chown -R www-data:www-data /var/run/uwsgi

Reference:
https://stackoverflow.com/questions/27791330/bind-no-such-file-or-directory-core-socket-c-line-230

CAP Theorem with examples

Hasanul Islam — Mon, 02 Aug 2021 16:39:51 +0000

CAP (Consistency, Availability, and Partition tolerance) theorem helps us to make our choice while designing distributed systems. We can incorporate any two of these 3 functionalities into our system, not all of them.

Consistency:

If we read immediately after writing to database, our system should return the updated result.

Available:

System should operate 24/7 without any downtime.

Partition Tolerance:

Tolerating partition means we are agreed to embrace distributed system - our machines are in different networks. There might have some sorts of network outage between different networks in distributed systems, but in that case our system should be operational. If our system does not tolerate network partition, then it is an application with one machine - not a distributed system.

If we choose to implement distributed system, that means, we are tolerating network partition. So, in the presence of P(Partition tolerance), we either choose C(Consistency) or A(Availability).

Scenerio 1:

A small system has one machine for database. No sharding, no replication.

Comment: This system ensures CA, i.e. Consistency and Availability compromising Partion tolerance.

Scenerio 2:

When we post to facebook, instagram, then instantly our friends might not see our post.

Comment: This system ensures AP, i.e Availability and Partition Tolerance comprosmising Consistency.

Scenerio 3:

When we write at google doc where lots of things can be handled at client side, and client can save the doc whenver the server is available.

Comment: This system ensures CP, i.e. Consistency and Partition tolerance compromising Availability.

Testing Environment variable in pytest

Hasanul Islam — Sat, 31 Jul 2021 12:52:23 +0000

Suppose, we have a method that uses environment variables.

import os
def get_env_vars():
    user = os.environ['USER']
    token = os.environ['TOKEN']
    return user, token

We need to test this method. We will see 3 ways to test this method.

Using `monkeypatch`

def test_get_env_vars_with_monkeypatch(monkeypatch):
    monkeypatch.setenv('USER', 'hasan')
    monkeypatch.setenv('TOKEN', 'xxx')
    u, t = get_env_vars()
    assert u == 'hasan'
    assert t == 'xxx'

Using `unittest.mock` module

from unittest import mock

def test_get_env_vars_with_mock_module():
    with mock.patch.dict(os.environ, {'USER': 'hasan', 'TOKEN': 'xxx'}):
        u, t = get_env_vars()
        assert u == 'hasan'
        assert t == 'xxx'

Using `pytest-env` plugin

We can use this plugin to set environment variables that don't really matter to the function implementations.

Install pytest-env plugin using pip install pytest-env.
Add the following portion to pytest.ini, setup.cfg etc files:

[pytest]
env =
    USER=hasan
    TOKEN=xxx

Then write test as following:

def test_get_env_vars_with_pytest_env_plugin():
    u, t = get_env_vars()
    assert u == 'hasan'
    assert t == 'xxx'

Serverless Python: Managing Python Dependencies in AWS Lambda Layer

Hasanul Islam — Fri, 30 Jul 2021 14:47:47 +0000

Usage:

 build_layer.sh <layer_name> <path/to/requirements/file> <python-version>

for example,

 scripts/build_layer.sh tensorflow requirements/tensorflow.txt 3.8

where requirements/tensorflow.txt contains:

tensorflow==2.4.0

build_layer.sh

#!/bin/bash
set -eo pipefail

function usage {
    file_name=$0
    echo "usage: ${file_name} <layer_name> <path/to/requirements.txt/file> <python_version>"
    exit 1
}

layer_name=${1}
requirements_file=${2}
python_version=${3}

if [[ -z "${requirements_file}" ]]; then
  requirements_file="requirements.txt"
fi

if [[ -z "${layer_name}" || ! -f "${requirements_file}" ]]; then
  usage
fi

layer_path=".layers/${layer_name}/python/lib/python${python_version}/site-packages"
rm -rf "$layer_path"

echo "Installing packages from ${requirements_file}"
docker run --rm \
           --user $UID \
           --volume "$PWD:/var/task" \
           --workdir /var/task \
           --env CV_PYPI_USERNAME \
           --env CV_PYPI_PASSWORD \
           "lambci/lambda:build-python${python_version}" \
            pip install -Ur "${requirements_file}" --target "${layer_path}"

echo "Removing unnecessary packages from ${layer_path}"
rm -rf "$layer_path"/{botocore,boto3}*

echo "Removing unnecessary files/folders from ${layer_path}"
find "$layer_path" -type f -name "*.pyc" -exec rm -rf {} +
find "$layer_path" -type d -name "__pycache__" -exec rm -rf {} +
find "$layer_path" -type d \( -name "tests" -o -name "test" \)  -exec rm -rf {} +

echo "Layer creation finished at ${layer_path}"

Create Tarfile(tar.gz) in Python

Hasanul Islam — Fri, 30 Jul 2021 06:29:11 +0000

Let's write some test-cases to test the behavior of create_tar_file function.

test_tarfile.py

import pytest

from tarfile import create_tar_file


def test_creates_tarfile_from_source_dir(tmpdir):
    source_dir = tmpdir.mkdir("source")
    file_path = source_dir.join("a.py")
    file_path.write("x=2")
    assert file_path.read() == "x=2"
    create_tar_file("tarfile.tar.gz", source_dir, ["a.py"])


def test_raises_exception_if_source_is_not_a_dir(tmpdir):
    with pytest.raises(ValueError, match="Not a directory"):
        create_tar_file("tarfile.tar.gz", "not_found_dir", ["a.py"])


def test_raises_exception_if_file_not_found_in_source_dir(tmpdir):
    source_dir = tmpdir.mkdir("source")
    file_path = source_dir.join("a.py")
    file_path.write("x=2")

    with pytest.raises(ValueError, match=f"b.py is not a file inside {source_dir}"):
        create_tar_file("tarfile.tar.gz", source_dir, ["b.py"])

Now, let's write the implementation of create_tar_file.

tarfile.py

import os
import tarfile
from typing import List


def create_tar_file(tarfile_path: str, source_dir: str, files: List[str]) -> None:
    if not os.path.isdir(source_dir):
        raise ValueError('Not a directory')

    os.chdir(source_dir)

    with tarfile.open(tarfile_path, "w:gz") as tar:
        for f in files:
            if not os.path.isfile(os.path.join(source_dir, f)):
                raise ValueError(f'{f} is not a file inside {source_dir}')
            tar.add(f)

GeoDjango Installation: PostGIS and GDAL

Hasanul Islam — Mon, 06 Jul 2020 06:00:27 +0000

GeoDjango is a world-class geographic Web framework. We will install GeoDjango step by step:

Installing Geospatial library: GDAL
Database installation: PostGIS
DATABASES configuration
Add django.contrib.gis to INSTALLED_APPS

Installing Geospatial library: `GDAL` :

GeoDjango provides interfaces for some source geospatial libraries e.g. GEOS, PROJ.4, GDAL, GeoIP, PostGIS, SpatiaLite. On ubuntu/Debian, We will install the following packages which will install, directly or by dependency, the required geospatial libraries:

sudo apt-get update
sudo apt-get install python3-pip python3-dev libpq-dev 
sudo apt-get install binutils libproj-dev gdal-bin
pip install GDAL

Database installation: `PostGIS` :

PostGIS adds geographic object support to PostgreSQL, turning it into a spatial database.

Install PostgreSQL
Create Database, User and Grant Privileges
Install PostGIS apt package
Install the package that contains postgis.control manually
Create PostGIS Extension

Install PostgreSQL :

sudo apt-get update
sudo apt install postgresql postgresql-contrib

Create Database, User and Grant Privileges :

Log into an interactive Postgres session by typing:

sudo -u postgres psql

First, create a database.

postgres=# CREATE DATABASE db_name;

Next, create a database user.

postgres=# CREATE USER db_user WITH PASSWORD 'password';

Then, give our new user access to administer our new database:

postgres=# GRANT ALL PRIVILEGES ON DATABASE db_name TO db_user;

Exit out of the PostgreSQL prompt by typing:

postgres# \q

Install PostGIS apt package :

sudo apt install postgis --no-install-recommends

Install the package that contains postgis.control manually :

sudo apt-get install postgis postgresql-<POSTGRESQL_VERSION>-postgis-scripts

Create PostGIS Extension :

sudo -u postgres psql
postgres=# \c db_name
db_name=# CREATE EXTENSION postgis;

`DATABASES` configuration :¶

Add the following ENGINE to DATABASES config at settings.py:

DATABASES = {
    'default': {
         'ENGINE': 'django.contrib.gis.db.backends.postgis',
          ....
    },
}

Add `django.contrib.gis` to INSTALLED_APPS :

INSTALLED_APPS = [
    'django.contrib.admin',
     .......,
    'django.contrib.gis',
]

Feel free to leave a comment.

Caching in Django REST Framework: Memcached

Hasanul Islam — Sat, 04 Jul 2020 16:48:18 +0000

Caching something is to save the result of an expensive calculation so that we don’t have to perform the calculation next time. We will discuss the caching in Django REST framework step by step:

Caching Concept
Setting up the Memcached
Adding Cache Config in settings.py
Caching API Response
Invalidating Cached Data

Caching Concept

This is the pseudocode of the general caching concept:

get data from the cache using a key
if the key is in the cache:
    return the cached data
else:
    generate data
    save the generated data in the cache with a key(for next time)
    return the generated data

Setting up the Memcached :

Install Memcached
Install a Python Client

Install Memcached :

We will install Memcached using docker image.

docker run -d memcached -p 11211:11211 -m 64

This would set the Memcached server to use 64 megabytes for storage running at 11211 port.

Install a Python Client :

We will install pylibmc, a python client to communicate with the Memcached server. But at first, we need to install the required dependencies of pylibmc,

For ubuntu or ubuntu-based docker image,

sudo apt-get install libmemcached-dev zlib1g-dev

For MacOS,

brew install libmemcached

Then install the PyPI package,

pip install pylibmc

Adding Cache Config in settings.py :

CACHE_HOST='127.0.0.1'
CACHE_PORT=11211
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': f'{CACHE_HOST}:{CACHE_PORT}',
    }
}

Caching API Response :

Suppose, We have a user app. We will have the following functionalities:

We will implement caching at user details API /api/user/{id}/.
Any request at this API will try to get from the cache. If no cached data exists for a user_id, user details data will be fetched from Database.
Any change or deletion on the User object will invalidate the cache. The following is the user details API,

from rest_framework.viewsets import ModelViewSet
from rest_framework.response import Response
from django.core.cache import cache

from apps.user.models import User
from apps.user.serializers import UserSerializer

class UserViewSet(ModelViewSet):
    queryset = User.objects.all()
    serializer_class = UserSerializer

    def retrieve(self, request, pk=None):
        cache_key = f"user_details_{pk}"

        data = cache.get(cache_key)
        if data:
            return Response(data)

        response = super().retrieve(request, pk=pk)
        cache.set(cache_key, response.data)
        return response

Invalidating Cached Data :

We will invalidate the cached data using django signal.

Creating Signal Receivers
Connecting Receivers with Signals
Changing default_app_config

Creating Signal Receivers :

We will add the following lines in user/signals.py,

def _invalidate_cached_data(user_id):
    cache_key = f"user_details_{pk}"
    cache.delete(cache_key)


def user_post_save_handler(sender, instance, created, **kwargs):
    if not created:
        _invalidate_cached_data(instance.id)


def user_post_delete_handler(sender, instance, **kwargs):
    _invalidate_cached_data(instance.id)

Connecting Receivers with Signals :

We will add the following line at user/apps.py inside user app,

from django.apps import AppConfig
from django.db.models.signals import post_save, post_delete


class UserAppConfig(AppConfig):
    name = "apps.user"

    def ready(self):
        from apps.user.models import User
        from apps.user.signals import (
            user_post_delete_handler,
            user_post_save_handler,
        )

        post_save.connect(user_post_save_handler, sender=User)
        post_delete.connect(user_post_delete_handler, sender=User)

Changing `default_app_config` :

We will add the following line at user/__init__.py in user app.

default_app_config = "apps.user.apps.UserAppConfig"

Feel free to leave a comment.

Elasticsearch: Snapshot and Restore with AWS S3

Hasanul Islam — Tue, 30 Jun 2020 10:04:45 +0000

Elasticsearch provides very easy solution to backup and restore. For this tutorial, we will store the backup at AWS S3. We will take snapshots, restore the snapshot, and create a cron job to take snapshot daily in this tutorial. This tutorial is divided into the following sections:

Snapshot
Snapshot Repository
Registering Snapshot Repository
Taking Snapshot
Restoring From Snapshot
Monitoring Snapshot and Restore Progress
Daily Backup

Snapshot :

Snapshot is a backup taken from a running Elasticsearch cluster. We can take a snapshot of individual indices or of the entire cluster. Snapshots are incremental, which means each snapshot of an index only stores data that is not part of an earlier snapshot.

Snapshot Repository :

Snapshot repository is a container that stores snapshot. Snapshots can be stored in either local or remote repositories. Remote repositories can reside on AWS S3, HDFS, Azure, Google Cloud Storage, and other platforms supported by a repository plugin.

To retrieve information about all registered snapshot repositories:

curl -X GET "localhost:9200/_snapshot/_all?pretty"

Registering Snapshot Repository :

We must register a repository to take snapshots and restore from it. To register AWS S3 as a snapshot repository, we will follow the following steps:

AWS Setup
S3 Elasticsearch Plugin Installation
Snapshot Repository Registration

AWS Setup :

S3 Bucket: In this guide, we will create an S3 bucket named S3-BUCKET-NAME.
Custom Policy: We will create a custom policy with the following policy document:

{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::S3-BUCKET-NAME"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::S3-BUCKET-NAME/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

IAM User: Then, we will create an IAM user attaching the custom policy. We need to collect the ACCESS_KEY_ID and SECRET_ACCESS_KEY.

S3 Elasticsearch Plugin Installation :

Install S3 plugin:

    cd /usr/share/elasticsearch
    sudo bin/elasticsearch-plugin install --batch repository-s3

For easy setup, set -Des.allow_insecure_settings=true to /etc/elasticsearch/jvm.options. For more secure setup, we can use elasticsearch-keystore.

Snapshot Repository Registration :
To store backup in this S3 bucket, we must need to register this bucket as a snapshot repository. We can register this bucket as snapshot registory from command line:

curl -X PUT "localhost:9200/_snapshot/REPOSITORY_NAME?pretty" -H 'Content-Type: application/json' -d'
{
 "type": "s3",
 "settings": {
   "bucket": "S3-BUCKET-NAME",
   "region": "AWS_REGION",
   "access_key": "ACCESS_KEY_ID",
   "secret_key": "SECRET_ACCESS_KEY"
 }
}
'

Taking Snapshot :

We can take a snapshot from running elasticsearch cluser by the following command, Here, SNAPSHOT_NAME is unique per REPOSITORY_NAME.

curl -X PUT "localhost:9200/_snapshot/REPOSITORY_NAME/SNAPSHOT_NAME?wait_for_completion=true&pretty" -H 'Content-Type:application/json' -d'
{
  "indices": "index_1,index_2",
  "ignore_unavailable": true,
  "include_global_state": false
 }
'

All snapshots currently stored in the repository can be listed using the following command:

curl -X GET "localhost:9200/_snapshot/my_backup/_all?pretty"

Restoring From Snapshot :

To restore indices from S3, we can do this following:

curl -X POST "localhost:9200/_snapshot/REPOSITORY_NAME/SNAPSHOT_NAME/_restore?pretty" -H 'Content-Type: application/json' -d'
{
  "indices": "index_1,index_2",
  "ignore_unavailable": true,
  "include_global_state": false,              
  "rename_pattern": "index_(.+)",
  "rename_replacement": "restored_index_$1",
  "include_aliases": false
}
'

Monitoring Snapshot and Restore Progress :

We can monitor the status of the snapshot by following command:

curl -X GET "localhost:9200/_snapshot/REPOSITORY_NAME/SNAPSHOT_NAME/_status?pretty"

Daily Backup :

Creating a Bash Script: We will create a bash script e.g. daily_elastic_search_backup.sh as the following:

#!/bin/bash

TODAY=$(date +'%Y.%m.%d')
echo Today $TODAY indices will be stored in S3.

ELASTIC_SEARCH_HOST="localhost"
ELASTIC_SEARCH_PORT="9200"
REPOSITORY_NAME="REPOSITORY_NAME"
SNAPSHOT_NAME="snapshot-"$TODAY

echo Starting Snapshot $SNAPSHOT_NAME

curl -X PUT "$ELASTIC_SEARCH_HOST:$ELASTIC_SEARCH_PORT/_snapshot/$REPOSITORY_NAME/$SNAPSHOT_NAME?wait_for_completion=true" -H 'Content-Type:application/json' -d'
{
  "indices": "index-1,index-2",
  "ignore_unavailable": true,
  "include_global_state": false
 }
'
echo Successfully completed storing "$SNAPSHOT_NAME" in S3

Adding the Script to Crontab: We can now add following line to crontab to backup every day 12am UTC:

0 0 * * * /home/ubuntu/daily_elastic_search_backup.sh > /home/ubuntu/daily_elastic_search_backup.log 2>&1