DEV Community: Idiomatic Programmers

Ultimate Redis Cheatsheet

Idiomatic Programmers — Thu, 27 Jul 2023 11:44:02 +0000

What is Redis?

Redis, which stands for Remote Dictionary Server, is an open-source, in-memory data structure store that serves as a highly efficient and versatile database solution. Developed by Salvatore Sanfilippo, Redis is designed to offer lightning-fast data access and retrieval by keeping all the data in memory rather than on disk, ensuring remarkable performance and responsiveness.

One of the key features that sets Redis apart is its support for various data structures, including strings, lists, sets, sorted sets, and hashes. This versatility enables Redis to serve multiple purposes, such as acting as a key-value store, a message broker, or a caching layer.

Redis utilizes an in-memory approach, making it exceptionally fast for read and write operations. With its optimized data structures and efficient algorithms, Redis can process data in sub-millisecond response times, making it an ideal choice for real-time applications and use cases that require high-performance data manipulation.

Furthermore, Redis offers persistence options, allowing users to store their data on disk periodically or whenever specific conditions are met. This feature ensures data durability and makes Redis suitable for applications that require both speed and data persistence.

In addition to its exceptional performance, Redis provides advanced features like pub/sub messaging, transactions, and Lua scripting, which empower developers to build complex applications and systems with ease.

Redis has become immensely popular due to its simplicity, versatility, and extensive support for multiple programming languages. It offers client libraries for various languages, making it accessible and easy to integrate into different software stacks.

Whether you need to cache frequently accessed data, build real-time applications, implement a messaging system, or store and process complex data structures, Redis provides a robust and efficient solution that continues to be widely adopted across industries.

In conclusion, Redis is a powerful and flexible database solution that excels in performance, versatility, and ease of use. Its in-memory nature, support for various data structures, and extensive feature set make it a go-to choice for developers looking to build high-performance applications and systems.

How Redis is different from other Databases?

Redis differs from other databases in several key aspects:

In-Memory Data Storage: Redis primarily stores data in memory, allowing for extremely fast read and write operations. This design choice sets Redis apart from traditional disk-based databases that incur disk I/O overhead. By keeping data in memory, Redis can achieve high throughput and low latency, making it well-suited for applications that require real-time data processing and high-performance caching.
Data Structures: Redis supports a wide range of data structures, including strings, lists, sets, hashes, and sorted sets. This flexibility allows developers to model complex data scenarios efficiently. Unlike many other databases that offer a fixed schema, Redis provides dynamic and versatile data structures that can be manipulated and combined to suit specific application needs.
Built-in Caching Mechanisms: Redis has native support for caching, making it an excellent choice for scenarios that require frequent access to frequently accessed data. With features like key expiration and eviction policies, Redis can efficiently manage cache storage and ensure that the most relevant data remains readily available.
Pub/Sub Messaging: Redis includes built-in publish/subscribe messaging capabilities. This feature allows applications to implement real-time messaging and event-driven architectures. Publishers can send messages to specific channels, and subscribers receive messages from those channels in real-time. This makes Redis a suitable choice for building chat systems, real-time analytics, and other applications that require event-based communication.
Lua Scripting: Redis supports Lua scripting, which enables developers to execute server-side scripts directly within the database. This allows for complex operations and transactional logic to be performed within Redis, reducing the need for round-trips between the application and the database. Lua scripting enhances Redis's flexibility and extensibility by enabling custom behaviour and complex data manipulations.
Replication and Persistence: Redis provides built-in replication mechanisms for creating multiple copies of the database. This replication ensures high availability and fault tolerance by propagating changes from a master node to one or more slave nodes. Redis also offers persistence options through snapshotting and append-only file (AOF) logging, allowing data to be saved to disk for recovery in the event of a system failure.
Simplicity and Performance: Redis is known for its simplicity and performance. It offers a straightforward set of commands that are easy to understand and use. The focus on in-memory storage and the absence of complex query languages or indexing mechanisms simplify the database's design and contribute to its exceptional performance.

Overall, Redis stands out among other databases due to its in-memory storage, versatile data structures, caching capabilities, pub/sub messaging, Lua scripting, replication, and performance-oriented design. These features make Redis a popular choice for applications that require speed, flexibility, and real-time data processing.

Installing Redis and Redis CLI

Here are the steps to install Redis on Linux, macOS, and Windows:

Linux:

Open a terminal window.
Update the package manager:
```
sudo apt update
```
Install Redis:
```
sudo apt install redis-server
```
Start the Redis service:
```
sudo systemctl start redis-server
```
To access the Redis command-line interface (redis-cli):
```
redis-cli
```

macOS:

Open a terminal window.

Install Homebrew (if not already installed):

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Install Redis using Homebrew:
```
brew install redis
```
Start the Redis service:
```
brew services start redis
```
To access the Redis command-line interface (redis-cli):
```
redis-cli
```

Windows:

Download the latest Redis for Windows from the Microsoft Open Tech GitHub repository: https://github.com/microsoftarchive/redis/releases
Extract the downloaded archive to a directory of your choice, e.g., C:\Redis.
Open a Command Prompt window.
Navigate to the Redis installation directory:
```
cd C:\Redis
```
Start the Redis server:
```
redis-server.exe
```
Open another Command Prompt window to access the Redis command-line interface (redis-cli):
```
cd C:\Redis
redis-cli.exe
```

Please note that for Linux and macOS, the installation steps assume the usage of package managers like apt and Homebrew, respectively. If you are using a different package manager or distribution, you may need to adjust the installation commands accordingly.

Commands

SET manipulation

SET
- Command: SET
- Syntax: SET key value
- Usage: SET myset "Hello Redis"
- Description: Sets the value of a key with the provided string value. If the key already exists, it overwrites the existing value. This command is also used to create a new key-value pair if the key doesn't exist.
GET
- Command: GET
- Syntax: GET key
- Usage: GET myset
- Description: Retrieves the value associated with the specified key. If the key exists, the command returns the value; otherwise, it returns nil.
SADD
- Command: SADD
- Syntax: SADD key member [member ...]
- Usage: SADD myset "value1" "value2"
- Description: Adds one or more members to a set. If the members already exist, they are ignored. If the key doesn't exist, a new set is created with the specified members.
SMEMBERS
- Command: SMEMBERS
- Syntax: SMEMBERS key
- Usage: SMEMBERS myset
- Description: Returns all the members of a set. If the key doesn't exist, an empty set is returned.
SISMEMBER
- Command: SISMEMBER
- Syntax: SISMEMBER key member
- Usage: SISMEMBER myset "value1"
- Description: Checks if a member exists in a set. Returns 1 if the member is present; otherwise, returns 0.
SREM
- Command: SREM
- Syntax: SREM key member [member ...]
- Usage: SREM myset "value1" "value2"
- Description: Removes one or more members from a set. If the members don't exist, they are ignored. If the key becomes empty after removing members, the key is also removed.
SCARD
- Command: SCARD
- Syntax: SCARD key
- Usage: SCARD myset
- Description: Returns the cardinality (number of members) of a set. If the key doesn't exist, 0 is returned.
SINTER
- Command: SINTER
- Syntax: SINTER key [key ...]
- Usage: SINTER myset1 myset2
- Description: Returns the intersection of multiple sets. Only the members that exist in all the specified sets are included in the result set.
SUNION
- Command: SUNION
- Syntax: SUNION key [key ...]
- Usage: SUNION myset1 myset2
- Description: Returns the union of multiple sets. All the members from the specified sets are included in the result set.
SDIFF
- Command: SDIFF
- Syntax: SDIFF key [key ...]
- Usage: SDIFF myset1 myset2
- Description: Returns the difference between two sets. The members that exist in the first set but not in any other specified sets are included in the result set.

Sorted Sets manipulation

ZADD
- Command: ZADD
- Syntax: ZADD key [NX|XX] [CH] [INCR] score member [score member ...]
- Usage: ZADD mysortedset 1 "value1" 2 "value2"
- Description: Adds one or more members with scores to a sorted set. If a member already exists, its score is updated. Additional options like NX (only add new elements), XX (only update existing elements), CH (return the number of changed elements), and INCR (increment the score of existing member) can be specified.
ZSCORE
- Command: ZSCORE
- Syntax: ZSCORE key member
- Usage: ZSCORE mysortedset "value1"
- Description: Returns the score of a member in a sorted set. If the member doesn't exist, nil is returned.
ZRANGE
- Command: ZRANGE
- Syntax: ZRANGE key start stop [WITHSCORES]
- Usage: ZRANGE mysortedset 0 -1
- Description: Returns a range of members from a sorted set based on their position. The start and stop arguments specify the inclusive range of positions. The optional WITHSCORES flag includes the scores in the result.
ZREVRANGE
- Command: ZREVRANGE
- Syntax: ZREVRANGE key start stop [WITHSCORES]
- Usage: ZREVRANGE mysortedset 0 -1 WITHSCORES
- Description: Returns a range of members from a sorted set in reverse order based on their position. The start and stop arguments specify the inclusive range of positions. The optional WITHSCORES flag includes the scores in the result.
ZREM
- Command: ZREM
- Syntax: ZREM key member [member ...]
- Usage: ZREM mysortedset "value1" "value2"
- Description: Removes one or more members from a sorted set. If a member doesn't exist, it is ignored. If the key becomes empty after removing members, the key is also removed.
ZCARD
- Command: ZCARD
- Syntax: ZCARD key
- Usage: ZCARD mysortedset
- Description: Returns the cardinality (number of members) of a sorted set. If the key doesn't exist, 0 is returned.
ZCOUNT
- Command: ZCOUNT
- Syntax: ZCOUNT key min max
- Usage: ZCOUNT mysortedset 1 10
- Description: Returns the count of members in a sorted set within the specified score range (min and max inclusive).
ZRANK
- Command: ZRANK
- Syntax: ZRANK key member
- Usage: ZRANK mysortedset "value1"
- Description: Returns the position (rank) of a member in a sorted set when sorted in ascending order. The lowest rank is 0. If the member doesn't exist, nil is returned.
ZREVRANK
- Command: ZREVRANK
- Syntax: ZREVRANK key member
- Usage: ZREVRANK mysortedset "value1"
- Description: Returns the position (rank) of a member in a sorted set when sorted in descending order. The highest rank is 0. If the member doesn't exist, nil is returned.
ZINCRBY
- Command: ZINCRBY
- Syntax: ZINCRBY key increment member
- Usage: ZINCRBY mysortedset 5 "value1"
- Description: Increments the score of a member in a sorted set by the specified increment. If the member doesn't exist, it is added with the initial score as the increment value.

String, Number and Bit Manipulation

APPEND
- Command: APPEND
- Syntax: APPEND key value
- Usage: APPEND mykey "world"
- Description: Appends the specified value to the string value already stored at the key. If the key doesn't exist, a new key is created with the specified value.
BITCOUNT
- Command: BITCOUNT
- Syntax: BITCOUNT key [start end]
- Usage: BITCOUNT mykey
- Description: Counts the number of set bits (1s) in a string value. Optionally, a range can be specified to count the bits within that range.
BITOP
- Command: BITOP
- Syntax: BITOP operation destkey key [key ...]
- Usage: BITOP AND destkey key1 key2
- Description: Performs a bitwise operation (AND, OR, XOR, NOT) between multiple keys and stores the result in the destination key.
BITPOS
- Command: BITPOS
- Syntax: BITPOS key bit [start] [end]
- Usage: BITPOS mykey 1 0 10
- Description: Finds the position of the first set bit (1) or clear bit (0) in a string value. Optionally, a range can be specified to search for the bit within that range.
DECR
- Command: DECR
- Syntax: DECR key
- Usage: DECR mykey
- Description: Decrements the value of the key by 1. If the key doesn't exist, it is initialized with the value of 0 before decrementing.
DECRBY
- Command: DECRBY
- Syntax: DECRBY key decrement
- Usage: DECRBY mykey 5
- Description: Decrements the value of the key by the specified decrement. If the key doesn't exist, it is initialized with the value of 0 before decrementing.
GET
- Command: GET
- Syntax: GET key
- Usage: GET mykey
- Description: Retrieves the value associated with the specified key. If the key exists, the command returns the value; otherwise, it returns nil.
GETBIT
- Command: GETBIT
- Syntax: GETBIT key offset
- Usage: GETBIT mykey 5
- Description: Returns the bit value at the specified offset in a string value. The offset is zero-based.
GETRANGE
- Command: GETRANGE
- Syntax: GETRANGE key start end
- Usage: GETRANGE mykey 0 3
- Description: Retrieves a substring of the string value stored at the key, starting from the specified start index to the end index (inclusive).
GETSET
- Command: GETSET
- Syntax: GETSET key value
- Usage: GETSET mykey "newvalue"
- Description: Sets the value of the key with the provided string value and returns the old value. If the key doesn't exist, it is initialized with the specified value.
INCR
- Command: INCR
- Syntax: INCR key
- Usage: INCR mykey
- Description: Increments the value of the key by 1. If the key doesn't exist, it is initialized with the value of 0 before incrementing.
INCRBY
- Command: INCRBY
- Syntax: INCRBY key increment
- Usage: INCRBY mykey 5
- Description: Increments the value of the key by the specified increment. If the key doesn't exist, it is initialized with the value of 0 before incrementing.
INCRBYFLOAT
- Command: INCRBYFLOAT
- Syntax: INCRBYFLOAT key increment
- Usage: INCRBYFLOAT mykey 2.5
- Description: Increments the value of the key by the specified float increment. If the key doesn't exist, it is initialized with the value of 0 before incrementing.
MGET
- Command: MGET
- Syntax: MGET key [key ...]
- Usage: MGET key1 key2 key3
- Description: Retrieves the values associated with multiple keys. It returns an array of values corresponding to the provided keys.
MSET
- Command: MSET
- Syntax: MSET key value [key value ...]
- Usage: MSET key1 "value1" key2 "value2"
- Description: Sets multiple keys to their respective values. It allows setting multiple key-value pairs in a single command.
MSETNX
- Command: MSETNX
- Syntax: MSETNX key value [key value ...]
- Usage: MSETNX key1 "value1" key2 "value2"
- Description: Sets multiple keys to their respective values, only if none of the keys already exist. It allows atomic setting of multiple keys.
PSETEX
- Command: PSETEX
- Syntax: PSETEX key milliseconds value
- Usage: PSETEX mykey 1000 "Hello"
- Description: Sets the value of the key with the provided string value and a specified expiration time in milliseconds.
SET
- Command: SET
- Syntax: SET key value [EX seconds|PX milliseconds] [NX|XX]
- Usage: SET mykey "Hello" EX 60 NX
- Description: Sets the value of a key with the provided string value. Additional options like expiration time, conditional set (NX - only if the key doesn't exist, XX - only if the key exists) can be specified.
SETBIT
- Command: SETBIT
- Syntax: SETBIT key offset value
- Usage: SETBIT mykey 5 1
- Description: Sets the bit at the specified offset in a string value to either 0 or 1.
SETEX
- Command: SETEX
- Syntax: SETEX key seconds value
- Usage: SETEX mykey 60 "Hello"
- Description: Sets the value of the key with the provided string value and a specified expiration time in seconds.
SETNX
- Command: SETNX
- Syntax: SETNX key value
- Usage: SETNX mykey "Hello"
- Description: Sets the value of a key with the provided string value, only if the key doesn't exist.
SETRANGE
- Command: SETRANGE
- Syntax: SETRANGE key offset value
- Usage: SETRANGE mykey 5 "Redis"
- Description: Overwrites a substring of the string value stored at the key, starting from the specified offset with the provided value.
STRLEN
- Command: STRLEN
- Syntax: STRLEN key
- Usage: STRLEN mykey
- Description: Returns length

Conclusion

In conclusion, we have discussed Redis, a popular in-memory data structure store, and explored its unique characteristics that set it apart from other databases. Redis offers exceptional performance and versatility, making it suitable for various use cases, such as caching, real-time analytics, messaging systems, and more.

We have also covered the installation process of Redis on Linux, macOS, and Windows, including how to access the Redis command-line interface (redis-cli) for interacting with the database.

Furthermore, we have delved into the major commands related to set manipulation, providing syntax, usage examples, and descriptions for each command. These commands enable users to perform various operations like adding, retrieving, modifying, and removing elements from sets in Redis.

Additionally, we have explored the major commands related to sorted set manipulation, including operations such as adding members with scores, retrieving ranges, finding positions, incrementing scores, and more. Sorted sets in Redis provide an ordered collection of unique elements, allowing efficient operations based on both element values and scores.

Redis commands like APPEND, GET, SET, INCR, and many others offer powerful functionality for string manipulation, bit operations, and key-value management.

Redis, with its versatility, performance, and rich set of commands, continues to be a preferred choice for developers and architects when it comes to handling data-intensive applications and solving various data storage and retrieval challenges.

Celery Alternative for Django - Huey

Idiomatic Programmers — Fri, 13 May 2022 13:56:49 +0000

Background

So today I will be talking about a Celery alternative named Huey, which comes with a much easier setup than Celery and is much smaller in size compared to Celery.

The reason why I decided to try out Huey is because I have faced some issues with Celery sometimes for doing some common tasks because the documentation isn't too great.

For those who don't know what Celery is or haven't used it before, Huey is an asynchronous task queue which allows you to perform scheduled tasks or long running tasks in the background.

Prerequisites

We will be installing the following packages:

redis
django
huey
requests (optional, needed for the demo)

Github Repo

The following blog comes accompanied with a github repo, that you can use to test out the demo project that we will be creating.

Click here to view repo.

Project Setup

Create Project Directory

Open the terminal and type the following to create a directory, you can skip this step and do it from File Explorer itself.

mkdir huey_demo

Virtual Environment

Let us create a virtualenv first to install our project dependencies:
```
python -m venv venv
```
Activate the virtualenv (Linux):
```
source venv/bin/activate
```

Installing dependencies

Type the following command in the terminal to install all the dependencies:

pip install Django==4.0.4 redis==4.2.2 huey==2.4.3

At the time of writing this article, these were the versions I tested out this setup with, keep an eye on the Github Repo for any updates as per the latest version in the future.

Create Project

Create the django project by typing the following command in terminal:
```
django-admin startproject django_huey_demo
```
Change directory into Django project directory:
```
cd django_huey_demo
```
Create the app under our project:
```
python manage.py startapp demo
```

Include the created app into project settings.py, make the following changes:

INSTALLED_APPS = [
# Existing Apps
"demo.apps.DemoConfig",  # <== Add this line
]

Set debug mode to False in settings.py:
```
DEBUG=False
```

Project Overview

Now that we are done setting up our project, it is a good time to take you over what we will be building today. We will fetch "Word of the Day" daily from Wordnik API. Then we will store the word, its definition, and an example of the word in a sentence to our database.

We will setup a periodic task using Huey which will fetch the Word of the Day and store it.

For storing the word we will be creating a Django Model of the same.

Getting Wordnik API Key

You can follow this guide to obtain the API key.

Coding our project

Add Huey to our Project

We need to add Huey to installed apps of our project, so make the following changes in settings.py file:

INSTALLED_APPS = [
    # Existing apps
    'huey.contrib.djhuey', # <== Add this line
]

Install Redis

We need to install Redis for Huey to store information about queued tasks in it, like we used to do with Celery as well. You can refer to the following link to install redis based on your specific operating system.

If you're comfortable with using Docker, you can use the following command:

docker run --name redis_huey -p 6379:6379 -d redis

By default, Huey will try connecting to Redis server running on localhost:6379. If it isn't present, it will raise an error.

Model Definition

Add the following code to your demo/models.py file:

from django.db import models

class Word(models.Model):
    word = models.CharField(max_length=200)
    part_of_speech = models.CharField(max_length=100)
    definition = models.TextField()
    example = models.TextField()

    def __str__(self):
        return self.word

Make migrations:
```
python manage.py makemigrations demo
```
Apply migrations:
```
python manage.py migrate demo
```

Task Definition

Create a file named tasks.py in demo app directory. The reason why we named our file tasks.py is to help Huey auto discover the tasks present in our registered apps, if we named our file anything other than that, we would have to manually register our task. If you would like to know more you can check out the Huey documentation here.

Before we write the task definition we need to install an additional dependency requests. Install it by typing the following in your terminal:

pip install requests==2.27.1

Now comes the code:

import requests
from django.conf import settings
from huey import crontab
from huey.contrib.djhuey import db_periodic_task

from demo.models import Word

@db_periodic_task(crontab(hour="18", minute="00"))
def fetch_daily_word():
    r = requests.get(
        f"https://api.wordnik.com/v4/words.json/wordOfTheDay?api_key={settings.WORDNIK_API_KEY}")
    data = r.json()
    Word.objects.get_or_create(
        word=data["word"],
        part_of_speech=data["definitions"][0]["partOfSpeech"],
        definition=data["definitions"][0]["text"],
        example=data["examples"][0]["text"]
    )

Add the following line in your project settings:

WORDNIK_API_KEY = "api-key-here"

This codeblock might be a lot to take in, so let's go over the things in it one by one:

Huey Decorator
```
from huey.contrib.djhuey import db_periodic_task
```
This is a decorator provided by Huey to register periodic task that involve working with database, this decorator automatically closes the database connection upon task completion, for more details, you can refer here.
Crontab Schedule
```
@db_periodic_task(crontab(hour="18", minute="00"))
```
We are passing the argument crontab(hour="18", minute="00") to our periodic task decorator, this tells Huey to run our task at 6pm everyday. You can make use of this website to create your crontab schedules, I use it everytime.
Wordnik API Key
```
from django.conf import settings

# Usage
## settings.WORDNIK_API_KEY
```
from django.conf import settings is the standard way to import any data from our project settings, it is useful in cases where we have multiple settings files setup for different environment so it will know which file to pick from without us having to worry about it. It finds out which settings file we are using from the DJANGO_SETTINGS_MODULE environment variable. But you don't have to worry about these details.

Then we are using the key in our wordnik API call.
Wordnik API Call
```
r = requests.get(
f"https://api.wordnik.com/v4/words.json/wordOfTheDay?api_key={settings.WORDNIK_API_KEY}")
```
Here we are making use of the requests module to make a GET request to the wordnik API while passing our API Key for authentication.
Storing word in database
```
data = r.json()
Word.objects.get_or_create(
    word=data["word"],
    part_of_speech=data["definitions"][0]["partOfSpeech"],
    definition=data["definitions"][0]["text"],
    example=data["examples"][0]["text"]
)
```
After parsing the API response we are storing the word definition in our database. We are making use of get_or_create method instead of create method here so that we don't create multiple copies of the same word in our database if that word is ever repeated by the Wordnik API.

Wordnik API Response
Here is what the Wordnik API response for Word of the Day endpoint looks like. Some of the irrelevant sections of the response has been truncated for brevity purposes.

{
"word": "stolon",
"definitions": [
    {
    "source": "ahd-5",
    "text": "A long thin stem that usually grows horizontally along the ground and produces roots and shoots at widely spaced nodes, as in a strawberry plant.",
    "note": null,
    "partOfSpeech": "noun"
    },
    // More definitions here...
],
"publishDate": "2022-05-08T03:00:00.000Z",
"examples": [
    {
    "title": "4.1 Nursery establishment",
    "text": "A stolon is a stem that grows along the ground, producing at its nodes new plants with roots and upright stems.",
    // Additional data here...
    },
    // More examples here...
],
    // Additional fields here...
}

Here is what the Wordnik API response for Word of the Day endpoint looks like. Some of the irrelevant sections of the response has been truncated for brevity purposes.

Running Huey Worker

You can start the Huey worker by typing the following command in your terminal:

python manage.py run_huey

You can pass multiple flags to the above command which will change what gets logged to the console, such as:

-v, --verbose - verbose logging (includes DEBUG level)
-q, --quiet - minimal logging
-S, --simple - simple logging format (“time message”)

To look at various other options for logging, checkout the docs here.

What else can you do with Huey?

Task Decorators

Huey comes with multiple task decorators depending on what operations you are performing within the task. I'll explain in brief what all of those do below. Here is the import statement for all the decorators:

from huey.contrib.djhuey import task, periodic_task, db_task, db_periodic_task

task: A regular task
periodic_task : When you want to run a task periodically based on a schedule
db_task : When you want to perform db operations within your task
db_periodic_task : When you want to perform db operations in a periodic task

Crontab Examples

Let me show you some more examples of how you can use crontab to schedule your tasks.

crontab(minute='*/3') would schedule the task to run every three minute
crontab(hour='*/3', minute='5') would create a task that will run at 5 minute past every third hour.
crontab(minute='00', hour='10', month='*/2', day_of_week='*/5') would create a task that would run on every 5th day of the week, of every 2nd month at 10:00 AM.

Scheduling Tasks

For example, you have the following task defined inside tasks.py:

from huey.contrib.djhuey import task

@task()
def count():
    for i in range(10):
        print(i)

Now you want to call this task, but want it to run after 5 seconds, you can do the following:

count.schedule(delay=5)

delay parameter takes values in seconds, so if you want it to execute after 5 minutes specify 300 seconds.

Retrying tasks that fail

Suppose you add the following logic to our existing task:

@db_periodic_task(crontab(hour="18", minute="00"), retries=2)
def fetch_daily_word():
    r = requests.get(
        f"https://api.wordnik.com/v4/words.json/wordOfTheDay?api_key={settings.WORDNIK_API_KEY}")
    if r.status_code != 200:
        raise Exception("Unable to fetch data from Wordnik API") ## Add this logic
    else:
        data = r.json()
        Word.objects.get_or_create(
            word=data["word"],
            part_of_speech=data["definitions"][0]["partOfSpeech"],
            definition=data["definitions"][0]["text"],
            example=data["examples"][0]["text"]
        )

So we added the logic to check for status code of the response, and if it is something other than 200, it will retry that task upto 2 times. But these retries would happen without any time gap between the two attempts. Now what if you want to delay out multiple attempts of this task, we can do that by passing the retry_delay argument, it accepts values in seconds.

@db_periodic_task(crontab(hour="18", minute="00"), retries=2, retry_delay=10)

This will cause a 10 second delay between multiple attempts.

Development Mode

Huey comes with a default setting which makes working with Huey during development in Django easier. So whenever you have DEBUG=True present in your settings.py file, tasks will be executed synchronously just like regular function calls. The purpose of this is to avoid running both Redis and an additional consumer process while developing or running tests. You can read more about this here.

For this, we need to add the following line in settings.py:

HUEY = {}

However, if you want to override this behaviour, you can add the following Huey config instead:

HUEY = {
    "immediate": False
}

If you have the above config mentioned in settings.py while having DEBUG=True Huey will require you to setup Redis and run huey worker using run_huey command.

Celery vs Huey

Some observations about Huey as compared to Celery are:

Smaller footprint of dependencies as compared to Celery. Celery installs kombu and billiards along with it. Meanwhile, huey doesn't have any dependencies.
Lesser services needed to be run for periodic tasks, Celery requires running beat service and a worker service to work with periodic task meanwhile we only need to run one service using the run_huey command.

References

How to save Base64 encoded image to Django ImageField?

Idiomatic Programmers — Sat, 17 Jul 2021 12:47:50 +0000

I had to create an API that had to accept an image in Base64 encoded format and save it to Django image field, which requires an image file. So today I learned how to do this.

Step 1: B64 to PIL

The first step will be to convert this B64 image to an image file in memory, I will be using Pillow for this. You can install Pillow using this bash command.

$ pip install pillow

Make sure you have pip installed. If you're using Linux or Mac OS, then you might have to use pip3 instead of pip. But I'm using virtual env, so who cares.

after that, write this function.

import base64
import io
from PIL import Image

def decodeDesignImage(data):
    try:
        data = base64.b64decode(data.encode('UTF-8'))
        buf = io.BytesIO(data)
        img = Image.open(buf)
        return img
    except:
        return None

This function will return a PIL image if the B64 data is valid, or it will return None.

Step 2: PIL to Django ImageField

After getting the PIL image, wrote a function that will take this PIL image, convert it into a JPEG file buffer and then inserted it to Database instance using InMemoryUploadedFile

sources: InMemoryUploadedFile

What is InMemoryUploadedFile? This is basically a representation of the image in memory, that's as simple as it is.

from django.core.files.uploadedfile import InMemoryUploadedFile

img = decodeDesignImage(data)
img_io = io.BytesIO()
img.save(img_io, format='JPEG')
design.image = InMemoryUploadedFile(img_io, field_name=None, name=token+".jpg", content_type='image/jpeg', size=img_io.tell, charset=None)
design.save()

The InMemoryUploadedFile class takes 6 parameters:

File buffer: This is the JPEG format IO buffer I created in line 5.
field_name: There wasn't much information about this, if you have any idea then please comment below.
name: This is the name of the file, make sure you add .jpg at the end.
content_type: This is pretty self explanatory.
size: The size in bytes of the file.
charset: The character set, the file data is using. In this case None.

Now if you check your Media folder, you'll see that image as if you've uploaded it via a form.

So that's it for today's "Today I Learned"

How to download Pandas Dataframe as Excel or CSV in Django?

Idiomatic Programmers — Sat, 27 Mar 2021 03:00:00 +0000

Introduction

Pandas is an essential tool used by Python developers used for data analysis purposes, but what is the point of the analysis if we are not able to provide that insight to the end user. When Django and Pandas are used in conjunction we can create applications that are not only developed faster, but are also smart in utilizing the data that is collected from the user. I recently stumbled on one such use case, where the user interacts with the frontend and turns some filters and then the backend processes that data, and then provide the aggregated data as an Excel or CSV file to the user. I will be describing how I went about doing that below.

Prerequisites

In order to follow along this article, I am assuming you already have a Django project that is using Pandas library ready.

Requirements

In addition to Django and Pandas project ready, you would need to install Openpyxl.

Openpyxl

Openpyxl is a tool that allows to read and write Open Office XML formats such as Excel 2010 xlsx/xlsm/xltx/xltm files.

pip install openpyxl

Once installed, you can create Excel files natively from Python using a simple code like this.

from openpyxl import Workbook

workbook = Workbook()
sheet = workbook.active

sheet["A1"] = "ID"
sheet["B1"] = "Key"

sheet["A2"] = 1
sheet["B2"] = "Test Key"

workbook.save(filename="test.xlsx")

This will create a file called "test.xlsx". Try it out yourself.

HTTP Response

As you know, Django uses HTTP Request and Response to communicate with the client. Therefore all data must be converted to Byte String before sending it over to the client.

A Typical HTTP Response looks something like this.

For more information, checkout this article by Mozilla about HTTP Messages.

As you can see in the above picture, every HTTP response has a body section which is a Byte Array and a Header called Content-Type specifies what type of content it is and which character set the browser should use in order to decode the bytes. In the above case, the server is sending a text/html data and the browser need to use iso-8859-1 charset.

In our case, we are sending bytes that are related to Excel spreadsheets. So googling for a bit I found the content-type header for that is this. Source: Mozilla article about Common MIME Types

application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

What are MIME Types?

MIME is short for Multipurpose Internet Mail Extensions which is a standard that indicates the nature and format of a document sent via the Internet.

Important : Browsers use the MIME type, not the file extension, to determine how to process a URL, so it's important that web servers send the correct MIME type in the response's Content-Type header. If this is not correctly configured, browsers are likely to misinterpret the contents of files and sites will not work correctly, and downloaded files may be mishandled.

For more information, checkout MDN Article about MIME Types

The structure of a MIME Type is according to IANA (Internet Assigned Numbers Authority) is like this.

type/subtype

Therefore, in the MIME Type for Excel files, the type is " application", which belongs to any kind of binary data that cannot be directly decoded to a human-readable form (text, HTML, etc.). Files with type application require some sort of external or third party software to decode and read such as PDFs, Zip files, Excel files, etc.

Download Excel Files using Django and Pandas

Finally, we will see how we can send the Pandas dataframe to client as an excel file.

I am assuming that you already have the code for pandas so I will do some abstraction in a variable called data.

First we will import to the inbuilt library, BytesIO so that we can write the excel file as a Byte array.

BytesIO is a library using which we can write data onto system memory instead of writing them as a file.

from io import BytesIO

Then we will use Python context manager to open a Byte buffer on which we can write the excel file. Context Managers allow you to allocate and release memory precisely when you want it.

with BytesIO() as b
    with pd.ExcelWriter(b) as writer:
        # You can add multiple Dataframes to an excel file
        # Using the sheet_name attribute
      data1.to_excel(writer, sheet_name="DATA 1", index=False)
        data2.to_excel(writer, sheet_name="DATA 2", index=False)

    filename = "analytics_data.xlsx"

    # imported from django.http
    res = HttpResponse(
        b.getvalue(), # Gives the Byte string of the Byte Buffer object
        content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
    )
    res['Content-Disposition'] = f'attachment; filename={filename}'
    return res

We will be using context managers to open and close files because that's more efficient than manually closing and releasing the memory after use. If you already know how to export a set of data frames as an excel file, that code should look like this.

with pd.ExcelWriter('data.xlsx') as writer:
    data1.to_excel(writer, sheet_name="DATA 1", index=False)
    data2.to_excel(writer, sheet_name="DATA 2", index=False)

We will simply take this code and wrap it around a BytesIO context manager which gives us a memory buffer b as a file to work with. We will simply take that memory buffer and pass it to pd.ExcelWriter() class and the rest of the code will be the same this that context manager.

with BytesIO() as b
    with pd.ExcelWriter(b) as writer:
      data1.to_excel(writer, sheet_name="DATA 1", index=False)
        data2.to_excel(writer, sheet_name="DATA 2", index=False)

Now that we have our Excel data written in a memory buffer, we can simply convert that to ByteArray using the method b.getvalue() which we will pass as a parameter in Django HttpResponse with appropriate content_type that we discussed earlier.

filename = "analytics_data.xlsx"

# imported from django.http
res = HttpResponse(
  b.getvalue(), # Gives the Byte string of the Byte Buffer object
  content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
)

In addition to the Content-Type header, we also need to provide another header called Content-Disposition that tells the browser if we want to show the data in the browser download and save it as a local file as an attachment. We can also pass a filename with this header.

res['Content-Disposition'] = f'attachment; filename={filename}'
return res

Here is the complete code snippet for Django view to download the excel file.

def download_analytics_endpoint(request):
    with BytesIO() as b:
        data = get_analytics_data()

        with pd.ExcelWriter(b) as writer:
            data.to_excel(writer, sheet_name="Data", index=False)

        filename = f"analytics_data.xlsx"
        res = HttpResponse(
            b.getvalue(),
            content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
        )
        res['Content-Disposition'] = f'attachment; filename={filename}'
        return res

That is all, now you just have to create a Django URL for this function and you can download any file you want, just convert that file to a Byte string and use the correct Content-Type header.

Generative Adversarial Networks

Idiomatic Programmers — Thu, 11 Feb 2021 08:56:15 +0000

In 2014, Ian Goodfellow, who is currently working as Director of Machine Learning for Apple, published a paper called "Generative Adversarial Networks" or GAN for short. Which basically talked about a system of two neural networks, Generator and Discriminator, that can generate images or any data that is similar to provided datasets from essentially random noise.

The diagram below shows the basic architecture of a GAN.

Source: https://developers.google.com/

Generative Network takes some random noise and outputs some random noise. This output noise is passed to a discriminator along with the real image or data as a ground truth based on that both Discriminator and Generator is trained.

As you can see, the concept of GAN is very simple. When I started learning about GANs I thought I can easily implement one of those. But boy I was wrong. In reality training a GAN is extremely hard both Generator and Discriminator must be trained side by side if one overpowers the other it won't work, we will talk about all the problems you might face while training a GAN. But now let's look at a simple GAN in Pytorch. We will be using the MNIST Dataset for this post.

Understanding the Dataset

The MNIST dataset is a huge database of handwritten numbers from 0 to 9 used for Optical Character Recognition or reading numbers from an image.

This dataset consists of 28x28 images of handwritten numbers where each pixel contains either a zero or a one.

The computer vision extension of PyTorch, Torchvision, provides this dataset which we can download using the following code snippet.

mnist = datasets.MNIST(root='datasets', train=True, transform=transformations, download=True)

But before we can run this code, we need to import some libraries.

import torchvision.datasets as datasets
from torch.utils.data import DataLoader
import torchvision.transforms as transforms

You can install the torchvision library using this pip command.

pip install torchvision

The DataLoader class is used to load the data to memory in batches, this prevents your system from running out of memory while training.

Transforms class is used to make random augmentations to the image such as random rotation, resize, crop, etc. but for now we will only normalise the images to range from -1 to 1.

The entire function that you can copy paste is this.

def load_dataset():
    transformations = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, ), (0.5, ))
    ])
    mnist = datasets.MNIST(root='datasets', train=True, transform=transformations, download=True)
    return DataLoader(mnist, batch_size=32, shuffle=True)

Implementing the Generator

In this section, we will implement a generator network which will take a random noise vector of size 100 and convert it into a vector of size 784 which we will then convert to 28x28

class Generator(nn.Module):
    def __init__ (self):
        super(). __init__ ()
        self.gen = nn.Sequential(
            nn.Linear(100, 256),
            nn.LeakyReLU(0.01),
            nn.Linear(256, 784),
            nn.Tanh(), # make outputs [-1, 1]
        )

    def forward(self, x):
        return self.gen(x)

We will be implementing the original GAN created by Ian Goodfellow in this paper Generative Adversarial Networks.

Implemeting the Discriminator

Next, we will implement a discriminator that will take the vector of size 784 that may be generated or from a real image.

class Discriminator(nn.Module):
    def __init__ (self):
        super(). __init__ ()
        self.disc = nn.Sequential(
            nn.Linear(784, 128),
            nn.LeakyReLU(0.01),
            nn.Linear(128, 1),
            nn.Sigmoid(), # make outputs [0, 1]
        )

    def forward(self, x):
        return self.disc(x)

Hyperparameters

As I mentioned earlier, GANs are extremely hard to train, one of the reasons is that a GAN is very sensitive to the initial values or hyperparameters. You have to follow the original papers to get the training right. Otherwise, you might have to spend many days optimizing the hyperparameters.

config = {
    'device': "cuda" if torch.cuda.is_available() else "cpu",
    'lr': 3e-4,
    'epochs': 50,
        'batch_size': 32
}

In our case, we will be using the same parameters as it was said in the paper, that is, a learning rate of 0.0003 and 50 epochs.

What is a learning rate?

Simple answer, it's the rate at which a machine learning model learns. Smaller the number, the slower it learns and higher the number the faster it learns. For more info, check out our post about Perceptrons.

Training Time

Finally, it's time to actually generate some handwritten numbers, that is, train our GAN. First let's create the objects for Generator, Discriminator, and their optimizers, we will be using the Adam optimizers.

ADAM or ADA ptive M oment optimiser is an algorithm used to update the weights such that the overall error goes down.

disc = Discriminator().to(config['device'])
gen = Generator().to(config['device'])

optimiser_g = optim.Adam(params=gen.parameters(), lr=config['lr'])
optimiser_d = optim.Adam(params=disc.parameters(), lr=config['lr'])

We also need to define a loss function before we train, we will be using the Binary Cross Entropy loss function to calculate the error of our model.

Binary Cross Entropy (BCE) Loss

This loss formula is used to calculate the distance between two probability distributions.

BCE=ℓ(x,y)=L=(l1,…,lN),ln=−wn[yn⋅logxn+(1−yn)⋅log(1−xn)]

BCE = ℓ(x,y)=L=(l_1,…,l_N), l_n=−w_n[y_n⋅logx_n+(1−y_n)⋅log(1−x_n)]

BCE=ℓ(x,y)=L=(l1,…,lN),ln=−wn[yn⋅logxn+(1−yn)⋅log(1−xn)]

loss_fn = nn.BCELoss()

Now we will write our training step which is one cycle of generating and discriminating a handwritten number compare with an original number and update the models.

for epoch in range(config['epoch']):
    for batch_idx, (real, label) in enumerate(train_data):

        noise = torch.randn(config['batch_size'], 100).to(config['device']) # Create a random probability distribution
        fake = gen(noise) # Generate a fake number
        disc_real = disc(real).view(-1) # pass the real number through the discriminator
        lossD_real = loss_fn(disc_real, torch.ones_like(disc_real)) # calculate the loss for real image
        disc_fake = disc(fake).view(-1) # pass the fake number through the discriminator
        lossD_fake = loss_fn(disc_fake, torch.zeros_like(disc_fake)) # calculate the loss for fake image
        lossD = (lossD_real + lossD_fake) / 2 # calculate the average loss

        # update the weights for the discriminator
        disc.zero_grad()
        lossD.backward(retain_graph=True)
        optimiser_d.step()

        output = disc(fake).view(-1)
        lossG = loss_fn(output, torch.ones_like(output)) # calculate the error between the fake image and the true image

        # update the weights for the generator
        gen.zero_grad()
        lossG.backward()
        optimiser_g.step()

This code is heavily inspired by a youtube video by Aladdin Persson.

Find the complete code at our Github Repo

If you have any questions, feel free to comment below.

References

Goodfellow, Ian J., et al. “Generative Adversarial Networks.” ArXiv.org, 10 June 2014, arxiv.org/abs/1406.2661v1.
Kingma, Diederik P., and Jimmy Ba. “Adam: A Method for Stochastic Optimization.” ArXiv.org, 30 Jan. 2017, arxiv.org/abs/1412.6980.
“THE MNIST DATABASE.” MNIST Handwritten Digit Database, Yann LeCun, Corinna Cortes and Chris Burges, yann.lecun.com/exdb/mnist/.
Understanding Categorical Cross-Entropy Loss, Binary Cross-Entropy Loss, Softmax Loss, Logistic Loss, Focal Loss and All Those Confusing Names, gombru.github.io/2018/05/23/cross_entropy_loss/.

How to implement Auto Expiring Token in Django Rest Framework

Idiomatic Programmers — Sun, 17 Jan 2021 03:00:00 +0000

Introduction

Today I will be talking about one of the crucial feature you might have found missing in the Django Rest Framework, and that is the ability to create authentication tokens that expire after a set period of time, let's say after 20-25 days, every user that had authenticated previously gets prompted to re-authenticate again, which means the client will have to use the REST API to obtain the auth token once again.

One way you might have tried to solve this problem is by reaching out for some JWT token package, but I thought there must be a way to do this without having to switch to an entirely different type of authentication token system. That's when I found the solution that I am going to share with you all today.

Background

First of all, when I utilize token authentication I usually make use of a package called dj-rest-auth which provides me with endpoints for password management (forgot password, reset and change) and authentication (login, logout and register). So part of my solution will include changes to the configuration of that package.

Let's start by describing our solution, by default the tokens generated by Django Rest Framework are valid for lifetime and a user can have only one authentication token at a time, now upon logging out you can either choose to just clear the cookies and still maintain that old authentication token in database, which is a security flaw in itself, or implement a logout endpoint that will delete that token. If you were to choose the later path, you will soon face another issue, which is, if the user decided to log into their account on multiple devices, then they will be using the same authentication token, and if you delete that token, their sessions on both the devices will get logged out, which obviously is something undesirable.

To be able to resolve the above mentioned problem, we will be modifying the relation between the Token and User to be a ForeignKey instead, that way, a user can have multiple tokens (equivalent to having multiple sessions on different devices).

So let's begin.

Custom Token Model

The following will be the definition of our Token model:

from django.conf import settings
from django.db import models
from rest_framework.authtoken.models import Token as AuthToken

class Token(AuthToken):
    key = models.CharField("Key", max_length=40, db_index=True, unique=True)
    user = models.ForeignKey(
        settings.AUTH_USER_MODEL,
        related_name="auth_token",
        on_delete=models.CASCADE,
        verbose_name="User",
    )

For your reference, I defined the above model in an app called accounts, which is where I usually define my custom user model as well.

Now since I use dj-rest-auth, inside by project settings, I will define the following:

REST_AUTH_TOKEN_MODEL = "accounts.models.Token"

This setting was described in the package's configuration docs here..

Next up we will be creating a utility function used by dj-rest-auth for creating tokens:

import datetime

import pytz
from django.utils import timezone

def custom_create_token(token_model, user, serializer):
    token = token_model.objects.create(user=user)
    utc_now = timezone.now()
    utc_now = utc_now.replace(tzinfo=pytz.utc)
    token.created = utc_now
    token.save()
    return token

I saved the above function inside accounts/utils.py.

After this, define the following inside your project settings:

REST_AUTH_TOKEN_CREATOR = "accounts.utils.custom_create_token"

The above setting is also a configuration required by the dj-rest-auth package.

Now let's pause and explain what we have achieved so far. We have implemented the functionality to be able to create multiple tokens for a single user, meaning they can log onto multiple devices with each session being unique.

Migrations

If you were using the Token model provided by Django Rest Framework earlier, then you will need to make sure you remove the installed app for the token provided by it, to do that, go inside your project settings, and remove the following line from the INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    'rest_framework.authtoken' # <== This line should be removed
]

When you're done with that, let's create the migrations for the custom Token model we defined earlier, to do that run the following command in the terminal:

python manage.py makemigrations accounts

We used accounts here because our model resides in accounts app, change it accordingly as per your implementation. This will create the migrations, now migrate using:

python manage.py migrate accounts

NOTE : If you had any tokens previously created using default Token model, then those tokens will become invalid later as we will be switching out the Authentication Backend as well.

Custom Authentication Backend

Now we are going to be defining a custom Authentication Backend which will be derived from the default TokenAuthentication available in Django Rest Framework, and have some extended functionality.

Following is the code for our authentication backend:

import datetime

import pytz
from django.conf import settings
from django.utils import timezone
from rest_framework.authentication import TokenAuthentication
from rest_framework.exceptions import AuthenticationFailed

from accounts.models import Token

class ExpiringTokenAuthentication(TokenAuthentication):
    """
    Expiring token for mobile and desktop clients.
    It expires every {n} hrs requiring client to supply valid username 
    and password for new one to be created.
    """

    model = Token

    def authenticate_credentials(self, key, request=None):
        models = self.get_model()

        try:
            token = models.objects.select_related("user").get(key=key)
        except models.DoesNotExist:
            raise AuthenticationFailed(
                {"error": "Invalid or Inactive Token", "is_authenticated": False}
            )

        if not token.user.is_active:
            raise AuthenticationFailed(
                {"error": "Invalid user", "is_authenticated": False}
            )

        utc_now = timezone.now()
        utc_now = utc_now.replace(tzinfo=pytz.utc)

        if token.created < utc_now - settings.TOKEN_TTL:
            raise AuthenticationFailed(
                {"error": "Token has expired", "is_authenticated": False}
            )
        return token.user, token

I wrote the above code inside accounts/authentication.py.

To be able to use the above mentioned authentication backend, you will need to make changes to your REST_FRAMEWORK config in your project settings as follows:

REST_FRAMEWORK = {
    "DEFAULT_AUTHENTICATION_CLASSES": (
        "rest_framework.authentication.TokenAuthentication", # Replace this line
        "accounts.authentication.ExpiringTokenAuthentication", # <-- with this line 
        "rest_framework.authentication.SessionAuthentication",
    ),
 # ...
}

If you noticed carefully, in the code for our custom authentication backend, I mentioned settings.TOKEN_TTL, which is another functionality that I have added to allow configuring the Token's Lifespan or Token - Time to Live through project settings instead of setting it to a fixed value.

Now under your project settings, define the following:

import datetime

TOKEN_TTL = datetime.timedelta(days=15)

I have the set the token lifetime to be 15 days, but you are free to configure it as per your choice.

That's it for this article, please let me know if you found this useful in the comments below.

References

Django in Production - III

Idiomatic Programmers — Sun, 08 Nov 2020 22:25:35 +0000

Introduction

This is the third and the last post in the series, where we configure our django application to run in production. In the last post, we discussed how to configure and run NGINX as reverse proxy in front of gunicorn workers, and to also serve static files.

Today, I want to cover an important thing that you'll definitely need while setting up your servers, but wouldn't realize that you actually need it.

The technology that I am talking about is process monitors. Let me tell you examples of use cases where you might need them:

You are running a web app, and due to some maintenance downtime on your VPS provider, your server gets restarted. Your web app stops running because of it, you would have preferred if your web app started automatically with your server.
In a complex web app architecture, you're not going to be running just a single web server, you might be running additional services for things such as Asynchronous Task Queue such as Celery, and a Task Scheduler like Celery Beat and a database server. Now to remember to restart them each separately whenever you pull any new changes from the production branch to your server gets too tedious.
What if the server gets terminated accidentally because it encounters an error while running in the middle of the night, and you're asked to fix it immediately, sounds scary, right?

So, there are multiple available utilities out there for doing this task such as systemd. But from a beginner's perspective, systemd would seem intimidating. That's why I decided to search for a tool that does the job for us without making us feel intimidated, so the one I finally decided to go with is Supervisor.

Let's begin setting things up.

Supervisord Installation

If you're on Ubuntu you can install supervisor on your system by running:

sudo apt-get install supervisor

To see if you installed it successfully type:

sudo supervisorctl version

that should return the version of supervisor.

Now to work with supervisor we'll have to write configuration files, these configuration files exist inside /etc/supervisor/conf.d/ on Linux, these files have the extension .conf.

Let's create a configuration file for running gunicorn using supervisor, to do so type the below commands in your terminal:

cd /etc/supervisor/conf.d/
sudo touch gunicorn.conf
sudo nano gunicorn.conf

On the last line I opened the file in nano i.e. a terminal based text editor, you can use vim or any other text editor of your choice.

Then paste the following configuration inside it:

[program:gunicorn]
directory=/home/ubuntu/my-django-project/
command=/home/ubuntu/my-django-project/venv/bin/gunicorn --access-logfile - --log-level DEBUG --workers 4 --bind 127.0.0.1:8000 my-django-project.wsgi:application
autostart=true
autorestart=true
stderr_logfile=/home/ubuntu/my-django-project/logs/gunicorn/gunicorn.err.log
stdout_logfile=/home/ubuntu/my-django-project/logs/gunicorn/gunicorn.out.log

Now, be sure to replace /home/ubuntu/my-django-project with path to your django project and /home/ubuntu/my-django-project/venv/ with the path to your virtualenv. Now, if you notice there are two config at the end, stderr_logfile and stdout_logfile respectively, what they do is that define the path of the file where the error and output logs of the service should be saved, so be sure replace it with path where you want to save your logs.

Tip : If you mention any non existing directories in your path, be sure to create them beforehand or else supervisor will fail to start our service.

Now let's restart supervisor for it to start running our service, for that type the following in your terminal:

sudo supervisorctl reload

Let's check whether the above started our gunicorn service or not, by typing:

sudo supervisorctl status

If you follow the above steps correctly, you can safely integrate it with our previous NGINX configuration we discussed in our previous article.

References

Supervisor Config

5 techniques to Idiomatic Python (Loops)

Idiomatic Programmers — Fri, 21 Aug 2020 06:44:53 +0000

Python is, in my opinion, one of the easiest and most versatile languages in computer science. If you write python code correctly, it is difficult to differentiate a python code from pseudo-code. But sometimes, in the process of writing the most beautiful python code, most developers forget one thing: the execution speed of the code.

You can write code that is so readable and a layman can confuse it as English written by someone with bad grammar skills, yet it is understandable. But that code takes more than 300ms to run. That might not be that much of a delay but in the programming world, it is a serious problem.

On the other hand, one can write the same code in a different idiom that it takes less than 10ms to run. But, even expert python developers undergo a hard time understanding it.

It is thus very imported, for a developer to strike a balance between these two extremes. These kinds of code are known as Idiomatic Code in the industry.

Idiomatic Code, by definition, is a code that does a common task in a common way for your language. In other words, idiomatic code is any code that is easy to read and at the same time, is extremely efficient.

I follow Raymond Hettinger (@raymondh) a lot on twitter, he is one of the core developers in the Python community and he has contributed a lot of code in the Python repository to make the python language more idiomatic.

In this blog, I will go through some of the techniques that will help you improve your python code. Starting with:

1. range()

Let's start from the obvious, say you need to iterate from 2 to 10. So if you're like an absolute beginner you might write something like this:

for i in [2,3,4,5,6,7,8,9,10]:
    print(i)

Something like this works and is very understandable, it is not a scalable approach. What if you want to loop through 2 to 2 million.

In situations like this, you can use the range(stop) or range(start, stop, [step,]) built-in function.

the range function automatically generates the list for you. For example, you can rewrite the above code as:

for i in range(2, 11):
    print(i)

Notice how I wrote start from 2 and end at 11 and not 10. This is because the range function loops till stop - 1. There is one more parameter in range function other than the start and stop, that is the step. Step determines how many numbers the range have to skip.

Let's say you need to print all even numbers from 2 to 10, in that case, the skip parameter will be 2. By default, it is 1.

for i in range(2,11,2):
    print(i)

# OUTPUT: 2 4 6 8 10

Now let's say that you need to loop through a list. With the knowledge of range(), you might do this:

cloths = ['shirt', 'hat', 'socks', 'shorts']

for i in range(len(cloths))
  print(cloths[i])

But in python, there is a better way:

cloths = ['shirt', 'hat', 'socks', 'shorts']

for cloth in cloths
  print(cloth)

2. zip()

Let's say you have two lists, colour and cloths, of different sizes and you want to pair them until the smaller list is over. In this case, you can write a code like this:

colours = ['pink', 'red', 'green']
cloths = ['shirt', 'hat', 'socks', 'shorts']

n = min(len(colours), len(cloths))
for i in range(n):
    print(colours[i], cloths[i])

# OUTPUT
# pink shirt
# red hat
# green socks

Which is good, but you can also use the in-built function zip(*iterables) the best thing about this function is that you can pass any number of lists, not just two. Let me rewrite the above code and then explain to you how zip works.

colours = ['pink', 'red', 'green']
cloths = ['shirt', 'hat', 'socks', 'shorts']

for colour, cloth in zip(colours, cloths):
    print(colour, cloth)

# OUTPUT
# pink shirt
# red hat
# green socks

You might have guessed, zip takes any number of lists and returns another list that contains one element from each list. As you can see, both version produces the same output but the second one looks cleaner.

3. reversed()

If you want to loop over a list in reverse order, the traditional method was something like this.

cloths = ['shirt', 'hat', 'socks', 'shorts']

for i in range(len(cloths)-1, -1, -1):
  print(cloths[i])


# Output
# -------
# shorts
# socks
# hat
# shirt

But with Python, you can use the builtin function called reversed(). Take a look at this example:

cloths = ['shirt', 'hat', 'socks', 'shorts']

for cloth in reversed(cloths):
  print(cloth)


# Output
# -------
# shorts
# socks
# hat
# shirt

The second one is cleaner and faster than the first one.

4. enumerate()

You want to loop through a list along with the index. It's pretty straight forward in traditional programming:

cloths = ['shirt', 'hat', 'socks', 'shorts']

for i in range(len(cloths)):
  print(i, cloths[i])


# Output
# -------
# 0 shorts
# 1 socks
# 2 hat
# 3 shirt

But there is a cleaner and more efficient way in python:

cloths = ['shirt', 'hat', 'socks', 'shorts']

for i, cloth in enumerate(cloths):
  print(i, cloth)


# OUTPUT
# -------
# 0 shorts
# 1 socks
# 2 hat
# 3 shirt

5. sorted()

You need to loop over a list in sorted order, instead of running a sorting algorithm, you can use the sorted() built-in method.

nums = [2,3,1,5,2]

for i in sorted(nums):
  print(i)

# OUTPUT
# ------
# 1
# 2
# 2
# 3
# 5

The python sorted function uses Tim Sort with an average complexity of n*log(n)

For reverse sorting, you can use the attribute

sorted(nums, reverse=True)

What you want to sort a list of strings.

cloths = ['shirt', 'hat', 'socks', 'shorts']

for cloth in sorted(cloths):
  print(cloth)

# OUTPUT
# ------
# hat
# shirt
# shorts
# socks

It will sort based on letters, but what if you want to sort based on length of the string, you can do that with key attribute. For example:

cloths = ['shirt', 'hat', 'socks', 'shorts']

for cloth in sorted(cloths, key=len):
  print(cloth)

# OUTPUT
# ------
# hat
# shirt
# socks
# shorts

So that was 5 techniques using with you can make your python code more idiomatic.

Reverse Engineering Netflix Falcor API

Idiomatic Programmers — Tue, 04 Aug 2020 21:23:48 +0000

Today, In this article I'll try to explain the Netflix API and how to get data about movies and tv shows :). I've been scraping a lot of popular streaming providers and Netflix was by the hardest one to get.

Falcor

Falcor is Netflix's JavaScript library, used for fetching data. If you want to read more about it, you can do it on their Github.

Inspecting the network

First, I'll look through the network and check what's happening, now if you've used Netflix you know they have auto playing videos everywhere, which causes a lot of requests as you scroll or click or whatever you're doing. I tried filtering down the endpoints/urls by typing "netflix".

At first you can see some of the auth links, but I'll ignore this for now. We need to find out how Netflix actually loads the videos and the data about them, so let's open this show for example and inspect the traffic.

We can see there is a POST request to this url, Checking the request data we can see what Netflix is sending. By inspecting the request headers we can see this is application/x-www-form-urlencoded content-type, so in this case, our key will videos, which probably means, that's just Netflix differentiates this data. Looking at the url (https://www.netflix.com/browse?jbv=80097140) we can see that the ID is being passed next, and finally we have a list of what we want to get in response. Cool, looks like we have some of the info we need, but we need to a find a way to get these video IDs, we're not going to manually go over and copy them..

Let's search through the genres and see what traffic we're getting, Almost same as the first one, I clicked to check Anime genre and found this POST data in request,

Well, well, same strategy as before, let's copy the request and check it out in Postman.

Analyzing the request headers

The headers are pretty much standard stuff, with some Netflix exceptions, however the most important is the Cookie. The cookie consists of 2 important token values, SecureNetflixId and NetflixID, these are passed in every request I've inspected and it looks like they're bound to the account, so you have to be subscribed to Netflix to hit those endpoints.

memclid={id}SecureNetflixId={SecureNetflixID}; NetflixId={NetflixId}; playerPerfMetrics=%7B%22uiValue%22%3A%7B%22throughput%22%3A6231.89%2C%22throughputNiqr%22%3A0.5994435164694499%7D%2C%22mostRecentValue%22%3A%7B%22throughput%22%3A6233.69%2C%22throughputNiqr%22%3A0.39654013276722966%7D%7D; OptanonConsent=isIABGlobal=false&datestamp=Wed+Jul+22+2020+11%3A06%3A12+GMT%2B0200+(Central+European+Summer+Time)&version=6.3.0&consentId=f0aa447b-7bc0-468f-a12e-e8f7f8acde23&interactionCount=1&landingPath=NotLandingPage&groups=C0001%3A1%2CC0002%3A1%2CC0004%3A1&hosts=H1%3A1%2CH2%3A1%2Cjyl%3A1%2CH7%3A1%2CH4%3A1%2CH6%3A1&AwaitingReconsent=false; _ga=GA1.2.616486459.1595596356; pas=%7B%22supplementals%22%3A%7B%22muted%22%3Atrue%7D%7D; profilesNewSession=0; clSharedContext=f89661b7-8d71-48cf-80ce-f53d4f1e959c; lhpuuidh-browse-ADZ4LPN6MZAN3MDBZ4OUFKDW6A=US%3AEN-US%3A648428ac-7104-4e30-83d9-b412c71e1910_ROOT; lhpuuidh-browse-ADZ4LPN6MZAN3MDBZ4OUFKDW6A-T=1596565541035

Looks like that's all we need for the headers.

Analyzing the URL

In the second image we saw the URL had some get queries, Mostly generic stuff, nothing catches eye here. So, we have everything we need, let's start making some requests and find more.

Making the first request

I copied the request as cURL and imported it in Postman, then I enabled only 1 key-value for our post data, because I don't want to get tons of data, baby steps.. Hit send and voila! We get this response,

{
   "jsonGraph":{
      "genres":{
         "6721":{
            "name":{
               "$type":"atom",
               "value":"Anime Series"
            }
         }
      }
   },
   "paths":[
      [
         "genres",
         6721,
         "name"
      ]
   ]
}

This looks right, right? Now let's play with the values and see if we can pull something else from that genre. Looks like there's a path that looks like this,

["genres",6721,"subgenres",{"from":0,"to":30},["id","name"]]

Let's try and modify it to our needs, now I'm not going to waste time how I found out the correct values and stuff, because that's all manual 'brute-forcing', if you want you can always brute force these endpoints.

["genres", 6721, {"from":0,"to":30}, "videos"]

so what does this means?Hey netflix, give me 30 videos from the genre ID 6721I'm not going to paste the whole response here, because it is quite long, but here's an example when fetching 3 videos,

{
   "jsonGraph":{
      "genres":{
         "6721":{
            "0":{
               "$type":"ref",
               "value":[
                  "videos",
                  "81002438"
               ]
            },
            "1":{
               "$type":"ref",
               "value":[
                  "videos",
                  "70205012"
               ]
            },
            "2":{
               "$type":"ref",
               "value":[
                  "videos",
                  "80050063"
               ]
            },
            "3":{
               "$type":"ref",
               "value":[
                  "videos",
                  "80095241"
               ]
            }
         }
      },
      "videos":{
         "70205012":{
            "videos":{
               "$type":"atom"
            }
         },
         "80050063":{
            "videos":{
               "$type":"atom"
            }
         },
         "80095241":{
            "videos":{
               "$type":"atom"
            }
         },
         "81002438":{
            "videos":{
               "$type":"atom"
            }
         }
      }
   },
   "paths":[
      [
         "genres",
         6721,
         {
            "from":0,
            "to":2
         },
         "videos"
      ]
   ]
}

Judging by the first request we made we can see we're getting actual video IDs! Awesome, so all that's left is to find all genres and pull every video id, after that it's a cake.

Pulling the genres

Now, we all know Netflix has tons of hidden genres, we've seen a lot of articles, plugins etc, about them, but how do we get them? Of course we're not going to copy-paste or even scrape those sites, we want fresh data. There are few ways to do this.

Brute forcing genre ids

Well, duh! we can just brute force with the same path from above,

["genres",{id_here},{"from":0,"to":30}, "videos"]

..but how would that work? Simple! if there are no videos under certain id, Netflix will send back response with $type: "error", which looks something like this.

{
  "jsonGraph":{
     "genres":{
        "4":{
           "0":{
              "$type":"error",
              "value":{
                 "message":"Cannot invoke method withAnnotation() on null object",
                 "cause":"java.lang.NullPointerException: Cannot invoke method withAnnotation() on null object"
              }
           },
           "1":{
              "$type":"error",
              "value":{
                 "message":"Cannot invoke method withAnnotation() on null object",
                 "cause":"java.lang.NullPointerException: Cannot invoke method withAnnotation() on null object"
              }
           }
        }
     }
  },
  "paths":[
     [
        "genres",
        4,
        {
           "from":0,
           "to":1
        },
        "videos"
     ]
  ]
}

Awesome, now it's matter of doing this till we hit 8 digits :). Which we can do quite fast, since we can put list of genre ids instead of just passing only 1 ID per request, should look something like this,

["genres",[1, 2, 3, 4, 5, 6, 7, 8],{"from":0,"to":30}, "videos"]

I actually did this at first with async requests and pulled around 90,000 different genres, and quite fast.

The other way

I was convinced brute forcing was the only way of pulling these genres, but I started playing with the API and managed to make a path that looks something like this,

["genreList",{"from":0,"to":1000}]

Hey Netflix, gimme genre list that has 1000 objects!!

This pulls total 137 genres! Will that be enough though? -nop_So what do we do? -_if you want reliable data, I'd say go with the brute forcing method, since some videos you can't find under some genres

Let's find some data now!

Pulling information about video titles

After playing with responses and looking at certain values, I constructed the following path,

["videos",{video_ids},["title", "runtime", "summary", "seasons", "releaseYear", "availability", "maturity", "mostWatchedData", "seasonCount", "episodeCount"]]

Same as genres we can pass multiple video ids and get data about them, max is 380, though. From the path you can see we're asking for some info like title of the show, the runtime etc.

Let's try for this ID,

["videos",80189685,["title", "runtime", "summary", "seasons", "releaseYear", "availability", "maturity", "mostWatchedData", "seasonCount", "episodeCount"]]

But also, let's pass another path,

["videos",80189685,"seasonList",{"from":0,"to":8},"id"]

Yep, that's right, we also want season list,_but simeon what if it's a movie, movies don't have seasons!!_yeah, well, remember the genres brute forcing method? if it's a movie, we simply don't care about it, as Netflix won't send us anything :). let's check the response,

{
   "jsonGraph":{
      "videos":{
         "80189685":{
            "title":{
               "$type":"atom",
               "value":"The Witcher"
            },
            "runtime":{
               "$type":"atom",
               "value":null
            },
            "summary":{
               "$type":"atom",
               "$size":41,
               "value":{
                  "id":80189685,
                  "type":"show",
                  "isOriginal":true
               }
            },
            "releaseYear":{
               "$type":"atom",
               "value":2019
            },
            "availability":{
               "$type":"atom",
               "$size":84,
               "value":{
                  "isPlayable":true,
                  "availabilityStartTime":1576828800000,
                  "availabilityDate":"December 20"
               }
            },
            "maturity":{
               "$type":"atom",
               "$size":591,
               "value":{
                  "rating":{
                     "value":"TV-MA",
                     "maturityDescription":"For Mature Audiences. May not be suitable for ages 17 and under.",
                     "maturityLevel":110,
                     "specificRatingReason":"violence, sex, nudity, language, gore, smoking",
                     "board":"US TV",
                     "boardId":10,
                     "reasons":[
                        {
                           "id":7415,
                           "reasonDescription":"violence",
                           "levelDescription":null
                        },
                        {
                           "id":7417,
                           "reasonDescription":"sex",
                           "levelDescription":null
                        },
                        {
                           "id":7421,
                           "reasonDescription":"nudity",
                           "levelDescription":null
                        },
                        {
                           "id":7426,
                           "reasonDescription":"language",
                           "levelDescription":null
                        },
                        {
                           "id":7429,
                           "reasonDescription":"gore",
                           "levelDescription":null
                        },
                        {
                           "id":7430,
                           "reasonDescription":"smoking",
                           "levelDescription":null
                        }
                     ]
                  }
               }
            },
            "mostWatchedData":{
               "$type":"atom",
               "value":null
            },
            "seasonCount":{
               "$type":"atom",
               "value":1
            },
            "seasonList":{
               "0":{
                  "$type":"ref",
                  "value":[
                     "seasons",
                     "80189598"
                  ]
               },
               "1":{
                  "$type":"atom"
               },
               "2":{
                  "$type":"atom"
               },
               "3":{
                  "$type":"atom"
               },
               "4":{
                  "$type":"atom"
               },
               "5":{
                  "$type":"atom"
               },
               "6":{
                  "$type":"atom"
               },
               "7":{
                  "$type":"atom"
               },
               "8":{
                  "$type":"atom"
               }
            },
            "episodeCount":{
               "$type":"atom",
               "value":8
            }
         }
      },
      "seasons":{
         "80189598":{
            "id":{
               "$type":"atom"
            }
         }
      }
   },
   "paths":[
      [
         "videos",
         80189685,
         [
            "availability",
            "episodeCount",
            "maturity",
            "mostWatchedData",
            "releaseYear",
            "runtime",
            "seasonCount",
            "seasons",
            "summary",
            "title"
         ]
      ],
      [
         "videos",
         80189685,
         "seasonList",
         {
            "from":0,
            "to":8
         },
         "id"
      ]
   ]
}

Awesome! Though we still need to find information about episodes, so let's use some of the data from the previous response,

["seasons",{season_list}, "episodes", {{"from":0,"to":{total_episodes}}},["title","runtime", "summary", "releaseYear", "availability", "maturity", "mostWatchedData"]]

Pretty simple, right? Same as videos and genres fetching, we can add list of seasons, not just 1! Here's an example response for an episode,

"80244465":{
   "title":{
      "$type":"atom",
      "value":"Betrayer Moon"
   },
   "runtime":{
      "$type":"atom",
      "value":4022
   },
   "summary":{
      "$type":"atom",
      "$size":89,
      "value":{
         "idx":3,
         "id":80244465,
         "type":"episode",
         "isOriginal":true,
         "episode":3,
         "season":1,
         "isPlayable":true
      }
   },
   "releaseYear":{
      "$type":"atom",
      "value":2019
   },
   "availability":{
      "$type":"atom",
      "$size":84,
      "value":{
         "isPlayable":true,
         "availabilityStartTime":1576828800000,
         "availabilityDate":"December 20"
      }
   },
   "maturity":{
      "$type":"atom",
      "$size":591,
      "value":{
         "rating":{
            "value":"TV-MA",
            "maturityDescription":"For Mature Audiences. May not be suitable for ages 17 and under.",
            "maturityLevel":110,
            "specificRatingReason":"violence, sex, nudity, language, gore, smoking",
            "board":"US TV",
            "boardId":10,
            "reasons":[
               {
                  "id":7415,
                  "reasonDescription":"violence",
                  "levelDescription":null
               },
               {
                  "id":7417,
                  "reasonDescription":"sex",
                  "levelDescription":null
               },
               {
                  "id":7421,
                  "reasonDescription":"nudity",
                  "levelDescription":null
               },
               {
                  "id":7426,
                  "reasonDescription":"language",
                  "levelDescription":null
               },
               {
                  "id":7429,
                  "reasonDescription":"gore",
                  "levelDescription":null
               },
               {
                  "id":7430,
                  "reasonDescription":"smoking",
                  "levelDescription":null
               }
            ]
         }
      }
   },
   "mostWatchedData":{
      "$type":"atom",
      "value":null
   }
}

Pretty cool, huh?

Coding it

I'm assuming your familiar with virtual environments and all, so I'm not going to explain that. Let's install requests,

(venv) $ pip install requests

Then, let's create config.py, where we'll copy our headers and urls. Next we need to somehow construct these 'paths' aka the POST body. Let's create a file path.py and write this class,

class Path(object):  
    def __init__ (self, name):  
        self.name = name  

    def __str__ (self):  
        return self.name  

    def __repr__ (self):  
        return "'" + self.name + "'"

Why do we need this? Well, if we want to build the path, we need to make a dictionary, and we need multiple paths, as we know, dictionaries can't have duplicate keys. This class allows us to create multiple keys, because we're going to pass an object, a string: 'path', which makes the keys 'different'. Let's create our main.py or whatever you want to call it. We need to import our Path class and our config, so let's add an empty __init__.py in our directory, For the sake of simplicity, I'm going to define one function only and put everything there. Also, I would recommend using proxies.

from .path import Path
from .config import url, PROXIES, headers
import requests

videos_id_list = []
# Note: We don't actually have to use the Path class here, since we have only one key-value pair.
payload = {  
   Path('path') : f'["genres", {list(range(1, 1000))}, {{"from":0, "to":300}}, "videos"]'  
}
   try:  
       response = requests.post(url, proxies=PROXIES, headers=headers, data=payload)  
       for obj in response.json().get('jsonGraph').get('videos'):  
       # NOTE: This most likely won't work, because `obj` is still json we need to parse.
               videos_id_list.append(obj)
   except Exception:  
    # Do something
       pass
videos_payload = {  
   Path('path'): f"["videos",{videos_id_list},["title", "runtime", "summary", "seasons", "releaseYear", "availability", "maturity", "mostWatchedData", "seasonCount", "episodeCount"]]",  
   Path('path'): f"["videos", {videos_id_list}, "seasonList",{{"from":0,"to":8}},"id"]",  
}

# After we get the video IDs we'll send

You can see the idea, I leave the rest to you.

Additional attributes we can get

availabilityEndDateNear
bookmarkPosition
boxArts
cast
copyright
creditsOffset
delivery
evidence
hasSensitiveMetadata
synopsis
isNSRE
tallBoxarts
writers

The End

I think Netflix has very a interesting API, it was fun working on this! If anyone has any questions feel free to drop a comment below.

Django in Production - II

Idiomatic Programmers — Sat, 20 Jun 2020 03:00:00 +0000

In the last post, we learnt how to make use of Gunicorn to run our Django application, and use Whitenoise to serve static files. Today, we are going to learn how to configure a reverse proxy in front of gunicorn to serve our website on port 80, and also use it as web server to serve our static files instead of using Whitenoise.

NGINX Installation

If you're using Ubuntu, run the following commands:

sudo apt-get update
sudo apt-get install nginx

Now you can check the status of nginx by running command:

sudo systemctl status nginx

sudo service nginx status

based on how your system is setup.

NGINX Configuration

So, for running NGINX you'll need to write configuration files. They're stored in the /etc/nginx/sites-available/ directory.

Let's first go into that directory by running:

cd /etc/nginx/sites-available/

Now let's create a file by doing:

sudo touch django.conf

And now let's open the file in the editor by doing:

sudo nano django.conf

Hint : We need to use sudo here because this directory is present in root and not inside our home directory.

Then paste the below config inside it.

server {                 
   server_name server_domain_or_ip;

   location / {
      include proxy_params;
      proxy_pass http://localhost:8000;
    }
}

In place of server_domain_or_ip you'll put either IP address or the domain of your website.

So NGINX uses this concept where you can have multiple configuration files as backup stored under /etc/nginx/sites-available/. But the active ones are present under /etc/nginx/sites-enabled/

So, you put your configuration files under sites-available and then create a soft symlink to that file inside sites-enabled

So let's create the symlink for this file under /etc/nginx/sites-enabled/.

To do that, run the following command:

cd sites-enabled
sudo ln -s /etc/nginx/sites-available/django.conf .

This will create the symlink of our config file under sites-enabled.

Now for our changes to come in effect, we'll need to instruct NGINX to read our config file once again and restart the server, we do this by running:

sudo systemctl restart nginx

sudo service nginx restart

Making our site secure

Right now, our site is being served at port 80, which means http, which isn't secure. But we want our website to be secure. So now let's install SSL certificate for our website using the certificate provided by Let's Encrypt. To do that, follow the steps below:

Let's update the packages list to the latest version, before we install our dependencies by running:

sudo apt-get update

Now let's install our dependencies.

sudo apt-get install software-properties

Let's add the repository needed for installing the certificate generating utility, by running:

sudo add-apt-repository universe
sudo add-apt-repository ppa:certbot/certbot

Now we need to refresh our local packages list, by running:

sudo apt-get update

Let's finally install the needed packages by running:

sudo apt-get install certbot python3-certbot-nginx

Now we can create and configure ssl certificates for our nginx configuration:

sudo certbot --nginx

When you run the last command, there'll be an interactive CLI program, that'll ask you for certain configuration specifics, one of them will be whether you want to redirect non secure traffic to https, then you should choose Yes.

If you aren't running Ubuntu on your server, then you can use this website to look for instructions for your own OS.

Serving Static Files using NGINX

Now that you have finally added NGINX to your stack, you can use it for serving your static files as well, as it can perform much better than the whitenoise alternative we discussed earlier.

First, we need to make changes to our Django settings.py file, add the below line inside it:

STATICFILES_STORAGE = "django.contrib.staticfiles.storage.ManifestStaticFilesStorage"
STATIC_ROOT = os.path.join(BASE_DIR, "staticfiles")
STATIC_URL = "/static/"

Make sure, the above settings are not defined elsewhere in your settings.

Now go in your project root directory and create the folder staticfiles by typing the following in your terminal:

mkdir staticfiles

If you have your own static files (CSS, JS etc.) used in your templates, then pass the path to files (in our case static) as follows:

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, "static"),
]

Now run the following command:

python manage.py collectstatic

Now that we are done setting up our django project, let's configure NGINX to serve these files, go back to the nginx directory by typing the below line:

cd /etc/nginx/sites-available/
sudo nano django.conf

Make the following changes to your config:

server {                 
   server_name server_domain_or_ip;

   location / {
      include proxy_params;
      proxy_pass http://localhost:8000;
   }

    # Below part is to be added for static files
    location /static/ {
        alias /home/ubuntu/path/to/django/project/staticfiles/;
        expires 365d;
        autoindex off;
    }

}

Add the newly added part to your configuration, and make sure to replace /home/ubuntu/path/to/django/project/ with the actual path where your project is stored.

When you're done, restart the nginx by running:

sudo systemctl restart nginx

sudo service nginx restart

And we're done serving static files using NGINX.

In the next post, we are going to configure a process control system, which will ensure that our gunicorn worker remains up all the time, and starts automatically at boot time.

Django in Production - I

Idiomatic Programmers — Thu, 18 Jun 2020 03:00:00 +0000

Introduction

So, you might have finished developing that project you were building for your client, and now it's time to deploy your project on a server and make it accessible to public. But you're confused on how to proceed further? I was too, but not anymore.

The reason why I decided to write this series is that when I had to deploy my project, I had to spend hours searching about it and then read up multiple articles about it, but even then they didn't cover all the essential details. So that's what prompted me to write this series, and I am going to go over my setup for running django in production.

First thing, I wanna clear up is that when you run your django server using the command:

python manage.py runserver

This starts up a development server, that is provided by the Django Team only for the purpose of development, you are not supposed to run your projects in production using it.

In production, you'll have to use a WSGI server such as Gunicorn or uWSGI, which I'm going to cover as well.

Deployment Checklist

Now, I would like to introduce you to a checklist utility for configuring django settings that you should setup properly in your project's settings.py.

To check whether you've configured those settings correctly in your project, run the following command:

python manage.py check --deploy

It'll display the settings that you should fix before going into production. For more details, refer to this section in Django documentation.

Turning Debug Mode Off

When you're done fixing those settings, you will see that you were asked to turn DEBUG mode off in production by setting DEBUG = False. Now that you've done that, if you tried running your server using python manage.py runserver now, and tried accessing your admin you'll realize that it doesn't load your CSS and you'll be greeted by a screen looking like this:

It's because Django advises you to serve your static files using something like NGINX, as mentioned in the docs here.

After hearing NGINX, you're probably feeling intimidated and you're thinking "oh no! now I need to learn another technology before I can run my app in production?", if not, atleast I felt like that. But worry not, I have got you covered, so instead of using NGINX, we are going to be using Whitenoise for our project. So, let's go ahead and install whitenoise by running:

pip install whitenoise

Now, you need to edit your django project settings, which will usually be present in settings.py. Add the following line to your settings:

STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')

Then you'll need to add whitenoise to your middlewares as follows, make sure to place it directly after the Django SecurityMiddleware and before all other middlewares:

MIDDLEWARE = [
  'django.middleware.security.SecurityMiddleware',
  'whitenoise.middleware.WhiteNoiseMiddleware',
  # ...
]

That's it now your static files will be served by whitenoise. If you want to go one step further and configure whitenoise to compress your static files before serving, then follow the next step.

Add this line inside your settings.py:

STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'

After you're done setting up whitenoise, if you start your server now and visit admin, you'll see that now the CSS files have loaded properly.

WSGI Server

Before getting started with setting up the WSGI server for our project, I'd like to talk a bit about them first.

WSGI stands for Web Server Gateway Interface. It is a standard interface for running Python code. It is what is implemented in applications like Gunicorn, uWSGI or mod_wsgi, out of which you're free to pick any based on your needs. Your web server is configured to pass requests to the WSGI container which runs your web application, then pass the response back to the requester.

Or to put it more simply, it processes the HTTP request and responses and gives them back to your application in a structured manner without them having to deal with raw HTTP headers and content.

If you wanna learn more about WSGI, I'd highly suggest you to watch this PyCon talk.

Enough theory, now let's jump into code. We'll be using gunicorn, let's start by installing it by running the command:

pip install gunicorn

Now, start gunicorn by running:

gunicorn myproject.wsgi

Here, you would replace myproject with your Django project name instead.

That's the most basic way of running your Django project using gunicorn, but you might need some other things like logging.

gunicorn myproject.wsgi --log-level DEBUG --access-logfile path/to/out.log --error-logfile path/to/error.log

The above command would setup error logs and access logs to your server, however make sure to replace path/to/out.log and path/to/error.log to actual file paths where you want to store your logs.

Further, in the above command, instead of debug, you can use several other values such as:

debug
info
warning
error
critical

In the next post, we are going to cover how to install and configure NGINX to run as reverse proxy to direct traffic to our WSGI server (gunicorn) and also serve our static files.