DEV Community: Idiomatic Programmers

Ultimate Redis Cheatsheet

Idiomatic Programmers — Thu, 27 Jul 2023 11:44:02 +0000

What is Redis?

Redis, which stands for Remote Dictionary Server, is an open-source, in-memory data structure store that serves as a highly efficient and versatile database solution. Developed by Salvatore Sanfilippo, Redis is designed to offer lightning-fast data access and retrieval by keeping all the data in memory rather than on disk, ensuring remarkable performance and responsiveness.

One of the key features that sets Redis apart is its support for various data structures, including strings, lists, sets, sorted sets, and hashes. This versatility enables Redis to serve multiple purposes, such as acting as a key-value store, a message broker, or a caching layer.

Redis utilizes an in-memory approach, making it exceptionally fast for read and write operations. With its optimized data structures and efficient algorithms, Redis can process data in sub-millisecond response times, making it an ideal choice for real-time applications and use cases that require high-performance data manipulation.

Furthermore, Redis offers persistence options, allowing users to store their data on disk periodically or whenever specific conditions are met. This feature ensures data durability and makes Redis suitable for applications that require both speed and data persistence.

In addition to its exceptional performance, Redis provides advanced features like pub/sub messaging, transactions, and Lua scripting, which empower developers to build complex applications and systems with ease.

Redis has become immensely popular due to its simplicity, versatility, and extensive support for multiple programming languages. It offers client libraries for various languages, making it accessible and easy to integrate into different software stacks.

Whether you need to cache frequently accessed data, build real-time applications, implement a messaging system, or store and process complex data structures, Redis provides a robust and efficient solution that continues to be widely adopted across industries.

In conclusion, Redis is a powerful and flexible database solution that excels in performance, versatility, and ease of use. Its in-memory nature, support for various data structures, and extensive feature set make it a go-to choice for developers looking to build high-performance applications and systems.

How Redis is different from other Databases?

Redis differs from other databases in several key aspects:

In-Memory Data Storage: Redis primarily stores data in memory, allowing for extremely fast read and write operations. This design choice sets Redis apart from traditional disk-based databases that incur disk I/O overhead. By keeping data in memory, Redis can achieve high throughput and low latency, making it well-suited for applications that require real-time data processing and high-performance caching.
Data Structures: Redis supports a wide range of data structures, including strings, lists, sets, hashes, and sorted sets. This flexibility allows developers to model complex data scenarios efficiently. Unlike many other databases that offer a fixed schema, Redis provides dynamic and versatile data structures that can be manipulated and combined to suit specific application needs.
Built-in Caching Mechanisms: Redis has native support for caching, making it an excellent choice for scenarios that require frequent access to frequently accessed data. With features like key expiration and eviction policies, Redis can efficiently manage cache storage and ensure that the most relevant data remains readily available.
Pub/Sub Messaging: Redis includes built-in publish/subscribe messaging capabilities. This feature allows applications to implement real-time messaging and event-driven architectures. Publishers can send messages to specific channels, and subscribers receive messages from those channels in real-time. This makes Redis a suitable choice for building chat systems, real-time analytics, and other applications that require event-based communication.
Lua Scripting: Redis supports Lua scripting, which enables developers to execute server-side scripts directly within the database. This allows for complex operations and transactional logic to be performed within Redis, reducing the need for round-trips between the application and the database. Lua scripting enhances Redis's flexibility and extensibility by enabling custom behaviour and complex data manipulations.
Replication and Persistence: Redis provides built-in replication mechanisms for creating multiple copies of the database. This replication ensures high availability and fault tolerance by propagating changes from a master node to one or more slave nodes. Redis also offers persistence options through snapshotting and append-only file (AOF) logging, allowing data to be saved to disk for recovery in the event of a system failure.
Simplicity and Performance: Redis is known for its simplicity and performance. It offers a straightforward set of commands that are easy to understand and use. The focus on in-memory storage and the absence of complex query languages or indexing mechanisms simplify the database's design and contribute to its exceptional performance.

Overall, Redis stands out among other databases due to its in-memory storage, versatile data structures, caching capabilities, pub/sub messaging, Lua scripting, replication, and performance-oriented design. These features make Redis a popular choice for applications that require speed, flexibility, and real-time data processing.

Installing Redis and Redis CLI

Here are the steps to install Redis on Linux, macOS, and Windows:

Linux:

Open a terminal window.
Update the package manager:
```
sudo apt update
```
Install Redis:
```
sudo apt install redis-server
```
Start the Redis service:
```
sudo systemctl start redis-server
```
To access the Redis command-line interface (redis-cli):
```
redis-cli
```

macOS:

Open a terminal window.

Install Homebrew (if not already installed):

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Install Redis using Homebrew:
```
brew install redis
```
Start the Redis service:
```
brew services start redis
```
To access the Redis command-line interface (redis-cli):
```
redis-cli
```

Windows:

Download the latest Redis for Windows from the Microsoft Open Tech GitHub repository: https://github.com/microsoftarchive/redis/releases
Extract the downloaded archive to a directory of your choice, e.g., C:\Redis.
Open a Command Prompt window.
Navigate to the Redis installation directory:
```
cd C:\Redis
```
Start the Redis server:
```
redis-server.exe
```
Open another Command Prompt window to access the Redis command-line interface (redis-cli):
```
cd C:\Redis
redis-cli.exe
```

Please note that for Linux and macOS, the installation steps assume the usage of package managers like apt and Homebrew, respectively. If you are using a different package manager or distribution, you may need to adjust the installation commands accordingly.

Commands

SET manipulation

SET
- Command: SET
- Syntax: SET key value
- Usage: SET myset "Hello Redis"
- Description: Sets the value of a key with the provided string value. If the key already exists, it overwrites the existing value. This command is also used to create a new key-value pair if the key doesn't exist.
GET
- Command: GET
- Syntax: GET key
- Usage: GET myset
- Description: Retrieves the value associated with the specified key. If the key exists, the command returns the value; otherwise, it returns nil.
SADD
- Command: SADD
- Syntax: SADD key member [member ...]
- Usage: SADD myset "value1" "value2"
- Description: Adds one or more members to a set. If the members already exist, they are ignored. If the key doesn't exist, a new set is created with the specified members.
SMEMBERS
- Command: SMEMBERS
- Syntax: SMEMBERS key
- Usage: SMEMBERS myset
- Description: Returns all the members of a set. If the key doesn't exist, an empty set is returned.
SISMEMBER
- Command: SISMEMBER
- Syntax: SISMEMBER key member
- Usage: SISMEMBER myset "value1"
- Description: Checks if a member exists in a set. Returns 1 if the member is present; otherwise, returns 0.
SREM
- Command: SREM
- Syntax: SREM key member [member ...]
- Usage: SREM myset "value1" "value2"
- Description: Removes one or more members from a set. If the members don't exist, they are ignored. If the key becomes empty after removing members, the key is also removed.
SCARD
- Command: SCARD
- Syntax: SCARD key
- Usage: SCARD myset
- Description: Returns the cardinality (number of members) of a set. If the key doesn't exist, 0 is returned.
SINTER
- Command: SINTER
- Syntax: SINTER key [key ...]
- Usage: SINTER myset1 myset2
- Description: Returns the intersection of multiple sets. Only the members that exist in all the specified sets are included in the result set.
SUNION
- Command: SUNION
- Syntax: SUNION key [key ...]
- Usage: SUNION myset1 myset2
- Description: Returns the union of multiple sets. All the members from the specified sets are included in the result set.
SDIFF
- Command: SDIFF
- Syntax: SDIFF key [key ...]
- Usage: SDIFF myset1 myset2
- Description: Returns the difference between two sets. The members that exist in the first set but not in any other specified sets are included in the result set.

Sorted Sets manipulation

ZADD
- Command: ZADD
- Syntax: ZADD key [NX|XX] [CH] [INCR] score member [score member ...]
- Usage: ZADD mysortedset 1 "value1" 2 "value2"
- Description: Adds one or more members with scores to a sorted set. If a member already exists, its score is updated. Additional options like NX (only add new elements), XX (only update existing elements), CH (return the number of changed elements), and INCR (increment the score of existing member) can be specified.
ZSCORE
- Command: ZSCORE
- Syntax: ZSCORE key member
- Usage: ZSCORE mysortedset "value1"
- Description: Returns the score of a member in a sorted set. If the member doesn't exist, nil is returned.
ZRANGE
- Command: ZRANGE
- Syntax: ZRANGE key start stop [WITHSCORES]
- Usage: ZRANGE mysortedset 0 -1
- Description: Returns a range of members from a sorted set based on their position. The start and stop arguments specify the inclusive range of positions. The optional WITHSCORES flag includes the scores in the result.
ZREVRANGE
- Command: ZREVRANGE
- Syntax: ZREVRANGE key start stop [WITHSCORES]
- Usage: ZREVRANGE mysortedset 0 -1 WITHSCORES
- Description: Returns a range of members from a sorted set in reverse order based on their position. The start and stop arguments specify the inclusive range of positions. The optional WITHSCORES flag includes the scores in the result.
ZREM
- Command: ZREM
- Syntax: ZREM key member [member ...]
- Usage: ZREM mysortedset "value1" "value2"
- Description: Removes one or more members from a sorted set. If a member doesn't exist, it is ignored. If the key becomes empty after removing members, the key is also removed.
ZCARD
- Command: ZCARD
- Syntax: ZCARD key
- Usage: ZCARD mysortedset
- Description: Returns the cardinality (number of members) of a sorted set. If the key doesn't exist, 0 is returned.
ZCOUNT
- Command: ZCOUNT
- Syntax: ZCOUNT key min max
- Usage: ZCOUNT mysortedset 1 10
- Description: Returns the count of members in a sorted set within the specified score range (min and max inclusive).
ZRANK
- Command: ZRANK
- Syntax: ZRANK key member
- Usage: ZRANK mysortedset "value1"
- Description: Returns the position (rank) of a member in a sorted set when sorted in ascending order. The lowest rank is 0. If the member doesn't exist, nil is returned.
ZREVRANK
- Command: ZREVRANK
- Syntax: ZREVRANK key member
- Usage: ZREVRANK mysortedset "value1"
- Description: Returns the position (rank) of a member in a sorted set when sorted in descending order. The highest rank is 0. If the member doesn't exist, nil is returned.
ZINCRBY
- Command: ZINCRBY
- Syntax: ZINCRBY key increment member
- Usage: ZINCRBY mysortedset 5 "value1"
- Description: Increments the score of a member in a sorted set by the specified increment. If the member doesn't exist, it is added with the initial score as the increment value.

String, Number and Bit Manipulation

APPEND
- Command: APPEND
- Syntax: APPEND key value
- Usage: APPEND mykey "world"
- Description: Appends the specified value to the string value already stored at the key. If the key doesn't exist, a new key is created with the specified value.
BITCOUNT
- Command: BITCOUNT
- Syntax: BITCOUNT key [start end]
- Usage: BITCOUNT mykey
- Description: Counts the number of set bits (1s) in a string value. Optionally, a range can be specified to count the bits within that range.
BITOP
- Command: BITOP
- Syntax: BITOP operation destkey key [key ...]
- Usage: BITOP AND destkey key1 key2
- Description: Performs a bitwise operation (AND, OR, XOR, NOT) between multiple keys and stores the result in the destination key.
BITPOS
- Command: BITPOS
- Syntax: BITPOS key bit [start] [end]
- Usage: BITPOS mykey 1 0 10
- Description: Finds the position of the first set bit (1) or clear bit (0) in a string value. Optionally, a range can be specified to search for the bit within that range.
DECR
- Command: DECR
- Syntax: DECR key
- Usage: DECR mykey
- Description: Decrements the value of the key by 1. If the key doesn't exist, it is initialized with the value of 0 before decrementing.
DECRBY
- Command: DECRBY
- Syntax: DECRBY key decrement
- Usage: DECRBY mykey 5
- Description: Decrements the value of the key by the specified decrement. If the key doesn't exist, it is initialized with the value of 0 before decrementing.
GET
- Command: GET
- Syntax: GET key
- Usage: GET mykey
- Description: Retrieves the value associated with the specified key. If the key exists, the command returns the value; otherwise, it returns nil.
GETBIT
- Command: GETBIT
- Syntax: GETBIT key offset
- Usage: GETBIT mykey 5
- Description: Returns the bit value at the specified offset in a string value. The offset is zero-based.
GETRANGE
- Command: GETRANGE
- Syntax: GETRANGE key start end
- Usage: GETRANGE mykey 0 3
- Description: Retrieves a substring of the string value stored at the key, starting from the specified start index to the end index (inclusive).
GETSET
- Command: GETSET
- Syntax: GETSET key value
- Usage: GETSET mykey "newvalue"
- Description: Sets the value of the key with the provided string value and returns the old value. If the key doesn't exist, it is initialized with the specified value.
INCR
- Command: INCR
- Syntax: INCR key
- Usage: INCR mykey
- Description: Increments the value of the key by 1. If the key doesn't exist, it is initialized with the value of 0 before incrementing.
INCRBY
- Command: INCRBY
- Syntax: INCRBY key increment
- Usage: INCRBY mykey 5
- Description: Increments the value of the key by the specified increment. If the key doesn't exist, it is initialized with the value of 0 before incrementing.
INCRBYFLOAT
- Command: INCRBYFLOAT
- Syntax: INCRBYFLOAT key increment
- Usage: INCRBYFLOAT mykey 2.5
- Description: Increments the value of the key by the specified float increment. If the key doesn't exist, it is initialized with the value of 0 before incrementing.
MGET
- Command: MGET
- Syntax: MGET key [key ...]
- Usage: MGET key1 key2 key3
- Description: Retrieves the values associated with multiple keys. It returns an array of values corresponding to the provided keys.
MSET
- Command: MSET
- Syntax: MSET key value [key value ...]
- Usage: MSET key1 "value1" key2 "value2"
- Description: Sets multiple keys to their respective values. It allows setting multiple key-value pairs in a single command.
MSETNX
- Command: MSETNX
- Syntax: MSETNX key value [key value ...]
- Usage: MSETNX key1 "value1" key2 "value2"
- Description: Sets multiple keys to their respective values, only if none of the keys already exist. It allows atomic setting of multiple keys.
PSETEX
- Command: PSETEX
- Syntax: PSETEX key milliseconds value
- Usage: PSETEX mykey 1000 "Hello"
- Description: Sets the value of the key with the provided string value and a specified expiration time in milliseconds.
SET
- Command: SET
- Syntax: SET key value [EX seconds|PX milliseconds] [NX|XX]
- Usage: SET mykey "Hello" EX 60 NX
- Description: Sets the value of a key with the provided string value. Additional options like expiration time, conditional set (NX - only if the key doesn't exist, XX - only if the key exists) can be specified.
SETBIT
- Command: SETBIT
- Syntax: SETBIT key offset value
- Usage: SETBIT mykey 5 1
- Description: Sets the bit at the specified offset in a string value to either 0 or 1.
SETEX
- Command: SETEX
- Syntax: SETEX key seconds value
- Usage: SETEX mykey 60 "Hello"
- Description: Sets the value of the key with the provided string value and a specified expiration time in seconds.
SETNX
- Command: SETNX
- Syntax: SETNX key value
- Usage: SETNX mykey "Hello"
- Description: Sets the value of a key with the provided string value, only if the key doesn't exist.
SETRANGE
- Command: SETRANGE
- Syntax: SETRANGE key offset value
- Usage: SETRANGE mykey 5 "Redis"
- Description: Overwrites a substring of the string value stored at the key, starting from the specified offset with the provided value.
STRLEN
- Command: STRLEN
- Syntax: STRLEN key
- Usage: STRLEN mykey
- Description: Returns length

Conclusion

In conclusion, we have discussed Redis, a popular in-memory data structure store, and explored its unique characteristics that set it apart from other databases. Redis offers exceptional performance and versatility, making it suitable for various use cases, such as caching, real-time analytics, messaging systems, and more.

We have also covered the installation process of Redis on Linux, macOS, and Windows, including how to access the Redis command-line interface (redis-cli) for interacting with the database.

Furthermore, we have delved into the major commands related to set manipulation, providing syntax, usage examples, and descriptions for each command. These commands enable users to perform various operations like adding, retrieving, modifying, and removing elements from sets in Redis.

Additionally, we have explored the major commands related to sorted set manipulation, including operations such as adding members with scores, retrieving ranges, finding positions, incrementing scores, and more. Sorted sets in Redis provide an ordered collection of unique elements, allowing efficient operations based on both element values and scores.

Redis commands like APPEND, GET, SET, INCR, and many others offer powerful functionality for string manipulation, bit operations, and key-value management.

Redis, with its versatility, performance, and rich set of commands, continues to be a preferred choice for developers and architects when it comes to handling data-intensive applications and solving various data storage and retrieval challenges.

Celery Alternative for Django - Huey

Idiomatic Programmers — Fri, 13 May 2022 13:56:49 +0000

Background

So today I will be talking about a Celery alternative named Huey, which comes with a much easier setup than Celery and is much smaller in size compared to Celery.

The reason why I decided to try out Huey is because I have faced some issues with Celery sometimes for doing some common tasks because the documentation isn't too great.

For those who don't know what Celery is or haven't used it before, Huey is an asynchronous task queue which allows you to perform scheduled tasks or long running tasks in the background.

Prerequisites

We will be installing the following packages:

redis
django
huey
requests (optional, needed for the demo)

Github Repo

The following blog comes accompanied with a github repo, that you can use to test out the demo project that we will be creating.

Click here to view repo.

Project Setup

Create Project Directory

Open the terminal and type the following to create a directory, you can skip this step and do it from File Explorer itself.

mkdir huey_demo

Virtual Environment

Let us create a virtualenv first to install our project dependencies:
```
python -m venv venv
```
Activate the virtualenv (Linux):
```
source venv/bin/activate
```

Installing dependencies

Type the following command in the terminal to install all the dependencies:

pip install Django==4.0.4 redis==4.2.2 huey==2.4.3

At the time of writing this article, these were the versions I tested out this setup with, keep an eye on the Github Repo for any updates as per the latest version in the future.

Create Project

Create the django project by typing the following command in terminal:
```
django-admin startproject django_huey_demo
```
Change directory into Django project directory:
```
cd django_huey_demo
```
Create the app under our project:
```
python manage.py startapp demo
```

Include the created app into project settings.py, make the following changes:

INSTALLED_APPS = [
# Existing Apps
"demo.apps.DemoConfig",  # <== Add this line
]

Set debug mode to False in settings.py:
```
DEBUG=False
```

Project Overview

Now that we are done setting up our project, it is a good time to take you over what we will be building today. We will fetch "Word of the Day" daily from Wordnik API. Then we will store the word, its definition, and an example of the word in a sentence to our database.

We will setup a periodic task using Huey which will fetch the Word of the Day and store it.

For storing the word we will be creating a Django Model of the same.

Getting Wordnik API Key

You can follow this guide to obtain the API key.

Coding our project

Add Huey to our Project

We need to add Huey to installed apps of our project, so make the following changes in settings.py file:

INSTALLED_APPS = [
    # Existing apps
    'huey.contrib.djhuey', # <== Add this line
]

Install Redis

We need to install Redis for Huey to store information about queued tasks in it, like we used to do with Celery as well. You can refer to the following link to install redis based on your specific operating system.

If you're comfortable with using Docker, you can use the following command:

docker run --name redis_huey -p 6379:6379 -d redis

By default, Huey will try connecting to Redis server running on localhost:6379. If it isn't present, it will raise an error.

Model Definition

Add the following code to your demo/models.py file:

from django.db import models

class Word(models.Model):
    word = models.CharField(max_length=200)
    part_of_speech = models.CharField(max_length=100)
    definition = models.TextField()
    example = models.TextField()

    def __str__(self):
        return self.word

Make migrations:
```
python manage.py makemigrations demo
```
Apply migrations:
```
python manage.py migrate demo
```

Task Definition

Create a file named tasks.py in demo app directory. The reason why we named our file tasks.py is to help Huey auto discover the tasks present in our registered apps, if we named our file anything other than that, we would have to manually register our task. If you would like to know more you can check out the Huey documentation here.

Before we write the task definition we need to install an additional dependency requests. Install it by typing the following in your terminal:

pip install requests==2.27.1

Now comes the code:

import requests
from django.conf import settings
from huey import crontab
from huey.contrib.djhuey import db_periodic_task

from demo.models import Word

@db_periodic_task(crontab(hour="18", minute="00"))
def fetch_daily_word():
    r = requests.get(
        f"https://api.wordnik.com/v4/words.json/wordOfTheDay?api_key={settings.WORDNIK_API_KEY}")
    data = r.json()
    Word.objects.get_or_create(
        word=data["word"],
        part_of_speech=data["definitions"][0]["partOfSpeech"],
        definition=data["definitions"][0]["text"],
        example=data["examples"][0]["text"]
    )

Add the following line in your project settings:

WORDNIK_API_KEY = "api-key-here"

This codeblock might be a lot to take in, so let's go over the things in it one by one:

Huey Decorator
```
from huey.contrib.djhuey import db_periodic_task
```
This is a decorator provided by Huey to register periodic task that involve working with database, this decorator automatically closes the database connection upon task completion, for more details, you can refer here.
Crontab Schedule
```
@db_periodic_task(crontab(hour="18", minute="00"))
```
We are passing the argument crontab(hour="18", minute="00") to our periodic task decorator, this tells Huey to run our task at 6pm everyday. You can make use of this website to create your crontab schedules, I use it everytime.
Wordnik API Key
```
from django.conf import settings

# Usage
## settings.WORDNIK_API_KEY
```
from django.conf import settings is the standard way to import any data from our project settings, it is useful in cases where we have multiple settings files setup for different environment so it will know which file to pick from without us having to worry about it. It finds out which settings file we are using from the DJANGO_SETTINGS_MODULE environment variable. But you don't have to worry about these details.

Then we are using the key in our wordnik API call.
Wordnik API Call
```
r = requests.get(
f"https://api.wordnik.com/v4/words.json/wordOfTheDay?api_key={settings.WORDNIK_API_KEY}")
```
Here we are making use of the requests module to make a GET request to the wordnik API while passing our API Key for authentication.
Storing word in database
```
data = r.json()
Word.objects.get_or_create(
    word=data["word"],
    part_of_speech=data["definitions"][0]["partOfSpeech"],
    definition=data["definitions"][0]["text"],
    example=data["examples"][0]["text"]
)
```
After parsing the API response we are storing the word definition in our database. We are making use of get_or_create method instead of create method here so that we don't create multiple copies of the same word in our database if that word is ever repeated by the Wordnik API.

Wordnik API Response
Here is what the Wordnik API response for Word of the Day endpoint looks like. Some of the irrelevant sections of the response has been truncated for brevity purposes.

{
"word": "stolon",
"definitions": [
    {
    "source": "ahd-5",
    "text": "A long thin stem that usually grows horizontally along the ground and produces roots and shoots at widely spaced nodes, as in a strawberry plant.",
    "note": null,
    "partOfSpeech": "noun"
    },
    // More definitions here...
],
"publishDate": "2022-05-08T03:00:00.000Z",
"examples": [
    {
    "title": "4.1 Nursery establishment",
    "text": "A stolon is a stem that grows along the ground, producing at its nodes new plants with roots and upright stems.",
    // Additional data here...
    },
    // More examples here...
],
    // Additional fields here...
}

Here is what the Wordnik API response for Word of the Day endpoint looks like. Some of the irrelevant sections of the response has been truncated for brevity purposes.

Running Huey Worker

You can start the Huey worker by typing the following command in your terminal:

python manage.py run_huey

You can pass multiple flags to the above command which will change what gets logged to the console, such as:

-v, --verbose - verbose logging (includes DEBUG level)
-q, --quiet - minimal logging
-S, --simple - simple logging format (“time message”)

To look at various other options for logging, checkout the docs here.

What else can you do with Huey?

Task Decorators

Huey comes with multiple task decorators depending on what operations you are performing within the task. I'll explain in brief what all of those do below. Here is the import statement for all the decorators:

from huey.contrib.djhuey import task, periodic_task, db_task, db_periodic_task

task: A regular task
periodic_task : When you want to run a task periodically based on a schedule
db_task : When you want to perform db operations within your task
db_periodic_task : When you want to perform db operations in a periodic task

Crontab Examples

Let me show you some more examples of how you can use crontab to schedule your tasks.

crontab(minute='*/3') would schedule the task to run every three minute
crontab(hour='*/3', minute='5') would create a task that will run at 5 minute past every third hour.
crontab(minute='00', hour='10', month='*/2', day_of_week='*/5') would create a task that would run on every 5th day of the week, of every 2nd month at 10:00 AM.

Scheduling Tasks

For example, you have the following task defined inside tasks.py:

from huey.contrib.djhuey import task

@task()
def count():
    for i in range(10):
        print(i)

Now you want to call this task, but want it to run after 5 seconds, you can do the following:

count.schedule(delay=5)

delay parameter takes values in seconds, so if you want it to execute after 5 minutes specify 300 seconds.

Retrying tasks that fail

Suppose you add the following logic to our existing task:

@db_periodic_task(crontab(hour="18", minute="00"), retries=2)
def fetch_daily_word():
    r = requests.get(
        f"https://api.wordnik.com/v4/words.json/wordOfTheDay?api_key={settings.WORDNIK_API_KEY}")
    if r.status_code != 200:
        raise Exception("Unable to fetch data from Wordnik API") ## Add this logic
    else:
        data = r.json()
        Word.objects.get_or_create(
            word=data["word"],
            part_of_speech=data["definitions"][0]["partOfSpeech"],
            definition=data["definitions"][0]["text"],
            example=data["examples"][0]["text"]
        )

So we added the logic to check for status code of the response, and if it is something other than 200, it will retry that task upto 2 times. But these retries would happen without any time gap between the two attempts. Now what if you want to delay out multiple attempts of this task, we can do that by passing the retry_delay argument, it accepts values in seconds.

@db_periodic_task(crontab(hour="18", minute="00"), retries=2, retry_delay=10)

This will cause a 10 second delay between multiple attempts.

Development Mode

Huey comes with a default setting which makes working with Huey during development in Django easier. So whenever you have DEBUG=True present in your settings.py file, tasks will be executed synchronously just like regular function calls. The purpose of this is to avoid running both Redis and an additional consumer process while developing or running tests. You can read more about this here.

For this, we need to add the following line in settings.py:

HUEY = {}

However, if you want to override this behaviour, you can add the following Huey config instead:

HUEY = {
    "immediate": False
}

If you have the above config mentioned in settings.py while having DEBUG=True Huey will require you to setup Redis and run huey worker using run_huey command.

Celery vs Huey

Some observations about Huey as compared to Celery are:

Smaller footprint of dependencies as compared to Celery. Celery installs kombu and billiards along with it. Meanwhile, huey doesn't have any dependencies.
Lesser services needed to be run for periodic tasks, Celery requires running beat service and a worker service to work with periodic task meanwhile we only need to run one service using the run_huey command.

References

How to save Base64 encoded image to Django ImageField?

Idiomatic Programmers — Sat, 17 Jul 2021 12:47:50 +0000

I had to create an API that had to accept an image in Base64 encoded format and save it to Django image field, which requires an image file. So today I learned how to do this.

Step 1: B64 to PIL

The first step will be to convert this B64 image to an image file in memory, I will be using Pillow for this. You can install Pillow using this bash command.

$ pip install pillow

Make sure you have pip installed. If you're using Linux or Mac OS, then you might have to use pip3 instead of pip. But I'm using virtual env, so who cares.

after that, write this function.

import base64
import io
from PIL import Image

def decodeDesignImage(data):
    try:
        data = base64.b64decode(data.encode('UTF-8'))
        buf = io.BytesIO(data)
        img = Image.open(buf)
        return img
    except:
        return None

This function will return a PIL image if the B64 data is valid, or it will return None.

Step 2: PIL to Django ImageField

After getting the PIL image, wrote a function that will take this PIL image, convert it into a JPEG file buffer and then inserted it to Database instance using InMemoryUploadedFile

sources: InMemoryUploadedFile

What is InMemoryUploadedFile? This is basically a representation of the image in memory, that's as simple as it is.

from django.core.files.uploadedfile import InMemoryUploadedFile

img = decodeDesignImage(data)
img_io = io.BytesIO()
img.save(img_io, format='JPEG')
design.image = InMemoryUploadedFile(img_io, field_name=None, name=token+".jpg", content_type='image/jpeg', size=img_io.tell, charset=None)
design.save()

The InMemoryUploadedFile class takes 6 parameters:

File buffer: This is the JPEG format IO buffer I created in line 5.
field_name: There wasn't much information about this, if you have any idea then please comment below.
name: This is the name of the file, make sure you add .jpg at the end.
content_type: This is pretty self explanatory.
size: The size in bytes of the file.
charset: The character set, the file data is using. In this case None.

Now if you check your Media folder, you'll see that image as if you've uploaded it via a form.

So that's it for today's "Today I Learned"

How to download Pandas Dataframe as Excel or CSV in Django?

Idiomatic Programmers — Sat, 27 Mar 2021 03:00:00 +0000

Introduction

Pandas is an essential tool used by Python developers used for data analysis purposes, but what is the point of the analysis if we are not able to provide that insight to the end user. When Django and Pandas are used in conjunction we can create applications that are not only developed faster, but are also smart in utilizing the data that is collected from the user. I recently stumbled on one such use case, where the user interacts with the frontend and turns some filters and then the backend processes that data, and then provide the aggregated data as an Excel or CSV file to the user. I will be describing how I went about doing that below.

Prerequisites

In order to follow along this article, I am assuming you already have a Django project that is using Pandas library ready.

Requirements

In addition to Django and Pandas project ready, you would need to install Openpyxl.

Openpyxl

Openpyxl is a tool that allows to read and write Open Office XML formats such as Excel 2010 xlsx/xlsm/xltx/xltm files.

pip install openpyxl

Once installed, you can create Excel files natively from Python using a simple code like this.

from openpyxl import Workbook

workbook = Workbook()
sheet = workbook.active

sheet["A1"] = "ID"
sheet["B1"] = "Key"

sheet["A2"] = 1
sheet["B2"] = "Test Key"

workbook.save(filename="test.xlsx")

This will create a file called "test.xlsx". Try it out yourself.

HTTP Response

As you know, Django uses HTTP Request and Response to communicate with the client. Therefore all data must be converted to Byte String before sending it over to the client.

A Typical HTTP Response looks something like this.

For more information, checkout this article by Mozilla about HTTP Messages.

As you can see in the above picture, every HTTP response has a body section which is a Byte Array and a Header called Content-Type specifies what type of content it is and which character set the browser should use in order to decode the bytes. In the above case, the server is sending a text/html data and the browser need to use iso-8859-1 charset.

In our case, we are sending bytes that are related to Excel spreadsheets. So googling for a bit I found the content-type header for that is this. Source: Mozilla article about Common MIME Types

application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

What are MIME Types?

MIME is short for Multipurpose Internet Mail Extensions which is a standard that indicates the nature and format of a document sent via the Internet.

Important : Browsers use the MIME type, not the file extension, to determine how to process a URL, so it's important that web servers send the correct MIME type in the response's Content-Type header. If this is not correctly configured, browsers are likely to misinterpret the contents of files and sites will not work correctly, and downloaded files may be mishandled.

For more information, checkout MDN Article about MIME Types

The structure of a MIME Type is according to IANA (Internet Assigned Numbers Authority) is like this.

type/subtype

Therefore, in the MIME Type for Excel files, the type is " application", which belongs to any kind of binary data that cannot be directly decoded to a human-readable form (text, HTML, etc.). Files with type application require some sort of external or third party software to decode and read such as PDFs, Zip files, Excel files, etc.

Download Excel Files using Django and Pandas

Finally, we will see how we can send the Pandas dataframe to client as an excel file.

I am assuming that you already have the code for pandas so I will do some abstraction in a variable called data.

First we will import to the inbuilt library, BytesIO so that we can write the excel file as a Byte array.

BytesIO is a library using which we can write data onto system memory instead of writing them as a file.

from io import BytesIO

Then we will use Python context manager to open a Byte buffer on which we can write the excel file. Context Managers allow you to allocate and release memory precisely when you want it.

with BytesIO() as b
    with pd.ExcelWriter(b) as writer:
        # You can add multiple Dataframes to an excel file
        # Using the sheet_name attribute
      data1.to_excel(writer, sheet_name="DATA 1", index=False)
        data2.to_excel(writer, sheet_name="DATA 2", index=False)

    filename = "analytics_data.xlsx"

    # imported from django.http
    res = HttpResponse(
        b.getvalue(), # Gives the Byte string of the Byte Buffer object
        content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
    )
    res['Content-Disposition'] = f'attachment; filename={filename}'
    return res

We will be using context managers to open and close files because that's more efficient than manually closing and releasing the memory after use. If you already know how to export a set of data frames as an excel file, that code should look like this.

with pd.ExcelWriter('data.xlsx') as writer:
    data1.to_excel(writer, sheet_name="DATA 1", index=False)
    data2.to_excel(writer, sheet_name="DATA 2", index=False)

We will simply take this code and wrap it around a BytesIO context manager which gives us a memory buffer b as a file to work with. We will simply take that memory buffer and pass it to pd.ExcelWriter() class and the rest of the code will be the same this that context manager.

with BytesIO() as b
    with pd.ExcelWriter(b) as writer:
      data1.to_excel(writer, sheet_name="DATA 1", index=False)
        data2.to_excel(writer, sheet_name="DATA 2", index=False)

Now that we have our Excel data written in a memory buffer, we can simply convert that to ByteArray using the method b.getvalue() which we will pass as a parameter in Django HttpResponse with appropriate content_type that we discussed earlier.

filename = "analytics_data.xlsx"

# imported from django.http
res = HttpResponse(
  b.getvalue(), # Gives the Byte string of the Byte Buffer object
  content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
)

In addition to the Content-Type header, we also need to provide another header called Content-Disposition that tells the browser if we want to show the data in the browser download and save it as a local file as an attachment. We can also pass a filename with this header.

res['Content-Disposition'] = f'attachment; filename={filename}'
return res

Here is the complete code snippet for Django view to download the excel file.

def download_analytics_endpoint(request):
    with BytesIO() as b:
        data = get_analytics_data()

        with pd.ExcelWriter(b) as writer:
            data.to_excel(writer, sheet_name="Data", index=False)

        filename = f"analytics_data.xlsx"
        res = HttpResponse(
            b.getvalue(),
            content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
        )
        res['Content-Disposition'] = f'attachment; filename={filename}'
        return res

That is all, now you just have to create a Django URL for this function and you can download any file you want, just convert that file to a Byte string and use the correct Content-Type header.

Generative Adversarial Networks

Idiomatic Programmers — Thu, 11 Feb 2021 08:56:15 +0000

In 2014, Ian Goodfellow, who is currently working as Director of Machine Learning for Apple, published a paper called "Generative Adversarial Networks" or GAN for short. Which basically talked about a system of two neural networks, Generator and Discriminator, that can generate images or any data that is similar to provided datasets from essentially random noise.

The diagram below shows the basic architecture of a GAN.

Source: https://developers.google.com/

Generative Network takes some random noise and outputs some random noise. This output noise is passed to a discriminator along with the real image or data as a ground truth based on that both Discriminator and Generator is trained.

As you can see, the concept of GAN is very simple. When I started learning about GANs I thought I can easily implement one of those. But boy I was wrong. In reality training a GAN is extremely hard both Generator and Discriminator must be trained side by side if one overpowers the other it won't work, we will talk about all the problems you might face while training a GAN. But now let's look at a simple GAN in Pytorch. We will be using the MNIST Dataset for this post.

Understanding the Dataset

The MNIST dataset is a huge database of handwritten numbers from 0 to 9 used for Optical Character Recognition or reading numbers from an image.

This dataset consists of 28x28 images of handwritten numbers where each pixel contains either a zero or a one.

The computer vision extension of PyTorch, Torchvision, provides this dataset which we can download using the following code snippet.

mnist = datasets.MNIST(root='datasets', train=True, transform=transformations, download=True)

But before we can run this code, we need to import some libraries.

import torchvision.datasets as datasets
from torch.utils.data import DataLoader
import torchvision.transforms as transforms

You can install the torchvision library using this pip command.

pip install torchvision

The DataLoader class is used to load the data to memory in batches, this prevents your system from running out of memory while training.

Transforms class is used to make random augmentations to the image such as random rotation, resize, crop, etc. but for now we will only normalise the images to range from -1 to 1.

The entire function that you can copy paste is this.

def load_dataset():
    transformations = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, ), (0.5, ))
    ])
    mnist = datasets.MNIST(root='datasets', train=True, transform=transformations, download=True)
    return DataLoader(mnist, batch_size=32, shuffle=True)

Implementing the Generator

In this section, we will implement a generator network which will take a random noise vector of size 100 and convert it into a vector of size 784 which we will then convert to 28x28

class Generator(nn.Module):
    def __init__ (self):
        super(). __init__ ()
        self.gen = nn.Sequential(
            nn.Linear(100, 256),
            nn.LeakyReLU(0.01),
            nn.Linear(256, 784),
            nn.Tanh(), # make outputs [-1, 1]
        )

    def forward(self, x):
        return self.gen(x)

We will be implementing the original GAN created by Ian Goodfellow in this paper Generative Adversarial Networks.

Implemeting the Discriminator

Next, we will implement a discriminator that will take the vector of size 784 that may be generated or from a real image.

class Discriminator(nn.Module):
    def __init__ (self):
        super(). __init__ ()
        self.disc = nn.Sequential(
            nn.Linear(784, 128),
            nn.LeakyReLU(0.01),
            nn.Linear(128, 1),
            nn.Sigmoid(), # make outputs [0, 1]
        )

    def forward(self, x):
        return self.disc(x)

Hyperparameters

As I mentioned earlier, GANs are extremely hard to train, one of the reasons is that a GAN is very sensitive to the initial values or hyperparameters. You have to follow the original papers to get the training right. Otherwise, you might have to spend many days optimizing the hyperparameters.

config = {
    'device': "cuda" if torch.cuda.is_available() else "cpu",
    'lr': 3e-4,
    'epochs': 50,
        'batch_size': 32
}

In our case, we will be using the same parameters as it was said in the paper, that is, a learning rate of 0.0003 and 50 epochs.

What is a learning rate?

Simple answer, it's the rate at which a machine learning model learns. Smaller the number, the slower it learns and higher the number the faster it learns. For more info, check out our post about Perceptrons.

Training Time

Finally, it's time to actually generate some handwritten numbers, that is, train our GAN. First let's create the objects for Generator, Discriminator, and their optimizers, we will be using the Adam optimizers.

ADAM or ADA ptive M oment optimiser is an algorithm used to update the weights such that the overall error goes down.

disc = Discriminator().to(config['device'])
gen = Generator().to(config['device'])

optimiser_g = optim.Adam(params=gen.parameters(), lr=config['lr'])
optimiser_d = optim.Adam(params=disc.parameters(), lr=config['lr'])

We also need to define a loss function before we train, we will be using the Binary Cross Entropy loss function to calculate the error of our model.

Binary Cross Entropy (BCE) Loss

This loss formula is used to calculate the distance between two probability distributions.

BCE=ℓ(x,y)=L=(l1,…,lN),ln=−wn[yn⋅logxn+(1−yn)⋅log(1−xn)]

BCE = ℓ(x,y)=L=(l_1,…,l_N), l_n=−w_n[y_n⋅logx_n+(1−y_n)⋅log(1−x_n)]

BCE=ℓ(x,y)=L=(l1,…,lN),ln=−wn[yn⋅logxn+(1−yn)⋅log(1−xn)]

loss_fn = nn.BCELoss()

Now we will write our training step which is one cycle of generating and discriminating a handwritten number compare with an original number and update the models.

for epoch in range(config['epoch']):
    for batch_idx, (real, label) in enumerate(train_data):

        noise = torch.randn(config['batch_size'], 100).to(config['device']) # Create a random probability distribution
        fake = gen(noise) # Generate a fake number
        disc_real = disc(real).view(-1) # pass the real number through the discriminator
        lossD_real = loss_fn(disc_real, torch.ones_like(disc_real)) # calculate the loss for real image
        disc_fake = disc(fake).view(-1) # pass the fake number through the discriminator
        lossD_fake = loss_fn(disc_fake, torch.zeros_like(disc_fake)) # calculate the loss for fake image
        lossD = (lossD_real + lossD_fake) / 2 # calculate the average loss

        # update the weights for the discriminator
        disc.zero_grad()
        lossD.backward(retain_graph=True)
        optimiser_d.step()

        output = disc(fake).view(-1)
        lossG = loss_fn(output, torch.ones_like(output)) # calculate the error between the fake image and the true image

        # update the weights for the generator
        gen.zero_grad()
        lossG.backward()
        optimiser_g.step()

This code is heavily inspired by a youtube video by Aladdin Persson.

Find the complete code at our Github Repo

If you have any questions, feel free to comment below.

References

Goodfellow, Ian J., et al. “Generative Adversarial Networks.” ArXiv.org, 10 June 2014, arxiv.org/abs/1406.2661v1.
Kingma, Diederik P., and Jimmy Ba. “Adam: A Method for Stochastic Optimization.” ArXiv.org, 30 Jan. 2017, arxiv.org/abs/1412.6980.
“THE MNIST DATABASE.” MNIST Handwritten Digit Database, Yann LeCun, Corinna Cortes and Chris Burges, yann.lecun.com/exdb/mnist/.
Understanding Categorical Cross-Entropy Loss, Binary Cross-Entropy Loss, Softmax Loss, Logistic Loss, Focal Loss and All Those Confusing Names, gombru.github.io/2018/05/23/cross_entropy_loss/.

5 techniques to Idiomatic Python (Loops)

Idiomatic Programmers — Fri, 21 Aug 2020 06:44:53 +0000

Python is, in my opinion, one of the easiest and most versatile languages in computer science. If you write python code correctly, it is difficult to differentiate a python code from pseudo-code. But sometimes, in the process of writing the most beautiful python code, most developers forget one thing: the execution speed of the code.

You can write code that is so readable and a layman can confuse it as English written by someone with bad grammar skills, yet it is understandable. But that code takes more than 300ms to run. That might not be that much of a delay but in the programming world, it is a serious problem.

On the other hand, one can write the same code in a different idiom that it takes less than 10ms to run. But, even expert python developers undergo a hard time understanding it.

It is thus very imported, for a developer to strike a balance between these two extremes. These kinds of code are known as Idiomatic Code in the industry.

Idiomatic Code, by definition, is a code that does a common task in a common way for your language. In other words, idiomatic code is any code that is easy to read and at the same time, is extremely efficient.

I follow Raymond Hettinger (@raymondh) a lot on twitter, he is one of the core developers in the Python community and he has contributed a lot of code in the Python repository to make the python language more idiomatic.

In this blog, I will go through some of the techniques that will help you improve your python code. Starting with:

1. range()

Let's start from the obvious, say you need to iterate from 2 to 10. So if you're like an absolute beginner you might write something like this:

for i in [2,3,4,5,6,7,8,9,10]:
    print(i)

Something like this works and is very understandable, it is not a scalable approach. What if you want to loop through 2 to 2 million.

In situations like this, you can use the range(stop) or range(start, stop, [step,]) built-in function.

the range function automatically generates the list for you. For example, you can rewrite the above code as:

for i in range(2, 11):
    print(i)

Notice how I wrote start from 2 and end at 11 and not 10. This is because the range function loops till stop - 1. There is one more parameter in range function other than the start and stop, that is the step. Step determines how many numbers the range have to skip.

Let's say you need to print all even numbers from 2 to 10, in that case, the skip parameter will be 2. By default, it is 1.

for i in range(2,11,2):
    print(i)

# OUTPUT: 2 4 6 8 10

Now let's say that you need to loop through a list. With the knowledge of range(), you might do this:

cloths = ['shirt', 'hat', 'socks', 'shorts']

for i in range(len(cloths))
  print(cloths[i])

But in python, there is a better way:

cloths = ['shirt', 'hat', 'socks', 'shorts']

for cloth in cloths
  print(cloth)

2. zip()

Let's say you have two lists, colour and cloths, of different sizes and you want to pair them until the smaller list is over. In this case, you can write a code like this:

colours = ['pink', 'red', 'green']
cloths = ['shirt', 'hat', 'socks', 'shorts']

n = min(len(colours), len(cloths))
for i in range(n):
    print(colours[i], cloths[i])

# OUTPUT
# pink shirt
# red hat
# green socks

Which is good, but you can also use the in-built function zip(*iterables) the best thing about this function is that you can pass any number of lists, not just two. Let me rewrite the above code and then explain to you how zip works.

colours = ['pink', 'red', 'green']
cloths = ['shirt', 'hat', 'socks', 'shorts']

for colour, cloth in zip(colours, cloths):
    print(colour, cloth)

# OUTPUT
# pink shirt
# red hat
# green socks

You might have guessed, zip takes any number of lists and returns another list that contains one element from each list. As you can see, both version produces the same output but the second one looks cleaner.

3. reversed()

If you want to loop over a list in reverse order, the traditional method was something like this.

cloths = ['shirt', 'hat', 'socks', 'shorts']

for i in range(len(cloths)-1, -1, -1):
  print(cloths[i])


# Output
# -------
# shorts
# socks
# hat
# shirt

But with Python, you can use the builtin function called reversed(). Take a look at this example:

cloths = ['shirt', 'hat', 'socks', 'shorts']

for cloth in reversed(cloths):
  print(cloth)


# Output
# -------
# shorts
# socks
# hat
# shirt

The second one is cleaner and faster than the first one.

4. enumerate()

You want to loop through a list along with the index. It's pretty straight forward in traditional programming:

cloths = ['shirt', 'hat', 'socks', 'shorts']

for i in range(len(cloths)):
  print(i, cloths[i])


# Output
# -------
# 0 shorts
# 1 socks
# 2 hat
# 3 shirt

But there is a cleaner and more efficient way in python:

cloths = ['shirt', 'hat', 'socks', 'shorts']

for i, cloth in enumerate(cloths):
  print(i, cloth)


# OUTPUT
# -------
# 0 shorts
# 1 socks
# 2 hat
# 3 shirt

5. sorted()

You need to loop over a list in sorted order, instead of running a sorting algorithm, you can use the sorted() built-in method.

nums = [2,3,1,5,2]

for i in sorted(nums):
  print(i)

# OUTPUT
# ------
# 1
# 2
# 2
# 3
# 5

The python sorted function uses Tim Sort with an average complexity of n*log(n)

For reverse sorting, you can use the attribute

sorted(nums, reverse=True)

What you want to sort a list of strings.

cloths = ['shirt', 'hat', 'socks', 'shorts']

for cloth in sorted(cloths):
  print(cloth)

# OUTPUT
# ------
# hat
# shirt
# shorts
# socks

It will sort based on letters, but what if you want to sort based on length of the string, you can do that with key attribute. For example:

cloths = ['shirt', 'hat', 'socks', 'shorts']

for cloth in sorted(cloths, key=len):
  print(cloth)

# OUTPUT
# ------
# hat
# shirt
# socks
# shorts

So that was 5 techniques using with you can make your python code more idiomatic.

Fixing Up Permissions After Data Migration in Django

Udit Mittal — Tue, 11 Aug 2020 14:06:07 +0000

Hello djangonauts, today I wanna talk about a problem that I faced while working with Django. So here's the thing I just recently performed a major release at work, the database had to be changed since we had newer migrations and database structure. Naturally I had to import some data from my older database to the new one, which for most of the part could safely be done using fixtures (or so I thought). So I would like to share some tips and things that I learned recently because of it.

Tip #1

Always exclude the Content Type table while exporting a fixture for data migration.

It can be done as follows:

python manage.py dumpdata --exclude=contenttypes

Why? You may ask? It's because Django automatically creates these records when you perform the migrations for your apps. These records are basically used by django internally for schema management. So if you import them from the older database, Django would raise errors.

Tip #2

Now if you were like me, you might have dumped your permissions records as well into the fixture, that's where the fun part comes in.

Now since content type was created afresh for the newer database, so their ids were different from the one in my previous database. And the permissions records which had the foreign key field to the Content Type model, used the older ids, so this essentially messed up all my permissions.

Usually if you're not making use of these, you can simply ignore them (which I wouldn't recommend personally). But since I was using these to provide staff accounts to people on the Django Admin with custom permissions, this meant that they might not have access to the right data or tables in the admin. So I thought, maybe I should manually fix these records and get done with it, but there were like 300 permissions in there, since there were many models, so I pondered whether there might be a better way to do it instead of doing it manually.

After a little bit of research, I came across update_permissions, a custom management command provided by django extensions, a very useful third party package I was already using in my project.

So let me demonstrate how to use it:

python manage.py update_permissions --apps news

Here we are passing the app name, for which we want to update permissions for, which being news in my case.

You can also update permissions for multiple apps at the same time, as shown below:

python manage.py update_permissions --apps news, articles

This will create the permission records which are automatically created for you by Django when you create a new model, but got messed up in our case.

In my case, I deleted all the records in the I had previously present in my Permissions table on the database and created them afresh using the command I just showed you above.

So that's it for today's "Today I Learned", until next time.