DEV Community: Peter van der Does

Wait for celery to finish its tasks

Peter van der Does — Tue, 18 Oct 2022 15:13:00 +0000

When deploying a Python application you sometimes have to restart celery. If there is a long-running task you might not want to kill that task, but rather wait 'till the tasks have finished.

This function will pause the accepting of new tasks and wait 'till all tasks are finished.

from celery import Celery
from celery.app.control import Control

def celery_check():
    app = Celery("app") # Name of the main module
    control = Control(app)
    control.cancel_consumer("celery")  # queue name, must probably be specified once per queue, but we use a single queue
    inspect = control.inspect()
    while True:
        active = inspect.active()
        if len(active.items()) > 0:
            seconds = 10
            time.sleep(seconds)
        else:
            break

Human Readable Passwords

Peter van der Does — Wed, 01 Jun 2022 14:09:46 +0000

Introduction

For a Django application, I had to create passwords for users who sign up on a website. Generating passwords can be relatively easy.

Take a look at this function.

def generate_password(length=10):
 return "".join(
        random.choice(string.ascii_letters + string.digits) for _ in range(length)
    )

This is the result

>>> print(generate_raw_password())
>>> B61CbF9GdW

This is secure but hard to remember.

To make them more user-friendly I thought I would create a function that is inspired by https://xkpasswd.net/s/. This website creates human-readable passwords which are secure. An example is 3~shoes~FOUND@ and this is much more human-readable than the previous method.

Details

How did I go about creating the human-readable passwords?

Word List

I had to start with a list of words. With Wordle being so popular finding a list of 5-letter words is not that hard. There are many Worlde clones and GitHub has a whole bunch of Wordle clones with such a list. But in all honesty why not go to the source of the original Wordle and see if you can get their list.

NY Times Wordle

The NY Times bought the original Wordle and the source is no longer available. You can still get the list with a little bit of hacking.
Open this link: https://www.nytimes.com/games/wordle/main.bd4cb59c.js. Search for ,Oa= and there you have the start of the list. There are about 10K words in this list. When I checked the list I did find some words that people might find slightly inappropriate, d*cks and that's not the bird, b**by.

On your computer

I found another way to generate a list of 5-letter words. Most Linux-based machines and Macs have a words list on the system, /usr/share/dict/words. To extract all 5-letter words we can write a simple one-liner in Perl.

perl -nle 'print if /^[a-z]{5}$/' /usr/share/dict/words > words5.txt

And there we have a list. Unfortunately, the full list contains words from 1934 and I found some words in the list that we no longer use.

The Stanford GraphBase

Donald Knuth created a list of five-letter words. This list is used to run various combinatoric experiments, graph algorithms, and other algorithms to explore the relationships among these words. The list is in the public domain and has 5757 words.

This is the list I'm using right now to create the human-readable password.

Using the list

Instead of putting the word list in a Python list, I decided to put the list in a DB instead. This makes it easier to add more words later if that is preferred.

The model is super easy.

# code/word_model.py

class Word(models.Model):
    word = models.CharField(max_length=10, unique=True)

    def __str__(self):
        return self.word

I will leave it up to you on how to fill the model.

Requirements

The password must meet the following requirements

Prepend and append the password with at least 1 digit and at most 2 digits.
Separate each word by a special character.
Each word will go through a transformation of being either all lower case, all upper case, or capitalized.
Do not repeat the words within the password.

I also want to have the flexibility of using as many words as I want, by default I'll use 3 and will not allow less than two words.

Code

# code/password.py

import random

from models import Word


def create_password(words=3):
    """
    Generate a password

    Parameters
    ----------
    words: int

    Returns
    -------
    str
    """
    if words < 2:
        raise ValueError
    numbers = random.sample(range(1, 99), 2)
    special_character = "!@$%^&*-_+=:|~?/.;"
    separator = random.sample(special_character, words - 1)
    selected_words = get_random_words(words)
    password = str(numbers[0])
    for step in range(words - 1):
        password = password + transform_word(selected_words[step]) + separator[step]
    password = password + transform_word(selected_words[words - 1]) + str(numbers[1])
    return password


def get_random_words(amount):
    """
    Get a list of unique words

    Parameters
    ----------
    amount: int

    Returns
    -------
    list of str

    """
    total_word_count = Word.objects.all().count()
    ids = random.sample(range(1, total_word_count), amount)
    selected_words = list(
        Word.objects.filter(id__in=ids).values_list("word", flat=True)
    )
    while len(selected_words) < amount:
        id = random.randint(1, total_word_count)
        if id in ids:
            continue
        selected_word = Word.objects.get(id=id)
        if selected_word:
            selected_words.append(selected_word.word)
    return selected_words


def transform_word(word):
    """
    Transform a string

    Parameters
    ----------
    word: str

    Returns
    -------
    str
    """
    transform = random.randint(1, 3)
    if transform == 1:
        transformed_word = word.capitalize()
    elif transform == 2:
        transformed_word = word.lower()
    elif transform == 3:
        transformed_word = word.upper()
    return transformed_word

Some examples of passwords generated by this function:

73PYGMY|Glens~Naiad33
29radar|bylaw%SWATH19
5plant.Cages+Dizzy54
72DOUGH!Yolks-hooch28
92FLESH^churl%genet69

Conclusion

The code allows you to create human-readable passwords. It is easy to expand the words used by adding them to the database.

Notes

I use my selection of special characters instead of string.punctuation. I did not want certain characters from that list like the quotation marks.

References

NY Times Wordle: https://www.nytimes.com/games/wordle/index.html
The Stanford GraphBase: https://www-cs-faculty.stanford.edu/~knuth/sgb.html

Found a typo?

If you've found a typo, a sentence that could be improved or anything else that should be updated on this blog post, you can access it through a git repository and make a pull request. Instead of posting a comment, please go directly to GitHub and open a new pull request with your changes.

Photo by Colin Lloyd - https://unsplash.com/photos/62OEfKjU1Vs

Snippet: Change timezone for DateTime object

Peter van der Does — Thu, 26 May 2022 02:21:18 +0000

Information

Convert a Python DateTime object from one timezone to another timezone. This snippet accepts either a naive or aware DateTime object.

Snippet

# code/timezone.py

import datetime as dt
import pytz


def to_timezone(datetime_value, tz):
    """
    Convert the given datetime object to a datetime object with the given timezone.
    Accepts both aware and naive datetime objects

    Parameters
    ----------
    datetime_value: datetime.datetime
    tz: pytz.timezone

    Returns
    -------
    datetime.datetime
    """
    if not isinstance(datetime_value, dt.datetime):
        raise SyntaxError

    if not hasattr(tz, "zone") or tz.zone not in pytz.all_timezones:
        raise SyntaxError

    if (
        datetime_value.tzinfo is not None
        and datetime_value.tzinfo.utcoffset(datetime_value) is not None
    ):
        datetime_value = datetime_value.astimezone(tz)
    else:
        datetime_value = tz.localize(datetime_value)

    return datetime_value

Found a typo?

Photo by Ijaz Rafi - https://unsplash.com/photos/L4hg5o67jdw

Django + memcached + namespace

Peter van der Does — Tue, 24 May 2022 13:06:16 +0000

Introduction

One of the drawbacks of using memcached is that you can not delete keys using a wildcard. With Django, you can delete many keys but you have to give it a list of keys to delete.

A way around this is by using namespaces. You will organize keys by namespace. For example, every key related to the products you sell falls in the namespace product. And your clients are in a namespace client. If you delete the namespace all keys within the namespace will be deleted as well. Unfortunately, we run into another snag, namespaces are not supported by memcached.

With the code presented in this article we fake namespaces and have a way to "delete" all keys in a namespace

TL;DR Check out the code.

How do we do this?

When we store data in memcached you need a key. What we will do is prepend this key with the namespace we want to use. For the key guitar, we will prepend the key like this product:guitar.

Wait! How will we invalidate all keys in the namespace product? Instead of using product as the prefix, we will store the namespace in memcached with a value. We use that value as the actual namespace.

The namespace key for memcached will be namespace:product and we store the value 1. I hear you say it: "Ha, but that will cause a conflict with other namespaces". You are right so let us prepend this value with the namespace itself like this product1. Now the guitar key will look like this product1:guitar.
When we change the value of the product value to product2, the guitar key will be product2:guitar. Now all keys in the namespace product1 will no longer be reachable and memcached will take care of removing them.

Code

Wrapping this all up in a class to be used within Django

# code/cache.py

import time

import xxhash
from django.core.cache import cache as django_cache


IS_DEVELOPMENT = False
CACHE_PREFIX = "MyApp"
HOUR = 3600
DAY = HOUR * 24


class MyCache:
    """
    This class is used to create a cache for MyApp.
    """

    def __init__(self, timeout=DAY):
        """
        Initialize the cache.
        """
        self.timeout = timeout if not IS_DEVELOPMENT else 20

    def __str__(self):
        """
        Return a string representation of the cache.
        """
        return "MyCache"

    def get(self, namespace, key):
        """
        Get a value from the cache.

        Parameters
        ----------
        namespace: str
        key: str
        """
        try:
            cache_key = self.safe_cache_key(namespace=namespace, key=key)
            return django_cache.get(cache_key)
        except Exception:
            return None

    def set(self, namespace, key, value, timeout=None):
        """
        Set a value in the cache.

        Parameters
        ----------
        namespace: str
        key: str
        value: str|int|dict|list
        timeout: int or None
        """
        try:
            cache_key = self.safe_cache_key(namespace=namespace, key=key)
            timeout = timeout or self.timeout
            django_cache.set(cache_key, value, timeout)
        except Exception:
            pass

    def delete(self, namespace, key):
        """
        Delete a value from the cache.

        Parameters
        ----------
        namespace: str
        key: str
        """
        try:
            cache_key = self.safe_cache_key(namespace=namespace, key=key)
            django_cache.delete(cache_key)
        except Exception:
            pass

    def delete_namespace(self, namespace):
        """
        Delete the namespace

        Parameters
        ----------
        namespace:str
        """
        self.update_cache_namespace(namespace=namespace)

    def safe_cache_key(self, namespace, key):
        """
        Create a key that is safe to use in memcached

        Parameters
        ----------
        namespace: str
        value: str

        Returns
        -------
        str
        """

        namespace = self.get_namespace(namespace=namespace)
        new_key = "{}:{}:{}".format(
            CACHE_PREFIX, namespace, xxhash.xxh3_64_hexdigest(key)
        )
        return new_key

    def get_namespace(self, namespace):
        """
        Get the namespace value for the given namespace

        Parameters
        ----------
        namespace: str

        Returns
        -------
        str
        """
        key = self.get_namespace_key(namespace=namespace)
        rv = django_cache.get(key)
        if rv is None:
            value = self.get_namespace_value(namespace=namespace)
            django_cache.add(key, value, DAY)
            # Fetch the value again to avoid a race condition if another
            # caller added a value between the first get() and the add()
            # above.
            return django_cache.get(key, value)

        return rv

    def update_cache_namespace(self, namespace):
        """
        Update the value for the namespace key
        Parameters
        ----------
        namespace: str
        """
        key = self.get_namespace_key(namespace=namespace)
        value = self.get_namespace_value(namespace=namespace)
        django_cache.set(key, value, DAY)

    def get_namespace_key(self, namespace):
        """

        Parameters
        ----------
        namespace: str

        Returns
        -------
        str
        """
        return xxhash.xxh3_64_hexdigest(f"namespace:{namespace}")

    def get_namespace_value(self, namespace):
        """
        Create value for the namespace value

        The namespace is used to make sure the hashed value is unique

        Parameters
        ----------
        namespace: str

        Returns
        -------
        str
        """
        namespace_value = namespace + str(int(round(time.time() * 1000)))
        return xxhash.xxh3_64_hexdigest(namespace_value)


cache = MyCache()

Usage

Set a value

cache.set("product", "combined_pricing", 5000)

Get a value

pricing = cache.get("product", "combined_pricing")

Delete single key

pricing = cache.delete("product", "combined_pricing")

Delete namespace

cache.delete_namespace("product")

Conclusion

Using this code allows you to group memcached keys and invalidate the keys in that namespace all at once.

Notes

I hash all memcached keys as well as the namespace value because it is part of a key. Parts of the key are not predictable and could potentially have characters that are incompatible with a key.
Using time for the namespace value eliminates the need for an increase method.
I prefer xxhash over any other hasher, it's super fast.

References

xxhash at PyPi: https://pypi.org/project/xxhash/

Found a typo?

Photo by Colin Lloyd - https://unsplash.com/photos/62OEfKjU1Vs