DEV Community: Azanul Haque

Cooking concurrently with Python

Azanul Haque — Sun, 14 Apr 2024 06:32:48 +0000

Concurrency refers to the ability of a program to manage multiple tasks seemingly at the same time. While a single CPU core can only execute one instruction at a time, concurrency allows programs to juggle multiple tasks by rapidly switching between them. This creates the illusion of parallelism, enhancing responsiveness and improving performance for I/O bound workloads.

There are two ways one can achieve concurrency in Python:

Multithreading
Asynchronous Programming (Asyncio)

Let's explore them one by one. This time we’re running a restraunt, we're extending the example used in article. We're going to have 3 chefs this time.

1. Multithreading

Threads are lightweight units of execution within a single process. They share the same memory space and resources but execute instructions independently.
Multiple threads can be running concurrently, but due to the Global Interpreter Lock (GIL) in Python's traditional CPython implementation, only one thread can execute Python bytecode at a time. There are other Python implementations that one can choose from, depending on the use case.

Let's redefine ordered_dishes to also print a message when an order is made.

def ordered_dishes():
    for _ in range(no_of_customers):
        order = random.choice(menu)
        print(f"I- Ordered {order}")
        yield order

Let's define a function prepare_dish wrapping our food pipeline logic.

def prepare_dish(dish):
    for op in serve(cook(get_ingredients(dish))):
        print(item)
    print(f"O- Served {dish}")
    return f"Served {dish}"

Let's create a thread pool with 3 workers and map prepare_dish with the dishes being ordered.

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=3) as executor:
    future_dishes = executor.map(prepare_dish, ordered_dishes())

for dish in future_dishes:
    print(dish)

Full script

from concurrent.futures import ThreadPoolExecutor
import random

ingredients = {
    "Pizza": ["Flour", "Yeast", "Tomato Sauce", "Mozzarella Cheese", "Pepperoni"],
    "Spaghetti": ["Pasta", "Ground Beef", "Tomato Sauce", "Onion", "Garlic"],
    "Chicken Stir Fry": ["Chicken", "Broccoli", "Peppers", "Soy Sauce", "Ginger", "Rice"],
    "Chicken Biryani": ["Chicken", "Onion", "Garlic", "Rice", "Yogurt", "Spices"],
    "Salad": ["Mixed Greens", "Tomatoes", "Cucumbers", "Carrots", "Dressing"]
}

menu = list(ingredients.keys())

no_of_customers = random.randint(1, 10)

def ordered_dishes():
    for _ in range(no_of_customers):
        order = random.choice(menu)
        print(f"I- Ordered {order}")
        yield order

def get_ingredients(dish):
    print(f"Get ingredients for {dish}...")
    yield from (ingredient for ingredient in ingredients[dish])

def cook(dish_ingredients):
    for ing in dish_ingredients:
        print(f"Cooking ingredient {ing}...")
        yield f"Cooked {ing}"

def serve(cooked_items):
    for item in cooked_items:
        print(f"Seasoning {item}...")
        yield f"Served {item} with a pinch of magic"

def prepare_dish(dish):
    for op in serve(cook(get_ingredients(dish))):
        print(op)
    print(f"O- Served {dish}")
    return f"Served {dish}"

with ThreadPoolExecutor(max_workers=3) as executor:
    future_dishes = executor.map(prepare_dish, ordered_dishes())

for dish in future_dishes:
    print(dish)

2. Asynchronous Programming (Asyncio)

Asynchronous programming is a programming paradigm that allows for non-blocking operations. Instead of waiting for a task to complete before moving on to the next one, asynchronous programming enables tasks to run in the background, freeing up resources and potentially improving performance.
Python's asyncio module provides a way to write concurrent code using the async and await syntax. It is particularly useful for I/O-bound and high-level structured network code.

Let's make prepare_dish an asynchronous function by prefixing the definition with async keyword.
```
async def prepare_dish(dish):
    ...
```
Let's create a pool with 3 workers and call prepare_dish with the dishes being ordered, similar to what we did with threads. To attain this, we're going to use a Semaphore. async and await can only be used inside an async function.
```
semaphore = asyncio.Semaphore(3)

async def chef_task(dish):
    async with semaphore:
        return await prepare_dish(dish)
```

Wrap it all in an async main function and run it with asyncio.run. We use asyncio.create_task to run tasks concurrently. asyncio.gather aggregates the results of every coroutine passed to it.

async def main():
    future_dishes = []
    for dish in ordered_dishes():
        task = asyncio.create_task(chef_task(dish))
        future_dishes.append(task)

    served_dishes = await asyncio.gather(*future_dishes)

    for dish in served_dishes:
        print(dish)

asyncio.run(main())

Full script

import asyncio
import random

ingredients = {
    "Pizza": ["Flour", "Yeast", "Tomato Sauce", "Mozzarella Cheese", "Pepperoni"],
    "Spaghetti": ["Pasta", "Ground Beef", "Tomato Sauce", "Onion", "Garlic"],
    "Chicken Stir Fry": ["Chicken", "Broccoli", "Peppers", "Soy Sauce", "Ginger", "Rice"],
    "Chicken Biryani": ["Chicken", "Onion", "Garlic", "Rice", "Yogurt", "Spices"],
    "Salad": ["Mixed Greens", "Tomatoes", "Cucumbers", "Carrots", "Dressing"]
}

menu = list(ingredients.keys())

no_of_customers = random.randint(1, 10)

def ordered_dishes():
    for _ in range(no_of_customers):
        order = random.choice(menu)
        print(f"I- Ordered {order}")
        yield order

def get_ingredients(dish):
    print(f"Get ingredients for {dish}...")
    yield from (ingredient for ingredient in ingredients[dish])

def cook(dish_ingredients):
    for ing in dish_ingredients:
        print(f"Cooking ingredient {ing}...")
        yield f"Cooked {ing}"

def serve(cooked_items):
    for item in cooked_items:
        print(f"Seasoning {item}...")
        yield f"Served {item} with a pinch of magic"

async def prepare_dish(dish):
    for op in serve(cook(get_ingredients(dish))):
        print(op)
    print(f"O- Served {dish}")
    return f"Served {dish}"

semaphore = asyncio.Semaphore(3)

async def chef_task(dish):
    async with semaphore:
        return await prepare_dish(dish)

async def main():
    future_dishes = []
    for dish in ordered_dishes():
        task = asyncio.create_task(chef_task(dish))
        future_dishes.append(task)

    served_dishes = await asyncio.gather(*future_dishes)

    for dish in served_dishes:
        print(dish)

asyncio.run(main())

Pros & Cons

Feature	Asyncio	Multithreading (CPython)
Concurrency Model	Single-threaded, event loop driven	Multi-threaded
CPU-bound tasks	Not ideal (overhead for context switching)	Limited benefit due to GIL (serializes execution)
I/O-bound tasks	Well-suited (handles waiting efficiently)	Less efficient (overhead for thread management)
Memory Usage	Lower (fewer threads)	Higher (more threads)
Complexity	Simpler (fewer potential race conditions)	More complex (requires synchronization)
Error Handling	Easier to reason about errors	Error handling can be trickier (race conditions)
Learning Curve	Steeper (different programming paradigm)	Easier to learn (familiar thread concepts)

Moving forward

If you're not using CPython (e.g., Jython, IronPython), multithreading can potentially utilize multiple cores for CPU-bound tasks more effectively, depending on the implementation.
For highly CPU-bound tasks in CPython, consider libraries like multiprocessing which can leverage multiple cores by spawning separate processes instead of threads (processes don't share the GIL). We'll look deeper into multiprocessing in a different post.
Till then here are a few good resources to go through:

Cooking with python generators

Azanul Haque — Fri, 05 Apr 2024 05:51:59 +0000

Generators are a powerful concept in Python that offer memory-efficient ways to create iterators. Unlike traditional iterables like lists or tuples that store all values in memory at once, generators produce values on demand, making them ideal for handling large datasets, data processing pipelines or infinite sequences.

Let's take an example to dive into generators. Imagine that you're a chef and you don't want to prepare the ingredients beforehand as you don't know how many and which dishes are going to sell today. We're going to create a pipeline from ordering random dishes to preparing and serving them.

Define a menu and the ingredients required for each dish first.

ingredients = {
    "Pizza": ["Flour", "Yeast", "Tomato Sauce", "Mozzarella Cheese", "Pepperoni"],
    "Spaghetti": ["Pasta", "Ground Beef", "Tomato Sauce", "Onion", "Garlic"],
    "Chicken Stir Fry": ["Chicken", "Broccoli", "Peppers", "Soy Sauce", "Ginger", "Rice"],
    "Chicken Biryani": ["Chicken", "Onion", "Garlic", "Rice", "Yogurt", "Spices"],
    "Salad": ["Mixed Greens", "Tomatoes", "Cucumbers", "Carrots", "Dressing"]
}

menu = list(ingredients.keys())

Let's say we get between 1 to a hundered customers each day and each of them orders a random dish from the menu. We'll store it in a list ordered_dishes.

import random

no_of_customers = random.randint(1, 100)

ordered_dishes = []
for _ in range(no_of_customers):
    ordered_dishes.append(random.choice(menu))

We get a maximum of 100 customers per day so it's fine to store all the dishes in a list but what if we were to get a million customers per day. We don't want to have to store so much data in-memory till we process each dish. To fix it let's convert this to a generator expression to process one ordered dish at a time.
```
ordered_dishes = (random.choice(menu) for _ in range(no_of_customers))
```

To present the final dish, we need to first get the ingredients, cook them and serve the dish. All these functions don't really need to be generators but how else are we going to learn about generators.

def get_ingredients(dish):
  print(f"Get ingredients for {dish}...")
  yield from (ingredient for ingredient in ingredients[dish])

def cook(dish_ingredients):
  for ing in dish_ingredients:
    print(f"Cooking ingredient {ing}...")
    yield f"Cooked {ing}"

def serve(cooked_items):
  for item in cooked_items:
    print(f"Seasoning {item}...")
    yield f"Served {item} with a pinch of magic"

Now creating the pipeline and capturing the output.

chef_pipeline = ((dish, serve(cook(get_ingredients(dish)))) for dish in ordered_dishes)
for final_dish in chef_pipeline:
    for item in final_dish[1]:
        print(item)
    print("Served", final_dish[0], "\n")

Final script

import random

ingredients = {
    "Pizza": ["Flour", "Yeast", "Tomato Sauce", "Mozzarella Cheese", "Pepperoni"],
    "Spaghetti": ["Pasta", "Ground Beef", "Tomato Sauce", "Onion", "Garlic"],
    "Chicken Stir Fry": ["Chicken", "Broccoli", "Peppers", "Soy Sauce", "Ginger", "Rice"],
    "Chicken Biryani": ["Chicken", "Onion", "Garlic", "Rice", "Yogurt", "Spices"],
    "Salad": ["Mixed Greens", "Tomatoes", "Cucumbers", "Carrots", "Dressing"]
}

menu = list(ingredients.keys())

no_of_customers = random.randint(1, 100)

ordered_dishes = (random.choice(menu) for _ in range(no_of_customers))

def get_ingredients(dish):
  print(f"Get ingredients for {dish}...")
  yield from (ingredient for ingredient in ingredients[dish])

def cook(dish_ingredients):
  for ing in dish_ingredients:
    print(f"Cooking ingredient {ing}...")
    yield f"Cooked {ing}"

def serve(cooked_items):
  for item in cooked_items:
    print(f"Seasoning {item}...")
    yield f"Served {item} with a pinch of magic"

chef_pipeline = ((dish, serve(cook(get_ingredients(dish)))) for dish in ordered_dishes)
for final_dish in chef_pipeline:
    for item in final_dish[1]:
        print(item)
    print("Served", final_dish[0], "\n")

This is very practical for a one man operation but what if it were a restraunt, multiple waiters, mutiple chefs, serving multiple. In that case we'll have to dive into concurrency, that's a story for another day. Till then check out these links for more on generators:

Python context managers

Azanul Haque — Wed, 03 Apr 2024 12:39:30 +0000

The with statement in Python is primarily used for resource management. It ensures that a resource is properly initialized and released when it's no longer needed, even if exceptions occur during the execution. It can also be used for suppressing certain exceptions or outputs.
Basically, context-managers can be leveraged to create a temporary state change and revert it to original once the intended work is done.
A few common use cases for context-managers are:

File I/O Operations: ensuring that file is properly closed after use.
Network Connections: ensuring the connection is properly closed after use.
Transaction Management: managing transactions - beggining, commiting, aborting, etc.
Resource Management: automatically releasing the lock, freeing the resource after use.

Let's take an example to explore context-managers further. Let's say that we have a function which prints a message to standard output. We want to capture this output instead of releasing it to standard output.
Imagine you're a super spy with a secret message, but you don't want anyone to see it! That's where our special code comes in. It's like a magic box that hides messages until you open it again.

We need a secret message to hide. To do this we simply define a function which will print a string.
```
def secret_message():
    print("This message is secret and should be suppressed!")
```
Let's create our message box. Our message box is going to be a context manager, so it has to implement the necessary enter() and exit() methods. enter() is called at the start of the context and exit() is called at the end, duh! enter() will return an object which can be accessed with as keyword. exit() is called with 3 arguments: exception type, exception value and traceback, these values are going to be None is no exception occured in the context. In enter() method, we're storing the stdout in an instance variable _original and setting the stdout to a StringIO() object. In exit() method, we're resetting the stdout and storing the captured output in _original.
```
from io import StringIO
import sys

class SecretMessageBox:
    def __enter__(self):
        self._original = sys.stdout
        sys.stdout = StringIO()
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        sys.stdout, self._original = self._original, sys.stdout
```
Now, let's use our box to hide a message. We're using with statement wo invoke our context manager and as to capture the object returned by enter() method of the context manager SecretMessageBox into a variable secret_box.
```
print("Start hiding messages!")
with SecretMessageBox() as secret_box:
    secret_message()
print("Stop hiding messages!\n")
```

Once we've hidden the message, we want to read the message later

hidden_message = secret_box._original.getvalue()
if hidden_message:
    print("This is what we hid:\n", hidden_message)
else:
    print("There are no hidden messages yet.")

Putting it all together

from io import StringIO
import sys

def secret_message():
    print("This message is secret and should be suppressed!")

class SecretMessageBox:
    def __enter__(self):
        self._original = sys.stdout
        sys.stdout = StringIO()
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        sys.stdout, self._original = self._original, sys.stdout

print("Start hiding messages!")
with SecretMessageBox() as secret_box:
    secret_message()
print("Stop hiding messages!\n")

hidden_message = secret_box._original.getvalue()
if hidden_message:
    print("This is what we hid:\n", hidden_message)
else:
    print("There are no hidden messages yet.")

In case we face an exception but want to ignore certain exception or certain type of exceptions.
We define a RuntimeError, raise it from secret_message() function and handle it in exit() method.
True returned from exit() implies the exception raised has been handled else the exception is raised further down the sttack.

NOT_REALLY_AN_EXCEPTION = RuntimeError("Just kidding")

def secret_message():
    print("This message is secret and should be suppressed!")
    raise NOT_REALLY_AN_EXCEPTION
...

class SecretMessageBox:
...
    def __exit__(self, exc_type, exc_value, traceback):
        sys.stdout, self._original = self._original, sys.stdout
        if exc_value == NOT_REALLY_AN_EXCEPTION:
            return True
...

try:
    with SecretMessageBox() as secret_box:
        secret_message()
except Exception as e:
    print("Exception occurred:", e)
    sys.exit()
finally:
    print("Stop hiding messages!\n")

Final script

from io import StringIO
import sys

NOT_REALLY_AN_EXCEPTION = RuntimeError("Just kidding")

def secret_message():
    print("This message is secret and should be suppressed!")
    raise NOT_REALLY_AN_EXCEPTION

class SecretMessageBox:
    def __enter__(self):
        self._original = sys.stdout
        sys.stdout = StringIO()
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        sys.stdout, self._original = self._original, sys.stdout
        if exc_value == NOT_REALLY_AN_EXCEPTION:
            return True

print("Start hiding messages!")
try:
    with SecretMessageBox() as secret_box:
        secret_message()
except Exception as e:
    print("Exception occurred:", e)
    sys.exit()
finally:
    print("Stop hiding messages!\n")

hidden_message = secret_box._original.getvalue()
if hidden_message:
    print("This is what we hid:\n", hidden_message)
else:
    print("There are no hidden messages yet.")

Also checkout contextlib, it provides some common utilities involving the with statement.

Time To Sync

Azanul Haque — Thu, 20 Jul 2023 22:40:58 +0000

What I built

A task management tool based on RICE prioritisation framework.

Category Submission:

Best Overall Project
Most Technical Impressive
Best Project built using Supabase as the main data provider for the refine app.
Best Project built using Material UI as the main UI framework for the refine app.

App Link

https://prismatic-selkie-9d9316.netlify.app

Screenshots

Description

A task management app with

Github OAuth and Email authentication.
Realtime multi-user task editing using Supabase DB subscriptions
Ability to create multiple categories and sort tasks by colour
API security and authorization using RLS policies
German translation available

Link to Source Code

https://github.com/Azanul/time-to-sync

Permissive License

MIT

Background

I've always had a difficult time with proper categorisation and prioritisation of my tasks, hence, a task management application based on a framework that I prefer.

How I built it

Used refine's auth provider, data provider and live provider APIs to connect supabase.
The app uses MUI as the primary UI framework utilizing MUI grid for presenting filterable and sortable tables.
I got to learn how to leverage RLS policies to secure APIs and to provide authorization based data fetching. It was my first time with Refine and MUI as well, I was genuinly impressed by the extensibility Refine & MUI provide.

What is next for the project

RLS policy based multi-tenancy
Task assignments

Additional Resources/Info

RICE Prioritisation method

Code Gladiator

Azanul Haque — Tue, 23 May 2023 23:20:17 +0000

What I built

A PoC multiplayer game

Category Submission:

Wacky Wildcards

Description

In the game, a Github user can attack other players in the arena by creating an attack PR in which they present a problem and the ones who are attacked can defend themselves by creating a defence PR by correctly solving the problem in attack PR.

Link to Source Code

https://github.com/Azanul/code-gladiator

Permissive License

MIT

Background (What made you decide to build this particular app? What inspired you?)

Seemed fun

How I built it (How did you utilize GitHub Actions or GitHub Codespaces? Did you learn something new along the way? Pick up a new skill?)

Used GitHub Actions to automate everything from greeting to validation to merging.

Django ORM's Hidden Magic: Advanced Uses and Techniques, Part-1

Azanul Haque — Thu, 09 Feb 2023 18:55:39 +0000

I'm assuming you already know what Django is and what is ORM. If not:-

Django is a high-level web framework for building web applications quickly and with less code. It is written in Python and follows the Model-View-Template (MVT) architectural pattern. Django provides many built-in features such as a user authentication system, admin interface, and database abstraction layer, which makes it easier to develop web applications.

An important part of the database abstraction layer is ORM, which stands for Object-Relational Mapping. ORM is a feature of Django that allows you to interact with your database as if you were working with Python objects. Instead of writing raw SQL queries, you can use the Django ORM to perform database operations in a more intuitive and Pythonic way. The ORM provides a high-level API for creating, retrieving, updating, and deleting records in your database. It also supports complex operations such as aggregations, relationships, and custom lookups, allowing you to write efficient and scalable database code for your Django projects.

We'll be using the official Django polls app example.

from django.db import models


class Question(models.Model):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')


class Choice(models.Model):
    question = models.ForeignKey(Question, on_delete=models.CASCADE)
    choice_text = models.CharField(max_length=200)
    votes = models.IntegerField(default=0)

Now let's jump into the cool stuff:

Custom Lookups

We can define custom lookups to perform complex database queries. For example, we can create a custom lookup to search for questions that lie within a certain length range. You can define the lookup as:

from django.db.models import Field, Lookup

@Field.register_lookup
class LengthWithinLookup(Lookup):
    lookup_name = 'length_within'

    def as_sql(self, compiler, connection):
        lhs, lhs_params = self.process_lhs(compiler, connection)
        rhs, rhs_params = self.process_rhs(compiler, connection)
        lower_limit, upper_limit = rhs_params[0][1:-1].split(", ")
        return f"LENGTH({lhs}) BETWEEN {lower_limit} AND {upper_limit}", []

...

We can query questions with length within the range [16, 40] by running:

Question.objects.filter(question_text__length_within=(16, 40))

For more info on custom lookups visit this link.

Aggregation & Annotation

Django ORM provides built-in support for database-level aggregation operations, such as counting, summing, averaging, grouping and more. These operations can be performed using the django.db.models API. The aggregate and annotate methods allow for powerful data analysis and manipulation and can help you write cleaner, more efficient code. Let's see a few examples:

# Total number of questions.
Question.objects.count()

# Max votes across all choices.
from django.db.models import Max
Choice.objects.aggregate(Max('votes'))

# Annotate each question with the no. of choices for it
Question.objects.annotate(num_choices=Count('choice'))

# Annotate each question with the choice_text of choice with highest votes
from django.db.models import Subquery, OuterRef
Question.objects.annotate(
    choice_text=Subquery(
        Choice.objects.filter(question=OuterRef('pk'))
        .order_by('-votes')
        .values('choice_text')[:1]
    )
)

I recommend that you take a look at the official documentation if you think you need either aggregation & annotation as it's much more detailed there, here's the link. Feel free to comment below if you have any queries not covered there. We need to be careful about yielding wrong results while aggregating & annotating, check out this link. In the last example we've used a subquery, let's look further into it.

Subqueries

Subqueries in Django ORM allow you to use the results of one query as input for another query. You can perform complex operations by chaining multiple queries together, which can lead to more efficient and readable code.

Subqueries can be used in several ways in the Django ORM, including as a filter, as an annotation, or as an expression. To use a subquery as a filter, you can use the Subquery expression in the filter method. To use a subquery as an annotation, you can use the Subquery expression in the annotate method. To use a subquery as an expression, you can use it as an input to other expressions, such as F expressions (we'll look into these in the next part). Lets take an example, let's say you want to find all questions that have at least one choice with a vote count greater than 5. One way to achieve this would be to use the exclude method along with a subquery:

from django.db.models import Subquery, OuterRef

question_with_high_votes = Choice.objects.filter(
    votes__gt=5, question=OuterRef('pk')
).values('question')

Question.objects.exclude(
    pk__in=Subquery(question_with_high_votes)
)

reference link.

Model Inheritance

This section can be an entire article all by itself and we still won't be able to cover it all. I hope you're already familiar with OOP. Django ORM supports several forms of model inheritance. The main forms are:

Abstract Base Classes: An abstract base class is a model that is meant to be used as a base class for other models, but it won't be instantiated itself. It provides fields that can be reused by other models but it is not stored in the database. Let's update our Question and Choice models to have created_at and updated_at times. We'll do so by making both of them extend an abstract class BaseModel.

# Abstract class
class BaseModel(models.Model):
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    class Meta:
        abstract = True

class Question(BaseModel):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')

class Choice(BaseModel):
    question = models.ForeignKey(Question, on_delete=models.CASCADE)
    choice_text = models.CharField(max_length=200)
    votes = models.IntegerField(default=0)

Both Question and Choice will have all their fields and those of BaseModel .

Multi-table Inheritance: Multi-table inheritance allows a model to inherit fields and methods from multiple parent models. The fields and methods of the parent models are stored in separate tables in the database, with the child model having its table for its fields. Let's extend our example for this:

class BaseModel(models.Model):
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    class Meta:
        abstract = True

class CommunityModel(models.Model):
    author = models.CharField(max_length=100)

class Question(BaseModel):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')

class Choice(BaseModel):
    question = models.ForeignKey(Question, on_delete=models.CASCADE)
    choice_text = models.CharField(max_length=200)
    votes = models.IntegerField(default=0)

class CommunityChoice(Choice, CommunityModel):
    pass

CommunityChoice will have all fields from Choice as well as from CommunityModel .

Proxy models: Proxy models are models that inherit from an existing model, but don't add any fields or methods. They are typically used to change the manager for a model, or change the default ordering of the model's objects.
```
class PublishedQuestion(Question):
    class Meta:
        proxy = True

    def was_published_recently(self):
        return self.pub_date >= timezone.now() - datetime.timedelta(days=1)
```
In this example, PublishedQuestion is a proxy model that extends the Question model. The PublishedQuestion model has a method was_published_recently that checks if the question was published on the last day. Because PublishedQuestion is a proxy model, it has the same fields and behavior as the Question model, but can have additional methods and attributes.

Let me clear up any dangling confusion among these three. Abstract models do not create DB tables, neither do Proxy models. Proxy models inherit a solid (has a DB table) model class, Abstract models are inherited for adding extra fields.

Raw SQL Query

If nothing else works for you, but you know SQL then you can run raw SQL queries directly to the connected database. We don't even have to create a new connection. We use the same persistent connection created by Django. Let's fetch all questions published before 1st Feb 2023.

from django.db import connection


cursor = connection.cursor()
cursor.execute("SELECT * FROM polls_question WHERE pub_date < %s", ['2023-02-01'])

result = cursor.fetchall()

cursor.close()

But what if we wanted Question objects instead of raw query results. We can get it like this:

Question.objects.raw("SELECT * FROM polls_question WHERE pub_date < %s", ['2023-02-01'])

We can even fetch question data from a different table, all we need is to provide the mapping from the source fields to destination fields if fields are differently numbered or named. Let's take a separate table

class NotQuestion(models.Model):
    question_text = models.CharField(max_length=200)
    due_date = models.DateTimeField('due date')
    category = models.CharField(max_length=100)

Now there are still two ways to do this:

field_map = {'due_date': 'pub_date'}
Question.objects.raw('SELECT * FROM polls_notquestion', translations=field_map)

This way we get the fields from NotQuestion in Question objects. The other way is to use the AS syntax in SQL.

Question.objects.raw('SELECT question_text, due_date AS pub_date FROM polls_notquestion')

Conclusion

Soon enough I'll add the next part to it, in which we'll cover Relationships, Prefetch, F expressions, Transactions and Indexing & Caching. To get notified when it releases, follow me on Twitter at AzanulZ or connect with me on LinkedIn, Azanul Haque.

MongoDB Atlas Hackathon 2022 on DEV HostelMan

Azanul Haque — Thu, 08 Dec 2022 23:14:53 +0000

What I built

HostelMan: A hostel allotment and management system. Digitised the process of hostel room allotment.

Category Submission:

Think Outside the JS Box

App Link

No Live Deployment

Screenshots

Description

3 microservices

Application: here students apply and upload their documents and then id of this form is sent for document verification through kafka.
Consumer: here we're consuming the Kafka messages and marking forms as verified.
Payment: here the student can pay for their form and finalize their allotment if it has been verified.

Link to Source Code

https://github.com/Azanul/HostelMan

Permissive License

MIT License

Background

(What made you decide to build this particular app? What inspired you?)
Mine and every other students' frustration and wastage of countless hours led me to develop this. I've created this to ease the process for both students and administration while also providing transparency.

How I built it

(How did you utilize MongoDB Atlas? Did you learn something new along the way? Pick up a new skill?)
Used MongoDB Atlas:

as primary database for storing students, staff and rooms info
for storing payment reference per form

Also learnt:

in depth docker networking
in depth Rust dependency management
hands on Kafka, plus, using it as a message queue

Additional Resources/Info

Currently uses stripe as payment provider
I intend to extend this project to

support multiple payment providers
enable AI based document verification
support real-time availablity update
support full text search over students collection for admins

The case of missing message ID

Azanul Haque — Fri, 04 Nov 2022 16:22:37 +0000

Intro

This article assumes that you know how modern email works. Even if you don't, you'll get what the issue was and you also might get some idea by the end of this article. The title might seem unrelated but the issue occured when google MTAs stopped putting message
Emails sent by Mattermost (An opensource Slack alternative) were being marked as spam by many spam filters. This was happening because the emails were missing message-id which according to the Internet Message Format specification, is optional but still should be present [mixed signals!].

What needed to be done

Add message-id to all outgoing email as specified in Identification Fields:

a good method is to put the domain name (or a domain literal IP address) of the host on which the message identifier was created on the right-hand side of the "@" (since domain names and IP addresses are normally unique), and put a combination of the current absolute date and time along with some other currently unique (perhaps sequential) identifier available on the system (for example, a process id number) on the left-hand side.

simply, the format was going to be:
<unique_string-date_time@host_name>

Write unit tests to verify that a message-id is always present.

How I did it

unique_string: A function was already written to generate a random string, so that's the one that I used. Although, a random string is not guaranteed to be unique as the same string can be generated again (the probability of repetition of a 16 char random string which has all 10 digits is ideally once out of 10^16).
date-time: Used Go's time package to generate timestamp from current time object.
host_name: There is a SMTPConfig object which has all the SMTP config including host name.
So, joining all these using fmt.Sprintf:

msgID := fmt.Sprintf("<%s-%d@%s>", model.NewRandomString(randomStringLength), time.Now().Unix(), config.Hostname)

Test case: The was a test case which ensured that emails without message-id are allowed. I just edited it to validate if every email has a message-id.

Next step

Have any questions/suggestions? leave a comment.
Never used mattermost? Give it a go.
Want to contribute like I did? Here's the repo.

References

Issue: Still having MTA for Googlemail with 7.2.0 #20828
Pull-request: Mm 46416 add default random message #21176

Hacktoberfest 2022 Rewind ▶▶▶

Azanul Haque — Fri, 04 Nov 2022 16:21:08 +0000

Introduction

Hacktoberfest 2022 just ended and I have decided to share this adventurous experience with you. I had contributed to open source in previous hacktoberfests as well but this time it was different. This time I was preparing for it in Preptember September.
This time, I wanted to contribute to as many different projects and as many different organizations as possible...

Swag

Although this wasn't the main aim, I collected a lot of swag this time. Especially because I used hacktoberfest-swag for scouting good organizations participating in hacktoberfest.

Contibutions

I'll be covering all my contributions in a tutorial like format. I'll be doing so in a series of articles which I'll keep linking here. I'll also be adding Github issue and PR references if you want to look into the entire process as well.

The Adventure

This time, my contributions ranged from writing documentation to learning a new programming language & writing a entire runtime in it.
I had to go Plus Ultra! [ Anime reference ] to make so diverse contributions. Making so many open source contributions while leading bigdata development at my day job and still keeping streaks at leetcode and duolingo was extremely difficult. My entire plan hinged on proper time management. I still fell a couple issues short as I wasn't able to do their PR on time.
I got to experience what it'd feel like to work in these organizations. From bad documentation & bad maintaining to awesome documentation & great maintaining. I'm going to keep contributing to some of the projects and I just want to wipe off the memory of a couple of other projects.
It's all part of the journey...

November & beyond

I plan on working on my website, writing tech related articles, pumping leetcode, working on my origami skills and ofcourse, contributing to open source.
A sword that I made right before writing this article

Here's my Github profile:

Azanul (Azanul Haque) · GitHub

WW91IGhhdmUgdG9vIG11Y2ggZnJlZSB0aW1l . Azanul has 115 repositories available. Follow their code on GitHub.

github.com