DEV Community: Dmitrii Doroshev

This Week in Changelogs: curl

Dmitrii Doroshev — Tue, 24 Mar 2026 00:23:17 +0000

Hey everyone, long time no see!

I started TWiC in 2023, and to be honest, mining diffs manually was exhausting; that's why the project faded away pretty quickly. Today, with a little bit of LLM help and automation, it has become much easier to find hidden gems in modern OSS and bring them to the audience. There's another problem though: sometimes there are too many gems, and I definitely don't want to restart a series of boring longreads.

So today, we're gonna cover recent changes in only one project, (arguably) the most popular library and command-line tool in the world - curl.

Zip bomb protection via delivered-bytes tracking

Commit, PR

A zip bomb is a relatively small compressed piece of data that is automatically decompressed on the client side into a giant blob, causing a DoS. One of the ways to protect against it is to set a limit on the incoming data size. Curl has had an option for it, CURLOPT_MAXFILESIZE, since 2003.

There was one small problem: before 8.20.0 it only affected the number of downloaded (compressed) bytes. The good news is that now they calculate delivered (decompressed) bytes separately, extending CURLOPT_MAXFILESIZE to act as zip bomb protection:

/* check that the 'delta' amount of bytes are okay to deliver to the
   application, or return error if not. */
CURLcode Curl_pgrs_deliver_check(struct Curl_easy *data, size_t delta)
{
  if(data->set.max_filesize &&
     ((curl_off_t)delta > data->set.max_filesize - data->progress.deliver)) {
    failf(data, "Would have exceeded max file size");
    return CURLE_FILESIZE_EXCEEDED;
  }
  return CURLE_OK;
}

Btw, check out the pull request itself: five AI bots and zero people besides the author (Daniel himself) were involved in the code review! And it actually helps a lot, because, well, check out the next one:

API bitmask copy-paste bug

Commit, PR

CURLMNWC_CLEAR_DNS and CURLMNWC_CLEAR_CONNS were both defined as (1L << 0) (or just 1), so you could never clear just DNS or just connections - always both.

The bug was introduced mid-2025 (PR, diff), and despite having 3 people involved as code reviewers and 2 approvals, it made it into 8.16.0 to 8.19.0:

Luckily, Codex Security found it, and the author of the original changeset came up with an elegant solution, introducing CURLMNWC_CLEAR_ALL as the new (1L << 0) (since that's what every existing caller was unknowingly doing), and reassigning DNS and CONNS to bits 1 and 2:

if(val & CURLMNWC_CLEAR_ALL)
  /* In the beginning, all values available to set were 1 by mistake. We
     converted this to mean "all", thus setting all the bits
     automatically */
  val = CURLMNWC_CLEAR_DNS | CURLMNWC_CLEAR_CONNS;

So yeah, if you can afford to involve complex static analyzers or modern AI-reviewers, you'd better do that, because even seasoned lead devs can miss such small issues.

Missing `return`

Commit, PR

Long story short, there was a helper called return_quote_error(). Thanks to its misleading name, the actual return statement was forgotten before the call, causing segfaults.

This is a classic case of a fix so small you're 100% sure you're right, you skip the review and just merge broken code:

I personally like that apart from fixing this problem, they removed the return_ prefix from the function name to avoid future confusion. Of course, such problems will also be caught by AI checkers, but here's another story:

PowerPC64 endianness AI-slop

Commit - proudly Co Authored By Claude Opus 4.6 (1M context) (not kher sobachiy!)
Revert

A PR from a user named AutoJanitor with a robot avatar and shitloads of contributions since the end of January 2026 added __powerpc64__ to the list of architectures that can use unaligned memory access for MD5/MD4 fast paths with the following description:

Currently the fast path only covers __i386__, __x86_64__, and __vax__. PowerPC64 (both LE and BE) (sic! - @dima) supports efficient unaligned memory access and should use the same optimization.

<...>

Verified on IBM POWER8 S824 (ppc64le) — unaligned 32-bit loads work correctly and produce single lwz instructions.

To understand what it means, let's check out the code itself:

/*
 * SET reads 4 input bytes in little-endian byte order and stores them
 * in a properly aligned word in host byte order.
 *
 * The check for little-endian architectures that tolerate unaligned memory
 * accesses is an optimization. Nothing will break if it does not work.
 */
#if defined(__i386__) || defined(__x86_64__) || \
    defined(__vax__) || defined(__powerpc64__)
                  /* NB ^^^^^^^^^^^^^^^^^^^^^^ */
#define MD4_SET(n) (*(const uint32_t *)(const void *)&ptr[(n) * 4])
/* ... */

So this is an optimization for little-endian architectures, and, as pointed out later, causes undefined behavior.

Anyway, the contribution looked rock-solid, got merged, and a few days later, a real person (the guy has a website with /~sam/ - he knows his shit) came into the original PR, a discussion happened, and the commit was reverted.

The thing is that while PowerPC can run in either endianness, the preprocessor check __powerpc64__ doesn't tell you which one. The optimization above is exclusively for little-endian cases (casting byte arrays directly to uint32_t *, causing unaligned memory access), so on big-endian PowerPC64 it produced wrong results or just failed.

Unfortunately, AI bots have once again ruined things slightly for Daniel Stenberg. Apparently, that's the fate of popular projects these days ¯\_(ツ)_/¯

Thank you, folks! I hope you enjoyed this "episode" of This Week in Changelogs! Next time, fewer AI slop problems and more things to actually learn from.

The Pitfalls of Reading User Input in C: a Story About scanf and Stdin

Dmitrii Doroshev — Thu, 11 Dec 2025 21:40:45 +0000

I recently had to write a piece of C code that takes some input from stdin, ignores the newline, discards whatever exceeds the buffer, and does this repeatedly in a loop.

Knowing something about scanf syntax (man) I came up with this:

#include <stdio.h>

void take_input(void)
{
    char buf[20];
    printf("> ");
    scanf("%19[^\n]", buf);
    printf("input=`%s`\n", buf);
}

int main(void)
{
    for (int i = 0; i < 5; i++) {
        take_input();
    }

    return 0;
}

(Note: The original code used an infinite loop, but a simple for is enough to demonstrate the behavior.)

When I ran it, the result was surprising:

$ gcc -o main main.c
$ ./main
> hello world↵
input=`hello world`
> input=`hello world`
> input=`hello world`
> input=`hello world`
> input=`hello world`
$ █

It consumed the string once and printed the same value 5 times. Why?

1. The Stack (Ghost Data)

Stack behavior explains the first confusing part. Although char buf[20] is a locally scoped variable, in this particular case, each invocation of take_input() ends up using the exact same stack memory address.

Printint the address makes this undeniable:

/* ... */
printf("address=%p, input=`%s`\n", (void*)buf, buf);
/* ... */

$ ./main
> hello world
address=0x7ffcb96c49d0, input=`hello world`
> address=0x7ffcb96c49d0, input=`hello world`
> address=0x7ffcb96c49d0, input=`hello world`
<...>

Because the buffer wasn't initialized, and because subsequent scanf calls failed to overwrite it, the old content (the "ghost data") remained unchanged. Initializing the buffer with zeroes fixes the specific issue:

/* ... */
char buf[20] = {0};
/* ... */

Now the buffer resets each time, but scanf still behaves oddly:

$ ./main
> hello world
address=0x7ffea5c2d840, input=`hello world`
> address=0x7ffea5c2d840, input=``
> address=0x7ffea5c2d840, input=``
<...>

2. Scanf, Scansets and Stdin

The real enemy here is the scanset %19[^\n]. It reads up to 19 characters (len(buf) - 1) that are not a newline, but it does not consume the newline itself.

Let's open gdb and inspect stdin to see the wreckage:

$ gcc -g3 -O0 main.c -o main
$ gdb main
(gdb) break 8
Breakpoint 1 at 0x11d3: file main.c, line 8.
(gdb) run
<...>
> hello world↵
<...>
(gdb) p *stdin
$1 = {
    _flags = -72539512,
    _IO_read_ptr = 0x55555555972b "\n",
    _IO_read_end = 0x55555555972c "",
    _IO_read_base = 0x555555559720 "hello world\n",
    <...>
}

stdin->_IO_read_ptr, the current position of input, still points to the newline. It was not consumed by scanf; it is still sitting in stdin, waiting to be digested.

When the loop runs again, scanf("%19[^\n]", buf) sees the newline immediately, matches zero characters, and aborts. The buffer stays empty (or zeroed), and the loop spins.

Unlike Python's expression input()[:19], which consumes the newline and truncates the string cleanly, C's scanf cannot express that behavior using a single scanset. I found three workable solutions.

Solution A: The "Double Tap" (Extra `scanf`)

We can use extra scanf calls to force the consumption of the remainder of the line and the newline:

void take_input(void)
{
    char buf[20];
    printf("> ");
    scanf("%19[^\n]", buf);
    /* discard the rest of line, if it's >19 chars */
    scanf("%*[^\n]");
    /* discard the newline */
    scanf("%*c");
    printf("input=`%s`\n", buf);
}

It looks messy, but it works reliably for whitespace, short inputs, and long truncated inputs:

$ ./main
> hey
input=`hey`
> hello world
input=`hello world`
> a b c d e f g h i j k l m o p q r s t u v w x y z
input=`a b c d e f g h i j`
> ␣␣␣␣␣    /* 5 whitespaces */
input=`     `
> abcdefghijklmnpoqrstuvwxyz
input=`abcdefghijklmnpoqrs`

Solution B: `fgets` (The Standard Way)

A more predictable and explicit approach is fgets:

#include <string.h>

void take_input_2(void)
{
    char buf[20];
    printf("> ");

    if (fgets(buf, sizeof(buf), stdin)) {
        char *newline_ptr = strchr(buf, '\n');
        if (newline_ptr) {
            /* replace \n with \0 to trim it */
            *newline_ptr = '\0';
        } else {
            /* no newline found, meaning the input
               was truncated; we must consume
               the rest of the line from stdin */
            int c;
            while ((c = getchar()) != '\n' && c != EOF);
        }
    }
    printf("input=`%s`\n", buf);
}

Solution C: `getline` (The Heap Way)

There is a handy getline function (man), which is not in ISO C but exists in POSIX. The difference is that getline scans the whole line and allocates the buffer on the heap. You don't have to care about buffer boundaries anymore, but you do need to manually free the memory:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

void take_input_3(void)
{
    /* initialize the pointer with NULL */
    char *line = NULL;
    /* getline will return the updated capacity of `line`
       - in practice it grows dynamically */
    size_t cap = 0;
    ssize_t n;

    printf("> ");

    /* n contains the actual length of the input line,
       or -1 on failure or EOF */
    n = getline(&line, &cap, stdin);
    if (n > 0) {
        /* remove newline */
        if (line[n-1] == '\n') {
            line[n-1] = '\0';
        }
        printf("input=`%s`\n", line);
    }

    free(line);  /* NB: always free the heap memory */
}

Outcomes

In the end, the exercise became a small reminder of how surprisingly tricky input handling in C can be. scanf looks convenient until you hit edge cases around whitespace and newlines. At that point, "convenience" becomes a liability.

The takeaway is simple: know your tools. If you need predictable, portable, line-oriented input with truncation handling, fgets is almost always the better choice. If you have the luxury of POSIX and heap allocation, getline offers freedom. But scanf? Save that for when you know exactly what the input looks like.

Update 2025-12-14

As Paul J. Lucas mentioned, there's no good solution for handling a large or endless line:

The problem with getline is that malicious input could send a very long line and crash your program due to running out of memory.

<...>

The only safe way is to use fgets <...> — but this assumes malicious input even has a newline, so you might not crash, but could be looping forever. You could also check the number of characters read and finally exit in defeat if you've exceeded some arbitrarily large number.
There's no perfect solution when you can't trust your input (which you never should).

So all the examples above will fail if you run something like this:

$ yes | tr -d '\n' | ./main

take_input (scanf) and take_input_2 (fgets + while ((c = getchar()) != '\n' <...>) will loop forever, and take_input_3 will eventually cause an OOM condition.

However, we can slightly change the while loop in the fgets implementation, as recommended, crashing the program if the length of the skipped "tail" is longer than we can reasonably expect:

#define MAX_TAIL_LENGTH (10 * 1024)

void take_input_2(void)
{
    char buf[20];
    printf("> ");

    if (fgets(buf, sizeof(buf), stdin)) {
        char *newline_ptr = strchr(buf, '\n');
        if (newline_ptr) {
            *newline_ptr = '\0';
        } else {
            int c;
            int i = 0;
            while ((c = getchar()) != '\n' && c != EOF) {
                i++;
                if (i > MAX_TAIL_LENGTH) {
                    fprintf(stderr,
                            "The input string is unexpectedly long.\n");
                    abort();
                }
            }
        }
    }
    printf("input=`%s`\n", buf);
}

$ yes | tr -d '\n' | ./main
The input string is unexpectedly long.
Aborted

This is an explicit, defensive failure mode rather than a silent hang.

Building a Toy Database: Concurrency is Hard

Dmitrii Doroshev — Sat, 16 Aug 2025 21:53:25 +0000

Hey folks, it's been a while. Here's some news on Bazoola, my precious toy database, which already reached version 0.0.7!

Demo App

Meet the "Task manager" - a flask-based app demonstrating current capabilities of Bazoola:

In order to run the task manager:

git clone https://github.com/ddoroshev/bazoola.git
cd bazoola
pip install bazoola
python demo/app.py

Then open http://127.0.0.1:5000 in your browser, and Bob's your uncle. The schemas and tables are all pre-created and filled with demo data, so you can browse and edit it, checking the explanation at the bottom of the page. For example, you can try to search design-related projects and tasks by "des" query:

If you scroll down a bit, there's an explanation of how it actually works:

I'm still working on the TEXT field implementation, so for now all descriptions and comments are basically CHAR(200), but this will change soon.

Also, cherry on top - this demo app has been almost entirely generated by ✨Claude Code! That was a huge boost for me, because it helped focus on the essentials, delegating the boring stuff to AI.

Concurrency

Finally, this thing can be run in concurrent environment: in parallel threads or even in parallel processes. There are no parallel operations though, but at least the data won't corrupt if you run two apps using the same data directory simultaneously.

The key is a global lock. In a nutshell:

class DB:
    # ...
    def insert(self, table_name, values):
        with self.storage.lock():
            # ...
            return tbl.insert(values)

    def find_all(self, table_name, *, joins=None):
        # ...
        with self.storage.lock():
            rows = self.tables[table_name].find_all()
            # ...
        return rows

    def find_by_id(self, table_name, pk, *, joins=None):
        # ...
        with self.storage.lock():
            row = self.tables[table_name].find_by_id(pk)
            # ...
        return row

    def delete_by_id(self, table_name, pk):
        with self.storage.lock():
            self.tables[table_name].delete_by_id(pk)

    def update_by_id(self, table_name, pk, values):
        with self.storage.lock():
            # ...
            return self.tables[table_name].update_by_id(pk, values)

Actually, there are two global locks under the hood:

A .lock file, essentially a global mutex shared between the processes via flock.
A threading.RLock object, shared between threads of a single process. However, flock is not enough, because it's basically useless within the same process.

# storage.py
class File:
    # ...
    @contextmanager
    def lock(self):
        try:
            fcntl.flock(self.f.fileno(), fcntl.LOCK_EX)
            yield
        finally:
            fcntl.flock(self.f.fileno(), fcntl.LOCK_UN)
# ...

class TableStorage:
    def __init__(self, cls_tables, base_dir: str):
        # ...
        self._threadlock = threading.RLock()
        self._lockfile = File.open(".lock", base_dir=base_dir)
        # ...

    @contextmanager
    def lock(self):
        with self._threadlock, self._lockfile.lock():
            yield

It's not about atomicity, it doesn't solve data consistency in general, i.e. if you unplug your computer in the middle of a write operation, you'll most likely lose some ¯\_(ツ)_/¯

Elimination of `find_by` and `find_by_substr`

Initially, I had two specific functions:

find_by(table_name, field_name, value) - an equivalent of WHERE field_name == value
find_by_substr(table_name, field_name, substr) - WHERE field_name LIKE substr

However, there's already find_by_cond(), allowing to search by arbitrary conditions, defined in cond.py.

So I just added EQ, SUBSTR and ISUBSTR and removed these two helpers in favor of find_by_cond(table_name, EQ(field=value)) and find_by_cond(table_name, SUBSTR(field=substr)).

One More Thing

One of the features of this database at the moment is it's size - its core implementation is less than 800 lines of Python code! I decided that it would be nice to highlight it in the Github description, and came up with a simple script, update_description.sh, which I run once in a while to count the lines of bazoola/*.py and update the repo description.

I didn't find out yet how to run it on CI, but I think this one is already good enough.

Conclusion

I treat this update as a huge leap: a month ago Bazoola was barely usable in real-life applications, and now you can build something on top of it. There's no VARCHAR or TEXT fields, the amount of conditions (EQ, LT, GT, ...) is quite limited, selecting particular fields and ordering is also not implemented, and that's why it's just a 0.0.7 release :)

I like the fact that the data files contain mostly plain text, it's easy to test and debug the logic and data consistency. Of course, I'm planning to go binary and to switch from Python to C, and those benefits disappear, but by that time I'll have a solid amount of tests and a bunch of tools for managing the data.

Stay tuned!

Building a Toy Database: Learning by Doing

Dmitrii Doroshev — Wed, 16 Jul 2025 12:49:18 +0000

Ever wondered how databases work under the hood? I decided to find out by building one from scratch. Meet Bazoola - a simple file-based database written in pure Python.

Why Build a Toy Database?

As a developer, I use relational databases every day, but I never truly understood what happens when I INSERT or SELECT. Building a database from scratch taught me more about data structures, file I/O, and system design than any tutorial ever could.

Plus, it's fun to implement something that seems like magic!

What is Bazoola?

Bazoola is a lightweight, educational database that stores data in human-readable text files. It's not meant to replace SQLite or PostgreSQL - it's meant to help understand how databases work.

Key Features:

Fixed-width column storage
CRUD operations - the basics every database needs
Foreign keys - because relationships matter
Automatic space management - reuses deleted record space
Human-readable files - you can literally cat your data

The Core Idea: Fixed-Width Records

Unlike CSV files where records have variable length, Bazoola uses fixed-width fields:

1     Alice         25    
2     Bob           30    
3     Charlie       35

This makes it trivial to:

Jump to any record by calculating its offset
Update records in-place without rewriting the file
Build simple indexes

Show Me the Code!

Here's how you use Bazoola:

from bazoola import *

# Define schemas
class TableAuthors(Table):
    name = "authors"
    schema = Schema([
        Field("id", PK()),
        Field("name", CHAR(20)),
    ])

class TableBooks(Table):
    name = "books"
    schema = Schema([
        Field("id", PK()),
        Field("title", CHAR(20)),
        Field("author_id", FK("authors")),
        Field("year", INT(null=True)),
    ])

# Create database instance and the files in `data/` subdir
db = DB([TableAuthors, TableBooks])

# Insert data
author = db.insert("authors", {"name": "John Doe"})
book = db.insert("books", {
    "title": "My Book",
    "author_id": author["id"],
    "year": 2024
})

# Query with joins
book_with_author = db.find_by_id(
    "books",
    book["id"],
    joins=[Join("author_id", "author", "authors")]
)
print(book_with_author)
# Output: {'id': 1, 'title': 'My Book', 'author_id': 1, 'year': 2024, 'author': {'id': 1, 'name': 'John Doe'}}

# Close the database
db.close()

Interesting Implementation Details

Space Management

When you delete a record, Bazoola fills it out with * symbols, maintaining a stack of free positions:

class Table:
    # ...
    def delete_by_id(self, pk: int) -> None:
        # ...
        self.f.seek(rownum * self.row_size)
        self.rownum_index.set(pk - 1, None)
        self.f.write(b"*" * (self.row_size - 1) + b"\n")
        self.free_rownums.push(rownum)
    # ...
    def seek_insert(self) -> None:
        rownum = self.free_rownums.pop()
        if rownum is not None:
            self.f.seek(rownum * self.row_size)
            return
        self.f.seek(0, os.SEEK_END)

This simple approach prevents the database from growing indefinitely.

Indexing

Every table automatically gets an index on its primary key in <table_name>__id.idx.dat file, so finding a record by ID is O(1) - just look up the position and seek!

Foreign Keys

Bazoola supports relationships between tables and joins:

# Query with joins
rows = db.find_all("books", joins=[
    Join("author_id", "author", "authors")
])

# Inverse joins (one-to-many)
author = db.find_by_id("authors", 1, joins=[
    InverseJoin("author_id", "books", "books")
])

However, it doesn't enforce referencial integrity yet, and you can basically delete anything you want ¯\_(ツ)_/¯

Human-Readable Storage

Data files are just formatted text:

$ cat data/books.dat
1     My Book             1     2024
************************************
3     My Book3            1     2024

Great for debugging and understanding what's happening!

What I Learned

Data format and file-based storages are tricky, that's why it's all just test-based.
Fixed-width has trade-offs - simple implementation, but wastes space. Real databases use more sophisticated formats.
Indexes are magic: the difference between O(n) table scan and O(1) index lookup is massive, even with small datasets.
Joins are hard.
Testing is crucial. Database bugs can corrupt data, and comprehensive tests saved me many times.

Limitations (and why they're OK)

The current implementation is intentionally simple:

no transactions (what if it fails mid-write?)
no concurrency (single-threaded only)
no query optimizer (every operation is naive)
no B-tree indices (just a simple index for primary keys)
no SQL (just Python API)
fixed-width wastes space These aren't bugs - they're learning opportunities! Each limitation is a rabbit hole to explore.

Try It Yourself!

pip install bazoola

The code is on GitHub. It's ~700 lines of Python (db.py) - small enough to read in one sitting!

What's Next?

Building Bazoola opened my eyes to database internals. Some ideas for further versions:

TEXT fields with a separate storage
strict foreign key constrainsts
unique constraints
B-tree indexes for range queries
ALTER TABLE
simple query planner
pure C implementation with Python bindings

But honestly, even this simple version taught me a ton.

Conclusion

Building a toy database is one of the best learning projects I've done. It's complex enough to be challenging but simple enough to finish. You'll understand your "real" database better, and you might even have fun!

The Cartesian Product Trap in Django ORM

Dmitrii Doroshev — Wed, 09 Jul 2025 21:16:23 +0000

I hit a performance wall at work recently after adding a single extra Count() on a Django queryset that looked harmless. Luckily, the problematic code didn't make it into the production environment, but I had to think hard to figure out what went wrong and find a solution.

The Demo Models

All the following models and entities are made up by me to illustrate the problem.

Imagine we have a PostgreSQL database storing information about separate stores, their departments, employees and products of the stores:

The DB tables can contain much more fields, but in our case we only care about relations between the models.

So in Django, models.py would look like this:

from django.db.models import Model, CharField, ForeignKey, CASCADE

class Store(Model):
    name = CharField(max_length=200)
    # ...

class Department(Model):
    store = ForeignKey(
        Store, related_name="departments", on_delete=CASCADE
    )
    # ...

class Employee(Model):
    department = ForeignKey(
        Department, related_name="employees", on_delete=CASCADE
    )
    # ...

class Product(Model):
    store = ForeignKey(
        Store, related_name="products", on_delete=CASCADE
    )
    # ...

Seed data (roughly what bit me):

2 stores,
10 departments per store,
500 employees per department - 10 000 employees overall,
2500 products per store.

In one of the places of our system, we show a list of the stores, including the amount of employees in each store. Pretty easy with Django ORM, right?

stores = Store.objects.annotate(
    total_employees=Count("departments__employees")
).values(
    "id",
    # ...
    "total_employees"
)

This query is relatively fast and works like a charm.

The Problem

Now let's imagine we were asked to add another counter: a number of products per store. We already have total_employees, why not just add total_products?

Well, most likely we already have some unit test for our piece of code, which checks the logic on a small amount of data, and before releasing the code, we can figure out that another JOIN was added, some data is duplicated, and instead of just COUNT(...), we switch to COUNT(DISTINCT ...), eventually coming up with something like this:

stores = Store.objects.annotate(
    total_employees=Count("departments__employees",
                          distinct=True),
    total_products=Count("products", distinct=True),
).values(
    "id",
    # ...
    "total_employees",
    "total_products",
)

Looks safe to commit, push, wait for the green tests on CI, merge and deploy.

However, after the deployment you'll almost immediately see that everything hangs. As I said, there's not that many stores, only 2 for now, not that many employees, and not that many departments.

And it takes, like, 10 seconds to fetch the numbers! What's wrong with it?

Let's take a closer look at the generated SQL for this seemingly innocent queryset:

SELECT 
    "shop_store"."id",
    COUNT(DISTINCT "shop_product"."id") AS "total_products", 
    COUNT(DISTINCT "shop_employee"."id") AS "total_employees"
FROM "shop_store"
  LEFT OUTER JOIN "shop_product"
    ON "shop_store"."id" = "shop_product"."store_id"
  LEFT OUTER JOIN "shop_department"
    ON "shop_store"."id" = "shop_department"."store_id"
  LEFT OUTER JOIN "shop_employee"
    ON "shop_department"."id" = "shop_employee"."department_id" 
GROUP BY "shop_store"."id"

And let's check the actual query plan:

GroupAggregate ... rows=2 ...
  ->  Nested Loop Left Join ... rows=25000000 ...
  ...
...
Execution Time: 11376.072 ms

Translation: these three JOINs turn into a 25-million-row cartesian mess before GROUP BY and COUNT(DISTINCT):

Products × Departments × Employees
= 2500 × 10 × 500
= 12 500 000  (per store)
× 2 stores
= 25 000 000 joined rows

The Fix

There are multiple ways of handling this case, but the easiest fix is to use subqueries:

subquery_products = Subquery(
    Product.objects.filter(store_id=OuterRef("pk"))
        .values("store_id")
        .annotate(count=Count("pk"))
        .values("count"),
    output_field=IntegerField()
)
subquery_employees = Subquery(
    Employee.objects.filter(department__store_id=OuterRef("pk"))
        .values("department__store_id")
        .annotate(count=Count("pk"))
        .values("count"),
    output_field=IntegerField()
)
stores = Store.objects.annotate(
    total_products=Coalesce(subquery_products, 0),
    total_employees=Coalesce(subquery_employees, 0),
).values("id", "total_products", "total_employees")

SQL query:

SELECT "shop_store"."id",
       COALESCE((
         SELECT COUNT(U0."id") AS "count"  
         FROM "shop_product" U0  
         WHERE U0."store_id" = "shop_store"."id"  
         GROUP BY U0."store_id"), 0
       ) AS "total_products",
       COALESCE((
         SELECT COUNT(U0."id") AS "count"
         FROM "shop_employee" U0
           INNER JOIN "shop_department" U1
             ON U0."department_id" = U1."id"
         WHERE U1."store_id" = "shop_store"."id"
         GROUP BY U1."store_id"), 0
       ) AS "total_employees"
FROM "shop_store";

Now this one takes a couple of milliseconds with pretty modest and predictable plan:

Seq Scan on shop_store ... rows=2 ...
  SubPlan 1
     -> Seq Scan on shop_product u0 ... rows=2500 ...
  SubPlan 2
     -> Hash Join ... rows=5000 ...
        -> Seq Scan on shop_employee u0_1 ... rows=10000 ...
        -> Hash ... rows=10 ...
...
Execution Time: 5.600 ms

No giant intermediate data sets, just two tiny scans:

before: 11 376 ms (~11 seconds)
after: 5.6 ms (2000x faster)

Takeaways

COUNT(DISTINCT) with multi-branch LEFT JOINs makes the database loop through the entire cartesian product.
Correlated subqueries aggregate each branch separately and scale linearly with data size.
Always test aggregate queries against production-sized data before you ship.

Test inheritance in unittest sounds neat - until it breaks your IDE, hides failures in CI, and kills debuggability. I wrote up why it's a trap and what to do instead.

Dmitrii Doroshev — Mon, 23 Jun 2025 11:15:31 +0000

Dmitrii Doroshev

Jun 23 '25

The Hidden Cost of Test Inheritance

#python #testing #unittest #pytest

Comments

3 min read

The Hidden Cost of Test Inheritance

Dmitrii Doroshev — Mon, 23 Jun 2025 08:45:51 +0000

I'm subscribed to Adam Johnson's blog and usually really enjoy his writing - it's practical, deep, and no-bullshit. But one recent post, Python: sharing common tests in unittest, caught me off guard.

It describes a "neat" pattern: write reusable test logic in a base class, subclass it to test multiple objects, hiding the base class from unittest discovery. While the intent is fine - DRYing out duplicated test code - the result is fragile, confusing, and just not worth it.

Here's why.

The Pattern: DRY Tests via Subclassing

# Sample units to test
class Armadillo:
    def speak(self) -> str:
        return "Hrrr!"

class Okapi:
    def speak(self) -> str:
        return "Gronk!"

# Test module
class BaseAnimalTests(TestCase):
    animal_class: type

    def test_speak(self):
        sound = self.animal_class().speak()
        self.assertIsInstance(sound, str)
        self.assertGreater(len(sound), 0)

class ArmadilloTests(BaseAnimalTests):
    animal_class = Armadillo

class OkapiTests(BaseAnimalTests):
    animal_class = Okapi

del BaseAnimalTests

Yes, it works and it reduces duplication. But it comes at the cost of everything else that makes tests maintainable.

The Problems

IDE and DX Pain

When a test fails, I want to jump to it in my IDE, set a breakpoint, and debug. With this pattern - good luck.

The method doesn't exist in ArmadilloTests, it's buried in a deleted parent class. You have to manually hunt it down, re-declare the test method just to put a breakpoint and debug it, and pray the animal_class setup matches what failed:

class ArmadilloTests(TestCase):
    animal_class = Armadillo

    def test_speak(self):
        super().test_speak()

It's tedious and wastes time. All this to avoid writing a 3-line test twice?

class ArmadilloTests(TestCase):
    def test_speak(self):
        sound = Armadillo().speak()
        self.assertIsInstance(sound, str)
        self.assertGreater(len(sound), 0)

Clear, simple, debug-friendly. Worth the few extra lines.

CI Failures Are Confusing

If a shared test fails in CI, you get something like:

test_speak (tests.ArmadilloTests.test_speak) ... FAIL
...
Traceback (most recent call last):
  File ".../tests.py", line 20, in test_speak
    self.assertGreater(len(sound), 0)
AssertionError: 0 not greater than 0

But the method isn't defined in ArmadilloTests, and Search everywhere won't help at all:

So now you have to reverse-engineer which base class it came from and how to recreate it locally.

This isn't clever. It's just fragile.

When It Kinda Makes Sense

There are rare cases:

dozens of classes implementing the same interface
you're the only one maintaining the codebase
you run everything headless in CI

But even then, you're building test framework plumbing to save what, a hundred lines?

The Clean Alternative: Parametrize It

Pytest Style

@pytest.mark.parametrize('animal_class', [Armadillo, Okapi])
def test_speak(animal_class):
    sound = animal_class().speak()
    assert isinstance(sound, str)
    assert len(sound) > 0

You see all the parameters. You see where the test lives. Failures are explicit:

test_speak[Armadillo] FAILED
test_speak[Okapi] PASSED

You can re-run just the failing test. You can debug with a conditional breakpoint. You don't need to explain how the tests are wired together - because they're not.

unittest Style (Optional, Not Ideal)

from parameterized import parameterized_class

@parameterized_class([
    {'animal_class': Armadillo},
    {'animal_class': Okapi},
], class_name_func=get_class_name)
class AnimalTests(TestCase):
    def test_speak(self):
        sound = self.animal_class().speak()
        self.assertIsInstance(sound, str)
        self.assertGreater(len(sound), 0)

Using parameterized_class from parameterized is still better than inheritance, but clunkier. Output is readable if you customize class_name_func. IDE support isn't great. Pytest remains the better option for anything dynamic.

Final Verdict

Tests should fail clearly, debug easily, and be readable years later. This pattern fails all three.

DRY is good. But in tests, visible duplication beats invisible abstraction.

Adam's trick technically works, but in practice, it makes tests harder to navigate, harder to trust, and harder to work with.

Stick to the boring version - you'll thank yourself later.

Why Django's override_settings Sometimes Fails (and How reload + patch Saved Me)

Dmitrii Doroshev — Fri, 13 Jun 2025 08:11:39 +0000

Sometimes @override_settings just doesn’t cut it.

I ran into a nasty issue while testing a Django module that relies on global state initialized during import. The usual test approach didn’t work. Here’s what happened and how I solved it.

The Setup

We had a module that builds a global dictionary from Django settings at import time. Let’s call it dragon.py, which takes settings.PUT_EGGS, which is False by default:

from django.conf import settings

DRAGON = {}
...
if settings.PUT_EGGS:
    DRAGON["eggs"] = "spam"

Another module uses DRAGON for core logic, e.g. mario.py:

from myproject.dragon import DRAGON

def find_eggs():
    if "eggs" in DRAGON:
        return "Found eggs!"
    return "Eggs not found"

Now I wanted to write a test that tweaks DRAGON and expects the logic to behave differently. Easy, right?

@override_settings(PUT_EGGS=True)
def test_find_eggs():
    assert find_eggs() == "Found eggs!"

Wrong. The test failed.

The Problem

override_settings works, but only for code that reads settings at runtime.

In my case, DRAGON was already built at import time , before the override kicked in. So it used the old value of PUT_EGGS, no matter what I did in the test.

This is the classic trap of global state baked during import. Welcome to pain town.

The Fix: reload + patch

Here's how I got out:

import importlib
from django.test import override_settings
from unittest.mock import patch
from myproject.mario import find_eggs

@override_settings(PUT_EGGS=True)
def test_find_eggs():
    # Reload the dragon module so DRAGON is rebuilt
    # with updated settings
    from myproject import dragon
    new_dragon = importlib.reload(dragon)

    # Patch the logic module to use the reloaded DRAGON
    with patch('myproject.mario.DRAGON', new_dragon.DRAGON):
        result = find_eggs()
        assert result == "Found eggs!"

Why This Works

importlib.reload(dragon) forces a fresh import of dragon, rebuilding DRAGON with the overridden settings;
dragon.DRAGON is updated in the scope of the test only, i.e. mario module still has the stale version of DRAGON;
patch(...) solves this problem by swapping the old DRAGON in mario with the freshly rebuilt one.

This is surgical. Ugly, but effective.

Lessons Learned

Avoid putting non-trivial logic at module scope, especially if it depends on Django settings. Wrap it in a function or lazy loader.
If you're stuck with global state, reload() and patch() give you a way out - just be careful about cascading dependencies.

If you’ve ever had a test mysteriously fail after overriding settings, this might be why.

Calculating the next run date of a Celery periodic task

Dmitrii Doroshev — Thu, 12 Jun 2025 06:28:07 +0000

The problem

You have a periodic task in Celery defined with a crontab(...) schedule, and you want to calculate the next time it's supposed to run.

Example: you want to find out when crontab(hour=12, minute=0) will trigger next after now.

Simple, right? There’s croniter library, which seems to be designed to solve this exact problem. Just use it, right?

Well.

First mistake: trying croniter with crontab

So my first instinct was to use croniter like this:

from celery.schedules import crontab
from croniter import croniter
from datetime import datetime

schedule = crontab(hour=12, minute=0)
cron = croniter(schedule, datetime.now())
next_run = cron.get_next(datetime)

Boom 💥 doesn’t work. Because Celery’s crontab is not a string and croniter expects a string like "0 12 * * *":

AttributeError: 'crontab' object has no attribute 'lower'

And no, crontab() does not have a nice .as_cron_string() method either.

So now you’re stuck parsing crontab's internal fields (._orig_minute, ._orig_hour, etc) just to reconstruct a string - and it starts to smell like overengineering for something that should be simple.

The right way (which I learned too late)

Turns out Celery’s crontab (and all schedules derived from celery.schedules.BaseSchedule) already has a method for this:

from datetime import datetime
from celery.schedules import crontab

schedule = crontab(hour=12, minute=0)
now = datetime.now()
# `now` is datetime.datetime(2025, 6, 11, 0, 16, 58, 484085)
next_run = now + schedule.remaining_delta(now)[1]
# `next_run` is datetime.datetime(2025, 6, 11, 12, 0)

That’s it. You don’t need croniter at all. Celery knows how to calculate the delta to the next run. It just doesn’t shout about it in the docs.

Summary

don’t reinvent Celery’s scheduling logic - it already has what you need;
crontab is not a cron string, don’t try to treat it like one;
use .remaining_delta(last_run) to calculate when the next run will happen.

Hope this saves someone the 2 hours I wasted trying to do it the wrong way.

Pytest Fish shell autocompletion

Dmitrii Doroshev — Sun, 29 Dec 2024 23:04:10 +0000

^TL;DR https://github.com/ddoroshev/pytest.fish

Typing repetitive commands or copying and pasting test names can eat up valuable time. To help, I've created pytest.fish - a Fish shell plugin that simplifies your pytest workflow. It's lightweight, simple to set up, and makes testing more efficient.

How to Use

Autocomplete test paths

Type pytest and hit TAB to get suggestions for test paths and functions:

Support for `-k` filter

Narrow down tests with -k and get name suggestions:

The plugin dynamically scans your project, so suggestions stay up-to-date.

Installation

Install with Fisher:

fisher install ddoroshev/pytest.fish

Or manually copy the files from the repository into your Fish configuration.

How It Works

The plugin doesn't rely on pytest directly (yet). Instead, it scans the current directory for test files and searches for test functions inside them, making the process relatively fast and efficient.

Other shells?

Since I primarily use Fish in my local development environment, I created a plugin specifically for this shell. However, if you use Bash or Zsh, feel free to create your own - or switch to Fish already. 😉

This Week in Changelogs: flask, pytest, IPython, etc

Dmitrii Doroshev — Tue, 07 Mar 2023 21:48:13 +0000

pyenv 2.3.13, 2.3.14

Highlights from the changelog:

added versions 3.10.10, 3.11.2, and 3.12.0a5;
fixed versions 3.5.10 and 3.6.15 for macOS and modern 64-bit platforms.

This one made me laugh a bit:

a7b181c introduce an indentation issue
3829742 fix the indendation issue.

That's how programming actually works!

TIL: head -n123 is a part of POSIX, head -123 is a shorthand that can be missing in some operating systems (pull request).

IPython 8.11.0

Highlights from the changelog:

%autoreload supports meaningful parameters (%autoreload all, %autoreload off, etc), not only numbers (%autoreload 0, %autoreload 2, etc).

I like the log of the pull request, it illustrates the approach of implementing a feature step-by-step, one frame at a time:

a7c76db get a dumb and working solution
d8bb14b optimize it
809ebdf and make it nice

Also, this fragment is quite interesting, print and logger.info need to be used carefully for logging and protected from being overwritten during hot-reload:

p = print
logger = logging.getLogger("autoreload")
l = logger.info

def pl(msg):
    p(msg)
    l(msg)

Everything you wanted to know about GitHub actions:

flask 2.2.3

Although the changelog is not that big, I like the thing about flask run --debug.

Previously, it was flask --debug run , and it was awkward. The fix itself is quite small, but there's a lot of changes in docs, and also a PyCharm screenshot was changed. Nice and pure!

pytest 7.2.1, 7.2.2

The changelogs contains mostly bug fixes. One of them is about pytest.approx() causing ZeroDivisionError on dicts.

Another one fixes type checkers behaviour for the following code, which I think should be illegal:

with pytest.raises(RuntimeError) if val else contextlib.nullcontext() as excinfo:

(Please, don't write the code like this.)

And they fixed a race condition when creating directories in parallel, using os.makedirs(..., exists_ok=True). Simple, but helpful.

whitenoise 6.4.0

The changelog mentions support for Django 4.2. It was good to know, by the way, that STATICFILES_STORAGE is going to be changed to STORAGES dict (pull request).

django-cors-headers 3.14.0

Changelog:

added support for Django 4.2,
switched from urlparse to urlsplit.

The latter is the most interesting, urlsplit is slightly faster. Also, it's cached, so sometimes you gain a huge performance.

The difference between these functions is that urlparse includes parsing of the "parameters" section of a URL:

scheme://netloc/path;parameters?query#fragment
                     ^ this

Since it's not widely used, in most cases it's safe to switch from urlparse to urlsplit.

This Week in Changelogs: Django and faker

Dmitrii Doroshev — Sat, 18 Feb 2023 11:02:14 +0000

Django 4.1.6, 4.1.7

4.1.6: release notes, blog post
4.1.7: release notes, blog post

9d7bd5a
An interesting bug of parsing the Accept-Language header. The format of the header value is complex, so there's a bunch of regular expressions and @functools.lru_cache(maxsize=1000) for caching the result. However, you can pass a huge header multiple times, causing DoS, so they added two if statements:

one that checks if the length is less than ACCEPT_LANGUAGE_HEADER_MAX_LENGTH
second - for checking the comma-separated strings. So they decided not to just raise an exception or truncate the string by [:ACCEPT_LANGUAGE_HEADER_MAX_LENGTH], but truncate the value in a safe way, so it can be parsed in a meaningful result. Good job!

26b7a25
There was a bug in generated SQL, caused by that .desc() in the model's Meta.constraints:

constraints = [
    UniqueConstraint(
        Lower("name").desc(), name="unique_lower_name"
    )
]

which resulted in <...> WHERE LOWER("myapp_foo"."name") DESC <...> when checking the uniqueness. Apparently, Django can check the constraints itself, not delegating it to the underlying database.

Although the fix is trivial, the case is not, and it wasn't covered in the initial implementation.

By the way, I like how they use typographic double quotes:

msg = "Constraint “name_lower_uniq_desc” is violated."

a637d0b f3b6a4f Those black updates are annoying, mainly because they make git blame misleading. However, there's a solution I didn't know about:

git blame --ignore-revs-file <file> - ignore commits listed in the file.
.git-blame-ignore-revs file - make GitHub ignore them as well.

590a92e
The bug was caused by the commit which we've already seen. Now you can safely raise ValidationError without the code.

628b33a
One more DoS fix, now it's about number of opened files when you put too many of them in one multipart payload. The fix introduces TooManyFilesSent exception, which results in HTTP 400 (DATA_UPLOAD_MAX_NUMBER_FILES = 100 by default).

I like this fragment:

try:
    return self._parse()
except Exception:
    if hasattr(self, "_files"):
        for _, files in self._files.lists():
            for fileobj in files:
                fileobj.close()
    raise

Beware of freeing your resources, garbage collector can't help you all the time!

faker 16.6.1..17.0.0

Their CHANGELOG is quite descriptive, so I'll just highlight something that I liked.

faker can generate valid image URLs using specific websites (TIL), and one of them, PlaceIMG, is shutting down, and they removed it from the list. The announcement is included in all the generated images:

Biased booleans introduced.
Added emoji provider 🎉 🥳
Added new es_AR provider, but for some reason it's not in the reflected in the CHANGELOG.
Black formatting - always beautiful.

In addition, it turned out that GitHub can put those linter errors from the actions right in the code. I don't know yet how to add this, but I definitely want it!

DEV Community: Dmitrii Doroshev

This Week in Changelogs: curl

Zip bomb protection via delivered-bytes tracking

API bitmask copy-paste bug

Missing return

PowerPC64 endianness AI-slop

The Pitfalls of Reading User Input in C: a Story About scanf and Stdin

1. The Stack (Ghost Data)

2. Scanf, Scansets and Stdin

Solution A: The "Double Tap" (Extra scanf)

Solution B: fgets (The Standard Way)

Solution C: getline (The Heap Way)

Outcomes

Update 2025-12-14

Building a Toy Database: Concurrency is Hard

Demo App

Concurrency

Elimination of find_by and find_by_substr

One More Thing

Conclusion

Building a Toy Database: Learning by Doing

Why Build a Toy Database?

What is Bazoola?

Key Features:

The Core Idea: Fixed-Width Records

Show Me the Code!

Interesting Implementation Details

Space Management

Indexing

Foreign Keys

Human-Readable Storage

What I Learned

Limitations (and why they're OK)

Try It Yourself!

What's Next?

Conclusion

The Cartesian Product Trap in Django ORM

The Demo Models

The Problem

The Fix

Takeaways

Test inheritance in unittest sounds neat - until it breaks your IDE, hides failures in CI, and kills debuggability. I wrote up why it's a trap and what to do instead.

The Hidden Cost of Test Inheritance

The Hidden Cost of Test Inheritance

The Pattern: DRY Tests via Subclassing

The Problems

IDE and DX Pain

CI Failures Are Confusing

When It Kinda Makes Sense

The Clean Alternative: Parametrize It

Pytest Style

unittest Style (Optional, Not Ideal)

Final Verdict

Why Django's override_settings Sometimes Fails (and How reload + patch Saved Me)

The Setup

The Problem

The Fix: reload + patch

Why This Works

Lessons Learned

Calculating the next run date of a Celery periodic task

The problem

First mistake: trying croniter with crontab

The right way (which I learned too late)

Summary

Pytest Fish shell autocompletion

How to Use

Autocomplete test paths

Support for -k filter

Installation

How It Works

Other shells?

This Week in Changelogs: flask, pytest, IPython, etc

pyenv 2.3.13, 2.3.14

IPython 8.11.0

flask 2.2.3

pytest 7.2.1, 7.2.2

whitenoise 6.4.0

django-cors-headers 3.14.0

This Week in Changelogs: Django and faker

Django 4.1.6, 4.1.7

Missing `return`

Solution A: The "Double Tap" (Extra `scanf`)

Solution B: `fgets` (The Standard Way)

Solution C: `getline` (The Heap Way)

Elimination of `find_by` and `find_by_substr`

Support for `-k` filter