DEV Community: rodbv

Djangonaut diaries, week 6: Charting the Unknown Unknowns of a PR

rodbv — Thu, 09 Apr 2026 15:57:13 +0000

Hi there!

This week's post is less technical, a reflection of the journey so far, while I celebrate my first PR merged and wait for two more to be reviewed.

More specifically, it's reflection on the PR for ticket #10919, which is in progress.

My first Django core PR

At my dayjob, we live and breathe the "Build, Measure, Learn" cycle. Our lead time from "ready for dev" to "deployed" in the Kanban board is measured in a few days, and we routinely deploy a few dozen times a week under CI/CD.

We rely heavily on feature flags to mitigate risk; if something doesn't pan out or a weird bug slips through, we simply flip a toggle, check the error logs, and iterate. Worst case (which is rare), we can rollback the deployment with a few clicks. It’s a nimble, high-velocity speedboat where the cost of failure is relatively low.

But stepping into Django core to tackle #10919 is an entirely different reality, and it's been a great and rewarding learning experience.

This issue opened back in 2009, regarding large browser memory usage and potentially inaccessible pages generated when deleting highly nested objects in Django Admin. See that red button all the way to the bottom of the page? there's where the "delete" button ends up when deleting objects with a lot of nested children.

Across the globe on a cargo ship

Contributing to Django isn't steering a speedboat; it is helping load and route a massive cargo ship.

Let me be clear: by comparing Django to a cargo ship, I am not saying it's slow or heavy. Django ships a new version every eight months like clockwork, constantly delivering a ton of great, innovative features while keeping very solid long-term support via its LTS program.

But its speed is measured in steady throughput and predictability, not rapid-fire pivots. It is a cargo ship because it delivers a massive amount of value across vast distances. Hundreds of thousands of businesses around the world depend on that cargo arriving safely. And when you are building for a ship this size, you quickly need to learn how to deal with its own momentum.

The Illusion of 98% Done

When I opened the PR in mid-March, I honestly thought the hardest part was behind me. I had an elegant solution: to prevent forcing an unexpected UI change on legacy projects, I proposed an opt-in, configurable attribute (ModelAdmin.delete_confirmation_max_display).

The interactions with the Django Fellows reviewing the code were fantastic. They asked incredibly well-considered questions that sharpened the implementation and deepened my understanding of the framework's constraints. They taught me the importance of well-crafted commits and smart use of rebase, to help them review new iterations and bissect any regressions.

I suddenly had to navigate the world of Sphinx and ReStructuredText (rst files). Where exactly does a new Admin setting belong in the massive official documentation? How do you format a .. versionadded:: 6.1 directive? You aren't just writing a quick explainer for your team; you are writing the manual that a ton of developers (and now, AI) is going to rely on.

After a few force pushes and refinements to keep the review process moving, the tests were green, all requests were considered, PR was marked as accepted! I felt like I was 98% done, merge was a few days away.

The Bucket of Cold Water (And Why We Need It)

By late March, the code was solid, but we needed to settle a specific UI detail: when truncating the list of deleted objects, should we cap the items globally across the whole page, or per nesting level? Better open up this decision to a larger audience, and following a suggestion by the reviewer, I headed over to the Django Forum, expecting a straightforward vote.

Instead, the very first response I received was a complete bucket of cold water: why not scrap the PR entirely and close this 17-year-old ticket?

The commenter argued that large-scale applications should just handle this themselves by overwriting some hooks, which is a totally feasible thing. From their perspective, adding this extra logic to the core was an unnecessary complication.

When your daily rhythm involves a handful of deployments and PRs merged, and the constant reward of shipping quickly, getting told to throw all your work away stings. This PR was my pet project for a month!

But once the initial sting faded, I realized that this pushback wasn't just necessary friction; it's very positive. Why?

In a massive framework like Django, every new feature, even a useful one, is a liability. It adds complexity and carries a maintenance burden that the community will have to support for years. The most vital stress-test in foundational open source isn't asking, "Is this useful?" It's asking, "Should we REALLY implement this?" Being deliberately sparse with new additions and ruthlessly questioning their necessity is the exact reason Django can continue shipping those reliable 8-month releases without collapsing under its own weight, for over 20 years.

The beauty of the open-source community is the sheer diversity of thought that this stress-test provokes. While I was still digesting that bucket of cold water and practicing some stoicism, a Django Fellow chimed in with the exact opposite view. Looking at the accessibility issues of hiding the "Delete" (actually, "Yes, I'm sure") button beneath thousands of rows, they saw great value in my PR. They argued that not only should the feature exist, but it might even need to be the default behavior out of the box!

I was suddenly standing between "throw this away" and "maybe we want to make this the default behavior, not opt-in".

I didn't want to go "defend" my position too fast (in fact, I haven't replied yet). I see the value of letting the discussion breathe, and ultimately, see where we find a consensus. In the meantime I decided to keep it an opt-in, configurable attribute, but use a global limit for a cleaner UI (it's much better indeed!).

Slow Coding and the Antidote to AI Fatigue

This meant I had to completely rewrite the core logic.

Instead of a simple per-level cap, I needed to revisit a recursive function with some weird data structure and maintain a global counter across deeply nested tree of deleted objects.

At work, to keep up with our velocity, I rely heavily on AI assistants. I can’t exactly tell my client, "Hey, I’m going to turn off my AI support for a few days just to exercise my brain, okay?"

But here, on a PR that might take days or weeks between checks? I had the ultimate luxury: time. This deliberate pace turned out to be the perfect antidote to AI fatigue.

Instead of prompting an LLM to rewrite the tree traversal, I spent a few days just turning the options over in my head while doing other things. For anyone who genuinely enjoys algorithms, stepping away from the prompt box and wrestling with the logic mentally is fun. At least for me.

In early April, I finalized the global counter logic, pushed the new commit, and hit 100% test coverage again.

The Rhythm of the Framework

As of writing this, the PR is still open, and that is exactly how it should be.

Right now, the Django Fellows are focused on the massive task of wrapping up the Django 6.1 release. A 17-year-old ticket can happily sit a little longer while the next major version of the project gets the focused stabilization it needs.

The reward of this process wasn't about getting a "Merged" badge on GitHub as fast as possible. The true value was uncovering those unknown unknowns...the rigor of the docs, the depth of the edge cases, the craftsmanship of commits, the importance community consensus, and the joy of good old algorithmic thinking. While I wait, I’m using this time to dive into other Trac issues and check pull requests from my fellow Djangonauts.

It’s just more time to learn the ropes! (and enough with nautical analogies now!)

See you!

Djangonaut diaries, week 4: Eliminating a Redundant Index in Django's ORM

rodbv — Sun, 29 Mar 2026 15:52:00 +0000

Hello again!

Last week, we discussed issue #22125, to prevent a redundant database index from being created on Many-to-many "through" tables. We geeked out a lot about composite indexes, reproduced the issue, and now it's time to send a patch for it.

One interesting thing about this ticket, is that there was in fact a PR created addressing it, 8 years ago! A few years later, it was closed due to inactivity.

That PR had some aspects that make a lot of sense, most importantly, it sets db_index=False in the function create_many_to_many_intermediary_model, which seems to be the change we need to do. Let's understand what happens there.

How are many-to-many models defined in Django?

As I've explained in the previous article, a many-to-many (m2m) relationship normally needs an intermediary table to store these links. For example if I have model Book and a model Author, and I want my app to allow a book to have several authors and an author to have several books, I will need a through-table to have the unique pairs (book_id, author_id).

Specifying this m2m relationship in code is straightfoward:

class Book(models.Model):
    # ...other book fields
    authors = models.ManyToManyField(Author, related_name="books")

What's curious here is that while every Django model normally has one backing database table, in this case we will have a third table besides books and authors which is not behind a model we declared. It will be a model and table defined by the framework itself, unless we tell it otherwise.

This is achieved using some Python metaprogramming, more specifically the type function which allows a class to be defined dynamically.

In create_many_to_many_intermediary_model:

def create_many_to_many_intermediary_model(field, klass):
    # ...some code
    name = "%s_%s" % (klass._meta.object_name, field.name)

    # Construct and return the new class.
    return type(
        name,
        (models.Model,),
        {
            "Meta": meta,
            "__module__": klass.__module__,
            from_: models.ForeignKey(
                klass,
                related_name="%s+" % name,
                db_tablespace=field.db_tablespace,
                db_constraint=field.remote_field.db_constraint,
                db_index=False, # <- our change
                on_delete=CASCADE,
            ),
            to: models.ForeignKey(
                to_model,
                related_name="%s+" % name,
                db_tablespace=field.db_tablespace,
                db_constraint=field.remote_field.db_constraint,
                on_delete=CASCADE,
            ),
        },
    )

This function receives two parameters, the field name which is defining the m2m relationship, authors, and the model class in which it was defined, Book. Based on this it defines a name for the class it wants to create dynamically, Book_authors, and the two fields it will need, from and to.

In the from field is where we can add our change to flip the default behavior of foreign keys, which is to create an index.

Does this change work?

Before we move forward, it's good to check: does this change actually work, at least on the happy path?

Let's run the migration with the change on and off, and compare the SQL produced. Without our change, we get this block when we run makemigrations and sqlmigrate over SQLite:

CREATE TABLE "blog_book_authors" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "book_id" integer NOT NULL REFERENCES "blog_book" ("id") DEFERRABLE INITIALLY DEFERRED, "author_id" integer NOT NULL REFERENCES "blog_author" ("id") DEFERRABLE INITIALLY DEFERRED);
CREATE UNIQUE INDEX "blog_book_authors_book_id_author_id_0a5bb3b3_uniq" ON "blog_book_authors" ("book_id", "author_id");
CREATE INDEX "blog_book_authors_book_id_35eae5ed" ON "blog_book_authors" ("book_id");
CREATE INDEX "blog_book_authors_author_id_fa034e3d" ON "blog_book_authors" ("author_id");

The second line from bottom is the interesting one, it's defining an index blog_book_authors_book_id_35eae5ed for book_id on our through-table, blog_book_authors.

Now with db_index=False:

CREATE TABLE "blog_book_authors" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "book_id" bigint NOT NULL REFERENCES "blog_book" ("id") DEFERRABLE INITIALLY DEFERRED, "author_id" bigint NOT NULL REFERENCES "blog_author" ("id") DEFERRABLE INITIALLY DEFERRED);
CREATE UNIQUE INDEX "blog_book_authors_book_id_author_id_0a5bb3b3_uniq" ON "blog_book_authors" ("book_id", "author_id");
CREATE INDEX "blog_book_authors_author_id_fa034e3d" ON "blog_book_authors" ("author_id");
COMMIT;

Nice, so our change works, the happy path is checked.

Updating tests

Now that we made the change and it seems to work, let's see the test coverage for it. I could search for the related functions and classes to run a subset of the tests, but since running the full test suite doesn't take that long, let's give a shot. Ideally, one or more tests will fail, and we can start by reviewing them. Let's run python tests/runtests.py and see how it goes

............................................................................s......................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 18412 tests in 220.346s

Interesting, all tests passed. So there's no specific test check for the creation of the FK index for the first field of an m2m table, which is fine, as the creation of FK indexes are well covered and this is just another FK index which happens to be in a m2m model.

Looking at that PR which was closed a few years ago, we get a good hint: In tests/schema/tests.py, there are lots of tests related to how m2m artifacts are created. We can add a new test there, similar to the one in the original PR, but that also looks similar to the current approach we see in tests nearby, which has evolved since.

Let's create a test helper _test_m2m_create_skips_redundant_source_fk_index(self, M2MFieldClass) that receives different variations of m2m fields that inherit from RelatedField, which is where we did our change: ManyToManyField, CustomManyToManyField and InheritedManyToManyField.

def _test_m2m_create_skips_redundant_source_fk_index(self, M2MFieldClass):
    class LocalBookWithM2M(Model):
        tags = M2MFieldClass("TagM2MTest", related_name="books")

        class Meta:
            app_label = "schema"
            apps = new_apps

    self.local_models = [LocalBookWithM2M]

    with connection.schema_editor() as editor:
        editor.create_model(TagM2MTest)
        editor.create_model(LocalBookWithM2M)

    through = LocalBookWithM2M._meta.get_field("tags").remote_field.through
    through_table = through._meta.db_table

    # ensure index for localbookwithm2m_id was NOT created
    self.assertNotIn("localbookwithm2m_id", self.get_indexes(through_table))
    self.assertIn("tagm2mtest_id", self.get_indexes(through_table))

    constraints = self.get_constraints(through_table).values()
    unique_columns = [
        constraint["columns"] for constraint in constraints if constraint["unique"]
    ]
    self.assertIn(["localbookwithm2m_id", "tagm2mtest_id"], unique_columns)

Our change is being checked in the line where we assert that an index for localbookwithm2m_id was NOT created in the through-table, check the comment.

To run the tests in this file only, we can pass the test class name using the -k param.

> python tests/runtests.py -k SchemaTests

sssssssssssssssssssssssssssssssssssssssssss....s.sss....................sss....s.ss.....s.ss.ss.sss..s......ss.s.s..................s........s......................ss..s.s......................s..s....s.s.ss.........................s.....ss.........s........s..sss..s...sss
----------------------------------------------------------------------
Ran 273 tests in 18.881s

Cool, test passed. Now it's crucial to "undo" our change and ensure the related tests fail, otherwise our test is a false-positive, i.e. a test that always passes and therefore has no value.

So we commented out the line with db_index=False...

from_: models.ForeignKey(
     klass,
     related_name="%s+" % name,
     db_tablespace=field.db_tablespace,
     db_constraint=field.remote_field.db_constraint,
     # db_index=False,
     on_delete=CASCADE,
),

... and re-run the tests:

    ^^^^^^^^^^^^^^^^^
  File "/Users/rodrigo/.pyenv/versions/3.12.8/lib/python3.12/unittest/case.py", line 715, in fail
    raise self.failureException(msg)
    ^^^^^^^^^^^^^^^^^
AssertionError: 'localbookwithm2m_id' unexpectedly found in ['tagm2mtest_id', 'localbookwithm2m_id']

----------------------------------------------------------------------
Ran 273 tests in 18.994s

FAILED (failures=3, skipped=90)

Cool! Tests failed, so they work as a regression test. Now we can reintroduce our code change, make sure tests pass again, and we're close to send our PR.

One thing I want to do is adding a comment on the line we're setting db_index=False. There's a chance the reviewers will ask to remove this comment, but personally I like to add comments on unusual or non-obvious code. In the future, one may find that line with db_index=False and wonder why that's the case; after all, we normally expect every FK to have an index for it. So pointing to the ticket can be helpful. We'll see if others agree, this kind of preference depends a lot on the culture of the team/project.

from_: models.ForeignKey(
    klass,
    related_name="%s+" % name,
    db_tablespace=field.db_tablespace,
    db_constraint=field.remote_field.db_constraint,
    on_delete=CASCADE,
    # Ticket #22125: index is redundant with
    # unique_together (from_, to).
    db_index=False,
),

Ok, time to send the PR Fixed #22125 - Unnecessary creation of index for ManyToManyField.! My second!

Uh oh, some tests failed on CI

Oh well. Remember we had all test passing locally? That's not what happened when the PR ran its CI pipeline. Some tests failed for Postgres and Postgis.

So what do we have here? On top, I see that the test that failed was running Postgres. Then, we see the failing test is called test_tablespace_for_many_to_many_field - name makes sense - and that it was expecting to find 2 instances of something but found just one.

Let's dig into this test. I added a comment on where they failed.

@skipUnlessDBFeature("supports_tablespaces")
def test_tablespace_for_many_to_many_field(self):
    sql = sql_for_table(Authors).lower()
    # ... more code

    sql = sql_for_index(Authors).lower()
    # The ManyToManyField declares no db_tablespace, its indexes go to
    # the model's tablespace, unless DEFAULT_INDEX_TABLESPACE is set.
    if settings.DEFAULT_INDEX_TABLESPACE:
        self.assertNumContains(sql, settings.DEFAULT_INDEX_TABLESPACE, 2) # <-- failed here
    else:
        self.assertNumContains(sql, "tbl_tbsp", 2) # <- and here
    self.assertNumContains(sql, "idx_tbsp", 0)

So this is why we had no failures running tests locally on the default database (SQLite), but it failed on CI: this test only runs when the database backend supports Tablespaces (check the decorator on top), which is the case for Postgres and Postgis in the CI.

We could just set the value to what's expected and see how it goes, but it's important to understand a bit more what's going on. First of all, what's a tablespace? From Postgres docs:

Tablespaces in PostgreSQL allow database administrators to define locations in the file system where the files representing database objects can be stored. Once created, a tablespace can be referred to by name when creating database objects.
By using tablespaces, an administrator can control the disk layout of a PostgreSQL installation.

So it's a way to tell the db manager to store some of its files in different disks, which seems handy for huge databases.

I'd love to debug these tests and see the values being passed for these assertions, to understand what's going on. I could run Postgres locally, or set it up to run on Docker, but there's an easier way...

Enter django-docker-box

Django-docker-box can be better explained by its README, which includes this:

🧪 Test supported core database backends — MariaDB, MySQL, Oracle, PostgreSQL, SQLite
🌍 Test supported core geospatial backends — MariaDB, MySQL, Oracle, PostGIS, SpatiaLite

Among many other awesome things, it allows us to easily run tests on all databases supported by Django, one docker-compose command away, lovely.

To run it with Postgres is pretty easy, and we can pass the same parameters we normally pass to runtests.py to limit to the tests we care about, with -k TablespacesTests. I also set --parallel 1, to avoid spawning lots of processes to run only 5 tests.

❯ docker compose run --rm postgresql --parallel=1 -k TablespacesTests
[+]  4/4t 4/44
...
Testing against Django installed in '/django/source/django'
Found 5 test(s).
Creating test database for alias 'default'...
System check identified no issues (46 silenced).
.F.ss
======================================================================
FAIL: test_tablespace_for_many_to_many_field (model_options.test_tablespaces.TablespacesTests.test_tablespace_for_many_to_many_field)
...
AssertionError: 1 != 2 : Found 1 instances of 'tbl_tbsp', expected 2

----------------------------------------------------------------------
Ran 5 tests in 0.891s

FAILED (failures=1, skipped=2)

Nice! Tests failed just like in CI. Now we can pass the parameter --pdb that will break into debugging mode when the assertion fails, and have a look around. Knowing how to use pdb or ipdb is a very useful skill, worth checking in case you don't know them.

At this point, it's good to know that in these tests, there's a through table named "Authors" to link the "from" model Article and the "to" model Scientist.

This is a bit of the pdb session I had:

Opening PDB: AssertionError("1 != 2 : Found 1 instances of 'tbl_tbsp', expected 2")
> /django/source/tests/model_options/test_tablespaces.py(52)assertNumContains()
-> self.assertEqual(
(Pdb) l
 47             apps.all_models["model_options"] = self._old_models
 48             apps.clear_cache()
 49
 50         def assertNumContains(self, haystack, needle, count):
 51             real_count = haystack.count(needle)
 52  ->         self.assertEqual(
 53                 real_count,
 54                 count,
 55                 "Found %d instances of '%s', expected %d" % (real_count, needle, count),
 56             )
 57
(Pdb) u
> /django/source/tests/model_options/test_tablespaces.py(116)test_tablespace_for_many_to_many_field()
-> self.assertNumContains(sql, "tbl_tbsp", 2)
(Pdb) sql
'create index "model_options_articleref_authors_scientist_id_4ad98b60" on "model_options_articleref_authors" ("scientist_id") tablespace "tbl_tbsp"'
(Pdb)

First, I pressed l (as in "list") to list the code around the failure, and since it's in the asssertNumEqual function itself, I pressed u to "step out" and going to the outer scope, the test that called assertNumEqual. There, I could type the name of the variable being tested, sql, and got this

-> self.assertNumContains(sql, "tbl_tbsp", 2)
(Pdb) sql
'create index "model_options_articleref_authors_scientist_id_4ad98b60" on "model_options_articleref_authors" ("scientist_id") tablespace "tbl_tbsp"'
(Pdb)

So it's getting the SQL that was produced for the "to" model, Scientist, and not the "from" model, Article. To make sure our understanding is correct, let's undo our change on related.py and run again

(Pdb) sql
'create index "model_options_articleref_authors_article_id_727a3ebf" on "model_options_articleref_authors" ("article_id") tablespace "tbl_tbsp"
create index "model_options_articleref_authors_scientist_id_4ad98b60" on "model_options_articleref_authors" ("scientist_id") tablespace "tbl_tbsp"'

In this case, I see two indexes, one for article_id and one for scientist_id. So it makes sense, we should really reduce the assertion from 2 to 1 occurence, as the index for article_id will be gone. Nice!

Lowering the values on the tests and running them again got the tests to pass:

Found 5 test(s).
Creating test database for alias 'default'...
System check identified no issues (46 silenced).
...ss
----------------------------------------------------------------------
Ran 5 tests in 0.872s

Time to push this and see if the CI is happy (that takes a while, so go have some dinner, pet the cat ;))

Great! I think this is it by now, now I can make sure the PR is according to the guidelines etc and wait for it to be reviewed.

See you!

Djangonaut diaries, week 3 - Working on an ORM issue

rodbv — Sat, 21 Mar 2026 00:39:26 +0000

Hi there!

The PR from the previous week is currently being reviewed, with some great suggestions from one of the Django fellows, Jacob Walls.

In the meantime I picked up one of the issues that were suggested by the navigator for this cohort, Akash, an ORM optimization issue titled "Unnecessary creation of index for ManyToManyField".

In summary, when Django creates the migrations for a many-to-many "through" table, it creates a database index for the leftmost column which is redundant as the database engine will favor the composite index.

Now, if you understood every word on this sentence and the ticket title, maybe this blog post is not for you, you can wait for the next one in which I actually fix the issue. I won't judge you.

...

What? you here still? Great!! Let's dig a bit on what are database indexes, how the most common index for relational databases work (balanced trees, aka b-trees), what are many-to-many (m2m) relations and junction tables (also called through-tables), and finally how "unique" indexes for m2m tables work.

Knowing the basics of all that, we will have the tools to work on a proper fix.

What is an index, anyway? Why not index everything?

So, we all know indexes from books: if I want to find a topic or a chapter in a book, instead of reading every page, I can check its index at the beginning or the end of the book, search for the topic there, get the page number and jump straight to that page. In a way, it's a trade-off: we're making the book a bit heavier and expensive (having more pages) in exchange for making it more practical to use.

Database indexes also help us find data quickly, they're shortcuts to find data and avoid table scans, which is when the database engine reads every row in a table to find what was requested.

An index is a separate data structure from the main table, optimized for speed and size. The most common type is the B-Tree (Balanced Tree), which maintains a consistent depth so that searching for any record takes roughly the same amount of time.

Just like book indexes, there's also a trade-off here: to make data retrieval faster, we will use a bit more disk space, and also make writing operations a bit slower, as we need to update not only the table but also the indexes affected. In other words, indexes are not "free", and having unnecessary indexes is a bad idea.

How indexes are used

In the diagram below, the database is searching for the book "Two Scoops of Django" (an excellent book btw!). It follows a specific logical path through the pages:

The search starts at the Root Page. The engine compares "Two Scoops" against the keys J and S. Since "T" comes after "S," it ignores the first two branches and follows the pointer to the third.

It lands on an Internal Page for the S-Z range. It sees another split and determines the title must be in the S-U section.

Finally, it arrives at the Leaf Page, which contains the actual index entries. it finds "Two Scoops of Django" and its associated rowid: 88, which is where the record is physically located on disk and can be retrieved for the user.

By traversing this tree, the database only reads a few pages from disk instead of scanning the entire table. Those who remember big-O notation, that's O(log N), which means much better :D

What about M2M relations, composite indexes and unique constraints?

Many-to-many relations, or M2M for short, are relationships where both parties involved can occur several times. For example, if you have a table for Books and a table for Authors, you need a third table to link them: a book can have one or more authors, and an author can have written one or more books. This middle table is called a through-table, or junction table, or a join table, or Larry (just kidding, not Larry).

One rule we usually want for our through-table is to enforce that the pairs (book_id, author_id) are unique. There is no reason to record twice that a specific book has a specific author!

To enforce this rule (aka constraint) efficiently, a Composite Index combining both IDs is needed. The database needs to check this index before every insert or update to ensure the pair doesn't already exist. It’s the same B-Tree structure we saw before, but instead of sorting by a single value, it sorts by tuples of (book_id, author_id).

The order of columns in this index is critical. Because the B-Tree is sorted, the intermediate pages will first split the data by the first value book_id. If there are multiple authors for that same book, it then splits those by the second value author_id. In the database world we call the first the "leftmost" column, and it's the most important of the pair, for the reasons we'll explain now.

Check the diagram below, see that when we search for Book ID 10 and author ID 5, it will first find the book, and then its authors.

So whenever I try to associate an author to a book, the database engine can use this index to quickly check if the relation already exists, and if so raise an error.

From this arrangement we may note two things:

If I want to search for all authors for a given book ID, this index is good enough, it can just get all nodes once it reaches the Book ID;
On the other hand, if I want to quickly search for books related to a specific author, this index won't help me at all. In this case, I would need a separate index on this table for author_id.

Back to the ticket

Ooookay, I hope you're still with me after this rabbit hole :D

Onto our ticket: it says that for M2M relations, Django is creating three indexes:

One for the first entity ID
One for the second entity ID
One for the pair (unique constraint index)

From the explanation above you can agree with the creator of the ticket that the first index is redundant, right?

But, let's see if this is actually true in practice, if a real database engine would optimize and just use the composite index if I search by book ID.

To do this, we can use a nice tool (mentioned in the ticket comments, which is how I learned about it) called SQL Fiddle. This tool allow us to provide code for tables and queries, pick a database engine, and by running a special SQL command called EXPLAIN QUERY, see how the database would perform the query, its "query plan". Very handy, especially because SQL Fiddle supports all relational databases supported by Django (SQLite, Postgres, MySQL, Oracle, MariaDB).

This is the test we did there. Note that, just like issue described, we created three indexes, one for each column and one for the composite index (unique constraint).

-- 1. Create the main tables
CREATE TABLE Book (
  ID INTEGER PRIMARY KEY AUTOINCREMENT,
  Title TEXT
);

CREATE TABLE Author (
  ID INTEGER PRIMARY KEY AUTOINCREMENT,
  Name TEXT
);

-- 2. Create the M2M through-table
CREATE TABLE Book_Authors (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  book_id INTEGER,
  author_id INTEGER,
  FOREIGN KEY (book_id) REFERENCES Book(ID),
  FOREIGN KEY (author_id) REFERENCES Author(ID)
);

-- 3. Add the indexes (Fixed Typo and Names)
CREATE INDEX idx_book_author_book_id ON Book_Authors(book_id);
CREATE INDEX idx_book_author_author_id ON Book_Authors(author_id);
CREATE UNIQUE INDEX idx_book_author_unique_book_id_author_id ON Book_Authors(book_id, author_id);

-- 4. Gather statistics
ANALYZE;

-- 5. THE TEST
EXPLAIN QUERY PLAN 
SELECT * FROM Book_Authors WHERE book_id = 10;

...and the result is...drumroll....

QUERY PLAN
`--SEARCH Book_Authors USING COVERING INDEX 
idx_book_author_unique_book_id_author_id (book_id=?)

Okay! So we can confirmed the thesis of the issue. We searched by book ID and SQLite opted to use the composite index despite having a dedicated index for book ID. If we use SQL Fiddle to toggle Postgres, Oracle and MariaDB, the same will happen (try it!)

If we search by Author, on the other hand, it will use the specific index for author:

EXPLAIN QUERY PLAN 
SELECT * FROM Book_Authors WHERE author_id = 5;

QUERY PLAN
`--SEARCH Book_Authors USING INDEX idx_book_author_author_id (author_id=?)

Is Django really producing the extra index?

Finally, it's good to "check the plug": is Django really creating three indexes, including the extra one for the leftmost column? if not, you may ask me why did I make you read such long article... well, at least you didn't have to write it!

To test on Django I will create the models. Note that the m2m relation is defined in the Book model.


class Author(models.Model):
    name = models.CharField(max_length=100)

    def __str__(self):
        return self.name


class Book(models.Model):
    title = models.CharField(max_length=200)
    authors = models.ManyToManyField(Author, related_name="books")
    published_date = models.DateField()

    def __str__(self):
        return f"{self.title}"

Now I can run the command to create the migration...

> python manage.py makemigrations

Migrations for 'blog':
  blog/migrations/0003_author_book.py
    + Create model Author
    + Create model Book

And use the command manage.py sqlmigrations to see the SQL commands that will produce the tables and indexes

... create table etc

CREATE UNIQUE INDEX "blog_book_authors_book_id_author_id_0a5bb3b3_uniq" ON "blog_book_authors" ("book_id", "author_id");
CREATE INDEX "blog_book_authors_book_id_35eae5ed" ON "blog_book_authors" ("book_id");
CREATE INDEX "blog_book_authors_author_id_fa034e3d" ON "blog_book_authors" ("author_id");

There it is! The redundant index on book_id, named blog_book_authors_book_id_35eae5ed.

Now that we know

Django creates 3 indexes for m2m through-tables, including one for the leftmost column
Database engines like SQLite, Postgres etc optimize usage of indexes and will prefer using the composite index when possible

...we can move on and find a fix to prevent the extra index from being created! But that's the for the next post, soon ;)

Cheers!

PS: If you want to know a bit more about other types of indexes, I have some slides for a talk I did for Python Cerrado (a regional conference here in Brazil) in 2024. This is how I got so nerdy about b-trees etc, to have a reason to travel and meet more Pythonistas ;)

PPS: Yes I used AI to draw these Mermaid diagrams, I'm not that good on Mermaid. And I learned that dev.to does not support Mermaid too :(

Djangonaut diaries, week 2: working on the first issue

rodbv — Tue, 17 Mar 2026 01:31:26 +0000

Hi there!

OK, so in parts 1 and 2 we prepared our development environment to contribute with Django. I guess now it's time to actually get stuff done!

The issue I picked up is 17 years old, which I'd love to say "nearly my age", but unfortunately not :P

In summary: when you select one or more items to be deleted in Django Admin, it rightly shows a confirmation page, like this

So, it shows the object(s) you're about to delete, and their children, recursively, which is great to avoid regrets.

But what if there are thousand of children objects? Suddenly the list may get reeeeeeaaaaaally long, taking several seconds to load, potentially slowing down the browser due to memory usage, while forcing the user to scroll a lot to reach the "Yes, I'm sure" button.

While the original request was to add the ability to hide the list of objects altogether, I really like a suggestion (comment #13) by user terminator14, to let one define the maximum number of items to show (say, 100) and then something like "...and 821 more", to give a sense of how much was truncated - note that there's also a summary with the total number on top.

Fine, now that the ideas are out there, let's get to the code. In part two of this series I already showed how I found the related code - there's no magic really, I just searched for a specific string, "Are you sure you want to delete".

1. Make it work

This is where everything starts, in the delete confirmation template:

<p>{% blocktranslate %}Are you sure you want to delete the {{ object_name }} “{{ escaped_object }}”? All of the following related items will be deleted:{% endblocktranslate %}</p>
{% include "admin/includes/object_delete_summary.html" %}
<h2>{% translate "Objects" %}</h2>
<ul id="deleted-objects">{{ deleted_objects|unordered_list }}</ul>

From this, I can see that the list with <li>s is rendered by

<ul id="deleted-objects">{{ deleted_objects|unordered_list }}</ul>

Curious. I was expecting to find a for loop with deleted_objects, but somehow[1] deleted_objects|unordered_list is producing the <li>s in it, so let's have a look.

This is what we see in the deleted_objects variable (thanks, awesome debugger from the previous post!)

[
    'Blog: <a href="/admin/blog/blog/.../">First Blog</a>', 
    [
        'Post: <a href="/admin/blog/post/54db9806.../">Consectetur labore...</a>', 
        ['Comment: Nice post!', 'Comment: Thanks for sharing'], # These are the [...]

        'Post: <a href="/admin/blog/post/dcbba30f.../">Porro dolorem...</a>', 
        ['Comment: Very informative'],

        'Post: <a href="/admin/blog/post/f80ecf48.../">Numquam voluptatem...</a>', 
        ['Comment: I disagree with this point'],

        'Post: <a href="/admin/blog/post/84438d87.../">Sed modi labore...</a>', 
        ['Comment: Great read'],

        'Post: <a href="/admin/blog/post/38505bb6.../">Dolor est dolorem...</a>', 
        ['Comment: Love the style'],

        'Post: <a href="/admin/blog/post/5dbb276c.../">Dolore amet...</a>', 
        ['Comment: This was helpful'],

        'Post: <a href="/admin/blog/post/b0eb6a30.../">Sed dolorem...</a>', 
        ['Comment: Can you explain more?'],

        'Post: <a href="/admin/blog/post/9153de52.../">Quaerat magnam...</a>', 
        ['Comment: +1'],

        'Post: <a href="/admin/blog/post/03c071e1.../">Quiquia quisquam...</a>', 
        ['Comment: Excellent summary'],

        'Post: <a href="/admin/blog/post/e4b7fcd0.../">Porro voluptatem...</a>', 
        ['Comment: Looking forward to the next one']
    ]
]

So, there are no <li>s here, just a recursive data structure. My first idea was to actually truncate the items at this level, which would have to be a recursive function too (to limit the number of comments for each post as well), but whenever we're getting into a very complex code change, it's good to take a step back and think..is it really the simplest thing I could do?

The answer is no. If there's some other code traversing this data structure to produce nested <li>s, that's probably where I can have some sort of counter, while the list items are being produced, and truncate them.

That's on the filter receiving deleted_objects, called unordered_list.

The documentation for unordered_list provides a great explanation of what it does, and how:

Recursively take a self-nested list and return an HTML 
unordered list -- WITHOUT opening and closing <ul> tags.

Inside this function there's another function which actually builds the list items, in the last line:

def list_formatter(item_list, tabs=1):
  indent = "\t" * tabs
  output = []
  for item, children in walk_items(item_list):
     sublist = ""
     if children:
       sublist = "\n%s<ul>\n%s\n%s</ul>\n%s" % (
          indent,
          list_formatter(children, tabs + 1),
          indent,indent)

       output.append("%s<li>%s%s</li>" % (indent, escaper(item), sublist))

Ok so I want to test a concept: we already have an empty list called output (line 3); can I just create a counter and have an if statement checking the length of output and breaking the loop when we reach the limit? Let's check!

def list_formatter(item_list, tabs=1):
    indent = "\t" * tabs
    output = []
    max_items = 3 # changed here...
    for item, children in walk_items(item_list):
        sublist = ""
        if children:
            sublist = "\n%s<ul>\n%s\n%s</ul>\n%s" % (
                indent,
                list_formatter(children, tabs + 1),
                indent,
                indent,
             )

        if len(output) >= max_items: # ...and here
            output.append("%s<li>...and N more.</li>" % indent)
            break
        output.append("%s<li>%s%s</li>" % (indent, escaper(item), sublist))

    return "\n".join(output)

Reloading the page...

...it worked!!!!!11

Now, I need "N" to be an actual number. The trouble is that this list is built iteratively (it's a recursive function), so I can't break the loop when I reach the limit, I gotta let it continue and keep counting how many were truncated.


def list_formatter(item_list, tabs=1):
    indent = "\t" * tabs
    output = []
    max_items = 3
    truncated_count = 0 # added a counter for truncated items
    for item, children in walk_items(item_list):
        sublist = ""
        if children:
            sublist = "\n%s<ul>\n%s\n%s</ul>\n%s" % (
                indent,
                list_formatter(children, tabs + 1),
                indent,
                indent,
            )

        if len(output) >= max_items: # changed a bit here...
            truncated_count += 1
        else:
            output.append("%s<li>%s%s</li>" % (indent, escaper(item), sublist))

    if truncated_count > 0: # and moved the truncated message outside the loop
        output.append("%s<li>...and %s more.</li>" % (indent, truncated_count))

    return "\n".join(output)

Drumroll...

Woohoo! We're basically there. 90% done (yeah not really, but that's how we work, right? always 90% done!)

To see if this works recursively, I can change max_items to 1 and see if they're applied on the comments too.

As an Australian friend of mine would say... "that's a beauty, mate!"

2. Make it right

Now that we made it work, we have to make it right, which means we need to get rid of that hard-coded max_items=3, it should be promoted to a parameter for the filter:

<ul id="deleted-objects">{{ deleted_objects|unordered_list:max_items }}</ul>

...where max_items is a context variable like 100.

To achieve this I just added a parameter on unordered_list. Since I want to keep this function backwards-compatible, it's a nullable param, with default None.

def unordered_list(value, autoescape=True, max_items: int | None = None):
    # ... previous code ...

    def list_formatter(item_list, tabs=1):
            # ... previous code ...

            # ...tweaked a bit here to handle None
            if max_items is not None and len(output) >= max_items:
                truncated_count += 1
            else:
                output.append("%s<li>%s%s</li>" % (indent, escaper(item), sublist))

        if truncated_count > 0:
            output.append("%s<li>...and %s more.</li>" % (indent, truncated_count))

        return "\n".join(output)

    return mark_safe(list_formatter(value))

Now, let's try with a value on the filter...

<ul id="deleted-objects">{{ deleted_objects|unordered_list:2 }}</ul>

Uh-oh, the template engine is not happy.

Basically, since unordered_filter already has a parameter autoescape, I can't add a second one, and pass a value for it. Bummer. I can't force max_items to be in front of autoescape as well, this would be a change of interface.

The simplest solution I could think of is to create a new filter, truncated_unordered_list, which will have the parameter max_items as its first parameter. This new filter is just thin layer for the original.

@register.filter(is_safe=True, needs_autoescape=True)
def truncated_unordered_list(value, max_items: int | None = None, autoescape=True):
    return unordered_list(value, max_items=max_items)

Maybe one of the reviewers will suggest a better way? we'll see :)

So now let's try our new filter:

<ul id="deleted-objects">{{ deleted_objects|truncated_unordered_list:2 }}</ul>

...the moment of truth...

We're back!

Let's try some variations. First, I don't pass any param, truncated_unordered_list should not break.

<ul id="deleted-objects">{{ deleted_objects|truncated_unordered_list }}</ul>

I won't put the screenshot here, but suffice to say, the whole list was rendered, un-truncated (is that a word? no) so we're good.

Next test, is if I pass a value of zero.

<ul id="deleted-objects">{{ deleted_objects|truncated_unordered_list:0 }}</ul>

It works, although it's a bit weird. We can do something better with it later.

To wrap up this, I just want to have a context variable, not a hard-coded number on the template, since this is a template that ships with Django. Something like this

<ul id="deleted-objects">{{ deleted_objects|truncated_unordered_list:delete_confirmation_max_objects }}</ul>

To get there, let's go from the bottom up and find our way.

We want to be able to add a parameter on an Admin class to define when to truncate, like this

@admin.register(Blog)
class BlogAdmin(admin.ModelAdmin):
    delete_confirmation_max_objects = 100
    # ...other code for this admin

This is cool because it's a backwards-compatible change: unless a value for delete_confirmation_max_objects is set, the delete confirmation page works just like it always worked, unchanged.

Aside: This also means that maybe this could be a change added to Django 6.1 or 6.2 (as of the time of writing, Django is on 6.0), since this change is opt-in - but honestly I don't know how this policy for backwards-compatible new features work..we'll see.

Anyway, having this variable there won't do anything by itself, I need to add it to ModelAdmin

class ModelAdmin(BaseModelAdmin):
    # ..lots of code...

    # Custom templates (designed to be over-ridden in subclasses)

    # ...other constants
    delete_confirmation_max_objects = None

Note that there's already a section for custom template variables, so it looks right to have our new constant there.

Finally, I need to pass this constant from admin to the template, via context. I searched for how and where other constants were passed, and found the place

# django/contrib/admin/actions.py
@action(
    permissions=["delete"],
    description=gettext_lazy("Delete selected %(verbose_name_plural)s"),
)
def delete_selected(modeladmin, request, queryset):
    # ... lots of code

    context = {
        **modeladmin.admin_site.each_context(request),
        # ... other values
        "delete_confirmation_max_objects": modeladmin.delete_confirmation_max_objects,
    }

Nice! We closed the loop.

Now when I do this...

<ul id="deleted-objects">{{ deleted_objects|truncated_unordered_list:delete_confirmation_max_objects }}</ul>

...and this

@admin.register(Blog)
class BlogAdmin(admin.ModelAdmin):
    delete_confirmation_max_objects = 2
    # ...other code for this admin

I see this

Mission accomplished! Time to prepare the PR!

For the PR, there was quite a lot bit and pieces I had to add. The most important bits being

Add tests for "happy path" and edge cases;
Add checks for invalid values like truncated_unordered_list:invalid_number

..and also some instructions from the contributor's guide, like release documentation and add my name to AUTHORS (which feels very...entitled? early? weird.)

Anyway.... Here is my first PR and I couldn't be more proud

I do expect this PR to require a few iterations before it's approved (if ever), but that's part of the process, and part of the experience I want to have as a Djangonaut.

Hope you enjoyed! Cheers from Floripa, Brazil!

[1] I know you thought "Palpatine returned" after "Somehow", if you're a Star Wars fan too.

Djangonaut Diaries: Week 1, part 2 - Creating and debugging a Django project

rodbv — Fri, 13 Mar 2026 16:45:24 +0000

Hi there!

In the previous post, we cloned a fork of Django locally, and ensure it's installed and running its tests. Now we can start using this local copy of Django to do whatever changes we want to do on Django, for a future contribution.

For some context, my goal is to eventually open a Pull Request (PR) to improve how the confirmation page in Admin for items we want to delete display these items. You can see a comment I added to this issue on Trac, which is Django's issue-tracking system.

To make the changes I want, I am planning to

Create a sample Django project with some simple models (Blog, Post and Comments), pointing to the local installation of Django we did on the first post;
Create a helper command to insert a few hundred blog posts and comments, to visualize the current behavior on Django Admin;
Set up Visual Studio Code's debugger to be able to debug the application, including the code in Django core, to better understand what is going on the admin template I want to change, especially how data is sent to the template, so I can start experimenting some changes and creating some tests (which will be a future post).

Cool! Let's get to it.

Creating a new project and pointing to our local Django install

For the new project, I will use uv, which is an extremely fast Python package manager and integrates seamlessly with pyproject.toml to manage the packages we will need to install as we go along.

First I will create a directory sample_blog, and type the following in it

❯ uv init .
❯ uv add --editable ../django
Using CPython 3.14.3 interpreter at: /usr/bin/python3.14
Creating virtual environment at: .venv
Resolved 5 packages in 1.95s
      Built django @ file:///home/rodrigo/code/django-test/django
Prepared 1 package in 1.93s
Installed 3 packages in 5ms
 + asgiref==3.11.1
 + django==6.1.dev20260311170544 (from file:///home/rodrigo/code/django-test/django)
 + sqlparse==0.5.5

As you can see, uv took care of creating a virtual env for me, which is great. We can activate it as usual with source command.

❯ source .venv/bin/activate

We can now use it to create our sample project, as usual.

❯ django-admin startproject sample_blog .

Nice! So we now have a local installation of Django and our own project, side-by-side. Next I will create an application, the models I want (Blog, Post, Comments). I will not cover every step here as it's the standard set of steps, basically what you'll find on Django's tutorial, but you can have a look at what I've done in this repository.

In summary what will be done is:

create an app called "blog"
create three models: Blog, Post and Comment
add these models to Admin, so I can manage them.

Once I make and run the migrations and create a superuser, we can see this by opening Admin

Adding sample data using commands

Back to the issue I want to send a PR, it's related to the number of items that we show when we confirm the deletion of one or more things.

To see the issue, I would like to add about 1000 blog posts, which is boring to do manually. I could open Django shell and run an ad-hoc script with a loop, but then, when I decide to run it again in the future, this code will be gone.

It's better to create a command, which will be part of my application.

Mine will be called create_posts and will have accept an argument --quantity (or -q) with the number of posts. For simplicity sake we will just add them to the first blog we find, and create a new one if none exists yet - we can make this more complicated later, but it's fine by now ;)

Since we want some "lorem ipsum" for the posts, let's add a library called ... lorem, which is great for this job. Let's also use a nice little lib called tdqm which shows a progress bar while the command is running, good for long-running stuff

uv add lorem tdqm

This is how the command looks like

# blog/management/commands/create_posts.py
import lorem
from django.contrib.auth import get_user_model
from django.core.management.base import BaseCommand
from tqdm import tqdm

from blog.models import Blog, Post

User = get_user_model()


class Command(BaseCommand):
    help = "Creates posts for the first available blog"

    def add_arguments(self, parser):
        parser.add_argument(
            "-q",
            "--quantity",
            type=int,
            default=5,
            help="Number of posts to create (default: 5)",
        )

    def handle(self, *args, **options):
        quantity = options["quantity"]

        user, _ = User.objects.get_or_create(
            username="admin",
            defaults={"is_staff": True, "is_superuser": True},
        )

        # use first blog or create one if it doesn't exist
        blog = Blog.objects.first()
        if not blog:
            blog = Blog.objects.create(
                        name=lorem.sentence().rstrip(".")[:120],
                        description=lorem.paragraph(),
                        created_by=user,
                    )
            self.stdout.write(f"Created new blog: {blog.name}")

        # create posts with progress bar
        for _ in tqdm(range(quantity), desc="Creating posts"):
            Post.objects.create(
                blog=blog,
                title=lorem.sentence().rstrip(".")[:300],
                content=lorem.text(),
                created_by=user,
            )

        self.stdout.write(self.style.SUCCESS(f"Created {quantity} posts in '{blog.name}'")

Running the command I will see something like this

❯ python manage.py create_posts -q 1000
Creating posts: 100%|██████████████████████████████████████████████████████████████████████| 1000/1000 [00:03<00:00, 290.33it/s]
Created 1000 posts in 'My Djangonaut Blog'

If I refresh the delete page for the blog, I will see the new posts there:

Using VS Code debugger

Ok, so eventually I'd like to send a PR in which this huge list is truncated, for example it shows the first 100 elements and adds something like "...and 998 more blog posts" at the end, so that the red delete button is not 10,000 pixels down the page.

For that I need to first find the template that renders this list, and then put a debugger somewhere in the view related to this template to see how the data is sent to it.

I am using VS Code, so I will open it on my sample_blog directory, and add the django directory to the workspace, so I can navigate seamlessly between my app code and Django's own code.

Now, to find the probable spot where the code is, I will just search for a string that is probably somewhere in a template, Are you sure you want to delete

I actually found 2 occurrences, and then I figured out that one is for the template when we're deleting one item, and one for "bulk delete" of several items. We may want to change both, but for now I will focus on the first one which is in django/contrib/admin/templates/admin/delete_confirmation.html

I will not put the full code of the template here, but you can check here.

The interesting part is the where I see a ul item that points to deleted_objects, which is probably the list I want:

    <p>{% blocktranslate %}Are you sure you want to delete the {{ object_name }} “{{ escaped_object }}”? All of the following related items will be deleted:{% endblocktranslate %}</p>
    {% include "admin/includes/object_delete_summary.html" %}
    <h2>{% translate "Objects" %}</h2>
    <ul id="deleted-objects">{{ deleted_objects|unordered_list }}</ul>

Let's search for deleted_objects. Note that I selected the options for case-sensitive search and match whole words, to reduce the number of matches.

Looks like the thing we want is on django/contrib/admin/options.py, around line 2230!

Ok, so to wrap up this session, I wonder how does this thing looks like? is it a string with <li>s?

Let's debug!

First thing to do is to put a breakpoint, that red dot you see on the screenshot above, which can be done either by clicking to the left of the line number, or press F9 when the cursor is on the line.

Then, pressing CTRL + SHIFT + P to open the command palette, or pressing F5, will start debugging. VS Code will present you with a few options, and you should choose

More Python debugger options -> Python debugger -> Django

You will know that everything is working fine when you see this toolbar showing up, which belongs to the debugger

If I reload the delete page on Admin, and go back to VS Code, I will see this

We're in! Note that I can see the value of deleted_objects and all the other variables in scope.

In particular, deleted_objects looks like this

['Blog: <a href="/admin/blog/blog/6b7346c7-42c0-4947-b8a9-be5f1d451baa/change/">My Djangonaut Blog</a>', 
['Post: <a href="/admin/blog/post/48200b32-a238-4325-981f-ff5a43eaf324/change/">First post</a>', [...], 
'Post: <a href="/admin/blog/post/42928075-1743-43c6-b1b4-f41ce6ad21c6/change/">Second post</a>', 
'Post: <a href="/admin/blog/post/43e02f24-0f13-4eb8-baa2-8e3434764148/change/">Labore modi amet magnam velit adipisci 
...]
]

So it's a list of strings with a mix of keys and links... interesting!

I guess we're ready to start playing with these values, see how we could truncate them, but that will be the next post, as this is obviously not a trivial thing, we have some sort of recursive data structure to deal with... I'll need some coffee for this :D

Good news is, the goal for this sessions is accomplished, we can now debug Django! Woot!

See you soon!

Djangonaut Diaries: Week 1 - Setting up the stage (and the machine)

rodbv — Wed, 11 Mar 2026 23:01:22 +0000

I am officially a Djangonaut! Joining the latest cohort (Session 6) of the Djangonaut Space program is very cool, and it means it's time to contribute back to the framework that I love and have been working with daily for the last 3.5 years: Django.

But, before I can start contributing with code, I need to get it running locally, setup a working local development environment and get its unit tests passing, to ensure that any future changes I make don't break something that I may not be aware of :D

While doing this, I encountered a few issues that I had to figure out, and this blog post is to discuss a bit how I overcame these issues on my machine which runs Fedora 43. If you're using another Linux-based distro, like Ubuntu, most of the tips here will help as well, you will just have to replace the dnf command with apt.

Step 0: Fork, clone and venv

To send code back to Django as future Pull Requests, first I need to create a "fork" of Django, which is a copy of its repository at https://github.com/django/django. To create the fork I just had to click on the "Fork" button on the top right of the django repo.

Once I had my own fork, I could clone it locally, by typing the following on my terminal

git clone git@github.com:rodbv/django.git

Note that rodbv refers to my fork, it's my username on github. Replace with your own username!

Now I can start the instructions on how to run Unit Tests on the official documentation.

To isolate the libraries we install in their own environment, and not mess up with other programs in my machine, it's good to create a virtual environment, or venv.

After entering the directory I just created via clone, I ran

python -m venv .venv
source .venv/bin/activate

Continuing with the instructions in the doc, I entered the tests directory where we'll run them:

cd tests
python -m pip install -e ..
python -m pip install -r requirements/py3.txt

It started well for a few seconds, but then I got this

building 'pywatchman.bser' extension
creating build/temp.linux-x86_64-cpython-314/pywatchman
gcc -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O3 -fPIC -I./pywatchman -I/usr/include/python3.14 -c pywatchman/bser.c -o build/temp.linux-x86_64-cpython-314/pywatchman/bser.o
error: command 'gcc' failed: No such file or directory

The Art of Deciphering the Terminal

When working on a large codebase like Django, the terminal provides a wealth of information. Learning to get a deep breath and "read between the lines" of a long error message is a great skill for any contributor.

The key is looking at the very last command attempted. Here, the system tried to call gcc (the GNU C Compiler). The error No such file or directory simply means the tool isn't installed yet. Once the compiler is available, this error should go away

The fix:

sudo dnf install gcc

That's sudo apt install gcc if you're on Ubuntu, Mint or Pop!OS

With the compiler ready, I tried again and encountered a new message:

Building wheel for pylibmc (pyproject.toml) ... error
  In file included from src/_pylibmcmodule.c:34:
  src/_pylibmcmodule.h:41:10: fatal error: Python.h: No such file or directory
     41 | #include <Python.h>
        |          ^~~~~~~~~~
  compilation terminated.
  error: command '/usr/sbin/gcc' failed with exit code 1

How to read this:
Now we see that gcc is working perfectly, but it’s looking for Python.h. In C development, .h files are "headers", the interface that allows C code to interact with Python. Seeing this means you have the Python runtime, but you need the files used for building Python-linked applications.

I didn't know exactly what is the name of the dependency to install that would give me these .h files, but a quick trip to Google informed me that I should install this

sudo dnf install python3-devel

Running the install command again, gave me this

src/_pylibmcmodule.c:36:12: fatal error: libmemcached/memcached.h: No such file or directory
     36 | #  include <libmemcached/memcached.h>
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~

Ok, one more Google search and I see that I need to install memcached-devel

sudo dnf install libmemcached-devel

I think you get the drift now. I had one more of these issues, in which the error looked like this

src/_pylibmcmodule.c:36:12: fatal error: zlib.h: No such file or directory
    36 | #  include <zlib.h>
       |            ^~~~~~~~
compilation terminated.

...and the fix (again, with the help of Google) was to install zlib-devel.

I later learned that Fedora has a "shortcut" for many of these tools. Instead of installing gcc and other basics one by one, you can install a whole suite of building tools at once using:
sudo dnf groupinstall "Development Tools"
This is the Fedora equivalent of Ubuntu's build-essential and is a great time-saver for a fresh install

Victory: The Green Dots

After resolving the system dependencies, I ran the install command one last time. This time, instead of errors, I saw a beautiful cascade of successful installations:

Installing collected packages: PyYAML, pywatchman, pylibmc, Pillow, numpy, ...
Successfully installed MarkupSafe-3.0.3 Pillow-12.1.1 PyYAML-6.0.3 pylibmc-1.6.3 pywatchman-3.0.0 ...

With the environment fully prepped, it was time for the moment of truth. I ran the test suite:

./runtests.py

And I could finally see the tests running! Watching the terminal fill with dots as over 19,000 tests passed was incredibly rewarding.

In the next post I will show you how I got to create a sample Django project and point to this local installation, and debug it using VS Code's awesome debugging tools :)

Cheers from Brazil!

Creating a To-do app with HTMX and Django, part 9: active search

rodbv — Sat, 11 Jan 2025 22:08:09 +0000

Welcome back! In the part 9, we will add one more feature of that showcases how HTMX can provide common rich web experience with very little code: active search.

This is how the result will look like, sending POST requests to /tasks/search and swapping the results in the list.

Adding the URL and view

We will need a new URL, /tasks/search, which will receive a POST request with a parameter called query.

In core/urls.py we add the route

# core/urls.py

from django.urls import path
from . import views

urlpatterns = [
    path("", views.index, name="index"),
    path("tasks/", views.tasks, name="tasks"),
    path("tasks/<int:task_id>/toggle/", views.toggle_todo, name="toggle_todo"),
    path("tasks/<int:task_id>/", views.task_details, name="task_details"),
    path("tasks/<int:task_id>/edit/", views.edit_task, name="edit_task"),
    path("tasks/search/", views.search, name="search"), # <-- NEW
]

In core/views.py we add the new method

# core/views.py

... previous code

@login_required
@require_http_methods(["POST"])
def search(request):
    query = request.POST.get("query")

    if not query:
        return redirect("tasks")

    results = request.user.todos.filter(title__icontains=query).order_by("-created_at")

    return render(
        request,
        "tasks.html#todo-items-partial",
        {"todos": results},
        status=HTTPStatus.OK,
    )

The first if is to handle empty queries; when the user clears the search field we can simply redirect to the tasks view, which returns all the todo items and preserve the infinite scrolling behavior.

When the query is not empty, we're filtering the todo items with icontains, which is a case-insensitive clause, and returning the matching items. Once again we're using our partial template, todo-items-partial, which accepts a list of todo's.

Adding the search input in the template

In the tasks.html template we'll add a new input on the top of the page, and connect it to our new url/view


..previous code

  <div class="w-full max-w-2xl">
    <div class="mb-4">

      <!-- NEW -->
      <div class="flex justify-center">
        <label class="rounded-full input my-4 w-1/2">
          <span class="icon-search text-gray-500"></span> 
          <input type="search" name="query" class="grow text-md" placeholder="Type to search..." 
          hx-post="{% url 'search' %}"
          hx-trigger="input changed delay:500ms, keyup[key=='Enter']"
          hx-target="#todo-items" />
        </label>
      </div>

   <form ...

The code to fire the search url on change, with debouncing, is straight from the examples section of the official HTMX site.

It's nice how self-explanatory the hx-trigger attribute reads: whenever the input is changed, with a delay (debounce) of 500 milliseconds, or when the ENTER key is pressed.

hx-target informs that we want to place the results in the #todo-items element (the ul), and since we want to change its children, we don't need to specify hx-swap, its default value is what we want already, innerHTML.

While we're checking the code working,let's have a look in developer tools/network, to see the requests being fired.

Testing the view

Let's add some tests to our new view, in test_view_tasks.py

The first test is our "happy path", we want to ensure we just return the items searched, which are two of the three fake items we have in the test. The other two test for no matches and empty searches.

@pytest.mark.django_db
def test_search_filtering(client, make_todo, make_user):
    user = make_user()
    client.force_login(user)

    make_todo(title="Todo 1", user=user)
    make_todo(title="Another Todo", user=user)
    make_todo(title="Something else", user=user)

    response = client.post(reverse("search"), {"query": "Todo"})
    content = response.content.decode()

    assert "Todo 1" in content
    assert "Another Todo" in content

    assert "Something else" not in content

@pytest.mark.django_db
def test_search_zero_matches_returns_empty_list(client, make_todo, make_user):
    user = make_user()
    client.force_login(user)

    make_todo(title="Todo 1", user=user)
    make_todo(title="Another Todo", user=user)
    make_todo(title="Something else", user=user)

    response = client.post(reverse("search"), {"query": "Nonexistent"})
    content = response.content.decode()

    assert not any(
        todo in content for todo in ["Todo 1", "Another Todo", "Something else"]
    )

@pytest.mark.django_db
def test_search_empty_query_redirects_to_all_tasks(client, make_todo, make_user):
    user = make_user()
    client.force_login(user)

    make_todo(title="Todo 1", user=user)
    make_todo(title="Another Todo", user=user)
    make_todo(title="Something else", user=user)

    response = client.post(reverse("search"), {"query": ""})
    assert response.status_code == HTTPStatus.FOUND  # redirect
    assert response.url == reverse("tasks")

Running all tests to ensure it's all good

❯ uv run pytest
Test session starts (platform: darwin, Python 3.12.8, pytest 8.3.4, pytest-sugar 1.0.0)
django: version: 5.1.4, settings: todomx.settings (from ini)
configfile: pyproject.toml
plugins: sugar-1.0.0, django-4.9.0
collected 10 items

 core/tests/test_todo_model.py ✓                                                                                                                                    10% █
 core/tests/test_view_tasks.py ✓✓✓✓✓✓✓✓✓                                                                                                                           100% ██████████

Results (0.53s):
      10 passed

A note on testing context and templates

Someone more experienced with testing Django templates may wonder why we're testing the content of the rendered template, and not the context being passed and the template invoked, which would be more assertive, as such:

   ...
   assertTemplateUsed(response, "tasks.html#todo-items-partial")
   todos = {todo.title for todo in response.context["todos"]}
   assert todos == {"Todo 1", "Another Todo"}

This is because of a current issue in django-template-partials, issue #54, which has not been fixed yet (I'm giving it a go, but it's not trivial..)

Due to this issue the response does not contain values for context and templates.

Anyway, this is it for active search! all in all, not much code for some awesome improvement on UX. As usual, the final version of this code can be found in its branch, https://github.com/rodbv/todo-mx/tree/part-9. Cheers!

Creating a To-do app with HTMX and Django, part 8: inline edit, and using Iconify

rodbv — Wed, 08 Jan 2025 21:34:42 +0000

Welcome back to the series of posts in which I'm documenting my learning process of HTMX with Django. So far we have implemented the ability to add todo items, delete them, toggle their completion status, and infinite scroll.

To see the whole series or articles, you can check dev.to/rodbv. The repository of the code so far is at github.com/rodbv/todo-mx.

The next goal is to implement inline edit of the todo-item: when we click on an "edit" button somewhere in the list item, it would show an input text with the task title, and save/cancel buttons.

As usual, there are a few possibilities here; one is to render both the <span> with the title, and a form with <input> for each row, and use plain Javascript (or AlpineJS) to toggle edit mode.

I'd rather keep following the hypermedia/server-oriented approach and return the form for the row when a request to edit and item is made to the server; in other words, add an "edit" button on the row that sends a request to tasks/:id/edit which in turn returns the form we want, replacing the whole line.

First, let's add a new button, "Edit", near the existing "Delete" button, configured with the GET request we want:

... previous code

{% for todo in todos %} 
  <li class="list-row">
    <input type="checkbox" class="checkbox checkbox-lg checkbox-info mr-4"
      hx-put="{% url 'toggle_todo' todo.id %}"
      hx-swap="outerHTML"
      hx-target="closest li"
      hx-on:click="this.setAttribute('disabled', 'disabled')"
      {% if todo.is_completed %} checked {% endif %}
    />

    <span class="flex-1 text-lg 
      {% if todo.is_completed %}text-gray-500{% endif %}"
    >{{ todo.title }}</span>

    <!-- THIS BUTTON IS NEW -->
    <button hx-get="{% url 'edit_task' todo.id %}" 
      class="btn btn-circle btn-sm btn-info btn-outline"
      hx-swap="innerHTML"
      hx-target="closest li"
      >Edit</button>

      <button hx-delete="{% url 'task_details' todo.id %}" 
        class="btn btn-circle btn-sm btn-error btn-outline"
        hx-on:todo-deleted="this.closest('li').remove()"
        hx-confirm="Delete this item?">x</button>
    </li>
  {% endfor %}

... previous code

This is how the button will look like:

Later, we will use Iconify to add some nice icons to the new buttons, but let's focus on the task now.

Having the button to request the edit form, it's time to add the new url and view. First, in core/urls.py:

# core/urls.py

from django.urls import path
from . import views

urlpatterns = [
    path("", views.index, name="index"),
    path("tasks/", views.tasks, name="tasks"),
    path("tasks/<int:task_id>/toggle/", views.toggle_todo, name="toggle_todo"),
    path("tasks/<int:task_id>/", views.task_details, name="task_details"),
    # NEW
    path("tasks/<int:task_id>/edit/", views.edit_task, name="edit_task"),
]

Then on core/views.py:


# core/views.py

...previous code

# NEW
@login_required
@require_http_methods(["GET"])
def edit_task(request, task_id):
    todo = request.user.todos.get(id=task_id)
    return render(
        request,
        "tasks.html#todo-item-edit",
        {"todo": todo},
    )

Note that the new url/view is returning a new partial name, tasks.html#todo-item-edit. It's time for us to define this partial, in tasks.html. Since this partial only rendered when requested by this view, we will add it to the bottom of the file, and do not set the inline attribute, which would cause it to be rendered by default.

In core/templates/tasks.html, at this to the very end of the file:


...previous code

{% partialdef todo-item-edit %}
<input type="checkbox" form="edit-form" name="is_completed" class="checkbox checkbox-lg checkbox-info mr-2"
  {% if todo.is_completed %} checked {% endif %}/>

<div class="flex space-x-4">

  <form id="edit-form" hx-post="{% url 'task_details' todo.id %}" hx-swap="outerHTML" hx-target="closest li" class="w-full flex items-center space-x-2">
    <input type="text"  name="title" required class="input input-bordered flex-1 text-lg" value="{{ todo.title }}" autofocus />
  </form>

  <button form="edit-form" class="btn btn-circle btn-sm btn-success" 
    type="submit"
  >Save</button>

  <button class="btn btn-circle btn-sm btn-error btn-outline" 
    hx-get="{% url 'task_details' todo.id %}" 
    hx-swap="outerHTML" 
    hx-target="closest li"
  >Cancel</button>

</div>
{% endpartialdef %}

We're adding the is_completed checkbox, to show the status of the todo, which will also be sent with the form, in case the user decides to change it. Note that we removed its hx-put trigger. We also have an input text, set as required, for the title, and two buttons:

The Save button will submit the form to /tasks/:id/details, with the new value of the title;
The Cancel button will send a GET request to the same route, which will simply return the todo item using the usual partial for todos, tasks.html#todo-items-partial. We'll get to this code soon, first let's see it in action to show what we just described.

One nice side-effect of using a form to edit the todo item is that the Enter/Return key also works.

We already have the url/view being used by both the save and cancel routes - they're both calling task_details, with the methods POST and GET respectively, so let's just extend the view function.

# core/views.py

... previous code

@login_required
@require_http_methods(["DELETE", "POST", "GET"])
def task_details(request, task_id):
    todo = request.user.todos.get(id=task_id)

    if request.method == "DELETE":
        todo.delete()
        response = HttpResponse(status=HTTPStatus.NO_CONTENT)
        response["HX-Trigger"] = "todo-deleted"
        return response

    if request.method == "POST":
        title = request.POST.get("title")
        if not title:
            raise ValueError("Title is required")

        todo.title = title
        todo.is_completed = bool(request.POST.get("is_completed"))
        todo.save()

    # both POST and GET will just return the todo details
    return render(
        request,
        "tasks.html#todo-items-partial",
        {"todos": [todo]},
    )

That's it for the changes on in the view.

Ideally, we should be using th PUT method, not POST. While this is possible, it would require much more code, we would have to drop the form and send hx-put straight from the Save button, using hx-vals to collect the values of is_completed and title. We would also lose two benefits of the form, which is the native required validation of the input, and autofocus for the title input, so I think it's a reasonable tradeoff from REST-ful purity.

Using Iconify for the icons

The last thing we want to do is to make the buttons nicer-looking. I've decided to use Iconify, an open-source, free alternative; specifically, their "Ming Cute - Line" set.

We can just copy the CSS for the icons we want, in put in a new file in our project, /core/static/css/styles.css (click here to see the code). I'm adding a second class name to each style, to make it simpler to refer to the class names. It's a good idea to keep the original names for reference in Iconify.

/* 
To get more icons, go to 
https://icon-sets.iconify.design/mingcute/page-3.html?suffixes=Fill 
*/

.icon-delete,
.mingcute--folder-delete-line {
    display: inline-block;
    width: 18px;
    height: 18px;

...rest of CSS

In core/templates/tasks.html we can add a reference to the CSS file and use them in each button, for example:

  <button class="btn btn-circle btn-sm btn-error btn-outline" 
    hx-get="{% url 'task_details' todo.id %}" 
    hx-swap="outerHTML" 
    hx-target="closest li"
  ><span class="icon-back"></span></button>

This is how it looks like. The whole code can be found in the part 8 branch. Feel free to ask for what you want in the next posts!

Creating a To-Do app with HTMX and Django, part 7: infinite scroll

rodbv — Mon, 06 Jan 2025 02:57:32 +0000

This is part 7 of the series in which I'm documenting my learning process of HTMX with Django, in which we will follow HTMX's documentation to implement an infinite scroll feature for the todo items.

If you want to check the rest of the series, have a look at dev.to/rodbv for the complete list.

Updating the partial template to load several items

When we implement infinite scroll, we will have to return several todo items (the next "page" of items) and load them in the partial template we currently have. This means changing a bit how our partial template is composed; it's currently set as described in the diagram below, in which the partial template is responsible for rendering a single todo item:

We want to invert the order, having the partial around the for loop:

Let's perform the swap in the template core/templates/index.html:

<ul id="todo-items" class="list bg-base-100 rounded-box shadow-md">
  {% partialdef todo-items-partial inline %}
    {% for todo in todos %} 
        <li class="list-row">
           ... code inside the li, unchanged
        </li>
      {% endfor %}
    {% endpartialdef %} 
</ul>

Soon we will get back to the template to add the hx-get ... hx-trigger="revealed" bit that performs the infinite scroll, but first let's just change the view to return several items instead of one on the toggle and create operations:

... previous code 

def _create_todo(request):
    title = request.POST.get("title")
    if not title:
        raise ValueError("Title is required")

    todo = Todo.objects.create(title=title, user=request.user)

    return render(
        request,
        "tasks.html#todo-items-partial", # <-- CHANGED
        {"todos": [todo]}, # <-- CHANGED
        status=HTTPStatus.CREATED,
    )

... previous code 


@login_required
@require_http_methods(["PUT"])
def toggle_todo(request, task_id):
    todo = request.user.todos.get(id=task_id)
    todo.is_completed = not todo.is_completed
    todo.save()

    return render(
        request,
        "tasks.html#todo-items-partial",  # <-- CHANGED
        {
            "todos": [todo], # <-- CHANGED
        },
    )

The tests checking for the content being still pass, and the page looks the same, so we're good to implement the infinite scroll itself.

Implementing infinite scroll

On the template, we need to setup a hx-get request to /tasks, with hx-trigger="revealed", which means the GET request is only fired when the element is about to enter become visible on screen; this means we want it to be set after the last element in the list, and we also need to indicate which "page" of data we want to load. In our case we'll show 20 items at a time.

Let's change the template accordingly:

    <ul id="todo-items" class="list bg-base-100 rounded-box shadow-md" hx-indicator=".indicator">
      {% partialdef todo-items-partial inline %}
        {% for todo in todos %} 
          <li class="list-row">
                ...code inside the li is unchanged
          </li>
        {% endfor %}
        {% if next_page_number %}
          <p id="loading" 
              hx-get="{% url 'tasks' %}?page={{next_page_number}}" 
              hx-trigger="revealed" 
              hx-swap="outerHTML" 
              class="mx-auto my-4 loading loading-dots"></p>
        {% endif %}
        {% endpartialdef %} 
    </ul>

There's an if next_page_number check around the "loading" icon at the bottom of the list, it will have two purposes: one is to indicate when we're loading more data, but more importantly, when the loader is revealed (it appears on the visible part of the page), it will trigger the hx-get call to /tasks, passing the page number to be retrieved. The attribute next_page_number will also be provided by the context

The directive hx-swap:outerHTML indicates that we will replace the outerHTML of this element with the set of <li>s we get from the server, which is great because not only we show the new data we got, but we also get rid of the loading icon.

We can now move to the views file.

As a recap, here's how the GET /tasks view looks like by now; it's always returning the full template.

@require_http_methods(["GET", "POST"])
@login_required
def tasks(request):
    if request.method == "POST":
        return _create_todo(request)

    # GET /tasks
    context = {
        "todos": request.user.todos.all().order_by("-created_at"),
        "fullname": request.user.get_full_name() or request.user.username,
    }

    return render(request, "tasks.html", context)

There's a change done in the code above already, which is to sort by newest todos first; now that we expect to have a long list, it doesn't make sense to add new items at the bottom and mix it with infinite scroll - the new item will end up mixed in the middle of the list.

We now need to differentiate regular GET requests from HTMX requests, for which we will return just a list of todos and our partial template. There is a library called django-htmx which is very handy, as it extends the request parameter with attributes like request.htmx and the values of all hx-* attributes, but that's overkill at the moment; let's just check for the HTMX header by now, and handle paging using Django's paginator.

# core/views.py

... previous code

PAGE_SIZE = 20

...previous code

@require_http_methods(["GET", "POST"])
@login_required
def tasks(request):
    if request.method == "POST":
        return _create_todo(request)

    page_number = int(request.GET.get("page", 1))

    all_todos = request.user.todos.all().order_by("-created_at")
    paginator = Paginator(all_todos, PAGE_SIZE)
    curr_page = paginator.get_page(page_number)

    context = {
        "todos": curr_page.object_list,
        "fullname": request.user.get_full_name() or request.user.username,
        "next_page_number": page_number + 1 if curr_page.has_next() else None,
    }

    template_name = "tasks.html"

    if "HX-Request" in request.headers:
        template_name += "#todo-items-partial"

    return render(request, template_name, context)

The first thing we do is to check the page param, and set it to 1 if it's not present.

We check for the HX-Request header in the request, which will inform us whether the incoming request is from HTMX, and lets us return the partial template or the full template accordingly.

This code requires some tests for sure, but before that let's just give it a go. Have a look at the network tool, how the requests are fired as the page is scrolled, until we reach the last page. You can also see the animated "loading" icon showing for a brief moment; I've throttled the network speed to 4g to make it visible for longer.

Adding tests

To wrap it up, we can add a test to ensure pagination is working as intended

# core/tests/test_view_tasks.py

... previous tests

@pytest.mark.django_db
def test_tasks_pagination(client, make_todo, make_user):
    user = make_user()
    client.force_login(user)

    # create 2 pages of data
    for i in range(PAGE_SIZE + 3):
        make_todo(title=f"Todo #{i}", user=user)

    response = client.get(reverse("tasks"))

    context = response.context
    assert context["next_page_number"] == 2
    assert len(context["todos"]) == PAGE_SIZE

    # add header to ensure this is processed as an HTMX request
    response = client.get(reverse("tasks") + "?page=2", HTTP_HX_Request="true")

    content = response.content.decode().strip()
    assert content.startswith("<li")
    assert content.endswith("</li>")

    assert ">Todo #22<" not in content
    assert ">Todo #1<" in content

That's it by now! This was by far the most fun I've had with HTMX so far. The full code for this post is here.

For the next post I'm considering adding some client state management with AlpineJS, or maybe add a "due date" feature. See you!

Creating a To-Do app with HTMX and Django, part 6: implementing Delete with tests

rodbv — Sat, 04 Jan 2025 21:28:30 +0000

This is part 6 of a set of posts in which I am documenting how I'm learning HTMX with Django. You can find the whole series on dev.to/rodbv.

In this post, we will implement the ability to delete todo items, and this time around we'll follow a test-driven approach, in short cycles of writing tests, seeing them fail, and then making them pass (Red-Green-Refactor).

Creating the view and url

First, let's implement the task details view and map it to the url, which will called with the DELETE method on /tasks/:id.

# core/tests/test_view_tasks.py

... previous tests

@pytest.mark.django_db
def test_delete_task(client, make_todo, make_user):
    user = make_user()
    client.force_login(user)

    todo = make_todo(title="New Todo", user=user)

    response = client.delete(reverse("task_details", args=[todo.id]))

    assert response.status_code == HTTPStatus.NO_CONTENT
    assert not user.todos.filter(title="New Todo").exists()

Note that this view will return 204 - No content, which indicated that the request was fulfilled but it has no return data, since the todo was deleted.

Running the test will make it fail, since we haven't implemented the url/view yet:

❯ uv run pytest
Test session starts (platform: darwin, Python 3.12.8, pytest 8.3.4, pytest-sugar 1.0.0)
django: version: 5.1.4, settings: todomx.settings (from ini)
rootdir: /Users/rodrigo/code/opensource/todo-mx
configfile: pyproject.toml
plugins: sugar-1.0.0, django-4.9.0, mock-3.14.0
collected 6 items

============= short test summary info =================================
FAILED core/tests/test_view_tasks.py::test_delete_task - django.urls.exceptions.NoReverseMatch: Reverse for 'task_details' not found. 'task_details' is not a valid view function or pattern name.

Results (0.45s):
       5 passed
       1 failed
         - core/tests/test_view_tasks.py:66 test_delete_task

Let's add the url:

# core/urls.py

from django.urls import path
from . import views

urlpatterns = [
    path("", views.index, name="index"),
    path("tasks/", views.tasks, name="tasks"),
    path(
        "tasks/<int:task_id>/toggle/",
        views.toggle_todo,
        name="toggle_todo",
    ),
    path(  # <-- NEW
        "tasks/<int:task_id>/",
        views.task_details,
        name="task_details",
    ),
]

Running the test again will yield a new error message. Note how the failing methods are guiding the development of the feature, which is a nice side-effect of TDD

FAILED core/tests/test_view_tasks.py::test_delete_task - AttributeError: module 'core.views' has no attribute 'task_details'

Results (0.45s):
       1 failed
         - core/tests/test_view_tasks.py:66 test_delete_task
       5 deselected

Let's follow what was asked in the error message and add a task_details view

# core/views.py

... previous code


@login_required
@require_http_methods(["DELETE"])
def task_details(request, task_id):
    todo = request.user.todos.get(id=task_id)
    todo.delete()

    return HttpResponse(status=HTTPStatus.NO_CONTENT)

Running the tests now will have them all passing

❯ uv run pytest
collected 6 items

 core/tests/test_todo_model.py ✓                                                                                                                                    17% █▋
 core/tests/test_view_tasks.py ✓✓✓✓✓                                                                                                                               100% ██████████

Results (0.37s):
       6 passed

Adding the delete button on the template

We can now move to our template.

My first approach was to use hx-target and hx-swap, similarly to how I've done with the toggle function

          <button hx-delete="{% url 'task_details' todo.id %}" 
          class="btn btn-circle btn-sm btn-error btn-outline"
          hx-swap="outerHTML" 
          hx-target="closest li">x</button>

This rendered the button, and when I clicked on it the request was fulfilled accordingly, but the row was not removed automatically.

Checking the HTMX documentation, I learned that 204 No-content responses do not fire swaps by default.

There are a few ways around it:

Change the configuration of HTMX to let 204 promote swaps (explained in the article above);
Change the response to 200 OK, which worked fine when I tested it
Fire an HTMX trigger and responding to it on the template with and hx-on:[trigger-name] handler.

Let's go for the third option as it's something new; it's pretty straightforward too.

First let's add the trigger to the response, as a header. We can first add the expected result to the test and see it fail

@pytest.mark.django_db
def test_delete_task(client, make_todo, make_user):
    user = make_user()
    client.force_login(user)

    todo = make_todo(title="New Todo", user=user)

    response = client.delete(reverse("task_details", args=[todo.id]))

    assert response.status_code == HTTPStatus.NO_CONTENT
    assert response["HX-Trigger"] == "todo-deleted" # <-- NEW
    assert not user.todos.filter(title="New Todo").exists()

...and then add the code to make the test pass:

# core/views.py

...previous code 

@login_required
@require_http_methods(["DELETE"])
def task_details(request, task_id):
    todo = request.user.todos.get(id=task_id)
    todo.delete()

    # CHANGED
    response = HttpResponse(status=HTTPStatus.NO_CONTENT)
    response['HX-Trigger'] = 'todo-deleted'
    return response

Now that we have the trigger header, we can respond to it in the template. If you want to see the whole code for the template, it's in the part 06 branch.

<button hx-delete="{% url 'task_details' todo.id %}" 
  class="btn btn-circle btn-sm btn-error btn-outline"
  hx-on:todo-deleted="this.closest('li').remove()"
>x</button>

Adding confirmation

Our delete code is working but it's nice to have a confirmation, since this is a destructive action.

The simplest solution is to use hx-confirm, but it's not very good-looking, but gets the work done.

<button hx-delete="{% url 'task_details' todo.id %}" 
  class="btn btn-circle btn-sm btn-error btn-outline"
  hx-on:todo-deleted="this.closest('li').remove()"
  hx-confirm="Delete this item?"
>x</button>

You can find the final version of the code so far in the part 6 branch.

In part 7 of this series, we will implement infinite scrolling.

Creating a To-Do app with HTMX and Django, part 5: Testing the views

rodbv — Fri, 03 Jan 2025 12:13:07 +0000

Before we add more features to the Todo app, let's have a look at ways to test the urls, views and the data being returned by them to ensure we're not breaking anything. This is part 5 in our series on how to use HTMX with Django and template partials. You can find the whole list of posts on dev.to/rodbv.

Testing GET /tasks

Our /tasks route supports two methods, GET to return the whole tasks.html template and all the tasks of the user, and POST that creates one task and returns a partial view.

Let's start with the GET route, by adding a new file tests/test_view_tasks.py:

# core/tests/test_view_tasks.py

from http import HTTPStatus
import pytest
from django.urls import reverse
from pytest_django.asserts import assertTemplateUsed


@pytest.mark.django_db
def test_get_tasks_returns_tasks_template_and_todo_items_of_the_user(
    client, make_user, make_todo
):
    user = make_user()
    client.force_login(user)

    make_todo(_quantity=3, user=user)

    response = client.get(reverse("tasks"))

    assertTemplateUsed(response, "tasks.html")

    assert response.status_code == HTTPStatus.OK
    assert len(response.context["todos"]) == 3
    assert all(todo.user == user for todo in response.context["todos"])

For these tests we opted to not wrap them around with a test class, since we don't need any specific fixtures or setup code. We're making use of the make_user and make_todo general fixtures we defined on conftest.py.

The first assertion we're making is using pytest_django's assertTemplateUsed, and we're also making sure we're returning all the Todo items of the user.

To run the test, we execute the uv command

❯ uv run pytest
Test session starts (platform: darwin, Python 3.12.8, pytest 8.3.4, pytest-sugar 1.0.0)
django: version: 5.1.4, settings: todomx.settings (from ini)
rootdir: /Users/rodrigo/code/opensource/todo-mx
configfile: pyproject.toml
plugins: sugar-1.0.0, django-4.9.0, mock-3.14.0
collected 2 items

 core/tests/test_todo_model.py ✓                                                                                                                                    50% █████
 core/tests/test_view_tasks.py ✓                                                                                                                                   100% ██████████

Results (0.33s):
       2 passed

Testing POST /tasks to create a Todo

To test the POST url, which returns a partial template with a single todo item, we won't be able to leverage assertTemplateUsed. The way template partials work, the response object does not have the templates and context properties assigned, it just returns the resulting HTML in the content property. That's good enough for our needs though. Let's add one more test in the file:

# core/tests/test_view_tasks.py

... previous tests

# NEW 
@pytest.mark.django_db
def test_create_todo_view_stores_todo_and_returns_todo_on_partial(client, make_user):
    user = make_user()
    client.force_login(user)

    response = client.post(reverse("tasks"), {"title": "New Todo"})

    assert user.todos.filter(title="New Todo").exists()
    assert response.status_code == HTTPStatus.CREATED

    content = response.content.decode()

    assert "New Todo" in content
    assert '<li class="list-row">' in content

While I was writing this test, I noticed that the POST /tasks route was not returning the expected 201 (resource created) response code; it was returning 200 OK instead. That's fixed in core/views.py:

def _create_todo(request):
    title = request.POST.get("title")
    if not title:
        raise ValueError("Title is required")

    todo = Todo.objects.create(title=title, user=request.user)
    return render(
        request,
        "tasks.html#todo-item-partial",
        {"todo": todo},
        status=HTTPStatus.CREATED, # <-- CHANGED
    )

Testing the toggle todo view

Finally, let's test the PUT /tasks/:id route, that toggles a todo item from complete to incomplete, or vice-versa.

In this test we will use the parametrize feature of pytest to test both ways, setting a todo item as completed and not completed.

# core/tests/test_view_tasks.py

...previous tests

# NEW
@pytest.mark.parametrize("is_completed", [True, False])
@pytest.mark.django_db
def test_put_todo_toggles_todo_status_and_returns_todo_on_partial(
    client, make_todo, make_user, is_completed
):
    user = make_user()
    client.force_login(user)

    todo = make_todo(title="New Todo", user=user, is_completed=is_completed)

    response = client.put(reverse("toggle_todo", args=[todo.id]))

    todo.refresh_from_db()

    assert response.status_code == HTTPStatus.OK
    assert todo.is_completed is not is_completed

    content = response.content.decode()

    assert "New Todo" in content
    assert '<li class="list-row">' in content

Let's run the tests again and ensure it's all good. Note that the number of tests went from 3 to 5, since our new test will run twice to cover both values of is_completed.

Now that we have a test for our PUT route, I'd like to change it a bit: a route like PUT /tasks/:id, with no extra parameters or payload, would be expected to support several types of updates on the todo with the given ID; toggling the completed status is unexpected and ultimately a bad API design decision.

There are two interesting solutions, one is to change the route to be tasks/:id/toggle, and the other to keep it as tasks/:id and pass the value of is_completed in the form payload.

We will implement the first solution, since we will have other opportunities to learn how to pass values using hx-vals in the near future.

The good news are that since we're using Django's url function in the template, as shown below, instead of hard-coding it, and also using the reverse function in the tests, we can just change the url definition in urls.py and everything will just work.

<input type="checkbox" hx-put="{% url 'toggle_todo' todo.id %}" ...

The change is

# core/urls.py

from django.urls import path
from . import views

urlpatterns = [
    path("", views.index, name="index"),
    path("tasks/", views.tasks, name="tasks"),
    path(
        "tasks/<int:task_id>/toggle/",  # <-- CHANGED
        views.toggle_todo,
        name="toggle_todo",
    ),
]

All tests are still passing, and when we run the application, we can check the new URL being used. Not bad for a single line of change!

You can find the version of the code so far in the part 05 branch.

In part 6, we will implement a DELETE route and add it to our frontend.

Creating a To-Do app with HTMX and Django, part 4: adding new Todos

rodbv — Thu, 02 Jan 2025 22:09:59 +0000

Now that we've added the ability to toggle todo items on and off on Part 3, let's add a form to create new Todo items, using HTMX and a route that accepts POST requests.

It's going to look like this:

Extending the view to accept POST requests

To create new posts, we have two common options for the POST route: one is to have a separate route, such as /tasks/create, and the other is to use the same route we already have to list the tasks, /tasks. We'll opt for the latter, is it adheres best to REST and hypermedia practices, but both are fine.

Since the URL is already defined we just need to change the tasks view in core/views.py. To keep the code cleaner, we'll keep the code that handles POST requests on a separate function.

def _create_todo(request):  # <-- NEW
    title = request.POST.get("title")
    if not title:
        raise ValueError("Title is required")

    todo = Todo.objects.create(title=title, user=request.user)
    return render(request, "tasks.html#todo-item-partial", {"todo": todo})


@require_http_methods(["GET", "POST"])  # <-- NEW
@login_required
def tasks(request):
    if request.method == "POST":  # <-- NEW
        return _create_todo(request)

    context = {
        "todos": get_user_todos(request.user),
        "fullname": request.user.get_full_name() or request.user.username,
    }

    return render(request, "tasks.html", context)

On templates/tasks.html we will add a new form

<!-- core/templates/tasks.html -->

{% extends "_base.html" %}
{% load partials %}

{% block content %}
<div class="flex flex-col items-center mx-10 md:mx-20">
  <h1 class="text-2xl font-bold m-4">{{ fullname }}'s Tasks</h1>

  <div class="w-full max-w-2xl">
    <!-- NEW -->
    <div class="mb-4">
      <form hx-post="{% url 'tasks' %}" 
        hx-swap="beforeend" 
        hx-target="#todo-items" 
        hx-on::after-request="this.reset()"
        class="flex items-center space-x-2">
        <input type="text" required name="title" placeholder="Add a new todo" class="input input-bordered flex-1 text-lg" />
        <button type="submit" class="btn btn-primary btn-info text-lg w-24">Add</button>
      </form>
    </div>
    <!-- END OF NEW CODE -->

    <ul id="todo-items" class="list bg-base-100 rounded-box shadow-md">
      {% for todo in todos %}
      {% partialdef todo-item-partial inline %}
      <li class="list-row">
        <input type="checkbox"
          hx-put="{% url 'toggle_todo' todo.id %}"
          hx-swap="outerHTML"
          hx-target="closest li" 
          {% if todo.is_completed %}checked{% endif %} class="checkbox checkbox-lg checkbox-info mr-4" />
            <span class="flex-1 text-lg {% if todo.is_completed %}text-gray-500{% endif %}">{{ todo.title }}</span>
      </li>
      {% endpartialdef %} 
      {% endfor %}
    </ul>
  </div>
</div>
{% endblock %}

All the interesting code is in the form tag:

hx-post="{% url 'tasks' %}" indicates that the form will make a POST request to the tasks url;
hx-swap="beforeend" and hx-target="#todo-items" work together; it sets the target of the response to be the #todo-items tag (the ul), and that it should add the new fragment before the end of the list. If we wanted to add new items at the top, we could use afterbegin instead.
hx-on::after-request="this.reset()" will reset the form (clean the input text) after the request is sent; this will allow us to add several tasks at once, by typing and pressing "Enter" to submit the form.

Let's see it in action!

Dealing with slow requests

Now that we can add new items , let's handle with a common UX issue: latency. While our tests on localhost are pretty fast, in the real world the requests to create and toggle todo items may take a second or two, and the user may be unsure what's going on.

For the submit button, we'll make use of the hx-disable-elt attribute, and set it to the submit button, in tasks.html

      <form hx-post="{% url 'tasks' %}" 
        hx-swap="beforeend" 
        hx-target="#todo-items" 
        hx-on::after-request="this.reset();"
        hx-disabled-elt="#submit" <!-- NEW -->
        class="flex items-center space-x-2">
        <input type="text" required name="title" placeholder="Add a new todo" class="input input-bordered flex-1 text-lg" />
        <button id="submit" type="submit" class="btn btn-primary btn-info text-lg w-24">Add</button>
      </form>

For the toggle function, we can disable the checkbox during the request using hx-on:click


      <li class="list-row">
        <input type="checkbox"
          hx-put="{% url 'toggle_todo' todo.id %}"
          hx-swap="outerHTML"
          hx-target="closest li" 
          hx-on:click="this.setAttribute('disabled', 'disabled')" <!-- NEW-->
          {% if todo.is_completed %}checked{% endif %} class="checkbox checkbox-lg checkbox-info mr-4" />
            <span class="flex-1 text-lg {% if todo.is_completed %}text-gray-500{% endif %}">{{ todo.title }}</span>
      </li>

We can force the requests to be slow by opening developer tools/network and setting the Throttling option to 3G (in Firefox you can choose even slower options)

Let's check how it goes:

We're done by now! In part 5, we'll add some tests to our views, which is probably a much nicer experience than testing client-side code :)

As usual you can check the code written in Part 4 in its branch in Github.