DEV Community: Syed Mubtada Ali

Tips and tricks for optimizing the performance of Django ORM

Syed Mubtada Ali — Wed, 03 May 2023 13:59:29 +0000

Django is a popular web framework for building complex, data-driven applications, and its Object-Relational Mapping (ORM) system is a crucial component for interacting with databases. However, as your application grows in complexity and scale, it’s essential to ensure that your ORM queries are optimized for performance. In this article, we’ll explore some tips and tricks for optimizing the performance of Django ORM so that your application can handle larger volumes of data and operate more efficiently. These tips will help you get the most out of your database queries and keep your application running smoothly.

Only()

In the above code, we are only using two fields. Still, when Django ORM translates Python code into SQL, it retrieves all fields from the model as a result, even if some data is not necessary. To optimize the query and prevent unnecessary data retrieval, developers can utilize the .only() method and explicitly indicate the specific fields that are required.

Prefetch_related / select_related

When Django evaluates a QuerySet, the relationship fields are not included in the query, so when you access them, Django runs the DB again to get the values. Accessing a related field in a loop results in N+1 queries. You can check the queries executed on the database using tools like django-debug-toolbar and django-silk.

In the above code, the first query is executed on the option queryset, and then for every option, three queries are executed: question, division, and concern. To reduce queries, we can add question and division in select_related and concern in prefetch_related.

This code will execute only 2 queries on the database regardless of the number of options: one for options and the second for concerns. In the previous version, if there were 100 options, it would perform 301 (3N + 1) queries on the database.

One important thing to understand is that after adding select_related and prefetch_related, you cannot use ORM functions such as first(), *last(), or *order_by(), as these will run the queries again. If you want the first object of concern, use option.concerns.all()[0] instead of option.concerns.all(). Use Prefetch() only when you want additional filtering and order_by; otherwise, simple prefetch_related(“concern”) is enough.

The basic difference between select_related and prefetch_related is that select_related performs a join with each lookup and gets the results back in the same query, but it extends the “select” to include the column of all joined tables. On the other hand, prefetch_related performs a separate query for each table to be joined. It filters each of these tables with a WHERE IN clause. The select_related can be used for one-to-one and one-to-many relations if the related field exists in the table where the query is being executed. However, if the related field exists in the other table, we can use prefetch_related. It’s also used for many-to-many relations.

Queryset Evaluation and Caching

Take a look at the following simple example to understand ORM evaluation:

It might look like it performed two database queries, but it actually performed only a single query, which was executed on the last line, “len(active_staff)”.

Creating a queryset doesn’t perform any database activity, not even stacking the filters. A queryset result is only fetched from the database when asked by performing certain actions, like iteration, len, slicing, repr, list, etc. This is called evaluation.

Caching allows you to avoid making multiple database queries when reusing the same queryset. The first time you run a QuerySet, Django saves the results in a temporary storage place called cache. Then, whenever you run that same QuerySet again, Django will use the cached results instead of running another query to the database. This makes the process of retrieving data faster and more efficient.

In the above example, the same query is executed twice on the database. To avoid this problem, save the QuerySet and reuse it:

In_bulk()

This function will make your life easier if you need a dict map of the queryset. It allows you to retrieve a dictionary of objects from the database based on a list of primary key values or the defined key in the function argument.

Instead of:

Use this:

The code is much cleaner and also provides a performance boost by leveraging the lazy nature of Django ORM.

For those who are passionate about writing neat and organized code and haven’t had the chance to go through “Clean Code” by Robert C. Martin, this article provides a helpful summary of the book’s first half and key takeaways. It’s worth reading if you’re looking to enhance your coding skills and principles.

Lessons from “Clean Code”

Syed Mubtada Ali ・ May 3 ・ 9 min read

#python #cleancode #programming #productivity

I hope that you have learned something fresh and insightful. If our interests align, please consider subscribing. You can also reach out to me on LinkedIn.

Thank you for your time and attention.

Lessons from “Clean Code”

Syed Mubtada Ali — Wed, 03 May 2023 13:46:54 +0000

So if you want to go fast, if you want to get done quickly, if you want your code to be easy to write, make it easy to read. — Robert C. Martin

Usually, there are two types of programmers clever ones and professionals. Both of them can get the job done but there are a lot of differences between the two. Professionals understand that clarity is the king. They use their powers to write clean code that others can understand and work on it by writing clean code. But what exactly is clean code? What are its characteristics and how can one write clean code? These were the questions in my mind when I started reading Clean Code written by Robert C. Martin and gladly this book didn’t disappoint me at all. This book has so many takeaways for clean code and I think every programmer should read it especially at the start of their career.

In this article, we will learn

variable names should reveal its intent and should not include data type
class and function name should be a noun and verb respectively
functions should do only one thing and number of arguments should be minimum
comments should be avoided as much as possible
parts of code linked together should be placed next to each other
the code should be formatted in a way that the eyeballing should be minimum while reading it

All the code in this article is written in Python so if you are unfamiliar with the language it might be a bit abstract for you.

Characteristics of Clean Code?

Robert C. Martin aka Uncle Bob asked some very experienced and well-known programmers that what they thought about clean code. As per them here are a few of the characteristics of clean code.

I like my code to be elegant and efficient. Logic should be straightforward to make it hard for bugs to hide. Clean code does one thing well. — Bjarne Stroustrup(inventor of C++)

Clean code can be read, and enhanced by a developer other than its original author. It has unit and acceptance tests. It has meaningful names — Dave Thomas

Clean code is simple and direct. Clean code reads like well-written prose — *Grady Booch* (author of Object-Oriented Analysis and Design with Applications)

So, If I outline these characteristics, the clean code should be

Elegant: It should be simple and easy to read. It should make you smile while reading it.
Focused: Each function/module should expose a single-minded attitude. It should only focus on getting one thing done.
Tests: If it doesn’t have tests then it’s not clean no matter how elegant/readable it is.

Writing Clean Code

Meaningful Name

If a name requires a comment, then the name does not reveal its intent. — Uncle Bob

Names are important as they reveal the purpose. Coming up with good and meaningful names takes time but believe me it’s worth it because it saves time we have to read/overview it later.

Variable Name

It should describe its nature without any comment and if it’s not, then the name should be changed. i.e.

The data type should not be part of a variable name even if it’s allowed in the language’s coding convention. Especially in Python at the time of variable definition, we don’t set data type. It will cause problems if we change the data type of a variable at some point as we have to replace it with the new data type. Let’s take a look at the example below

Changing the above variable into float would be very hectic as we will have to replace it wherever we have used it. However, it’s allowed in *Hungarian notation* but now it’s not recommended to use it and Microsoft also doesn’t recommend this notation. (reference)

The variable names should be pronounceable. As Uncle Bob said, If you can’t pronounce it, you can’t discuss it without sounding like an idiot. “Well, over here on the ‘bee cee arr three cee enn tee’ (bcr3cnt) we have a ‘pee ess zee kyew’ (pszq) int, see?”

A name should also be searchable, for that, longer names are preferred over shorter ones. Also, a single character name should not be used for a large scope. It should only be used in a shorter scope like a small function/loop.

Function & Class Name

A function name should be a verb and it should also reveal the action along with the value, like extract_age, delete_user_info. *On the other hand, a class name should not be a verb. It should be a noun or a noun phrase name. i.e. *Customer, Account and WeatherRecord.

Don’t make it hard for yourself by adding unnecessary context. Don’t prefix every class/function with context unless they represent a different context. For example in an application of “Green Pizza Hut”, it will be really bad to prefix every function/class with “gph” as there will be a lot of redundant characters. Shorter names are better as far as they are clear.

Cuteness

If you are cute, great that’s good for you but please don’t make the code cute make it rather professional. I know it sounds very cute to name a variable after your pet or favorite character’s name but how on earth one can know what is “Bubbles” or “Dexter”.

Don’t use slang words while naming your variables, functions or classes. For example, don’t use merk() to mean kill().

One Word per Concept

Pick one word for a concept and stick with it. For example, we should not use *get, retrieve, extract *and *fetch as a function name *for the same concept of different classes. It will be very difficult to remember which class has which function name. In the same way, it’s also confusing to have *Manager, Controller *in the codebase as both serve the same purpose.

Functions

Functions should do one thing. They should do it well. They should do it only.— Uncle Bob

Size does matter in functions, the first rule of a function is that it should be small and then it should be smaller than that.

Do One Thing

There is no advantage of writing a function if it’s doing more than one thing. As the main purpose of writing functions is to decompose a larger concept into a set of steps at the next level of abstraction. Functions should not be large enough to hold nested structures. Also, the indent level of a function should not be greater than one or two. It will make the function easier to read and understand.

The function above, even though it’s not very long, is doing more than one thing. The block before the loop should be a separate function.

Uncle Bob mentioned a way to know that a function is doing more than “one thing” is if you can extract another function from it with a name that is not simply a rephrasing of its implementation.

Arguments

There should be no more than three arguments for a function. An ideal number of arguments should be zero. More arguments take more conceptual power. A lesser number of arguments make easier to write test cases.

Flag arguments are ugly. Passing a boolean into a function is a truly terrible practice. It immediately complicates the signature of the function, loudly proclaiming that this function does more than one thing. It does one thing if the flag is true and another if the flag is false.

Comments

Nothing can be quite so helpful as a well-placed comment. Nothing can be quite so damaging as an old crufty comment that propagates lies and misinformation. — Uncle Bob

Comments are always failures. We have them because we cannot always figure out how to express ourselves without them, but their use is not a cause for celebration. The purpose of a comment should not be to explain the intent. We write a module and we know it is confusing and disorganized. We know it’s a mess. So we say to ourselves, “Ooh, I’d better comment that!” No! You should better clean it!

There are two ways in the following code; a messy code with a comment to explain it and other is clean code explaining itself without any comment.

So, don’t use comments when you can use a variable/function.

Unfortunately we, the programmers, don’t realistically maintain the comments. Requirement changes and so does the code, a chunk of it moves from here and there but the comment stays there and creates the mess and misunderstanding. There should be no commented code or position markers. It’s very rare when position markers make sense to gather certain function together but in general, they should be removed.

An example of position maker is given below.

*# Actions #######*

Good Comments

Not all comments are bad, sometimes it’s necessary to add comments. For example for legal reasons, it’s good to add comments for copyrights, authorship at the start of the file. It’s also reasonable to add a warning of consequences and “to do” comments.

Sometimes a comment goes beyond just useful information about the implementation and provides the reason for writing this code. For example, in the following example the developer expressing the reason for adding download delay.

Formatting

Code formatting is important. It’s too important to ignore and it’s important to treat religiously.

The source file should be like a newspaper article. The filename should be simple but meaningful enough to tell us whether we are in the right file or not just like a newspaper article’s title. As the start of an article gives an abstract of the article and details increases as we go in. In the same way, high-level concepts should be at the top of a file and details should increase as we go down. Low-level function/details should be at the end of the file.

Vertical Formatting

Similar logical lines should be separated from other by a single blank line. The blank line is a visual cue that identifies a new and separate concept.

Just like adding a blank line separates concepts, vertical density implies close association. Lines of code that are tightly related should appear next to each other. Because it’s very frustrating if you are trying to locate a chain of inheritance for the definition of a function/variable, spending your time and energy just to find and remember where the pieces are.

If one function calls another, they should be vertically close, and the caller should be above the callee, if at all possible. The stronger that affinity, the less vertical distance there should be between them. This gives the program a natural flow. Also, variables should be declared as close to their usage as possible.

Horizontal Formatting

We use horizontal white space to associate things that are strongly related and disassociate things that are weakly related. We surround the assignment operators with white space to accentuate them. The spaces make that separation obvious. But we don’t put spaces between the function names and the opening parenthesis. This is because the function and its arguments are closely related.

In python, we also wrap comparisons, mathematical and boolean operators with a single space but if there are multiple operators either in a single line then we wrap the operator with the lower priority. An example is given below.

The following lines are not inspired by the “clean code” instead this is something that I have learned in my professional life. At the start of my career, I was often told by my team lead and seniors to split a line into multiple and vice versa while code reviewing. At that time I was confused by this splitting and merging but then my team lead explains this “*eyeballing” *effect and then it made sense to me.

The size of a line should be near to the length of its neighboring lines. So that the eyeballing should be minimized as much as possible while reading the code. It will make reading easier a lot.

For example in the code below, the size length is not near to its neighboring lines

The eyeballing here should be minimized, the refactored code is given below.

So, to conclude the whole thing, here are your takeaways from this:

Variable name should be meaningful, pronounceable, searchable and should reveal its intent. The data type should not be a part of it.
Class names should be a noun and have a clear description of the things inside it.
Function names should be a verb and do only one thing and should not be too long.
Functions should have the minimum number of arguments.
Comments can be written as an absolute last way of explaining what you are trying to accomplish and should be well thought.
Parts of codes linked together should be formatted next to each other.
The code should have as less eye-balling as possible.

If you have some questions or your perspective of this please do reach out to me through LinkedIn.

Happy Clean Coding