DEV Community: David

How I Speed up My Django Test Suite by 200%

David — Sat, 14 Dec 2019 17:04:34 +0000

tl;dr: Add --parallel at the end of the test command

Background

Earlier this week I was deploying a new feature on a legacy codebase. The whole test took 60 minutes before it was done. If this was going to be a one-time deployment, I was gonna let it slide off. I will have to work with this codebase for a little bit longer however, so I need to find a quick way to speed up the test suite.

I then looked for how to make Django test run in parallel and found it really quickly. It's achieved just by adding --parallel option in the test command, e.g. python manage.py test --parallel.

So problem solved right? The test ended in 5 minutes. Unfortunately, it ended in 5 minutes because the test failed. After some time figuring out why it failed, I realized that the failing tests were using Redis Queue (django-rq). Apparently running the --parallel command didn't isolate it.

I googled how to isolate the django-rq when running the test in parallel, but unfortunately, I wasn't able to find the solution. I then had 2 choices: abandon my plan on speeding the test suite, or writing a custom test runner. I chose the latter.

Writing a custom Django Test Runner to isolate Redis Queue

Before writing the test, I read up on how Django isolate the test database. I knew I would either extend or copied most of the logic from that. After digging around, I found the code where the database isolation happens.

Now let's write the custom test runner. Please note that I am using django-rq library. The way to set the Redis connection might be different. Still, the idea to isolate it remains the same. We will achieve isolation by assigning a unique database index to the Redis configuration for each worker.

Firstly, we need to supply a different function to init_worker in Django's ParallelTestSuite. To achieve that, we will write our custom test suite called CustomParallelTestSuite that extends ParallelTestSuite.

from django.test.runner import _init_worker
from django.test.runner import ParallelTestSuite

def redis_parallel_init_worker(counter):
    _init_worker(counter)

class CustomParallelTestSuite(ParallelTestSuite):
    init_worker = redis_parallel_init_worker

We still need to isolate the database however, that's why we will still call the _init_worker function inside our function.

Now we also need to modify the default base runner to use our CustomParallelTestSuite as well. We will do this by extending the DiscoverRunner class.

...
from django.test.runner import DiscoverRunner
...
class CustomDiscoverRunner(DiscoverRunner):
    parallel_test_suite = CustomParallelTestSuite

Sweet! All we've done up till now is only creating new classes, but the behavior of the test runner remains the same. Now it's time to modify the redis_parallel_init_worker to add the functionality of isolating the Redis. I copied most of the worker counter from _init_worker.

from django.conf import settings
from django.test.runner import _init_worker
from django.test.runner import DiscoverRunner
from django.test.runner import ParallelTestSuite

_worker_id = 0

def redis_parallel_init_worker(counter):
    from django_rq.settings import QUEUES
    _init_worker(counter)

    """
    Switch redis databases dedicated to this worker.
    This helper lives at module-level because of the multiprocessing module's
    requirements.
    """

    global _worker_id

    with counter.get_lock():
        _worker_id = counter.value - 1

    settings_dict = getattr(settings, 'RQ_QUEUES', None).copy()

    QUEUES = settings_dict
    QUEUES['default']['DB'] = _worker_id

With that, we've assigned a unique Redis database index based on their _worker_id, and Redis isolation is achieved.

Full code

Here's the full code for the custom test runner we just wrote.

# myapp/test/runner.py

from django.conf import settings
from django.test.runner import _init_worker
from django.test.runner import DiscoverRunner
from django.test.runner import ParallelTestSuite

_worker_id = 0


def redis_parallel_init_worker(counter):
    from django_rq.settings import QUEUES
    _init_worker(counter)

    """
    Switch redis databases dedicated to this worker.
    This helper lives at module-level because of the multiprocessing module's
    requirements.
    """

    global _worker_id

    with counter.get_lock():
        _worker_id = counter.value - 1

    settings_dict = getattr(settings, 'RQ_QUEUES', None).copy()

    QUEUES = settings_dict
    QUEUES['default']['DB'] = _worker_id


class CustomParallelTestSuite(ParallelTestSuite):
    init_worker = redis_parallel_init_worker


class CustomDiscoverRunner(DiscoverRunner):
    parallel_test_suite = CustomParallelTestSuite

Adding to settings to refer to use the custom test suite

Now the last thing we need to do is to change our settings to use the test runner we've just created. To do this add TEST_RUNNER = 'path.to.custom_runner.CustomRunnerClass' in settings.py, for example:

# settings.py
TEST_RUNNER = 'myapp.test.runner.CustomDiscoverRunner'

Wrapping up

Now, calling the command with --parallel runs the test correctly as intended. The test eventually ended in about 17 minutes, a speedup of around 3 times! Some of the tests can still be optimized if I wanted to increase the speed further, but I'm satisfied with this result for now.

Thanks for reading, I hope this post gives you a basic idea on how to write your own custom test runner if you encountered a similar problem. See you next time.

Why Learning Programming is Hard and Tips for Learning it Better

David — Sat, 23 Nov 2019 16:16:13 +0000

Have you ever encountered any difficulties when you first started programming? Some people are going to say learning programming is not hard. I can think of a few reasons why they say programming is easy. Maybe they are geniuses, who have a strong analytical background. Perhaps they just forgot the hardships they had when they started. The last reason I can think of is, they simply haven’t reached that point (or pointers in C) yet.

A lot of people are going to have difficulties when they started programming, or even after they have learned it for a while, so don’t be discouraged if you just started and find it difficult.

Maybe you know someone who can program a complex system easily. Most likely it looks easy because they spent a lot of time mastering it. They might also have encountered similar problems in the past. You can after all say, programming is about making a complex problem into smaller problems that can be solved easily by itself.

Programming learning curve

I believe the learning curve of programming is not linear. Once you reach a certain level, you will hit a wall. You will feel like you are unable to get better, not knowing how to improve. This will keep happening until you learn (or unlearn in some cases) a skill/knowledge. If you are familiar with games that have ranking/rating systems, you sometimes will feel like getting stuck in certain ranks despite you keep playing it. The only way to get past that is to figure out what you are missing and work on that to reach the next level. Once you’ve worked on that, you will feel like you are making a lot of progress, until you hit the next walls.

Phases of learning programming

This is just an example of learning phases based on my own experience and observations. This might differ from how you learn. I try to identify problems that are frequently blocking people’s learning progresses and try to come up with a solution.

First phase

This phase is where people get familiar with programming languages syntaxes and concepts: variables, if/else, for loop, functions. Once they learn this, there’s a first wall they have to overcome. How do you combine these concepts to create a running program?

For those who have a mentor guiding them, they should be able to get passed through this quickly. For those who don’t have one, I suggest following tutorials on how to create an app and learn from the patterns on how to build one.

People who can’t pass this probably won’t stay for long and usually gives up on programming.

Second phase

Those who make it to this phase should be able to create an application by now. The next wall to overcome is the difficulty to create a larger-scale application. Things get harder to maintain, changing one part of the app is causing bugs on another. This is another wall programmers have to overcome before they can advance. The solution to this is to learn how to write clean code, having knowledge about design patterns and principles. I recommend reading these or lookup for a summary:

Third phase

This phase happens when people start to work as a team. This is skippable because some people work alone and have never worked as a team. I suggest trying to work in a team at least once. It’s going to teach you some skills that you will never learn otherwise. You don’t have to be working in a company, you can just try and work on a project with a small number of people.

There might be another obstacle here for people to overcome, or probably not depending on the individuals. Some people might find it hard to work with other people. Some problems that might occur are communication problems, unable to read others’ code, etc.

I can’t really give any good advice, but here’s my thought. For communication, try and sit down with your teammates and keep evaluating how you work as a team. What works for a team doesn’t guarantee it’s going to work in another. For reading code, try to simulate the code on a piece of paper. Reading the test specs might help to explain what the code is trying to achieve, so you should start from there.

Fourth phase

Programmers at this phase should now know a few design patterns or principles, and how to write a clean code. Often times, I see programmers fall into the trap of trying to apply a design pattern they’ve learned to every problem. Sometimes using a design pattern is just adding complexity to the code. Your app is probably not needing that much flexibility yet (check out YAGNI principle).

So I’ve come to the conclusion that the biggest wall for this phase is not being able to understand trade-offs. These programmers tend to try and adhere to all of the principles/patterns they’ve learned. I personally think they are just a guide to help, and it’s all about trade-offs. Implementing a principle will be good in one situation, but it might be bad in another. That’s the difficulty of trying to level up in this phase, knowing when to pick a solution over another.

One way to sharpen your skills in making these kinds of decisions is, try to learn about anti-patterns before going on with patterns. Anti-patterns will give you a general idea of what not to do. Just keep yourself reminded that at the end of the day, you shouldn’t be adhering to all anti-patterns all the time. Remember, it’s about trade-offs. It might actually be good in certain situations!

There are probably a lot more phases after this, but I haven’t observed a lot to write about it. If you have any stories, please share it in the comments!

Make learning programming easier

So why do I think learning programming is hard? Most of the time it’s because people don’t know what to learn. What to learn largely depends on your goal and how experienced you are. I don’t believe just trying to code for 10,000 hours is gonna help. Instead, try to learn efficiently. Here are a few tips to make you learn more efficiently:

Have a goal in mind

Don’t just learn randomly. Build a roadmap for learning. For example, if you want to be a web developer, first find out what people usually learn to achieve that. You can check out the roadmap for being a web developer in this article: The 2019 Web Developer.

If you have other roadmaps of learning, please share it in the comments.

Read other people’s code

Do you remember when I said programming is about making a complex problem into smaller problems? Doing so requires experience, and by keep writing codes, you will have little difficulty solving the same problem the next time you encounter it. There’s a way to speed it up though. By reading other people’s code, you save the time needed to think up the solutions for yourself. Just remember to try to solve a problem by yourself from time to time in order to keep your problem-solving skills sharp.

Find a mentor

Most of the time people don’t improve because they don’t know what to learn. Having a mentor will help to identify what to learn next. Mentors can also help to provide the moral support you need in case you are giving up.

Recently I found Coding Coach, a platform where you can find a mentor for free. It’s a cool community! If you are an experienced developer, I suggest you apply as a mentor too. You might learn a thing or two becoming one.

Wrapping up

Personally, I think programming is hard. The reason being, programming is not just about writing a bunch of codes to solve a problem. Being a good programmer requires other skills such as collaborating, maintaining, ensuring codes don’t break, writing codes that other people understand and so on.

If you are currently trying to improve by trying to code repeatedly, please take a step back. Try to learn more efficiently instead. If you have any other learning tips, please share it in the comments.