Running your pytests faster

Open your conftest.py and type the following lines:

import gc
# Like gc.disable() but overrides your dependencies that do
# gc.disable() and gc.enable()
gc.set_threshold(0)

Depending on the project size and number of tests there is a speedup from a few percent in smaller projects up to 10% in a project with 30,000 tests. Your experience may vary.

I went down the Python GC rabbit hole and there will be a long story about it so here is a short one.

Most of the memory in Python is released when the reference count reaches zero. For a few cases when you might get cyclic references, there is a stop-the-world garbage collector. However, the trigger condition for starting the GC is when allocated but not released instances of non-atomic types reach a small threshold. That leads to a lot of GC cycles when you load something into memory for legitimate reasons: your Django QuerySet caching, file records, and collecting unit tests to run.

For my test case, 2nd generation gets more than 400,000 objects while pytest is running the tests. It takes some time to get there from the 0th generation and it just increases the n in the complexity of the collection algorithm.

If Instagram can run services without GC in production so can our tests.