Python Imports: The Most Unintuitive

Have you ever felt like a programming language was punishing you for trying to clean up your code? To make it more modular? To make it follow the SRP? Well, if you have tried to do this in Python I can confidently say you have run into this very common, and very annoying error message

ModuleNotFoundError: no module named "example" 

Let's say you encounter this error and see people say "Oh, just do absolute imports to avoid that issue!" and while yes that may solve the error, it is a messy solution. So, when you look into Python's relative imports instead and see that you just need to put an '__ init __.py' file in every directory your relatives should just magically work! Well, again, no. This same path of learning is what led me into the unintuitive, sloppy, and cumbersome world of Python imports. After finally dragging myself out of the other side of the abyss, I am hopeful this blog will be able to help someone else from having to stumble and trip through learning Python's import system.

First, let's discuss what kind of imports you can do in Python. For all of these examples, we will be using the following example tree

├── __init__.py
├── package1
   ├── package3
       ├── __init__.py
       ├── module7.py
   ├── __init__.py
   ├── module1.py
   ├── module2.py
   ├── module3.py
├── package2
   ├── package4
       ├── __init__.py
       ├── module8.py
   ├── __init__.py
   ├── module4.py
   ├── module5.py
   ├── module6.py

The first major import method to know is the absolute import. An absolute import does not require the '__ init _.py' files in your directories. While this may seem nice as it leads to less bloat by not requiring a bunch of what _can be empty files there is a very important caveat. When you do an absolute import, the file you import will have access to every variable and object that file has within its scope. Again, this may seem nice but it leads to a lot of extra code if you do use it. This is what it would look like if you wanted to try and do an absolute import from module1 to module4, thus allowing module4 to use the function module1() in module4. First, the import

#in module4.py

import package1.module1

And then to be able to call that specific function

#in module4.py

#some code...
package1.module1.module1()

Well that doesn't seem to bad. But what if wanted to also call on module2's module2() and module3's module3()? Well, we still only have one import statement, but this is where you see the bloat start to come in.

#in module4.py

import package1.module1

#some code...
package1.module1.module1()
package1.module2.module2()
package1.module3.module3()

While this is nice in fact that you know what you're getting and you can avoid circular imports (we'll come back to this later), it can quickly lead to a lot of extra code. Oh, you wanted to use module7's module7() in module4? Oh no...

#in module4.py

import package1.package3.module7

#some code
package1.package3.module7.module7()

You can see how messy this can begin to get if you have multiple levels of nested folders. You know, a really common file structure when working with larger projects. Not all hope is lost though! We know there is another main method of imports in Python. The relative import.

The relative import is the method that requires us to have all of those __ init __.py files, but why? Well, if you read through deep enough into the late 1990s web-styled documentation, you will see that these files tell the Python interpreter "Hey! Look over here for packages and modules too!" Once you have these in place you can create much cleaner and clearer import methods. If you were to import module2's module2() into module1 it would look like this

#in module1.py

from package1.module2 import module2

#some code

module2()

Now that looks much nicer! We can choose the one object we want from module2 also allowing other engineers to know where that function is coming from and it avoids the repetitive long strings of dot notation in both the import and in the function call itself. Well, everything is pointing to relative imports just being better for clear and clean code, right? Well, there are some caveats. Yes, more caveats, Python imports have a lot of those. Let's say you for some reason needed to also import module1's module1() into module2. A very common case for this is if module1() is a main menu for a CLI, and module2() is a sub-menu of module1(). Well, if you want a user to be able to navigate back from module2()'s menu to the main menu in module1(), you would need to call module1(). Okay, let's go ahead and set that up!

#in module1

from package1.module2 import module2

def module1():
    #some code
    if(choice == 'a'):
        module2()

#in module2

from package1.module1 import module1

def module2():
    #some code
    if(choice == 'back'):
        module1()

Hey, that's nice! We can have some clean navigation functionality for our code! Nope! You just caused a circular import! This is another one of Python's infamous error codes. And with Python's, again, unintuitive import system, it is tied to it. We have already discussed the solution to it though! Those extremely bloated and cumbersome absolute imports! Let's change our code to the import system that would work for this use case.

#in module1

import package1.module2

def module1():
    #some code
    if(choice == 'a'):
        package1.module2.module2()

#in module2

import package1.module1

def module2():
    #some code
    if(choice == 'back'):
        package1.module1.module1()

Does anyone else's stomach start to churn a little bit when having to see the way to avoid circular imports? We will come back to my extremely mixed feelings on Python, for now, let's look at yet another unintuitive aspect of Python imports. Relative imports from other file directories.

We already saw the syntax for relative imports within the same directory

from package1.module2 import module2

Seems simple enough, now let's look at the Python documentation on relative imports from different folders (the packages in our example).

Relative imports use leading dots. A single leading dot indicates a relative import, starting with the current package. Two or more leading dots indicate a relative import to the parent(s) of the current package, one level per dot after the first

Okay, so following this logic, if we wanted to import module1's module1() to module4 we would need to have two leading dots to reach our parent directory. Then step back down with the dot notation we have already done. Well, let's try it!

#in module4

from ..package1.module1 import module1()

Again, nope! Another one of Python's infuriating import errors! Yet another way to run into the ModuleNotFoundError. This method of running into that error is the one that broke me mentally. Everywhere I searched, every documentation I read, all led to me (and another experienced dev I had asked for assistance) being completely lost. After way too many Stack Overflow questions and Google searches about this, I decided to just look at an older project I had access to from using it as a lab. I checked how they would handle this same
import and the moment I saw it, I was ready to put my forehead through my desk. Want to guess the solution? I will give you a moment to think about it based on the documentation given to us by Python.

...........

#in module 4
from package1.module1 import module1

Well, that seems to suggest maybe the Python documentation is just blatantly wrong. Okay, what about if it's another directory level down to module7? Again, no. Again, the documentation is just flat-out wrong. The syntax to bring module7's module7() to module4?

#in module4
from package1.package3.module7 import module7

Yeah, this was the moment I gave up on believing anything the Python documentation stated. This was the same moment I decided Python had the most unintuitive, cumbersome, slow, and dated import system of any language used in modern programming. Even now that I have finished the project that required me to have a complex import system, even now after having to drag myself through the Python import trenches, I am fuming about it. How can such a beautiful, beginner-friendly, and powerful (but very slow) language still have such a horrible import system? While this blog has been both an exploration and a venting session for myself, I hope it will save someone else from the abyss that Python imports.

Comments

[matej]

Entirely idiotic system with no explanation that helps.