Navigating Python's import system and namespace packages

The Hidden Teaching: The Nature of the Path

The student believes that a village must be a single, bounded place. But the master knows that a village can be a concept , defined by its connections and the paths that lead to it.

Thanks for reading Python Koans! If you enjoyed this post, consider sharing with your friends or subscribing below:

Python Koans | Vivis Dev | Substack

Python lessons wrapped in koans. Small puzzles, deep truths. Not your usual tutorial thread. Click to read Python Koans, by Vivis Dev, a Substack publication with hundreds of subscribers.

pythonkoans.substack.com

In Python, the import system follows a similar principle. You may think of packages as walled villages with a clear boundary. Yet the true nature of the path (sys.path) allows for the creation of unburdened villages: namespace packages, where a single logical package is assembled from components scattered across the filesystem.

---

Part 1: The Walled Village and the Path

The Python interpreter, when asked to import a package, follows a simple rule: it searches along a predefined path. This path, known as sys.path, is a list of directories.

A traditional package , or a "walled village," is a directory containing an __init__.py file. This file marks the directory as a package and provides its clear boundary. When you import this package, Python finds the directory on its path and all its contents are then accessible.

Consider a simple structure:

To use this, the directory containing my_village must be on sys.path. If you run Python from the directory above my_village, it is on the path by default.

The __init__.py is the village gate; it tells the interpreter, "This is the start of the journey."

Part 2: The Unseen Path and the Root of Confusion

The monk's confusion was not about the village itself, but about the assumption of a single starting point. The same confusion arises when a package is not directly on the path.

Consider the following:

If you run Python from a location that contains both project1 and project2, the my_village directories are not directly on sys.path. The path only contains the top-level directory.

The interpreter's path is not a tangled vine that seeks out all nested directories. It is a straight road, and it only knows about project1 and project2. It does not know to look inside them for other packages.

---

Part 3: The Unburdened Village: A Shared Destination

The solution lies in understanding the path as a shared space. A namespace package is created when multiple directories with the same name, but without an__init__.pyfile , are all placed on sys.path.

By adding the parent directories (project1 and project2) to the path, you instruct Python to consider them as valid starting points for imports.

The interpreter first looks on the path, finds project1, and sees a my_village directory inside. It then continues its search, finds project2, and sees another my_village directory. It then merges these two directories into a single logical package, the "unburdened village."

Part 4: The Path of the Confused Traveller

What if one part of the village has a wall and the other does not?

Consider if project1's my_village directory contains an __init__.py file, but project2's does not.

In this scenario, my_village is no longer a namespace package. Python's import system will find the first my_village directory on its search path (sys.path) that contains an __init__.py file, and it will treat that directory as the entire package.

When you run import my_village.temple, the interpreter will find the my_village directory in project1, but it will not continue to search for other my_village directories. It will only look for temple.py inside project1/my_village. The file will not be found, and you will get a ModuleNotFoundError.

The presence of the __init__.py file effectively "closes the gate" on the search for other parts of the package, turning it back into a traditional, "walled" package.

Part 5: The Power of the Unburdened Village

Why would one choose to build a village without walls?

• Extensibility : Frameworks like pytest and zope use this pattern. The core library defines a namespace, for example my_app.plugins. Developers can then create their own packages, like my_app.plugins.my_feature, and the core application will automatically find and load them without any modification to the main codebase.

• Decomposition : A large project can be broken into smaller, independent libraries that are developed and versioned separately but still function as a single logical package.

• Modularity : It allows for a more distributed and modular architecture, where a project's components can be managed and installed from different locations.

---

Leaving the Village

The master's reply was a lesson in perspective. The village is not the directory itself but the concept of a shared space. The path is not a single location but a collection of all the roads one can travel.

Your confusion, like the monk's, arose from seeking a single physical truth. But Python's import system, like the master’s village, shows us that meaning can be constructed from disparate parts as long as you follow the right path.

Thanks for reading Python Koans! If you enjoyed this post, consider sharing with your friends or subscribing below:

Python Koans | Vivis Dev | Substack

Python lessons wrapped in koans. Small puzzles, deep truths. Not your usual tutorial thread. Click to read Python Koans, by Vivis Dev, a Substack publication with hundreds of subscribers.

pythonkoans.substack.com