Dynamic importing stuff in Python

In my artificial intellingence project's prototype Okai (which is completely written in python), I did some dynamic importing which surprisingly was unknown to many. I think you might be interested in knowing about it as well.

Before I explain dynamic importing, lemme explain traditional import too for the sake of readers of all levels. To follow along, you must be atleast familiar with basics of Python. If you don't understand anything, read anyway... you will learn something new. Let's start with the most basic example and increment on it. If you feel bored, skip to next section.

import module

import os

Very familiar import. Here, we are simply importing os module. os is now available in the scope as a "module" object (Python treats literally everything as an object) identified as os.

import module as something

import random as r

Again, we are simply importing a module. But now, the module will be available as given identifier (here it is r) in the scope.

import module's module

import urllib.request

Note that you can't import methods and properties of a class without importing class.

. is the scope resolution operator in python. It simply allows you to go a level deeper in the namespace and find the next identifier ie. request in above example. but urllib whole is imported and request is not defined! Why is that?

Something seems to be wrong here. But it is how it should be. To keep things interesting, let's continue our journey and pin it for later.

import module's module as something

import urllib.request as request

We have fixed the buggy import we did before. Now the urllib is not available in the scope, request is the scope and refers to urllib.request.

from module import module

from urllib import request

It works exactly like the above fix but more self explanatory and simpler. Remember, we still have a question pinned. We need to find the answer.

from module import all

from urllib import *

This one will import modules that have their identifiers listed in **all** list or simply, exist within the namespace of urllib's **init**. Since we used from keyword, urllib does not exist in the scope. But all the modules from within urllib 's namespace now exist with their default identifiers! So many identifiers within same scope can be dangerous. Consider the following example:

walk = 'Some string'

from os import *

I just committed a crime! walk has been overridden by os.walk ! To avoid such incidents, avoid importing *. As a thumb rule, do imports at the beginning of your code.

from module's module import module

from urllib.request import Request

It works as expected. Nothing new. You can keep going deeper in the modules as long as you want. Anything that comes inbetween from and import does not exist in global scope.

import self's module

Let's say that you wanted a module named as sys in your package, and you made one. But now, how do you import it when there is already a standard module with similar name? You use relative importing.

from . import sys

This will import sys from within current package.

from .sys import mod

This will import mod from within sys module that resides in current package.

from .. import sys

This imports sys from the parent package. But we still have that question pinned. Let's find the answer to that question. We need to find how import works behind the scene first.

How import works

We can always look in the documentation and find how import works. If not willing to do so, you can always dump global identifiers available with globals(). Other than globals(), __builtins__ list usually contains most of the built in classes and functions (as strings). If you dir( __builtins__ ), you will find __import__. This is invoked by import keyword. if you look in the help,

help( __import__ )

it explains what __import__ accepts as argument:

Help on built-in function __import__ in module builtins:

__import__ (...)
    __import__ (name, globals=None, locals=None, fromlist=(), level=0) -> module

    Import a module. Because this function is meant for use by the Python
    interpreter and not for general use it is better to use
    importlib.import_module() to programmatically import a module.

    The globals argument is only used to determine the context;
    they are not modified. The locals argument is unused. The fromlist
    should be a list of names to emulate ``from name import ...'', or an
    empty list to emulate ``import name''.
    When importing a module from a package, note that __import__ ('A.B', ...)
    returns package A when fromlist is empty, but its submodule B when
    fromlist is not empty. Level is used to determine whether to perform 
    absolute or relative imports. 0 is absolute while a positive number
    is the number of parent directories to search relative to the current module.

It also answered our question! When importing modules with scope resolution and fromList is empty, it returns the topmost object.

Note that is also indicates that you can use importlib.import_module() to programmatically import modules. Or should I say dynamically. As **import** provides functionality to import the modules, you can utilize it for dynamic importing. But that would be a bad practice as the documentation depreciates its direct use. So let's see what importlib can do for us.

Hey import library, import module named something...

Let's checkout how importlib.import_module() works:

import importlib
help(importlib.import_module)

As you can see, in the following, import_module accepts name of the module and package.

Help on function import_module in module importlib:

import_module(name, package=None)
    Import a module.

    The 'package' argument is required when performing a relative import. It
    specifies the package to use as the anchor point from which to resolve the
    relative import to an absolute import.

If you dir() on importlib, you will see that it also contains default **import** as well. Obviously, it is just a wrapper over **import** .

But it allows us to import modules by their name. Following is the code snippet of okai. Highlighted line is responsible for dynamic importing.

import importlib
from mind.thought import thought

class action(thought):
  def do(action, act_on):
    if(type(act_on) != thought):
      act_on = thought(act_on)

    todo = action.split('.')[-1]

    if( not callable(action) ):
      try:
        action = importlib.import_module('actions.'+ action)

      except ImportError:
        print("\n[ok]: I don't know how to - " + action + '.\n')
        # return thought('raise', 'EXIT_FAILURE')
        return thought('')

      try:
        toAct = getattr(action, todo)()
        return toAct.do(act_on)

      except Exception as e:
        print("[ok]: I can - " + action. __name__ + " but cannot - " + todo)
        print("[...] Because " + e.args[0])
        print()
        print(e)

    else:
      return action(act_on)

    return thought('')
# return thought('raise', 'EXIT_FAILURE')

That's that. You now know how import works and how to do dynamic importing. I hope you learned something new.

This article first appeared on my old blog, then dev.to. Since both are now dead, reposting here.