Speed Up Your Python Codebases With C Extensions

Nicholas Obert — Sat, 23 Apr 2022 17:28:31 +0000

Bring the speed of C to your Python programs

Python, being both easy and powerful, has become one of the most popular programming languages. Nonetheless, it sometimes lacks the much-valued speed of statically typed and precompiled programming languages like C and Java.

Why is Python Slow?

As you may know, Python is generally implemented through an interpreter. This can cause the code execution to be rather slow compared to languages, like C and Java, with compiled implementations and whose source code is compiled in advance into machine/byte code. However, this topic is beyond the scope of the article.

How Can You Speed Up Python Code?

Unless you have to perform computationally heavy operations, Python’s speed is not usually a problem. This is where C extensions come in handy.

C extensions are a way to code functions in C, compile them into a Python module and use them in your source code as a normal Python library.

Many popular modules are actually written in C or C++ (e.g. numpy, pandas, tensorflow…) for better performance and/or low-level functionalities.

A few quick disclaimers:

C extensions do not work for Python implementations other than Cpython. This should not be a problem since Cpython is the default one.
It is recommended you have basic knowledge of the C programming language. However, if you only know Python you should be able to follow along without any problems.

How to Build a C Extension

As an example, let’s implement the classic fib(n) function. fib(n) takes in a number, n, and returns the corresponding number in the Fibonacci sequence. Then we will compare the performance of the Python and the C version.

First of all, you need the Python C API Python.h. It’s a C header file that contains everything that’s needed to interface with Python.

Python API installation

On Linux you usually have to install the python-dev or python3-dev package, if not already present. (Note that on some distros the package name may be different)
By default, if installed with the default installer, Windows should come with Python.
MacOs should also come with Python. If it does not, brew reinstall python should do the trick.

Now, open the code editor of your choice and create the C module file. It should be named by convention — something along the lines of module_name.c, although you can name it whatever you want. Here we’ll call it c_module.c.

Before you start coding your extension, you need to include some core definitions and declarations.

It is recommended that you put these lines at the beginning of your file for compatibility purposes.

Since in Python everything is an object, our c_fib(n) function should return one, precisely a PyObject pointer (defined in Python.h).

Then it’s necessary to declare which functions to export from the module to make them accessible to Python.

Defining the module methods

Every exported method is represented as a struct comprised of:

The exported method name (in this case "c_fib”)
The actual method to be exported (c_fib)
The type of arguments the method takes (in this case METH_VARARGS). This is from the documentation on METH_VARARGS: “This is the typical calling convention, where the methods have the type PyCFunction. The function expects two `PyObject values. The first one is the self object for methods; for module functions, it is the module object. The second parameter (often called args`) is a tuple object representing all arguments.”*
A const char* describing the method

Defining the module

The module is represented as a struct, as shown in the code above. It’s self-documenting, except for the m_size argument, which we set to -1. From the documentation:

Setting m_size to -1 means that the module does not support sub-interpreters, because it has global state.

Module initialization function

The PyMODINIT_FUNC gets called when the module is imported, initializing it. Note that the function name must start with PyInit_ and end with your module’s name, hence PyInit_c_module().

These are just a few of the Python API’s functionalities. For more information and features visit the API’s documentation page.

Compile Your Extension to a Python Module

Once the C code is complete, you have to compile it to a Python module. Luckily there are a bunch of built-in tools that allow you to do exactly that.

Create a Python script, traditionally named setup.py, and insert the following code:

This script has many functionalities, but we’ll be using only the build and install commands. For more info, take a look at the documentation or just run the script with the help flag:

 python3 setup.py --help

From the command line run the following:

python3 setup.py build

This will create a directory named build and put the compiled libraries in it. When done, run this:

python3 setup.py install

This will install the just-built libraries on your system, making them accessible from anywhere.

Note that you might need root/admin privileges in order to do that. You don’t have to install it system-wide, but if you skip the install process you’ll have to use relative imports in order to use the extension.

Use C Extensions Inside a Python Program

Inside your Python file, import the newly created module using the name you chose. In our case, it is c_module:

As you can see, the extension can be used the same way as any other Python module would.

How Does It Compare With the Plain Python Version?

Now let’s compare the c_fib function with its plain Python counterpart. We’ll use the built-in time module:

Output:

Input: 5
py_res=5, py_time=5.245208740234375e-06
c_res=5, c_time=1.6689300537109375e-06

As expected, the C function is faster.

Note that you might get different timings on a different machine, but the C version of the same code will always be faster.

Let’s now try with some larger numbers:

Input: 10
py_res=55, py_time=3.147125244140625e-05
c_res=55, c_time=2.6226043701171875e-06

Input: 30
py_res=832040, py_time=0.40490126609802246
c_res=832040, c_time=0.004115581512451172

Input: 40
py_res=102334155, py_time=50.17047834396362
c_res=102334155, c_time=0.4414968490600586

The C version clearly outperforms the Python one when it comes to large numbers. If you just have to do a few simple calculations, it might not be worth implementing them in C as the performance difference would be minimal. However, given a really time-consuming operation or a function that has to be repeated many times, Python’s speed might not be enough.

This is where C extensions really shine. You can leave all the heavy lifting to C, but still use Python as your main language.

Real-Life Use-Cases for C Extensions

Say you have to perform some heavy calculations, be it for a cryptographic algorithm, deep learning model training, or processing large amounts of data. C extensions can relieve the burden from Python’s vertebrae and speed up your application.

What if you instead wanted to build a low-level interface or work directly on memory from Python? C extensions are the way to go, given you know how to work with raw pointers.

What about optimizing an already-existing Python application that performs badly but you don’t want to (or can’t) rewrite it in another language? C extensions are the answer.

Or what if you’re just an optimization diehard who wants his code to run as fast as possible, but you still want some high-level abstractions for networking, a GUI, and so on. In this case, C extensions are definitely your best friend.

Time is the thing we always wish we had more of. Invest it wisely.

Conclusion

Whether you are a Python developer addicted to performance and efficiency, someone who likes mixing in different technologies, or you just want to experiment with something new, C extensions for Python are a great addition to your developer toolbox. Not only do they provide you with virtually free performance, but they can extend the functionalities of Python, saving it from the obsolete technologies stack.

Thank you for reading.

DEV Community: Nicholas Obert