Python C/C++ binding - Some notes

Python binding is the connection between Python and external libraries or languages, allowing Python code to interface with C/C++, Java, or system-level libraries for extended functionality.

Source code

But why should we use Python binding

There are many cases where we may want to integrate C/C++ code with Python program, for example:

1. You already have a large, tested, stable library written in C++ that you’d like to take advantage of in Python. This may be a communication library or a library to talk to a specific piece of hardware. What it does is unimportant.

2. You want to speed up a particular section of your Python code by converting a critical section to C. Not only does C have faster execution speed, but it also allows you to break free from the limitations of the GIL, provided you’re careful.

3. You want to use Python test tools to do large-scale testing of their systems.

C vs C++ binding

• The binding concept and principles applied similarly for both C and C++, in most cases.

Binding concepts

Marshalling Data Types

• marshalling: The process of transforming the memory representation of an object to a data format suitable for storage or transmission.

In this case, we prepare data to move it from Python to C or vice versa
C stores data in the most compact form in memory possible. If you use an uint8_t, then it will only use 8 bits of memory total. In Python, on the other hand, everything is an object.

Understanding Mutable and Immutable Values

• C has pass-by-value or pass-by-reference

Managing Memory

• Python have garbage collector, C don't
• You’ll need to be aware of where the memory for each object was allocated and ensure that it’s only freed on the same side of the language barrier.

Setting Up Your Environment

• A C++ library installed and knowledge of the path for command-line invocation
• Python development tools (python3-dev)
• Python 3.6 or greater
• The invoke tool

Using the invoke Tool

• invoke is the tool you’ll be using to build and test your Python bindings in this tutorial (similar to make)

python3 -m pip install invoke

# build-cmult is build task
invoke build-cmult

# list available tasks
invoke --list

• invoke all runs the build and test tasks for all tools.
• invoke clean removes any generated files.

C or C++ Source

ctypes

• ctypes is a tool in the standard library for creating Python bindings

Calling the Function

The steps are as following:

• Load your library.
• Wrap some of your input parameters.
• Tell ctypes the return type of your function.

Library Loading

• ctypes provides several ways for you to load a shared library

libname = pathlib.Path().absolute() / "libcmult.so"
c_lib = ctypes.CDLL(libname)

Calling Your Function

float cmult(int int_param, float float_param);

x, y = 6, 2.3

# Below command fail. Except for int, ctypes don't know how to convert other param types.
# answer = c_lib.cmult(x, y)

# below line let ctypes know that returned type is float
c_lib.cmult.restype = ctypes.c_float

answer = c_lib.cmult(x, ctypes.c_float(y))
print(f"    In Python: int: {x} float {y:.1f} return val {answer:.1f}")

CTypes Strengths and Weaknesses

• Strength: ctypes is built-in
• Weakness: manual works have to be done

CFFI

• CFFI is the C Foreign Function Interface for Python
• More automated approach to generate Python bindings

CFFI has multiple ways in which you can build and use your Python bindings:

• ABI vs API: API mode uses a C compiler to generate a full Python module, whereas ABI mode loads the shared library and interacts with it directly.
• in-line vs out-of-line: trade-off between speed and convenience
  • In-line mode compiles the Python bindings every time your script runs
  • Out-of-line mode requires an extra step to generate the Python bindings a single time and then uses them each time the program is run

This example use API out-of-line mode.

How It’s Installed

pip install cffi

Calling the Function

• Write some Python code describing the bindings.
• Run that code to generate a loadable module.
• Modify the calling code to import and use your newly created module.

Write the Bindings

# tasks.py
import cffi

""" Build the CFFI Python bindings """
print_banner("Building CFFI Module")

# 1. Create FFI object
ffi = cffi.FFI()

# 2. Reading and processing the header file
ffi.cdef(h_file.read())

# 3. Describe the source file that CFFI will generate
ffi.set_source(
    "cffi_example",
    # Since you're calling a fully-built library directly, no custom source
    # is necessary. You need to include the .h files, though, because behind
    # the scenes cffi generates a .c file that contains a Python-friendly
    # wrapper around each of the functions.
    '#include "cmult.h"',
    # The important thing is to include the pre-built lib in the list of
    # libraries you're linking against:
    libraries=["cmult"],
    library_dirs=[this_dir.as_posix()],
    extra_link_args=["-Wl,-rpath,."],
)

# 4. Build the Python Bindings
ffi.compile()

# 5. Calling Your Function
import cffi_example
answer = cffi_example.lib.cmult(x, y)

PyBind11

• Install: python3 -m pip install pybind11

• Build the cppmult lib
  

invoke.run(
    "g++ -O3 -Wall -Werror -shared -std=c++11 -fPIC cppmult.cpp "
    "-o libcppmult.so "
)

• Writing the Bindings:

#include <cppmult.hpp>

PYBIND11_MODULE(pybind11_example, m) {
    m.doc() = "pybind11 example plugin"; // Optional module docstring
    m.def("cpp_function", &cppmult, "A function that multiplies two numbers");
}

• Build Python bindings

# tasks.py
invoke.run(
    "g++ -O3 -Wall -Werror -shared -std=c++11 -fPIC "
    "`python3 -m pybind11 --includes` "
    "-I /usr/include/python3.7 -I .  "
    "{0} "
    "-o {1}`python3.7-config --extension-suffix` "
    "-L. -lcppmult -Wl,-rpath,.".format(cpp_name, extension_name)
)

• Calling Your Function

import pybind11_example
answer = pybind11_example.cpp_function(x, y)

Strengths and Weaknesses

• PyBind11 is focused on C++ instead of C
• It supports classes.
• It handles polymorphic subclassing.
• It allows you to add dynamic attributes to objects from Python and many other tools, which would be quite difficult to do from the C-based tools you’ve examined.

Cython

• The approach Cython takes to creating Python bindings uses a Python-like language to define the bindings and then generates C or C++ code that can be compiled into the module.

• Cython can support both C and C++

• Install:
  

python3 -m pip install cython

• Write the Bindings

# cython_example.pyx
""" Example cython interface definition """

cdef extern from "cppmult.hpp":
    float cppmult(int int_param, float float_param)

def pymult( int_param, float_param ):
    return cppmult( int_param, float_param )

• The language used here is a special mix of C, C++, and Python

• Build the Python Bindings:
  

# tasks.py
def compile_python_module(cpp_name, extension_name):
    invoke.run(
        "g++ -O3 -Wall -Werror -shared -std=c++11 -fPIC "
        "`python3 -m pybind11 --includes` "
        "-I /usr/include/python3.7 -I .  "
        "{0} "
        "-o {1}`python3.7-config --extension-suffix` "
        "-L. -lcppmult -Wl,-rpath,.".format(cpp_name, extension_name)
    )

def build_cython(c):
    """ Build the cython extension module """
    print_banner("Building Cython Module")
    # Run cython on the pyx file to create a .cpp file
    invoke.run("cython --cplus -3 cython_example.pyx -o cython_wrapper.cpp")

    # Compile and link the cython wrapper library
    compile_python_module("cython_wrapper.cpp", "cython_example")
    print("* Complete")

• Calling Your Function

import cython_example
answer = cython_example.pymult(x, y)

Strengths and Weaknesses

• Cython is a relatively complex tool that can provide you a deep level of control when creating Python bindings for either C or C++.

Other Solutions

• There are many other tools that we can use for Python/C-C++ binding: PyBindGen, Boost.Python, SIP, Cppyy, Shiboken, SWIG

References

• Walkthrough of Python/C++ Binding
• realpython