DEV Community

loading...
Cover image for Extending Python with Go

Extending Python with Go

astagi profile image Andrea Stagi ・5 min read

One of the most interesting features of CPython is the ability to add new built-in modules to Python written in C and C++. Python provides its API to do that, a set of headers and core types for writing extensions.

In this tutorial I’ll give you an overview of how to extend Python 3 with C, and then how to do the same thing using a modern language like Go.

Go is a valid alternative to C if you want to extend Python: is easier than C, has a Garbage Collector, provides great performances and executing code in parallel is really straightforward with Go routines.

Setup your environment

This tutorial is based on:

  • Python 3.8.1
  • Golang 1.13.5

Setup LIBRARY_PATH and PKG_CONFIG_PATH on your system. I run the code under macOS Catalina 10.15.1 so I have to set

export PKG_CONFIG_PATH=/Library/Frameworks/Python.framework/Versions/3.8/lib/pkgconfig/
export LIBRARY_PATH=/Library/Frameworks/Python.framework/Versions/3.8/lib
Enter fullscreen mode Exit fullscreen mode

Extending Python with C

Suppose you want to create a new module to make sums, something easy like

from newmath import sum

print (sum(5,4))
Enter fullscreen mode Exit fullscreen mode

Following the Python section in the docs, you can write the first extension newmath.c

#define PY_SSIZE_T_CLEAN
#include <Python.h>

static PyObject *sum(PyObject *self, PyObject *args) {
    const long a, b;

    if (!PyArg_ParseTuple(args, "LL", &a, &b))
        return NULL;

    return PyLong_FromLong(a + b);
}

static PyMethodDef MathMethods[] = {
    {"sum", sum, METH_VARARGS, "Add two numbers."},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef newmathmodule = {
    PyModuleDef_HEAD_INIT, "newmath", NULL, -1, MathMethods
};

PyMODINIT_FUNC PyInit_newmath(void) {
    return PyModule_Create(&newmathmodule);
}
Enter fullscreen mode Exit fullscreen mode

And compile it as a shared library

gcc -shared -o newmath.so `pkg-config --cflags --libs python3` `python3-config --libs --embed` newmath.c
Enter fullscreen mode Exit fullscreen mode

The result is a new module newmath.so that can be used in Python code as a module.

Do the same using Go

Go SDK comes with an amazing toolset called cgo which allows Go programs to interoperate with C and enables you to build shared libraries from Go.

Thanks to the magic C.* namespace is possible to use anything from the C world, so the C code above can be rewritten in Go in this way

package main

// #cgo pkg-config: python3
// #cgo LDFLAGS: -L/Library/Frameworks/Python.framework/Versions/3.8/lib -lpython3.8 -ldl -framework CoreFoundation
// #define PY_SSIZE_T_CLEAN
// #include <Python.h>
import "C"

//export sum
func sum(self, args *C.PyObject) *C.PyObject {
    var a, b C.longlong
    return C.PyLong_FromLongLong(a + b)
}

func main() {}
Enter fullscreen mode Exit fullscreen mode

In this case cgo needs a preamble before import “C”: it may contain any C code, functions, includes statements, variables declarations and definitions and may then be referred to from Go code as though they were defined in the package “C”. Compile everything with

go build -buildmode=c-shared -o newmath.so
Enter fullscreen mode Exit fullscreen mode

And newmath module is ready to be used in Python

from newmath import sum

print(sum(2, 40))
Enter fullscreen mode Exit fullscreen mode

It works! ✌🏻

Anyway this is not the method I prefer to extend Python with Go because there are some limitations: cgo doesn’t support variadic functions! In this piece of code I don't use PyArg_ParseTuple because variadic functions like that need to be wrapped in another function.

int PyArg_ParseTuple_LL(
    PyObject * args,
    long long * a,
    long long * b
) {
    return PyArg_ParseTuple(args, "LL", a, b);
}
Enter fullscreen mode Exit fullscreen mode

Macro functions need to be wrapped as well, e.g. PyLong_Check (source: include/python3.8/listobject.h).

int is_a_long(PyObject * p) {
    return PyLong_Check(p);
}
Enter fullscreen mode Exit fullscreen mode

Another way to extend Python with Go is moving all the Python stuff into C and just call the Go function inside C. To do that, define sum function as a normal Go function using export sum preamble.

package main

import "C"


//export sum
func sum(a, b int) int {
    return (a + b)
}

func main() {}
Enter fullscreen mode Exit fullscreen mode

and then compile everything with

go build -buildmode=c-archive -o libnewmath.a
Enter fullscreen mode Exit fullscreen mode

This command generates two files:

  • libnewmath.a we need to link later in the final step
  • libnewmath.h containing some definitions, including
extern GoInt sum(GoInt p0, GoInt p1);
Enter fullscreen mode Exit fullscreen mode

Now rewrite _newmath.c (prepend _ to exclude this file from go build) including libnewmath.h and using sum function we created with Go

#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "libnewmath.h"


static PyObject *sum_wrapper(PyObject *self, PyObject *args) {
    const long a, b;

    if (!PyArg_ParseTuple(args, "LL", &a, &b))
        return NULL;

    return PyLong_FromLong(sum(a, b));
}

static PyMethodDef MathMethods[] = {
    {"sum", sum_wrapper, METH_VARARGS, "Add two numbers."},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef newmathmodule = {
    PyModuleDef_HEAD_INIT, "newmath", NULL, -1, MathMethods
};

PyMODINIT_FUNC PyInit_newmath(void) {
    return PyModule_Create(&newmathmodule);
}
Enter fullscreen mode Exit fullscreen mode

And then generate the module newmath.so, linking libnewmath

gcc _newmath.c -shared -o newmath.so `pkg-config --cflags --libs python3` `python3-config --libs --embed` -L . -lnewmath
Enter fullscreen mode Exit fullscreen mode

Easier! 😉

Performances

Having to deal with different universes has a cost in terms of performances: when a Go function is used from another runtime, it spins up the Go runtime in parallel with the caller's runtime getting the goroutine threads, GC and all that other nice stuff that would normally be initialized up when running a Go program on its own.

Once you cross the boundary, try to do as much on the other side as you can! If you call a Go function inside a Python or C "for" cycle you spin up and destroy Go env on any iteration, it's more performant writing a single Python interface that wraps the execution of the entire "for" cycle inside Go.

When extending Python with Go is faster than using Python? The answer is parallel execution!

I tried to make a simple application to countdown from 25000000 to 0 with a thread and do the same thing with another thread. In Python I get the following results:

  • Using simple threads ~3.7 sec.
  • Using Multiprocessing Pools ~2.4 sec.
  • Using Python JobLib ~2.2 sec.

Extending Python with Go and doing the parallel computation inside Go takes only ~0.02 sec.! Python is not performant running CPU-bound multithread programs, it's limited by Global Interpreter Lock - GIL.

You can check the code on Github

More on cgo

Discussion (3)

Collapse
unacceptable profile image
Robert J.

I tried following this example for python 3.9.1 instead of 3.8.1.

For some reason when I do a go build I get an error that it's trying to use 3.8 even though I set the variables (PKG_CONFIG_PATH & LIBRARY_PATH) to 3.9. I am utterly confused by this. Any ideas what I am doing wrong here?

Collapse
unacceptable profile image
Robert J.

Never mind I sorted it out. I mentally ad-blocked comments when I was reading the code and missed the following:

// #cgo LDFLAGS: -L/Library/Frameworks/Python.framework/Versions/3.8/lib -lpython3.8 -ldl -framework CoreFoundation
Enter fullscreen mode Exit fullscreen mode

I changed it to

// #cgo LDFLAGS: -L/Library/Frameworks/Python.framework/Versions/3.9/lib -lpython3.9 -ldl -framework CoreFoundation
Enter fullscreen mode Exit fullscreen mode

and was able to build it in python 3.9.

Fantastic tutorial! Thanks for sharing!

Collapse
astagi profile image
Forem Open with the Forem app