Implementing a Plugin System in Python

This post was originally published on Siv Scripts

In July, I released my first open source project. It's an apispec plugin that generates OpenAPI Specification (aka Swagger docs) for Falcon web applications.

Apispec's design made it easy to extend core functionality for a specific use case. I extended the apispec.BasePlugin class, overrode a couple of methods, and I was done. Had to dig into apispec internals to figure out how things were wired together, but it was easy to build upon what was already there

This got me thinking about how to implement a plugin system in one of my own applications. Creating a plugin architecture? It sounded hard. An advanced computer science concept, if you will.

I mulled over a few different implementations based on software I had previously used. I realized this wasn't a difficult problem. Like everything else in programming, once we deconstruct the problem into smaller chunks, we can reason about implementation details clearly.

We assume things are more difficult than they appear. This is especially true for problems we have not seen before. Take a step back. Breathe. Break the problem down into smaller pieces. You got this.

In this Quick Hit, we will walk through the implementation of a simple plugin system.

---

Background

A plugin is a software component that adds a specific feature to an existing computer program. When a program supports plug-ins, it enables customization (Wikipedia)

There are many benefits to building apps with a plugin framework:

• 3rd party developers can create and extend upon your app
• new features are easier to developer
• your application becomes smaller and easier to understand

Sample Application Flow

We have a program that starts, does a few things, and exits.

Plugin Architecture Workflow

We refactored our Business Logic into a Plugin Framework that can run registered plugins. The plugins will need to meet the specifications defined by our framework in order to run.

---

Toy Example

Let's implement a toy example where we have a plugin framework to print to the console. Our project will have the following structure:

.
├── internal.py  # internal business logic
├── external.py  # contains user-created plugins
└── main.py      # initialize and run application

Internal

This module contains the application class and an internal plugin.

# internal.py

class InternalPrinter:
    """Internal business logic"""
    def process(self):
        print("Internal Hello")


class MyApplication:
    """First attempt at a plugin system"""
    def __init__(self, *, plugins: list=list()):
        self.internal_modules = [InternalPrinter()]
        self._plugins = plugins

    def run(self):
        print("Starting program")
        print("-" * 79)

        modules_to_execute = self.internal_modules + self._plugins
        for module in modules_to_execute:
            module.process()

        print("-" * 79)
        print("Program done")
        print()

External

# external.py

class HelloWorldPrinter:
    def process(self):
        print("Hello World")


class AlohaWorldPrinter:
    def process(self):
        print("Aloha World")

Main

In this module, we run instances of our application with the external plugins we want to enable.

# main.py

from internal import MyApplication
from external import HelloWorldPrinter, AlohaWorldPrinter


if __name__ == "__main__":
    # Run with one plugin
    app = MyApplication(plugins=[HelloWorldPrinter()])
    app.run()

    # Run with another plugin
    app = MyApplication(plugins=[AlohaWorldPrinter()])
    app.run()

    # Run with both plugins
    app = MyApplication(plugins=[HelloWorldPrinter(), AlohaWorldPrinter()])
    app.run()

Discussion

The Application's plugin framework works for both internal and external plugins. We define internal plugins in the application code. External plugins are initialized and passed into the application at runtime.

Each plugin inherits from the base Python object and has a process() method. Nothing complex, want to keep this example as simple as possible.

We can run our plugins by calling the application's run() method. This method loops over all the plugins and calls each instance's process() function. As we see from the output above, the plugins are executed in the same order as the list.

Running Toy Application

$ python main.py
Starting program
------------------------------------------------------------------------------------
Internal Hello
Hello World
------------------------------------------------------------------------------------
Program done

Starting program
------------------------------------------------------------------------------------
Internal Hello
Aloha World
------------------------------------------------------------------------------------
Program done

Starting program
------------------------------------------------------------------------------------
Internal Hello
Hello World
Aloha World
------------------------------------------------------------------------------------
Program done

Real World Example

This pattern can be used in conjunction with the Adapter pattern to simplify application development.

Let's say we have a large number of external clients we want to interface with. Each API is different, but the tasks we need to perform for each client are the same.

One possible implementation of this is to write an adapter around each of the client APIs, resulting in a common interface. Next, we can leverage the plugin framework to solve our business problem, and then we can use plugins to make it work for all of our clients.

This is a very high level description of the solution. I leave implementation as an exercise to the reader.

---

Additional Resources

• A Simple Plugin Framework
• Plugin libraries
  • pluggy
  • yapsy
  • straight.plugin

Comments

[Jan-Philipp]

I am currently working with a plugin system within a daemon and I really love the abilitty to quickly add plugins in runtime or just fix them. You broke down the core of it really good and I will take that to use within my personal project soon :) Thanks!