Plugin for Building and Managing Plugins!

Author: Jacob Marks (Machine Learning Engineer at Voxel51)

Use this plugin to build your own FiftyOne plugins

Welcome to the tenth and final week of Ten Weeks of Plugins. During these ten weeks, we have built a FiftyOne plugin (or multiple!) each week and shared the lessons learned!

If you’re new to them, FiftyOne Plugins provide a flexible mechanism for anyone to extend the functionality of their FiftyOne App. You may find the following resources helpful:

• FiftyOne Plugins Repo
• FiftyOne Plugin Docs
• Plugins Channel in the FiftyOne Community Slack

What we’ve built so far:

• Week 0: 🌩️ Image Quality Issues & 📈 Concept Interpolation
• Week 1: 🎨 AI Art Gallery & Twilio Automation
• Week 2: ❓Visual Question Answering
• Week 3: 🎥 YouTube Player Panel
• Week 4: 🪞Image Deduplication
• Week 5: 👓Optical Character Recognition (OCR) & 🔑Keyword Search
• Week 6: 🎭Zero-shot Prediction
• Week 7: 🏃Active Learning
• Week 8: ⏪Reverse Image Search
• Week 9: 🌌Concept Space Traversal

Ok, let’s dive into this week’s FiftyOne Plugin - a Plugin-Builder/Manager Plugin!

💪This plugin was co-developed with Voxel51 CEO Brian Moore, and will be available as a Voxel51 core plugin!

The Plugin Plugin 🧩🛠️🧩

Over the past ten weeks, I’ve built a LOT of plugins. Just look at the list above and you’ll see plugins spanning every stage of the data lifecycle, from ingestion to annotation, cleaning, conversing with, and using as a queryable database.

Some of these plugins were simpler than others. On one end, the Twilio automation plugin consists of a single Python file without bells and whistles. On the opposite extreme, plugins like Active Learning, which required multiple operators, caching, and special handling for many different scenarios. Plugins like Reverse Image Search and Concept Space Traversal were challenging in a different way, mostly because I am new to JavaScript. But that is for another day.

After building more than a dozen plugins, I decided it was time to get meta. Working together, Brian Moore and I built a plugin for managing and building plugins. This plugin provides a simple UI that makes it easy to:

1. Manage your existing plugins
2. Install new plugins
3. Construct a component
4. Generate the scaffolding for a new Python operator/plugin

This plugin is of course not a silver bullet — it is not going to write a plugin for you from start to finish. But it is a great way to outsource boilerplate so you can focus on the core logic! 

💡There are infinitely many ways to improve this plugin. If there’s an addition or modification that would make plugin management/development easier, submit a pull request! Community contributions are more than welcome — that’s who the plugin is for.

Plugin Overview & Functionality

The Plugin Plugin is a Python plugin with four operators, which streamline your plugin workflows as follows:

• install_plugin: install new plugins
• manage_plugins: manage your installed plugins
• build_component: design the UI for input components
• build_operator_skeleton: create the scaffolding for a Python plugin

Installing Plugins

The install_plugin operator allows you to install FiftyOne plugins directly from the FiftyOne App. No need to use the command line. You can install any Python plugin by selecting the first tab (GITHUB) in the operator’s modal and typing or pasting in information detailing where the plugin can be found on GitHub. This can be the URL, ref, or ref string of the GitHub repo. Here’s what it looks like to install the line2d plugin by @wayofsamu:

On the backend, this operator is essentially a wrapper around FiftyOne’s plugin installation methods:

import fiftyone.plugins as fop

# Plugin installation
fop.download_plugin(...)

For plugins registered (AKA listed) in the FiftyOne Plugins GitHub README, the process is even simpler — you can just select the plugin from the dropdown menu! The second tab (VOXEL51) contains all of the plugins included in the Core Plugins, Voxel51 Plugins, and Example Plugins tables. The third tab (COMMUNITY) contains all of the plugins listed in the Community Plugins table. Here’s an alternative method for installing the line2d plugin:

All of the plugins listed in the dropdowns in the second and third tabs are pulled in real-time from the FiftyOne Plugins Repo’s README. This means that these will stay up to date as more plugins are added.

🚀If YOU add your plugin as a FiftyOne Community Plugin (just submit a PR!) then everyone who uses this Plugin Plugin will see your plugin and be able to install it directly from the FiftyOne App!

Managing Plugins

Once you have plugins downloaded and installed, it is only natural to want a streamlined interface for managing those plugins. For instance, you may want to enable some, disable others, and ensure that all of the requirements for a given plugin are met.

FiftyOne also has methods for doing these management operations:

import fiftyone.plugins as fop

# Plugin enablement
fop.list_plugins()
fop.enable_plugin(name)
fop.disable_plugin(name)

# Plugin package requirements
fop.load_plugin_requirements(name)
fop.ensure_plugin_requirements(name)

The manage_plugins operator wraps all of these methods in a simple UI so you can accomplish any of these tasks from within the FiftyOne App!

The first tab, ENABLEMENT, allows you to enable or disable any of your installed plugins. The switch in the right column represents what state that plugin is currently in. When you change one or more of the switches, you will have the option to set the enablement status of the respective plugins:

The second tab, REQUIREMENTS, makes it easy to check whether all of the dependencies for a given plugin are met:

Creating Components

The build_component operator makes it easy to create input components for operators in Python plugins (the contents within the operator’s resolve_input() method). When I began building FiftyOne plugins, I found myself living in the
FiftyOne Operator Types documentation. In particular, I was constantly looking through the list of operator types to find allowed view types for a given type of data.

Here’s an example: how do you visually represent and capture user input for a boolean (True/False) variable? One option is to use a checkbox, where the user can check or uncheck the box to specify the value of the variable. Another option is a switch, which the user can toggle to the on or off state.

Each data type has its own set of allowed or sensible view types. For some types, such as boolean data, the difference in code is simply changing view=types.CheckboxView()to view=types.SwitchView(). However, for certain input types the syntax can change from one view to another.

Take floats for example. If we use an input field (FieldView) to capture user input, and we want to specify a maximum allowed value of 10., we would write something like this:

inputs.float(
    "my_float",
    label="My float label",
    description="My float description",
    view=types.FieldView(componentProps={'field': {'max': 10}}),
)

But if instead we wanted to use a slider (SliderView), we would need to substitute this:

view=types.SliderView(componentProps={'slider': {'max': 10}}),

The difference is small, but over the course of writing an entire plugin (or multiple plugins!), these subtleties can add up.

The build_component operator saves you these headaches by providing the code for your component, constructed entirely via UI in the FiftyOne App. 

The operator supports the following types:

• Choice: where one option out of a list can or must be selected
• Boolean: True or False
• Number: an integer or float
• Message: a message to the user, which doesn’t take any inputs

Scaffolding a Plugin

The final operator is the one which will likely save you the most time: build_operator_skeleton. After I built my first few FiftyOne plugins, every time I went to create a new plugin I found myself copying and pasting code from previous plugins into a new directory. I used these previous plugins as a template for the new plugin.

Why? Because along with the flexibility of FiftyOne’s plugin system comes a certain level of verbosity in plugin code. In other words, there is a good amount of boilerplate code! The code is all necessary, but the volume of code doesn’t always match up with the amount of information encoded.

To solve this problem, I created the build_operator_skeleton operator! This operator turns the high-level design of Python plugins into a point and click UI experience. In particular, it divides the process up into a few steps:

1. Config & Placement: input the operator’s basic info, choose which flags you want turned on in the operator’s config, and specify how you want the operator to appear as a button in the app, if at all.
2. Input & Output: set whether the operator captures input from the user and if it prints output after execution. 
3. Execution & Delegation: specify which trigger happens on operator execution, if any, and configure the delegation of execution. Triggers include reloading samples, reloading the entire dataset, setting the view, and opening a panel. Delegation options include True (delegate), False (don’t delegate), and User Choice (give user the choice whether or not to delegate)
4. Preview Code: scroll through the dynamically updated code which would populate the __init__.py file of the plugin to be generated. If you want to make changes to 1-3, you can do so at any time.
5. Create: choose the directory where you want the plugin to be generated, provide the information necessary to fill out the fiftyone.yml file, and generate this code on execution of the operator.

After executing the code, you will see the new operator appear in your operators list when you press " ` ", so long as you didn’t set unlisted=True! However, this generated code is not the end of the story. If you look at the code, you may see blocks that look like this:

def execute(self, ctx):
        ### Your logic here ###
        ### Create your view here ###
        view = ctx.dataset.take(10)
        ctx.trigger(
            "set_view",
            params=dict(view=serialize_view(view)),
        )
        return {}

And this:

def resolve_input(self, ctx):
        inputs = types.Object()
        ### Add your inputs here ###
        return types.Property(inputs)

This plugin just creates the scaffolding or “skeleton” of the plugin so that you can focus on the plugin’s logic: input, output, and execution. You will need to define any views that you want to set (the placeholder is view = ctx.dataset.take(10)), and you will need to specify the paths to any SVG files you want to include as icons.

💡At the moment, this operator only supports single-operator plugins, and it does not integrate directly with the component builder operator. If you are excited about streamlining the FiftyOne plugin developer experience, reach out and/or submit a PR to expand this plugin’s functionality!

Lessons Learned

Dynamically Updating Code Blocks

An essential part of the build_component and build_operator_skeleton operators was the dynamically updating code previews. To make the building process interactive, having the code update each time the user makes a change to one of the inputs/arguments.

At first I tried passing dynamic=True into these operators, but this was not sufficient: the code would not reliably update. The solution to this problem (credit to Ritchie Martori) was to make the identifier for code block component dependent on each of the inputs. This way, the app would act as if the old component was removed and a new component matching the new input specifications had been created in its place, achieving the effect of dynamic updating.

This is most easily demonstrated with an example. Let’s go back to the booleans in the component builder. The code looks like this:

code = f"""
    inputs.bool(
        "my_boolean",
        label="My boolean label",
        description="My boolean description",
        view={view_text}(),{default_code}
    )"""

Where view_text and default_code are generated programmatically based on user choices for the view type (CheckboxView or SwitchView) and whether or not there should be a default.

The preview for the code is represented as a string with a CodeView view type with language set to Python:

inputs.str(
        f"boolean_code_{view_type}_{default}",
        label="Boolean Code",
        default=dedent(code),
        view=types.CodeView(language="python"),
    )

The identifier for this code block depends on the view type and the default, so the code ends up being dynamically regenerated each time a change is made to either of them!

Plugin Management Deserves a Panel

The plugin management and creation operators that Brian and I built are powerful as currently constructed. But one of the primary points this process made clear is that plugin management and creation utilities deserve their own panel. 

For plugin management, my dream is to have a panel similar to the VSCode extension management sidebar, where you can see all of the plugins that you have enabled, disabled, and which plugins need updating. Like VSCode, the panel would allow you to browse officially supported and third-party plugins, viewing rating scores and detailed information about their functionality.

For plugin creation, this could enable a more full-fledged plugin builder UI with drag and drop components, diagrammatic representations for inputs and outputs, and editing the plugin’s code in real-time.

If you are interested in building this, reach out!

It’s Plugins All The Way Down

When building the build_component and build_operator_skeleton operators, one of the hardest parts was limiting scope. Plugins are insanely flexible, giving you the power to build custom computer vision applications for almost any workflow. This same flexibility makes it possible to build a plugin-builder plugin. But by the same token, it is impossible to bottle up the full richness of the plugin system and provide this via a crisp UI.

There are infinitely many ways to extend the plugin building functionality, from including icon selection/generation to integrating the component builder with the operator skeleton builder, and even using a large language model to create the README (or build the plugin for you!). I decided to keep it simple as a starting point that I know will be useful to other plugin developers, because it is already useful for me!

If you have ideas for how to improve or extend the plugin builder operators, I encourage you to reach out and/or submit a PR :)

Conclusion

The Plugin Plugin or “metaplugin” represents the culmination of our Ten Weeks of Plugins journey. But it is really just the beginning! These ten weeks have just been a taste of what is possible with FiftyOne’s extensible plugin framework — and soon it will be easier than ever to define custom UIs using just Python.

The fun is only just beginning. On November 15th, we’re hosting a workshop on all things plugins; we’ve got some bonus rounds of plugins in store for you over the next few weeks; and stay tuned for the first ever FiftyOne Plugins Hackathon!