Build Custom Computer Vision Applications

Author: Jacob Marks (Machine Learning Engineer at Voxel51)

Embed Custom UIs Directly into Data-Centric Workflows

Welcome to week three of Ten Weeks of Plugins. During these ten weeks, we will be building a FiftyOne plugin (or multiple!) each week and sharing the lessons learned!

If you’re new to them, FiftyOne Plugins provide a flexible mechanism for anyone to extend the functionality of their FiftyOne App. You may find the following resources helpful:

• FiftyOne Plugins Repo
• FiftyOne Plugin Docs
• Plugins Channel in the FiftyOne Community Slack

Here’s what we’ve built so far:

• Week 1: AI Art Gallery & Twilio Automation
• Week 2: Visual Question Answering

Ok, let’s dive into this week’s FiftyOne Plugin!

Custom YouTube Player Panel 📺🔧🔘

Over the first two weeks of this 10 Weeks of Plugins journey, all three of the plugins that I built were pure Python plugins plugins that allow you to create operators, which execute custom code behind the scenes. These plugins enable a plethora of workflows, and as primarily a Python programmer myself, I find the process for creating Python plugins to be quite intuitive, especially after creating a few.

Sometimes, however, it’s nice to be able to create a custom user interface. Take VoxelGPT for instance: the chatbot-like interface (plus easy statefulness!) is really only possible when given its own devoted space within the FiftyOne App. In FiftyOne, creating custom interfaces like this is possible via JavaScript Plugins, which give you blank canvases on which to design workflows and experiences.

In FiftyOne, these “canvases” are called Panels, and when you write a JavaScript plugin, you can build your own panels from scratch. Using JavaScript (and React Material UI components) you can essentially design a custom UI for yourself and/or others, which you can pull up at the click of a button. You’ll see what I mean shortly!

For the third week of 10 Weeks of Plugins, I built a YouTube Player Panel Plugin. For this plugin, I built a relatively simple UI consisting of a YouTube video player and a few selectors to make it interactive. While this plugin itself is not terribly complex, and my UI is not terribly artistic, it showcases the basic elements required to build a panel to your liking. 

The best part: I used ChatGPT to write the JavaScript for me! 

Plugin Overview & Functionality

The YouTube Player Panel Plugin is a joint Python/JavaScript plugin with a single operator: open_youtube_player_panel, which opens a custom JavaScript component named YouTubePlayerPanel. You can choose to play a video from a preset list of YouTube videos from the Voxel51 YouTube channel, or you can play a YouTube video from URL.

There are three ways to execute the  open_youtube_player_panel operator and open the panel:

1 - Press the YouTube player button in the Sample Actions Menu:

2 - Click on the + icon next to the Samples tab and select YouTube Player from the dropdown menu:

3 - Press " ` " to pull up your list of operators, and select "open_youtube_player_panel":

The Python portion of the plugin is exceedingly simple:

• Storing an SVG icon for YouTube in an assets folder within the plugin’s directory, I specified this as the icon to use to represent the operator in two ways: first, in the operator’s config() method, so that this would be the icon in the operators list, and second, in its resolve_placement() method, so that the button in the action menu is also the YouTube icon.

• In the execute() method, the operator triggers an open_panel action, and the params passed into trigger() instruct that the panel to be opened is named YouTubePlayerPanel.

The JavaScript portion is a bit more involved. It consists of:

• YouTubeIcon: the SVG icon that appears next to the label YouTube Player on the tab in the panel.
• YoutubeEmbed: an iframe component (basically an embedded live view of HTML loaded on another page — in this case YouTube). This component takes as an argument an embedId, which is the unique identifier for YouTube videos.
• YouTubePlayerPanel: the custom panel containing the YouTube player iframe, a list of preset videos, a text field for inputting a different video link, and the logic for updating the display based on values of the associated variables.
• registerComponent: this registers the YouTubePlayerPanel we defined as a Panel with the name “YouTubePlayerPanel”, so that we can reference it (and open it) from our Python operator.

Installing the Plugin

If you haven’t already done so, install FiftyOne:

pip install fiftyone

Then you can download this plugin from the command line with:

fiftyone plugins download https://github.com/jacobmarks/vqa-plugin

Refresh the FiftyOne App, and you should see the YouTube button show up in the actions menu. 

Lessons Learned

Icon Rendering

FiftyOne plugins are very flexible when it comes to icon rendering. If you desire, you can have distinct icons for an operator in the action menu, the operator list, and at the top of its panel. Even though this particular plugin only uses one icon, I try to stick to the convention of putting all icons in an assets folder, and then referencing the svg file via absolute path within the plugin’s directory. For example:

icon="/assets/youtube.svg"

For the icon in the panel itself, I had to create a separate icon component in JavaScript. The paths in the svg were the same, but I had to use different viewBox coordinates to make all of the icons line up nicely in their respective contexts.

Trigger Composes Operators

In the AI Art Gallery and Twilio Automation plugins, I had used the ctx.trigger() method to perform operations like reloading samples (ctx.trigger(“reload_samples”)), and reloading the dataset (ctx.trigger(“reload_dataset”)). I was even aware from VoxelGPT that you could use ctx.trigger() to set the session’s view.

But it wasn’t until this YouTube Player Plugin that it clicked for me that ctx.trigger() allows you to compose operators. You can trigger a Python operator from JavaScript, or a JavaScript operator from Python. You can even trigger operators defined in other plugins!

In this plugin the Python operator’s execute() method looks like:

def execute(self, ctx):
        ctx.trigger(
            "open_panel",
            params=dict(
                name="YouTubePlayerPanel", isActive=True, layout="horizontal"
            ),
        )

What this means is that when executed, the open_youtube_player_panel operator is triggering another operator, open_panel, with the specific input parameters. In this case, the open_panel operator is opening the panel named YouTubePlayerPanel, which we defined in our YouTubePlayerPlugin.tsx file, and is splitting the canvas of the FiftyOne App in two horizontally (as opposed to vertically).

You can do so much with these operator compositions. I’m excited to explore this further in coming weeks!

No JavaScript?; No Problem! 

My biggest takeaway from building this plugin is that you don’t actually need to know any JavaScript, React, or Typescript to build a “JavaScript” plugin in FiftyOne. The greatest strength of FiftyOne’s JavaScript plugins is their straightforward interface. 

Because FiftyOne plugin components map fairly directly onto React Material UI components, I found that the FiftyOne Plugin docs plus the Material UI docs plus ChatGPT was more than enough to turn my vision into reality. If you’d like to dive deeper into this, let me know and I’ll put together some resources :)

Conclusion

FiftyOne plugins are incredibly flexible. They give you the blank canvas you need to create tailor-made data-centric machine learning applications. Playing YouTube videos in the FiftyOne App is cool, but the real value is the ability to educate, to showcase workflows, and to create a compelling story — all with data at the forefront!

Stay tuned for the remainder of these ten weeks as we continue to pump out a killer lineup of plugins! You can track our journey in our ten-weeks-of-plugins repo — and I encourage you to fork the repo and join me on this journey!