FiftyOne Computer Vision Tips and Tricks - March 8, 2024

Welcome to our weekly FiftyOne tips and tricks blog where we recap interesting questions and answers that have recently popped up on Slack, GitHub, Stack Overflow, and Reddit.

As an open source community, the FiftyOne community is open to all. This means everyone is welcome to ask questions, and everyone is welcome to answer them. Continue reading to see the latest questions asked and answers provided!

Wait, what’s FiftyOne?

FiftyOne is an open source machine learning toolset that enables data science teams to improve the performance of their computer vision models by helping them curate high quality datasets, evaluate models, find mistakes, visualize embeddings, and get to production faster.

• If you like what you see on GitHub, give the project a star.
• Get started! We’ve made it easy to get up and running in a few minutes.
• Join the FiftyOne Slack community, we’re always happy to help.

Ok, let’s dive into this week’s tips and tricks!

Extracting the SimilarityIndex

Community Slack member Rafi asked:

Suppose I've computed similarity using fob, for instance:

results = fob.compute_similarity(
    dataset,
    embeddings="embedding",
    brain_key="sim",
    metric="cosine",
)

If I aim to later extract the SimilarityIndex using the brain_key, how can I achieve this?

This should be straight-forward when you use:

results = dataset.load_brain_results(brain_key)

Learn more about the FiftyOne Brain’s “similarity” features in the Docs.

Applying prediction labels to video samples

Community Slack member ZKW asked:

Can I apply prediction labels to video samples and display the video with dynamic labels (like moving a bbox) in FiftyOne App?

Check out the “Video Labels – FiftyOne Tips and Tricks” blog. Frame level detections allow you to apply detection labels to a video, in case the bounding_boxes move as the objects move when the video is displayed. The relevant snippet is below:

dataset.ensure_frames()
sample.frames

The “identity” of boxes are preserved across frames via the index attribute.

Modifying sample fields in batches

Community Slack member Clement asked:

I am looking for a clever way to modify a batch of sample fields all at once, based on their Ids. Specifically, I want to modify a particular bounding box in a frame. What’s the most best way to go about this in FiftyOne?

You'll want to take a look at the “Setting Values” section of the Docs. Using dataset.values() and dataset.set_values()is the recommended way to retrieve and set up large amounts of field elements at a single time across your dataset.

For example, getting a dictionary mapping sample id to field value for each sample can be done with:

d = zip(*dataset.values(["id", "my_field"]))

Then check out set_values() in the Docs for how to add new fields or update existing fields similarly.

Rendering Alpha Matts

Community Slack member Shuo asked:

I have prepared an ImageSegmentationDirectory and successfully imported it, but alpha matts are rendered as binary masks. See the screenshot below. How can I achieve the desired results?

Use the Heatmap label. The Segmentation label will be rendered as a binary mask while the Heatmap label will render an alpha matt. Once you have the heatmap in FiftyOne, try playing around with the opacity of labels in the FiftyOne App!

Clipping Video Samples

Community Slack member Daniel asked:

I have a video sample with 18,000 frames and I want to copy the first 9,000 to another sample. How can I accomplish this?

If you are working with FFMPEG and want to add frames or modify the actual video files, It may be easy to retrieve the  file names and do the editing programmatically with FiftyOne. Check out the “Clips Views” section of the Docs. 

You can use to_clips() to create views into your video datasets that contain one sample per clip defined by a specific field or expression in a video collection.

For example, if you have temporal detection labels on your dataset, then you can create a clips view that contains one sample per temporal segment by simply passing the name of the temporal detection field to to_clips().