Astrodevil

Posted on Jan 13

How to Implement Frame.io style Comments in Your Video Player App📹

#webdev #javascript #programming #tutorial

When working with media, precise frame or time-based feedback is crucial for streamlining the production workflow. Timestamped comments enable pinpoint accuracy, eliminating ambiguity. For instance, instead of vague notes like "fix the part with the guy in the yellow hat," collaborators can specify, "At 01:23, the audio is too loud."

These timestamps are automatically attached to comments, providing valuable context without requiring manual input from users.

However, building an automatic timestamped commenting feature in a video application can be challenging. It typically involves setting up a backend system to store and manage comments, creating API endpoints for real-time communication, implementing logic to add timestamps to comments, and designing a user interface for displaying and posting comments.

In this tutorial, we'll demonstrate how to use Velt SDK to implement Frame.io style comments in a video application. By integrating Velt, you'll be able to add interactive, real-time comments with timestamps.

Challenges of Building Frame.io Style Comments from Scratch

Implementing Frame.io-style commenting involves several technical challenges:

Timestamped Commenting: Enabling users to click on a comment and navigate directly to the corresponding point in the video requires precise synchronization between the comment system and the video player.
Real-Time Collaboration: Ensuring that comments appear in real-time for all users viewing the video necessitates a robust real-time communication infrastructure.
User Interface Complexity: Developing an intuitive and responsive UI for displaying and interacting with comments, including overlays and drag-and-drop functionality, adds to the development effort.

Traditionally, developers had to write extensive code to handle these interactions, manage performance, and address UI challenges.

Fortunately, tools like Velt simplify this process by providing pre-built components and APIs that handle the complexities of real-time collaboration and commenting.

What is Velt?

Velt is a powerful tool that offers pre-built components and APIs via it’s SDK, allowing developers to integrate real-time collaborative features into their applications. It removes the need to build a collaborative commenting system from the ground up.

Instead of writing thousands of lines of custom logic, Velt’s components handle the core behavior for you. By simply importing and positioning these components, you can add collaborative commenting in minutes. They’re also customizable, so you can tailor them to match your product’s design and workflow.

With Velt, you can build features like Frame.io-style video commenting, where users annotate specific segments of a video and attach time-based comments. This makes it easy for editors or reviewers to pinpoint exact frames and understand the context of the feedback. Clicking on a comment navigates directly to the mentioned timestamp, helping teams move faster and avoid confusion.

Building the Video Application

In this segment, we demonstrate an implementation of Frame.io-style commenting using the Velt SDK within a video application. This Next.js app allows users to comment on videos with timestamps. The code for this application is already available on GitHub, so we won’t have to build it from scratch.

Upon completion, our video app will have the following features:

Comments with automatic timestamps
Real-time appearance of comments

Prerequisites

To follow along with the tutorial, you need the following:

A Velt account - create a Velt account
Knowledge of Next.js and TypeScript
Node.js and NPM installed

Before you start coding, you'll need to configure your Velt environment.

Get Velt API key

The next step is to safelist your domains. However, we will only add a local host since this is a demo application. Domains can be safelisted through the Velt console by adding the app deployment URL to the "Allowed Domains" section. This step is crucial while setting up your application with Velt and must be completed before going live with Velt features.

Project Setup and Installations

To get started quickly, clone the project I already built and install the necessary packages for the application:

Clone the project from GitHub:

git clone https://github.com/Studio1HQ/frame.io-style-video-app.git && cd frame.io-style-video-app

Install the necessary dependencies, i.e, Velt for timestamped commenting and Video-React, a web video player for React applications. To do so, open the project in your code editor and run the following command:

npm install

In your code editor, create a .env.local file and paste your API key in the following format:

NEXT_PUBLIC_VELT_API_KEY=your-api-key

Setup and Configure Velt

Wrap your app: In the app folder, in the page.tsx file, we use <VeltProvider apikey= "Your_API_KEY"> to wrap all the components. The VeltProvider is a component in the Velt React SDK that wraps your application and provides the necessary context for Velt’s functionality.
Identify user: To identify users, we call useIdentify(user) within a VeltProvider child, which is the AuthComponent.tsx file.
Set document ID: In VeltDocument.tsx, use useSetDocumentId("my-document-id") (from @veltdev/react).Wrap the App with VeltProvider

Wrap your App

Navigate to the root page.tsx file and in the app function, wrap its content with VeltProvider and pass your API key as props.

"use client";

import { VeltProvider } from "@veltdev/react";
import VideoComponent from "../components/VideoComponent";

export default function App() {
  if (!process.env.NEXT_PUBLIC_VELT_API_KEY) {
    throw new Error("VELT_API_KEY environment variable is required");
  }

  return (
    <VeltProvider apiKey={process.env.NEXT_PUBLIC_VELT_API_KEY!}>
      <VideoComponent />
    </VeltProvider>
  );
}

Identify Users

This AuthComponent is responsible for identifying the current user to the Velt SDK.

//Make sure this is a child component to VeltProvider and not within the same file where VeltProvider is placed.

"use client";
import { useIdentify } from "@veltdev/react";
import { useState, useRef, useEffect } from "react"; // Import useRef and useEffect
import Image from 'next/image';

export default function AuthComponent() {
  // Define multiple user profiles
  const userProfiles = {
    johnDoe: {
      uid: "123",
      organizationId: "organizationId123",
      displayName: "John Doe",
      email: "johndoe@gmail.com",
      photoURL: "/video/profile-pic.jpeg", // Make sure this path is correct or accessible
      color: "#008000",
      textColor: "#FFFFFF",
    },
    janeSmith: {
      uid: "456",
      organizationId: "organizationId123",
      displayName: "Jane Smith",
      email: "janesmith@example.com",
      photoURL: "/video/profile-pic-jane.jpg", // Ensure you have this image or change to a valid path
      color: "#FF5733",
      textColor: "#FFFFFF",
    },
    // Add more user profiles here if needed
  };

  // State to manage which user is currently "logged in"
  const [currentUserKey, setCurrentUserKey] = useState<'johnDoe' | 'janeSmith'>('johnDoe');
  // State to manage dropdown visibility
  const [isDropdownOpen, setIsDropdownOpen] = useState(false);

  // Ref for the dropdown container to detect clicks outside
  const dropdownRef = useRef<HTMLDivElement>(null);

  const yourAuthenticatedUser = userProfiles[currentUserKey];

  // 2) Fetch the relevant User info from yourAuthenticatedUser
  const {
    uid,
    displayName,
    email,
    photoURL,
    organizationId,
    color,
    textColor,
  } = yourAuthenticatedUser;

  // Create the Velt user object
  const veltUser = {
    userId: uid,
    organizationId: organizationId,
    name: displayName,
    email: email,
    photoUrl: photoURL,
    color: color,
    textColor: textColor,
  };

  //3) Pass the user object to the SDK
  useIdentify(veltUser);

  // Function to switch between users
  const switchUser = (key: 'johnDoe' | 'janeSmith') => {
    setCurrentUserKey(key);
    setIsDropdownOpen(false); // Close dropdown after selection
  };

  // Close dropdown when clicking outside
  useEffect(() => {
    const handleClickOutside = (event: MouseEvent) => {
      if (dropdownRef.current && !dropdownRef.current.contains(event.target as Node)) {
        setIsDropdownOpen(false);
      }
    };

    document.addEventListener("mousedown", handleClickOutside);
    return () => {
      document.removeEventListener("mousedown", handleClickOutside);
    };
  }, []);

  return (
    <div style={{ position: 'relative', display: 'flex', alignItems: 'center' }} ref={dropdownRef}>
      {/* Clickable user display */}
      <div
        onClick={() => setIsDropdownOpen(!isDropdownOpen)}
        style={{
          display: "flex",
          alignItems: "center",
          gap: "10px",
          cursor: "pointer", // Add cursor pointer
          padding: "5px", // Add some padding for better click area
          borderRadius: "5px",
          // You can add hover effects here if you want
        }}
      >
        <span>{yourAuthenticatedUser?.displayName}</span>
        <Image
          src={yourAuthenticatedUser?.photoURL}
          alt={yourAuthenticatedUser?.displayName}
          width={50}
          height={50}
          style={{
            border: "1px solid black",
            borderRadius: "50%",
            objectFit: "cover",
          }}
        />
      </div>

      {/* Dropdown menu */}
      {isDropdownOpen && (
        <div
          style={{
            position: 'absolute',
            top: 'calc(100% + 10px)', // Position below the user display
            right: 0, // Align to the right
            backgroundColor: 'white',
            border: '1px solid #ccc',
            borderRadius: '5px',
            boxShadow: '0 2px 8px rgba(0,0,0,0.15)',
            zIndex: 1000, // Ensure it's above other content
            minWidth: '180px',
            padding: '5px 0',
          }}
        >
          {Object.entries(userProfiles).map(([key, user]) => (
            <div
              key={key}
              onClick={() => switchUser(key as 'johnDoe' | 'janeSmith')} // Cast key to the union type
              style={{
                display: 'flex',
                alignItems: 'center',
                gap: '10px',
                padding: '8px 15px',
                cursor: 'pointer',
                backgroundColor: currentUserKey === key ? '#f0f0f0' : 'transparent', // Highlight active user
              }}
              onMouseEnter={(e) => (e.currentTarget.style.backgroundColor = '#e0e0e0')}
              onMouseLeave={(e) => (e.currentTarget.style.backgroundColor = currentUserKey === key ? '#f0f0f0' : 'transparent')}
            >
              <Image
                src={user.photoURL}
                alt={user.displayName}
                width={30}
                height={30}
                style={{
                  borderRadius: "50%",
                  objectFit: "cover",
                }}
              />
              <span>{user.displayName}</span>
            </div>
          ))}
        </div>
      )}
    </div>
  );
}

The useIdentify() hook communicates with the Velt SDK to inform it about the currently logged-in user. Once identified, any subsequent actions, such as creating comments, will be associated with this user within the Velt system.

Ordinarily, you’d need to pass in your already authenticated user to useIdentify() hook. However, for this demo, we have hardcoded two users, John and Jane, to save time.

If you need to update user information or change their access, you can recall the **identify* method with the forceReset option set to true. This ensures that any changes to user metadata or permissions are immediately reflected.*

Set document ID

This VeltDocument.tsx component integrates document-level commenting functionality from the Velt SDK.

"use client";
import { useSetDocumentId, VeltCommentTool } from "@veltdev/react";
import { MessageSquare } from "lucide-react";

export default function YourDocument() {
  useSetDocumentId("my-document-id");

  return (
    <div>
      <VeltCommentTool>
        <MessageSquare className="h-5 w-5 cursor-pointer text-amber-50 mr-3" />
      </VeltCommentTool>
    </div>
  );
}

useSetDocumentId('my-document-id'): This hook call sets the unique identifier for the current page or view to 'my-document-id'
<VeltCommentTool/>: Renders the Velt comment tool UI, allowing users to add comments associated with `'my-document-id*'*`

Now that the Velt SDK is configured and your users are identified, let's explore how to integrate Velt Comments into our application.

Add Velt Comments

Now that Velt has been set up, let’s examine the core of the video commenting system. We will discuss the VideoComponent.tsx file, where all commenting and user engagement occur, and analyze the key functions that enable this functionality.

We will split this component into four separate chunks and explain each of the sections in detail:

Import the neccessary Velt components
Synchronizing Video Position and Comment Mode Handling
Synchronizing Comment Clicks from Sidebar/Timeline
Handling Timeline Comment Clicks (onTimelineCommentClick)

Now let's set the stage for exploring the heart of Velt's commenting system. We'll now break down its key functionalities, starting with the essential imports.

Import the necessary Velt components:

import React, { useEffect, useState, useCallback, useRef } from "react";
import {
  VeltCommentsSidebar,
  VeltCommentPlayerTimeline,
  VeltSidebarButton,
  VeltComments,
  useCommentModeState,
  useVeltClient,
} from "@veltdev/react";
import VeltDocument from "./VeltDocument";
import AuthComponent from "./AuthComponent";

VeltCommentsSidebar: Renders the sidebar UI where comments are displayed
VeltCommentPlayerTimeline: A component to show comment bubbles on the video player seek bar
VeltSidebarButton: Renders a button used to toggle the visibility (open/close) of the VeltCommentsSidebar
VeltComments: The main comment component that manages the comment UI. This component accepts an allowedElementQuerySelectors prop, with which you can specify the HTML elements you want the comments on. For example: allowedElementQuerySelectors={["#videoPlayerId"]}. This element id (#videoPlayerId) is used to specify which ****HTML elements on your page Velt's commenting functionality should be active on.
useCommentModeState: A Hook to track the state of comment mode. It returns a boolean value indicating whether comment mode is active or not.
Synchronizing Video Position and Comment Mode Handling:

 const setLocation = useCallback(() => {
    if (videoRef.current && client) {
      const location: VeltLocation = {
        id: secondsToReadableTime(videoRef.current.currentTime),
        locationName: secondsToReadableTime(videoRef.current.currentTime),
        currentMediaPosition: videoRef.current.currentTime,
        videoPlayerId: "videoPlayerId",
      };
      client.setLocation(location);
    }
  }, [client, videoRef]);

  useEffect(() => {
    if (commentModeState && videoRef.current && client) {
      videoRef.current.pause();
      setIsPlaying(false);
      setLocation();
    }
  }, [commentModeState, client, setLocation]);

The setLocation() function updates Velt's location data and synchronizes it with the video's current playback position. It contains a location object, a JSON object representing a specific area or context within the application, containing the video's current time. It then uses client.setLocation() method to send this location data to the Velt SDK.
The secondsToReadableTime function converts seconds to a human-readable time.
The useEffect() hook monitors changes to the commentModeState. When comment mode is active, it pauses the video and calls setLocation(). This ensures that the video's current position is synchronized with Velt when a user enters comment mode.
The locationName property on the location object sets the video's timestamp in the sidebar comments.
Synchronizing Comment Clicks from Sidebar/Timeline

  const handleCommentClick = useCallback(
    (
      event: React.MouseEvent<HTMLElement> & {
        detail?: CommentClickDetail;
      }
    ) => {
      const video = videoRef.current;
      const customDetail = (event as any).detail as
        | CommentClickDetail
        | undefined;

      if (customDetail?.location?.currentMediaPosition !== undefined && video) {
        if (video?.currentTime !== customDetail.location.currentMediaPosition) {
          (video as HTMLVideoElementWithCurrentTime).currentTime =
            customDetail.location.currentMediaPosition;
          setLocation();
          updateCustomTimeline();
        }
      }
    },
    [videoRef, setLocation, updateCustomTimeline]
  );

The handleCommentClick() function responds to clicks on comments within the Velt sidebar. When clicked, it extracts the comment's timestamp and navigates the video to that point.

Here’s how it works:

It reads the time (currentMediaPosition) associated with the clicked comment.
If the video's current time differs from the comment's time, it sets the video's playback to the comment’s time. This synchronizes the video player with Velt's comment locations.
It then calls setLocation() to update the application's record of the current video time.
Finally, it calls updateCustomTimeline() to refresh the video’s custom timeline visual. We have made this custom bar to show the video's current position.
Handling Timeline Comment Clicks (onTimelineCommentClick)

  const onTimelineCommentClick = useCallback(
    (
      event: React.MouseEvent<HTMLElement> & {
        detail?: CommentClickDetail;
        location: VeltLocation;
      }
    ) => {
      if (!!event) {
        // Destructure location from the event detail
        const { location } = event;

        const video =
          videoRef.current as HTMLVideoElementWithCurrentTime | null;
        if (video && location) {
          console.log(location);
          video.pause();
        }

        if (video?.paused) {
          (video as HTMLVideoElementWithCurrentTime).currentTime =
            location.currentMediaPosition;

          if (location && client) {
            client.setLocation(location as VeltLocation);
          }
        }
      }
    },
    [client, videoRef]
  );

The onTimelineCommentClick function is passed as props to the VeltCommentPlayerTimeline, a prebuilt Velt component that renders on the timeline. It's designed to display comment bubbles on top of the video player timeline at the correct positions corresponding to the frames or timestamps where comments were made.
The onTimelineCommentClick function is triggered when a user clicks a comment marker on the video timeline. It retrieves the comment's timestamp from the event data, pauses the video, and seeks the comment's timestamp to reflect the new position.
It uses client?.setLocation() to tell Velt to update its context to the new location, which will then display any comments associated with that specific timestamp on the video.

Woo-hoo! 🎉 Your application is now fully functional!

Live Demo and Preview

To test the application locally, run:

npm run dev

Then open your browser and navigate to http://localhost:3000 to interact with the app.

You can view the live demo here: https://frame-io-style-video-app.vercel.app/

Want to see how it works? Here's a quick preview:

Explore the full source code on GitHub to dive deeper into the implementation.

Conclusion: Making Video Collaboration Easy

In this tutorial, we've demonstrated how to implement Frame.io-style video commenting using the Velt SDK, achieving:

Timestamped Comment: Allowing users to add comments linked to specific moments in the video.
Video Timeline Sync: Ensuring comments correspond accurately with video playback.
Comment Management UI: Providing an intuitive interface for users to interact with comments.

This implementation demonstrates how modern SDKs, such as Velt, can significantly reduce development time while delivering enterprise-level functionality. What traditionally required a whole development team and months of work is now achievable in a single day.

Next Steps:

Explore Advanced Features: Utilize Velt's capabilities such as @mentions, comment threading, and real-time presence indicators to enhance collaboration.
Customize the UI: Tailor the commenting interface to align with your application's design system for a seamless user experience.
Integrate Additional Collaborative Tools: Consider adding features such as screen sharing, live cursors, and in-app notifications to further enhance user interaction.

By leveraging Velt's comprehensive suite of components, you can transform your simple video application into a more dynamic app with better collaborative functionalities.

Get started with Velt SDK today and take your video collaboration to the next level.

DEV Community