How to Implement Picture-in-Picture(PIP) Mode in React Native App for Android?

📌 Introduction

Picture-in-Picture (PiP) is a commonly used feature in video conferencing software, enabling users to simultaneously engage in a video conference and perform other tasks on their devices. With PiP, you can keep the video conference window open, resize it to a smaller size, and continue working on other tasks while still seeing and hearing the other participants in the conference. This feature proves beneficial when you need to take notes, send an email, or look up information during the conference.

PiP mode is very effective in apps that require users to watch a video while engaging with other information or features. For example, in a video conferencing program, users can continue their video call in a tiny window while accessing other elements of the application, such as chat or file sharing. Students on an e-learning platform can view a lecture video while taking notes or accessing other information.

PiP mode is effective because it improves the user experience without interfering with their workflow. PiP mode increases the flexibility and usability of applications by allowing users to multitask and access additional functions while watching a video. This can result in greater user engagement, satisfaction, and, ultimately, success for your application.

In this post, we'll explore how to implement Picture-in-Picture(PIP) into a React Native video calling app. We will guide you through the steps of installing VideoSDK, integrating it into your project, and adding PiP mode to improve the video viewing experience in your application.

🚀 Getting Started with VideoSDK

To take advantage of the chat functionality, we will need to use the capabilities that the VideoSDK offers. Before we dive into the implementation steps, let's make sure you complete the necessary prerequisites.

🔐 Create a VideoSDK Account

Go to your VideoSDK dashboard and sign up if you don't have an account. This account gives you access to the required Video SDK token, which acts as an authentication key that allows your application to interact with VideoSDK functionality.

🔑 Generate your Auth Token

Visit your VideoSDK dashboard and navigate to the "API Key" section to generate your auth token. This token plays a crucial role in authorizing your application to use VideoSDK features.

For a more visual understanding of the account creation and token generation process, consider referring to the provided tutorial.

🛠️ Prerequisites and Setup

Make sure your development environment meets the following requirements:

• Node.js v12+
• NPM v6+ (comes installed with newer Node versions)
• Android Studio or Xcode installed

🔌 Integrate VideoSDK​

Install the VideoSDK by using the following command. Ensure that you are in your project directory before running this command.

• For NPM

npm install "@videosdk.live/react-native-sdk" "@videosdk.live/react-native-incallmanager"

• For Yarn

yarn add "@videosdk.live/react-native-sdk" "@videosdk.live/react-native-incallmanager"

Project Configuration

Setting up your project correctly is the next essential phase in ensuring smooth operation and integration. Setting up permissions, updating required files, and connecting dependencies are all part of project configuration, which helps to make sure your application can make the most out of VideoSDK capabilities.

Android Setup

• Add the required permissions in the AndroidManifest.xml file.

<manifest
  xmlns:android="http://schemas.android.com/apk/res/android"
  package="com.cool.app"
>
    <!-- Give all the required permissions to app -->
    <uses-permission android:name="android.permission.INTERNET" />
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
    <!-- Needed to communicate with already-paired Bluetooth devices. (Legacy up to Android 11) -->
    <uses-permission
        android:name="android.permission.BLUETOOTH"
        android:maxSdkVersion="30" />
    <uses-permission
        android:name="android.permission.BLUETOOTH_ADMIN"
        android:maxSdkVersion="30" />

    <!-- Needed to communicate with already-paired Bluetooth devices. (Android 12 upwards)-->
    <uses-permission android:name="android.permission.BLUETOOTH_CONNECT" />

    <uses-permission android:name="android.permission.CAMERA" />
    <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
    <uses-permission android:name="android.permission.RECORD_AUDIO" />
    <uses-permission android:name="android.permission.SYSTEM_ALERT_WINDOW" />
    <uses-permission android:name="android.permission.FOREGROUND_SERVICE"/>
    <uses-permission android:name="android.permission.WAKE_LOCK" />

    <application>
   <meta-data
      android:name="live.videosdk.rnfgservice.notification_channel_name"
      android:value="Meeting Notification"
     />
    <meta-data
    android:name="live.videosdk.rnfgservice.notification_channel_description"
    android:value="Whenever meeting started notification will appear."
    />
    <meta-data
    android:name="live.videosdk.rnfgservice.notification_color"
    android:resource="@color/red"
    />
    <service android:name="live.videosdk.rnfgservice.ForegroundService" android:foregroundServiceType="mediaProjection"></service>
    <service android:name="live.videosdk.rnfgservice.ForegroundServiceTask"></service>
  </application>
</manifest>

• Update your colors.xml file for internal dependencies.

<resources>
  <item name="red" type="color">
    #FC0303
  </item>
  <integer-array name="androidcolors">
    <item>@color/red</item>
  </integer-array>
</resources>

• Link the necessary VideoSDK Dependencies.

  dependencies {
   implementation project(':rnwebrtc')
   implementation project(':rnfgservice')
  }

include ':rnwebrtc'
project(':rnwebrtc').projectDir = new File(rootProject.projectDir, '../node_modules/@videosdk.live/react-native-webrtc/android')

include ':rnfgservice'
project(':rnfgservice').projectDir = new File(rootProject.projectDir, '../node_modules/@videosdk.live/react-native-foreground-service/android')

import live.videosdk.rnwebrtc.WebRTCModulePackage;
import live.videosdk.rnfgservice.ForegroundServicePackage;

public class MainApplication extends Application implements ReactApplication {
  private static List<ReactPackage> getPackages() {
      @SuppressWarnings("UnnecessaryLocalVariable")
      List<ReactPackage> packages = new PackageList(this).getPackages();
      // Packages that cannot be autolinked yet can be added manually here, for example:

      packages.add(new ForegroundServicePackage());
      packages.add(new WebRTCModulePackage());

      return packages;
  }
}

/* This one fixes a weird WebRTC runtime problem on some devices. */
android.enableDexingArtifactTransform.desugaring=false

• Include the following line in your proguard-rules.pro file (optional: if you are using Proguard)

-keep class org.webrtc.** { *; }

• In your build.gradle file, update the minimum OS/SDK version to 23.

buildscript {
  ext {
      minSdkVersion = 23
  }
}

Register Service

Register VideoSDK services in your root index.js file for the initialization service.

import { AppRegistry } from "react-native";
import App from "./App";
import { name as appName } from "./app.json";
import { register } from "@videosdk.live/react-native-sdk";

register();

AppRegistry.registerComponent(appName, () => App);

Essential Steps to Add Video

By following essential steps, developers can seamlessly incorporate PiP mode into their applications, enriching user experiences.

Step 1: Get started with api.js​

Before moving on, you must create an API request to generate a unique meetingId. You will need an authentication token, which you can create either through the videosdk-rtc-api-server-examples or directly from the VideoSDK Dashboard for developers.

export const token = "<Generated-from-dashbaord>";
// API call to create meeting
export const createMeeting = async ({ token }) => {
  const res = await fetch(`https://api.videosdk.live/v2/rooms`, {
    method: "POST",
    headers: {
      authorization: `${token}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({}),
  });

  const { roomId } = await res.json();
  return roomId;
};

Step 2: Wireframe App.js with all the components​

To build up a wireframe of App.js, you need to use VideoSDK Hooks and Context Providers. VideoSDK provides MeetingProvider, MeetingConsumer, useMeeting, and useParticipant hooks.

First, you need to understand the Context Provider and Consumer. Context is primarily used when some data needs to be accessible by many components at different nesting levels.

• MeetingProvider : This is the Context Provider. It accepts value config and token as props. The Provider component accepts a value prop to be passed to consuming components that are descendants of this Provider. One Provider can be connected to many consumers. Providers can be nested to override values deeper within the tree.
• MeetingConsumer : This is the Context Consumer. All consumers that are descendants of a Provider will re-render whenever the Provider’s value prop changes.
• useMeeting : This is the meeting hook API. It includes all the information related to meeting such as join, leave, enable/disable mic or webcam etc.
• useParticipant : This is the participant hook API. It is responsible for handling all the events and props related to one particular participant such as name, webcamStream, micStream etc.

The Meeting Context provides a way to listen for any changes that occur when a participant joins the meeting or makes modifications to their microphone, camera, and other settings.

Begin by making a few changes to the code in the App.js file.

import React, { useState } from "react";
import {
  SafeAreaView,
  TouchableOpacity,
  Text,
  TextInput,
  View,
  FlatList,
} from "react-native";
import {
  MeetingProvider,
  useMeeting,
  useParticipant,
  MediaStream,
  RTCView,
} from "@videosdk.live/react-native-sdk";
import { createMeeting, token } from "./api";

function JoinScreen(props) {
  return null;
}

function ControlsContainer() {
  return null;
}

function MeetingView() {
  return null;
}

export default function App() {
  const [meetingId, setMeetingId] = useState(null);

  const getMeetingId = async (id) => {
    const meetingId = id == null ? await createMeeting({ token }) : id;
    setMeetingId(meetingId);
  };

  return meetingId ? (
    <SafeAreaView style={{ flex: 1, backgroundColor: "#F6F6FF" }}>
      <MeetingProvider
        config={{
          meetingId,
          micEnabled: false,
          webcamEnabled: true,
          name: "Test User",
        }}
        token={token}
      >
        <MeetingView />
      </MeetingProvider>
    </SafeAreaView>
  ) : (
    <JoinScreen getMeetingId={getMeetingId} />
  );
}

Step 3: Implement Join Screen​

The join screen will serve as a medium to either schedule a new meeting or join an existing one.

function JoinScreen(props) {
  const [meetingVal, setMeetingVal] = useState("");
  return (
    <SafeAreaView
      style={{
        flex: 1,
        backgroundColor: "#F6F6FF",
        justifyContent: "center",
        paddingHorizontal: 6 * 10,
      }}
    >
      <TouchableOpacity
        onPress={() => {
          props.getMeetingId();
        }}
        style={{ backgroundColor: "#1178F8", padding: 12, borderRadius: 6 }}
      >
        <Text style={{ color: "white", alignSelf: "center", fontSize: 18 }}>
          Create Meeting
        </Text>
      </TouchableOpacity>

      <Text
        style={{
          alignSelf: "center",
          fontSize: 22,
          marginVertical: 16,
          fontStyle: "italic",
          color: "grey",
        }}
      >
        ---------- OR ----------
      </Text>
      <TextInput
        value={meetingVal}
        onChangeText={setMeetingVal}
        placeholder={"XXXX-XXXX-XXXX"}
        style={{
          padding: 12,
          borderWidth: 1,
          borderRadius: 6,
          fontStyle: "italic",
        }}
      />
      <TouchableOpacity
        style={{
          backgroundColor: "#1178F8",
          padding: 12,
          marginTop: 14,
          borderRadius: 6,
        }}
        onPress={() => {
          props.getMeetingId(meetingVal);
        }}
      >
        <Text style={{ color: "white", alignSelf: "center", fontSize: 18 }}>
          Join Meeting
        </Text>
      </TouchableOpacity>
    </SafeAreaView>
  );
}

Step 4: Implement Controls​

The next step is to create a ControlsContainer component to manage features such as Join or leave a Meeting and Enable or Disable the Webcam/Mic.

In this step, the useMeeting hook is utilized to acquire all the required methods such as join(), leave(), toggleWebcam and toggleMic.

const Button = ({ onPress, buttonText, backgroundColor }) => {
  return (
    <TouchableOpacity
      onPress={onPress}
      style={{
        backgroundColor: backgroundColor,
        justifyContent: "center",
        alignItems: "center",
        padding: 12,
        borderRadius: 4,
      }}
    >
      <Text style={{ color: "white", fontSize: 12 }}>{buttonText}</Text>
    </TouchableOpacity>
  );
};

function ControlsContainer({ join, leave, toggleWebcam, toggleMic }) {
  return (
    <View
      style={{
        padding: 24,
        flexDirection: "row",
        justifyContent: "space-between",
      }}
    >
      <Button
        onPress={() => {
          join();
        }}
        buttonText={"Join"}
        backgroundColor={"#1178F8"}
      />
      <Button
        onPress={() => {
          toggleWebcam();
        }}
        buttonText={"Toggle Webcam"}
        backgroundColor={"#1178F8"}
      />
      <Button
        onPress={() => {
          toggleMic();
        }}
        buttonText={"Toggle Mic"}
        backgroundColor={"#1178F8"}
      />
      <Button
        onPress={() => {
          leave();
        }}
        buttonText={"Leave"}
        backgroundColor={"#FF0000"}
      />
    </View>
  );
}

function ParticipantList() {
  return null;
}
function MeetingView() {
  const { join, leave, toggleWebcam, toggleMic, meetingId } = useMeeting({});

  return (
    <View style={{ flex: 1 }}>
      {meetingId ? (
        <Text style={{ fontSize: 18, padding: 12 }}>
          Meeting Id :{meetingId}
        </Text>
      ) : null}
      <ParticipantList />
      <ControlsContainer
        join={join}
        leave={leave}
        toggleWebcam={toggleWebcam}
        toggleMic={toggleMic}
      />
    </View>
  );
}

Step 5: Render Participant List​

After implementing the controls, the next step is to render the joined participants.

You can get all the joined participants from the useMeeting Hook.

function ParticipantView() {
  return null;
}

function ParticipantList({ participants }) {
  return participants.length > 0 ? (
    <FlatList
      data={participants}
      renderItem={({ item }) => {
        return <ParticipantView participantId={item} />;
      }}
    />
  ) : (
    <View
      style={{
        flex: 1,
        backgroundColor: "#F6F6FF",
        justifyContent: "center",
        alignItems: "center",
      }}
    >
      <Text style={{ fontSize: 20 }}>Press Join button to enter meeting.</Text>
    </View>
  );
}

function MeetingView() {
  // Get `participants` from useMeeting Hook
  const { join, leave, toggleWebcam, toggleMic, participants } = useMeeting({});
  const participantsArrId = [...participants.keys()];

  return (
    <View style={{ flex: 1 }}>
      <ParticipantList participants={participantsArrId} />
      <ControlsContainer
        join={join}
        leave={leave}
        toggleWebcam={toggleWebcam}
        toggleMic={toggleMic}
      />
    </View>
  );
}

Step 6: Handling Participant's Media​

Before Handling the Participant's Media, you need to understand a couple of concepts.

1. useParticipant Hook

The useParticipant hook is responsible for handling all the properties and events of one particular participant who joined the meeting. It will take participantId as argument.

const { webcamStream, webcamOn, displayName } = useParticipant(participantId);

2. MediaStream API

The MediaStream API is beneficial for adding a MediaTrack into the RTCView component, enabling the playback of audio or video.

<RTCView
  streamURL={new MediaStream([webcamStream.track]).toURL()}
  objectFit={"cover"}
  style={{
    height: 300,
    marginVertical: 8,
    marginHorizontal: 8,
  }}
/>

Rendering Participant Media

function ParticipantView({ participantId }) {
  const { webcamStream, webcamOn } = useParticipant(participantId);

  return webcamOn && webcamStream ? (
    <RTCView
      streamURL={new MediaStream([webcamStream.track]).toURL()}
      objectFit={"cover"}
      style={{
        height: 300,
        marginVertical: 8,
        marginHorizontal: 8,
      }}
    />
  ) : (
    <View
      style={{
        backgroundColor: "grey",
        height: 300,
        justifyContent: "center",
        alignItems: "center",
      }}
    >
      <Text style={{ fontSize: 16 }}>NO MEDIA</Text>
    </View>
  );
}

Congratulations! By following these steps, you're on your way to unlocking the video within your application. Now, we are moving forward to integrate the feature that builds immersive video experiences for your users!

Integrate Picture-in-picture (PiP) Feature

Picture-in-picture (PiP) is a commonly used feature in video conferencing software, enabling users to simultaneously engage in a video conference and perform other tasks on their devices. With PiP, you can keep the video conference window open, resize it to a smaller size, and continue working on other tasks while still seeing and hearing the other participants in the conference. This feature proves beneficial when you need to take notes, send an email, or look up information during the conference.

This guide explains the steps to implement the Picture-in-Picture feature using VideoSDK.

Step 1: Install Package​

To begin with, you need to install a third-party package react-native-pip-android to achieve PiP mode in Android.

npm install react-native-pip-android

Step 2: Setup​

Include the following attribute in /android/app/src/main/AndroidManifest.xml file.

  <activity
    ...
      android:supportsPictureInPicture="true"
      android:configChanges=
        "screenSize|smallestScreenSize|screenLayout|orientation"
        ...

Step 3: Import Activity in MainActivity.java file​

...
import com.reactnativepipandroid.PipAndroidModule;

public class MainActivity extends ReactActivity {

...

@Override
  public void onPictureInPictureModeChanged (boolean isInPictureInPictureMode) {
    PipAndroidModule.pipModeChanged(isInPictureInPictureMode);
  }

Step 4: Setup PiP for Rendering participant media​

To set up Picture-in-Picture (PiP) mode for rendering participant media, you can utilize the usePipModeListener hook to control the rendering. The example below demonstrates how to render a participant list.

import PipHandler, { usePipModeListener } from "react-native-pip-android";

function ParticipantView({ participantId, inPipMode }) {
  const { webcamStream, webcamOn } = useParticipant(participantId);

  return webcamOn && webcamStream ? (
    <RTCView
      streamURL={new MediaStream([webcamStream.track]).toURL()}
      objectFit={"cover"}
      style={{
        height: inPipMode ? 75 : 300,
        marginVertical: 8,
        marginHorizontal: 8,
      }}
    />
  ) : null;
}

function MeetingView() {
  // Get `participants` from useMeeting Hook
  const { participants } = useMeeting({});

  const inPipMode = usePipModeListener();

  // Use this boolean to show / hide ui when pip mode changes
  if (inPipMode) {
    // Render the participant in PiP Box

    return [...participants.keys()].map((participantId, index) => (
      <ParticipantView
        key={index}
        participantId={participantId}
        inPipMode={true}
      />
    ));
  }

  return (
    <View style={{ flex: 1 }}>
      // ..
      // ..
    </View>
  );
}

Step 5: Enter in PiP mode​

To enter Picture-in-Picture (PiP) mode, use the enterPipMode method of PipHandler. Provide the PiP box's height and width as parameters to this method.

function ControlsContainer() {
  return (
    <View
      style={{
        padding: 24,
        flexDirection: "row",
        justifyContent: "space-evenly",
      }}
    >
      <Button
        onPress={() => {
          PipHandler.enterPipMode(300, 500);
        }}
        buttonText={"PiP"}
        backgroundColor={"blue"}
      />
    </View>
  );
}

By following these essential steps, you can successfully integrate VideoSDK into your application and enhance it with the Picture-in-Picture (PiP) feature.

Conclusion

Congratulations You’ve now learned how to implement Picture-in-Picture mode in a React Native Android app! By following these systematic guidelines, developers can unlock the full potential of video communication with the Picture-in-Picture (PiP) feature within their applications. This feature can offer a great user experience and enable users to multitask more effectively. VideoSDK provides a powerful set of tools and APIs that make it easier than ever to create immersive video experiences for your users. Whether you're building a video conferencing app, a live streaming service, or a virtual event platform, VideoSDK has everything you need to succeed.