DEV Community: Dyte

Adding Live Video Calling to Tic-Tac-Toe With Flutter and Dyte

Govind Maheshwari — Tue, 09 Jan 2024 19:32:30 +0000

Introduction

Welcome to an exciting journey into the world of interactive gaming! In this blog post, we'll explore how to develop a captivating multiplayer Tic-Tac-Toe game using Flutter, a powerful and versatile framework for building cross-platform applications. But that's not all — we'll take it a step further by integrating the Dyte video SDK, allowing users to connect with their friends and engage in real-time battles of strategy and wit.

Engage in the timeless classic of Tic-Tac-Toe, reimagined with a modern twist. With Flutter's flexibility and Dyte's seamless video calling capabilities, you can create an immersive gaming experience that brings people together, no matter the distance. So, get ready to delve into the exciting world of game development as we guide you through building a multiplayer Tic-Tac-Toe game with the added thrill of live video communication.

Throughout this tutorial, we'll provide step-by-step instructions, code snippets, and valuable insights to empower you in crafting your Flutter application. Whether you're a seasoned developer or just starting your programming journey, this guide will equip you with the knowledge and tools to create an engaging multiplayer game that captivates and entertains your users.

So, let's dive in and discover how you can leverage the power of Flutter and the Dyte video SDK to breathe new life into the classic game of Tic-Tac-Toe, bringing joy and connection to players worldwide.

Prerequisites

You'll need a few key elements to start creating your multiplayer Tic-Tac-Toe game using the Dyte video SDK in Flutter. Firstly, make sure you have Flutter installed on your system. You can follow your operating system's official Flutter installation guide to set it up successfully.

To help you hit the ground running, we've prepared an initial project setup with a pre-built UI for the Tic-Tac-Toe game. You can access the project code and user interface files here. This initial setup provides a solid foundation to build upon, saving you time and effort in creating the game's basic structure.

Now visit the official Dyte website at dyte.io and navigate to the signup page to create an account and explore the various features and functionalities offered by Dyte.

Next, you must incorporate the Dyte video SDK into your Flutter project. The Dyte SDK allows seamless integration of video calling functionality, enabling real-time communication between players.

Using the following command, you can add the dyte_core plugin in the pubspec.yaml file.

flutter pub add dyte_core

We are going to use Getx for state management. GetX is a powerful state management solution for Flutter that offers simplicity, performance, and scalability. With GetX, you can easily manage the state of your Tic-Tac-Toe game and handle navigation, dependencies, and more.

In the initial screen, you will find three files.

The first one is the main.dart file, which contains the landing page.
The second file is game_screen.dart — this will hold the UI part for our Tic-Tac-Toe game.
The third file is room_state.dart. This dart file holds all the logic for dyte video calling and the Tic-Tac-Toe game state.

Open project is terminal and runs the flutter pub get command to load all the required plugins.

Currently, the game is not in a proper working state, but we will make it happen by the end of this blog.

To proceed, let's open the room_state.dart file. Inside this file, you will find a set of code snippets.

Initially, you will come across variable initializations accompanied by comments that provide insights into their purpose and usage.

To receive updates from the Dyte video SDK, we need to add some listeners to the RoomStateNotifier class. This can be accomplished by implementing the following code:

class RoomStateNotifier extends GetxController
implements
DyteMeetingRoomEventsListener,
DyteParticipantEventsListener,
DyteChatEventsListener {
...

By registering these listeners, we ensure that our application is notified of any changes or updates in the room state from the Dyte video SDK.

Upon adding the listener to the RoomStateNotifier class, you may encounter errors indicating that specific override methods are missing. To resolve these errors, you can utilize the quick-fix functionality provided by your IDE.

Initiating the Dyte meeting

Dyte video SDK offers override methods that allow us to receive updates on our call. In this application, we will be utilizing the following override methods:

onMeetingInitCompleted(): This method is called when the meeting initialization is completed, providing us with relevant information about the meeting.
onMeetingRoomJoinCompleted(): When the user successfully joins a meeting room, this method is invoked, allowing us to handle any necessary actions or updates.
onMeetingRoomLeaveCompleted(): Invoked when the user leaves the meeting room. This method enables us to perform any required cleanup or follow-up tasks.
onParticipantJoin(DyteJoinedMeetingParticipant participant): This method is called when a new participant joins the meeting, providing information about the joined participant.
onParticipantLeave(DyteJoinedMeetingParticipant participant): When a participant leaves the meeting, this method is triggered, supplying details about the departing participant.
onNewChatMessage(DyteChatMessage message): Whenever a new chat message is received, this method is invoked, allowing us to process and handle the message accordingly.

First, attach the listener to the RoomStateNotifier class to start receiving updates from the Dyte video SDK. Furthermore, we initiate the video call by invoking the init(meetingInfo) method provided by the dyteClient instance. This action triggers the video call, enabling users to join the meeting and engage in real-time communication with other participants. The dyteClient variable is of type DyteMobileClient, which serves as a manager for interacting with a Dyte server.

Here is the code snippet for adding the listener and initiating a Dyte meeting.

RoomStateNotifier(String name) {
username = name;
dyteClient.value.addMeetingRoomEventsListener(this);
dyteClient.value.addParticipantEventsListener(this);
dyteClient.value.addChatEventsListener(this);
final meetingInfo = DyteMeetingInfoV2(
authToken:'---TOKEN HERE—--',
enableAudio: false,
enableVideo: true);
dyteClient.value.init(meetingInfo);
}

Joining the Dyte meeting

Once the Dyte meeting is successfully initialized, a callback will be triggered in the onMeetingInitCompleted() function. This callback indicates that the meeting has been set up correctly and is ready for joining the room.

The following code needs to be implemented within the onMeetingInitCompleted() function to facilitate joining the call and configuring the username within the room.

@override
void onMeetingInitCompleted() {
dyteClient.value.localUser.setDisplayName(username);
dyteClient.value.joinRoom();
}

The onMeetingRoomJoinCompleted() callback function is triggered when the room join process is completed. In this callback, we set the roomJoin variable to true, indicating that the local user has successfully joined the room.

@override
void onMeetingRoomJoinCompleted() {
roomJoin.value = true;
}

Similarly, in the onMeetingRoomLeaveCompleted() callback, we handle the removal of listeners from this class. This ensures that the necessary cleanup tasks are performed when leaving the Dyte meeting, preventing any potential memory leaks or unwanted behaviors. By implementing this callback, we can properly manage the lifecycle of the class and maintain a streamlined and efficient application flow.

@override
void onMeetingRoomLeaveCompleted() {
roomJoin.value = false;
dyteClient.value.removeMeetingRoomEventsListener(this);
dyteClient.value.removeParticipantEventsListener(this);
dyteClient.value.removeChatEventsListener(this);
Get.delete<RoomStateNotifier>();
Get.back();
}

When an opponent joins the room, the onParticipantJoin() function is triggered, providing us with a callback. At this point, we can proceed to initiate the game. However, before starting, it is necessary to assign symbols for local and remote peers. We will call the existing assignSymbol() function available within the same file to accomplish this. By invoking this function, we can ensure that each player is assigned their respective symbols, setting the stage for an engaging and fair gameplay experience.

@override
void onParticipantJoin(DyteJoinedMeetingParticipant participant) {
   if (participant.userId != dyteClient.value.localUser.userId) { //not a local user
       remotePeer.value = participant;
       assignSymbol();
   }
}

We use their unique user IDs to assign local and remote peers symbols. We can ensure consistent symbol assignment across the room by sorting these IDs. In this implementation, we designate the character "O" to the user at the 0 index and "X" to the other peer. As user IDs are unique and consistent within the room, this approach guarantees that each player receives their designated symbol, enhancing clarity and fairness during gameplay.

Leaving the Dyte meeting

If a remote peer leaves the room during an ongoing game, we must ensure that the local peer also leaves to maintain a synchronized and consistent experience. To handle this scenario, we will utilize the onParticipantLeave override function.

When a remote peer leaves the room, we will display a dialog to the user, providing them with a clear choice to leave the room. The dialog will have the barrierDismissible parameter set to false, meaning the user cannot dismiss the dialog by tapping outside.

Once the user clicks the "Leave" button within the dialog, a method will be invoked to trigger the local peer's departure from the room. This ensures that both participants gracefully exit the room, maintaining the integrity of the game session.

By implementing this flow, we provide a seamless and intuitive experience for users, allowing them to easily handle situations where a remote peer leaves the game unexpectedly.

dyteClient.value.leaveRoom();

Here is the code for onParticipantLeave:

@override
void onParticipantLeave(DyteJoinedMeetingParticipant participant) {
  if (participant.userId != dyteClient.value.localUser.userId &&
    remotePeer.value != null) {
      remotePeer.value = null;
      Get.defaultDialog(
        barrierDismissible: false,
        onWillPop: () async {
          return false;
        },
        title: "Opponent Left this game.",
        textConfirm: "Leave",
        middleText: "",
        confirmTextColor: Colors.white,
        onConfirm: () {
          dyteClient.value.leaveRoom();
          Get.back();
        });
      }
}

Using Dyte’s chat component

To facilitate real-time updates of the game whenever the user takes their turn, we will leverage the chat feature provided by the Dyte video SDK. In this approach, we will send the displayExOh array as text through the chat functionality, ensuring it is transmitted to the remote peer's device. Utilizing the Dyte SDK's chat feature, we establish a seamless communication channel between the two peers, enabling the synchronization of game progress and retrieving the updated displayExOh array on the remote peer's side. This mechanism ensures a consistent and interactive gaming experience, allowing both players to sync throughout the game.

sendMessage() {
dyteClient.value.chat.sendTextMessage(displayExOh.toString());
}

Lastly, we make use of the onNewChatMessage override function. This function lets us receive the displayExOh array in text format through the chat functionality. We then convert this text to the original array format, enabling us to update the user interface (UI) accordingly. By implementing this functionality, we ensure that any updates made by the remote peer are received and reflected in real-time on the local user's UI.

@override
void onNewChatMessage(DyteChatMessage message) {
if (message.userId == dyteClient.value.localUser.userId) {
     return;
}
DyteTextMessage textMessage = message as DyteTextMessage;
List<String> temp = textMessage.message
.replaceFirst("[", "")
.replaceFirst("]", "")
.split(", ");
displayExOh.value = temp;
localUserTurn.toggle();
boxCount.value = displayExOh.where((p0) => p0!="").length;
checkWinner();
}

Managing audio/video of participants

We will be utilizing two additional functions, toggleVideo() and toggleAudio(), to manage the local peer's video and audio within the room. These functions allow us to dynamically control the starting and stopping of the local peer's video and audio streams during the video call session. By invoking toggleVideo(), we can initiate or pause the local peer's video transmission, while toggleAudio() enables us to start or mute the local peer's audio feed. These functionalities provide flexibility and control to the user, allowing them to manage their video and audio presence according to their preferences and requirements within the room.

toogleVideo() {
     if (isVideoOn.value) {
 dyteClient.value.localUser.disableVideo();
     } else {
       dyteClient.value.localUser.enableVideo();
     }
       isVideoOn.toggle();
}
toogleAudio() {
     if (isAudioOn.value) {
        dyteClient.value.localUser.disableAudio();
     } else {
        dyteClient.value.localUser.enableAudio();
     }
        isAudioOn.toggle();
}

Setting up the Tic-Tac-Toe game

Now, let's navigate to the gameScreen.dart file. This file contains the UI code for our game and the video call controls UI. However, to display the video tiles correctly, we need to add some additional code. This code will handle the rendering and layout of the video tiles, ensuring they are displayed correctly on the game screen. By incorporating this code, we can seamlessly integrate the game UI and the video call interface, providing users with a comprehensive and immersive experience.

Let's search for the comment "// Dyte Meeting Video View" in the gameScreen.dart file. Within the corresponding section, we will proceed to check whether the user has joined the Dyte meeting or not by using the following code:

if (roomStateNotifier.roomJoin.value)

By implementing this code snippet, we can differentiate between the scenarios where the user has successfully joined the Dyte meeting and where they have not. This conditional logic allows us to handle each situation appropriately, ensuring a smooth and coherent user experience throughout the game.

If the roomJoin variable is true, it indicates that the user has successfully joined the Dyte meeting. To render the user's video tile, we can utilize the VideoView widget provided by the Dyte video SDK. This widget lets us display the video feed for local and remote peers within the game interface. By incorporating the VideoView widget, we can seamlessly integrate the video streams from the local and remote participants, enriching the interactive experience and fostering real-time communication between the players.

// Local user video tile and name
Column(
   mainAxisAlignment: MainAxisAlignment.center,
   children: [
      SizedBox(
        height: 100,
        width: 100,
        child: ClipRRect(
           borderRadius: BorderRadius.circular(17),
           child: const VideoView(
              isSelfParticipant: true,
           ),
        ),
     ),
    Text(roomStateNotifier.dyteClient.value.localUser.name,style: const    TextStyle(color: Colors.white),)
  ],
),

To render the local peer's video feed, we must pass the isSelfParticipant parameter as true, as shown in the code snippet above. This informs the VideoView widget that the video being rendered is from the local participant.

On the other hand, for rendering the remote peer's video feed, we can utilize the following code:

if (roomStateNotifier.remotePeer.value != null)
// Remote user video tile and name
Column(
mainAxisAlignment: MainAxisAlignment.center,
children: [
SizedBox(
 height: 100,
 width: 100,
  child: ClipRRect(
   borderRadius: BorderRadius.circular(17),
   child: VideoView(
    meetingParticipant:
    roomStateNotifier.remotePeer.value,
    ),
   ),
  ),
    Text(roomStateNotifier.remotePeer.value!.name,style: const TextStyle(color:        Colors.white),)
   ],
),

To fulfill the requirement, we need to include a meetingParticipant parameter with the value of remotePeer, which we have stored in the onParticipantJoin override function.

Utilizing the meetingParticipant parameter and accessing the remotePeer value ensures that the correct participant's video stream is rendered within the VideoView widget.

And here’s the source code for the project.

Throughout this blog post, we explored the seamless integration of these powerful tools, enabling you to create an immersive and interactive gaming experience.

Conclusion

You've crafted a visually appealing and responsive user interface by leveraging Flutter's cross-platform capabilities. The Dyte video SDK has brought players closer together, allowing them to enjoy real-time Tic-Tac-Toe battles of strategy and wit, regardless of their physical location. With GetX, you've efficiently managed the game's state, providing users with a smooth and enjoyable gaming experience.

But this is just the beginning of your game development journey. There are endless possibilities for expanding and enhancing your Tic-Tac-Toe game. Consider implementing game statistics, player rankings, or additional game modes to keep your users engaged and entertained.

As you continue to explore and experiment with Flutter and the Dyte video SDK, remember to embrace the joy of learning and be open to trying new ideas. The world of game development is constantly evolving, offering limitless opportunities for innovation.

Get better insights on leveraging Dyte's technology and discover how it can revolutionize your app's communication capabilities with its SDKs.

Congratulations! You've reached the end of this exciting journey into building a multiplayer Tic-Tac-Toe game using Flutter, the Dyte video SDK, and GetX for state management.

Add Video Calling to Chess With Flutter, Dyte, and GetX

Govind Maheshwari — Tue, 09 Jan 2024 19:26:04 +0000

Discover the fusion of tradition and innovation as we explore the world of chess, reimagined through Dyte video calling in the Flutter framework. Break the distance barriers and get your tactical hat on because it's getting real, in real-time, like never before.

In this blog, we'll (grand)master adding video calling to chess, all within the flexible Flutter framework. Prepare to take your strategic battles beyond the board, embracing virtual gameplay and personal connections.

We'll capture your pieces and walk you through every step of this tutorial. You can expect clear, easy-to-follow instructions, short code examples, and valuable tips. We aim to give you the confidence and skills to create your unique Flutter app. Whether you're an experienced coder or starting out doesn't matter. This guide is designed for all sorts of players.

So, let's jump right in and harness the capabilities of Flutter and Dyte to give a fresh perspective to the timeless board game. Our journey will infuse joy and connection into players all around the globe.

Requirements

You'll need critical elements to create your multiplayer chess game using the Dyte video calling API in Flutter. Firstly, make sure you have Flutter installed on your system. You can follow your operating system's official Flutter installation guide to set it up successfully.

To help you hit the ground running, we've prepared an initial project setup with a pre-built user interface (UI) for the chess game. You can access the project code and user interface files here. This initial setup provides a solid foundation to build upon, saving you time and effort in creating the game's basic structure.

Next, you must visit the official Dyte website and navigate to the signup page to create an account and explore the various features and functionalities Dyte offers.

You'll then need to incorporate the Dyte video calling SDK into your Flutter project. The Dyte plugin will seamlessly integrate video calling functionality, enabling real-time communication between players.

You can add the dyte_core plugin in the pubspec.yaml file using the following command.

flutter pub add dyte_core

To add chess functionality, we are using the bishop, squares, and square_bishop plugins.

We are going to use Getx for state management. GetX is a powerful state management solution for Flutter that offers simplicity, performance, and scalability. With GetX, you can easily manage the state of your chess game and handle navigation, dependencies, and more.

Basic chess concepts

Before we add video calling to our game of wits, we need to understand some chess concepts before writing code.

1. Chessboard and Squares: The chessboard consists of 64 squares arranged in an 8x8 grid. Each square alternates between light and dark colors.

2. Pieces and their movement:

King: Moves one square in any direction.
Queen: Moves diagonally, horizontally, or vertically any number of squares.
Rook: Moves horizontally or vertically any number of squares.
Bishop: Moves diagonally any number of squares.
Knight: Moves in an L-shape; two squares in one direction and one square perpendicular to it.
Pawn: Moves forward one square but captures diagonally.

3. Checkmate: If a player's king is in check and has no legal moves to escape, it's checkmate, and the game ends.

4. FEN: In (Forsyth-Edwards Notation) FEN notation, each chessboard square is represented by a character, with uppercase letters denoting white pieces and lowercase letters denoting black pieces. The characters used are as follows:

K/k: King
Q/q: Queen
R/r: Rook
N/n: Knight
B/b: Bishop
P/p: Pawn
1-8: Empty squares (number of empty squares in a row)

FEN also includes information about the player to move, castling rights, en passant target square, halfmove clock (the number of halfmoves since the last pawn move or capture), and fullmove number (the number of the current move).

FEN notation is commonly used in various chess-related applications, such as recording and sharing positions, analyzing games, and setting up positions for practice or analysis. It provides a standardized way to communicate chess positions regardless of language or platform.

**Note:** For rendering pieces on board we will require to generate FEN after every move.

Building Game

Now that you understand the chess terminology, let's move in the right direction and navigate to the lib folder. There, you will discover four files playing crucial roles to bring your game to life. Each file contributes uniquely to the seamless functionality and immersive experience you're crafting. Let's explore the significance of these files in shaping your game's architecture and interaction.

We have the main.dart file starting at the beginning. This Dart file serves as the entry point for your application. It's the first thing that gets executed when your app launches. In this file, you'll typically find the setup for your app's main structure, including the landing page. The landing page is what users initially see when they open your app. It's the gateway that sets the tone for the entire user experience (UX), providing a decisive first impression of what your app offers.

The second file is named game_screen.dart. This Dart file takes center stage in our chess game's UI aspect. Here, you'll find the design and layout components that visually represent the chess game to the user. This file shows how the game appears on the screen from the chessboard to the player interface. It's the bridge between the underlying game logic and what the user sees, ensuring a visually appealing and user-friendly experience for your players.

Moving on to the third file, called room_state.dart. This Dart file is the home for all the essential logic related to Dyte video calling and the state of the chess game. Here, you'll find the backbone of your application's functionality, connecting the dots between video communication and the intricate workings of chess. This file is a pivotal hub where the magic of video interaction and the strategy of chess converge to create a seamless and engaging UX.

The fourth is the http_request.dart file. The HttpRequest class in this Dart file is designed to manage API requests. It's responsible for handling tasks such as creating rooms and adding participants to these rooms. This integral class plays a crucial role in facilitating interactions with the backend of your application.

To get your project up and running, open your terminal and execute the command flutter pub get. This command is essential as it fetches and loads all the required plugins and dependencies for your project. It ensures your project has everything needed to function correctly so you can seamlessly build your engaging multiplayer chess game.

While the game might not be fully functional yet, we're committed to transforming it into a fully operational masterpiece by the time you complete this blog.

Let's focus on two essential features: room creation and participant addition. These functionalities serve as the cornerstone of our game, enabling us to establish fresh gaming rooms and effortlessly invite friends to join in on the fun. Through room creation, we can set the stage for exciting matches. At the same time, the participant addition feature lets us share these rooms with our buddies, fostering a sense of camaraderie and friendly competition.

API Request

Let's navigate to the http_request.dart file. Within this file, you'll discover the code responsible for room creation and participant addition through API requests. It's crucial to note that you'll need to include your organization ID and API key from the Dyte dashboard in this process. These credentials are essential for establishing a secure and authenticated connection with the Dyte services.

Code snippet for room creation

static Future<String?> createMeeting() async {
 String id = DateTime.now().millisecondsSinceEpoch.toString();
 try {
  Response response = await post(Uri.parse("$_baseUrl/meetings"),
   headers: {'Authorization': _token,"Content-Type": "application/json"},
   body: json.encode({"title":"chess-$id"})
  );
  if (response.statusCode == 201) {
    Map map = jsonDecode(response.body);
    return map["data"]["id"];
   }
  } catch (e) {
  Get.showSnackbar(GetSnackBar(
   title: e.toString(),
  ));
 }
 return null;
}

Code snippet for add participant

static Future<String?> addParticipant(String name, String roomId) async {
 String id = DateTime.now().millisecondsSinceEpoch.toString();
 try {
  Response response = await post(
  Uri.parse("$_baseUrl/meetings/$roomId/participants"),
  headers: {'Authorization': _token,"Content-Type": "application/json"},
  body: json.encode({"name":name,
           "Preset_name":"group_call_host",
           "custom_participant_id":"player-$id"
})
  );
 if (response.statusCode == 201) {
  Map map = jsonDecode(response.body);
  return map["data"]["token"];
 }
 } catch (e) {
  Get.showSnackbar(GetSnackBar(
  title: e.toString(),
 ));
 }
 return null;
}

For a more comprehensive understanding of the API requests and responses, please refer to the Dyte API guide. This guide provides detailed insights into working with the Dyte API for various operations.

You can explore Dyte API: Create a meeting for room creation and Dyte API: Add a participant for, well, adding participants.

These resources will provide in-depth information on the API endpoints, parameters, and expected responses, enabling you to seamlessly integrate these functionalities into your Flutter application for a fully-fledged multiplayer chess experience.

Now, let’s open the main.dart file for connecting room creation and adding participants to the UI.

Our initial focus is on establishing a room for the game. You'll encounter two buttons in the code and UI. One button lets you join an existing game, while the other enables you to create a new one.

When the "Create Game" button is pressed, we execute the following code to create a room:

String? roomId = await HttpRequest.createMeeting();

This code snippet triggers the creation of a room where the chess game will unfold. It's the starting point for crafting the environment where players engage in strategic battles on the virtual chessboard.

After successfully creating the room, we get the room ID. We will then pass this room ID to the addParticipant function, which will facilitate the token acquisition. This token is a vital element for the video calling functionality. Here's how you achieve this:

String? token = await HttpRequest.addParticipant(roomId);

We will transmit this acquired token to the RoomStateNotifier.

When joining an existing game, our approach involves prompting the user for a room ID. This room ID will then be utilized in the addParticipant function to obtain the necessary token. Once secured, this token is subsequently passed to the RoomStateNotifier for effective integration into the ongoing gaming experience.

Let's open the room_state.dart file to move forward. Once inside, you'll find a collection of code snippets pivotal to our progress. These snippets hold the key to implementing the Dyte video calling functionality and managing the state of our chess game. So, let's dive in and dissect these snippets to uncover the magic that will bring our multiplayer chess game to life!

As you begin exploring, you'll notice a series of variable initializations. These are accompanied by insightful comments that shed light on their intended roles and how they contribute to the program's functionality.

Initiating the Dyte video meeting

To keep track of updates from the Dyte plugin, we must incorporate listeners into the RoomStateNotifier class. Achieving this involves implementing the following code snippets:

class RoomStateNotifier extends GetxController
implements
DyteMeetingRoomEventsListener,
DyteParticipantEventsListener,
DyteChatEventsListener {
...

By incorporating these listeners, we establish a mechanism where our application stays informed about any alterations or updates occurring in the room state through the Dyte plugin. This real-time awareness is crucial to ensure our game stays in sync with the dynamic changes within the video communication framework.

Once you've integrated the listener into the RoomStateNotifier class, you might encounter errors that signal the absence of specific override methods. You can use the quick fix feature available in your integrated development environment (IDE) to tackle these errors. This tool can assist you in swiftly resolving the issues by generating the required override methods, ensuring your code remains error-free and fully functional.

The Dyte plugin equips us with various essential override methods that enable us to effectively capture updates concerning our calls. In the context of our application, we will be leveraging the following override methods to enhance our UX:

1. onMeetingInitCompleted():

This method triggers when the initialization of a meeting concludes. It furnishes pertinent details about the meeting, facilitating informed handling of meeting-related information.

2. onMeetingRoomJoinCompleted():

This method springs into action when a user successfully enters a meeting room. It manages any imperative actions or updates required upon room entry.

3. onMeetingRoomLeaveCompleted():

This method is the moment a user departs from a meeting room, offering the opportunity to execute any essential cleanup or follow-up procedures.

4. onParticipantJoin(DyteJoinedMeetingParticipant participant):

This method is triggered whenever a new participant joins the meeting, delivering comprehensive information about the recently joined participant.

5. onParticipantLeave(DyteJoinedMeetingParticipant participant):

This method comes into play as a participant exits the meeting, providing vital insights into the departing participant's details.

6. onNewChatMessage(DyteChatMessage message):

If a fresh chat message arrives, this method is invoked. It empowers us to process and manage incoming messages effectively.

Harnessing these override methods allows us to capture key events within our video calls and enhance our application's overall interactivity and engagement.

Let's attach the listener to the RoomStateNotifier class to start receiving updates from the Dyte API. Furthermore, we initiate the video call by invoking the init(meetingInfo) method provided by the dyteClient instance. This action triggers the process of starting the video call, enabling users to join the meeting and engage in real-time communication with other participants. The dyteClient variable is of type DyteMobileClient, which serves as a manager for interacting with a Dyte server.

Here is the code snippet for adding the listener and initiating the meeting:

RoomStateNotifier(String name, String token, String meetingId) {
username.value = name;
roomId.value = meetingId;
dyteClient.value.addMeetingRoomEventsListener(this);
dyteClient.value.addParticipantEventsListener(this);
dyteClient.value.addChatEventsListener(this);
final meetingInfo = DyteMeetingInfoV2(
authToken: token,
enableAudio: false,
enableVideo: true);
dyteClient.value.init(meetingInfo);
}

Once the meeting is successfully initialized, a callback will be triggered in the onMeetingInitCompleted() function. This callback indicates that the meeting has been set up properly and is ready for joining the room.

Joining the Dyte meeting

To join the call and configure the username within the room, the following code must be implemented within the onMeetingInitCompleted() function.

@override
void onMeetingInitCompleted() {
dyteClient.value.localUser.setDisplayName(username.value);
dyteClient.value.joinRoom();
}

When the room join process is completed, the onMeetingRoomJoinCompleted() callback function is triggered. In this callback, we set the roomJoin variable to true, indicating that the local user has successfully joined the room.

@override
void onMeetingRoomJoinCompleted() {
roomJoin.value = true;
}

Leaving the Dyte meeting

Similarly, in the onMeetingRoomLeaveCompleted() callback, we handle the removal of listeners from this class. This ensures that the necessary clean-up tasks are performed when leaving the meeting, preventing any potential memory leaks or unwanted behaviors. By implementing this callback, we can adequately manage the class's lifecycle and maintain a streamlined and efficient application flow.

@override
void onMeetingRoomLeaveCompleted() {
roomJoin.value = false;
dyteClient.value.removeMeetingRoomEventsListener(this);
dyteClient.value.removeParticipantEventsListener(this);
dyteClient.value.removeChatEventsListener(this);
Get.delete<RoomStateNotifier>();
Get.back();
}

Remote peer(s) join the Dyte meeting

When an opponent joins the room, the onParticipantJoin() function is triggered, providing us with a callback. At this point, we can proceed to initiate the game. However, it is necessary to assign color to local and remote peers before starting.

To accomplish this, we will call the existing assignColour() function available within the same file. By invoking this function, we can ensure that each player is assigned their respective symbols, setting the stage for an engaging and fair gameplay experience.

@override
void onParticipantJoin(DyteJoinedMeetingParticipant participant) {
   if (participant.userId != dyteClient.value.localUser.userId) { //not a local user
       remotePeer.value = participant;
       assignColour();
   }
}

We use their unique user IDs to assign local and remote peers colors. We can ensure consistent symbol assignment across the room by sorting these IDs. In this implementation, we designate the symbol "White" to the user at the 0 index and "Black" to the other peer. As user IDs are unique and consistent within the room, this approach guarantees that each player receives their designated symbol, enhancing clarity and fairness during gameplay.

If a remote peer leaves the room during an ongoing game, we must ensure that the local peer also leaves to maintain a synchronized and consistent experience. We will use the onParticipantLeave override function to handle this scenario.

When a remote peer leaves the room, we will display a dialog to the user, giving them a clear choice to leave the room. The dialog will have the barrierDismissible parameter set to false, meaning the user cannot dismiss the dialog by tapping outside.

dyteClient.value.leaveRoom();

Here is the code for onParticipantLeave:

@override
void onParticipantLeave(DyteJoinedMeetingParticipant participant) {
  if (participant.userId != dyteClient.value.localUser.userId &&
    remotePeer.value != null) {
      remotePeer.value = null;
      Get.defaultDialog(
        barrierDismissible: false,
        onWillPop: () async {
          return false;
        },
        title: "Opponent Leave this game.",
        textConfirm: "Leave",
        middleText: "",
        confirmTextColor: Colors.white,
        onConfirm: () {
          dyteClient.value.leaveRoom();
          Get.back();
        });
      }
}

makes their move, we are harnessing the chat feature provided by the Dyte plugin. Within this strategy, we employ the makeSquareMove(move) function from the bishop library, a built-in tool. This function possesses a boolean return type; if the move is invalid, it yields false. Otherwise, it is true. Upon confirming a valid move, we update the ChessBoard's state by fetching the latest squaresState via the bishop library.

To synchronize the remote user's experience, we transmit the most recent FEN (Forsyth-Edwards Notation) obtained from the bishop library through the Dyte plugin's chat functionality. This data is seamlessly sent to the remote peer's device.

onUserMove(Move move, {bool? isDrop, bool? isPremove}) {
 if (game.value.makeSquaresMove(move)) {
   state.value = game.value.squaresState(playerColor.value);
   sendMessage(game.value.fen);
   localUserTurn.value = false;
 }else{
   Get.snackbar("Invalid move","",snackPosition: SnackPosition.BOTTOM);
}
 if (game.value.checkmate) {
  Get.defaultDialog(
   barrierDismissible: false,
   title: "You won the game",
   textConfirm: "Leave",
   middleText: "",
   confirmTextColor: Colors.white,
   onConfirm: () {
    dyteClient.value.leaveRoom();
    Get.back();
   });
 }
}

Using Dyte’s chat component

To ensure real-time updates of the game whenever a user makes their move, we are harnessing the chat feature provided by the Dyte plugin. Within this strategy, we employ the makeSquareMove(move) function from the bishop library, a built-in tool. This function possesses a boolean return type; if the move is invalid, it yields false, otherwise, it is true. Upon confirming a valid move, we update the ChessBoard's state by fetching the latest squaresState via the bishop library.

To synchronize the remote user's experience, we transmit the most recent FEN obtained from the bishop library through the Dyte plugin's chat functionality. This data is seamlessly sent to the remote peer's device.

onUserMove(Move move, {bool? isDrop, bool? isPremove}) {
 if (game.value.makeSquaresMove(move)) {
   state.value = game.value.squaresState(playerColor.value);
   sendMessage(game.value.fen);
   localUserTurn.value = false;
 }else{
   Get.snackbar("Invalid move","",snackPosition: SnackPosition.BOTTOM);
}
 if (game.value.checkmate) {
  Get.defaultDialog(
   barrierDismissible: false,
   title: "You won the game",
   textConfirm: "Leave",
   middleText: "",
   confirmTextColor: Colors.white,
   onConfirm: () {
    dyteClient.value.leaveRoom();
    Get.back();
   });
 }
}


sendMessage(String message) {
dyteClient.value.chat.sendTextMessage(message);
}

Lastly, we make use of the onNewChatMessage override function. This function lets us receive the chess FEN in text format through the chat functionality. We will now load the bishop variable (game) with a new FEN and update the chessBoard state using the squareState function, which enables us to update the UI accordingly.

By implementing this functionality, we ensure that any updates made by the remote peer are received and reflected in real-time on the local user's UI.

@override
void onNewChatMessage(DyteChatMessage message) {
 if (message.userId == dyteClient.value.localUser.userId) {
  return;
 }
 DyteTextMessage textMessage = message as DyteTextMessage;
 final fen = textMessage.message;
 game.value.loadFen(fen);
 state.value = game.value.squaresState(playerColor.value);
 if (game.value.checkmate) {
  Get.defaultDialog(
   barrierDismissible: false,
   title: "You lose the game",
   textConfirm: "Leave",
   middleText: "",
   confirmTextColor: Colors.white,
   onConfirm: () {
    dyteClient.value.leaveRoom();
    Get.back();
   });
  }
 localUserTurn.value = true;
}

Managing audio/video of participants

We will utilize two additional functions — toggleVideo() and toggleAudio() — to manage the local peer's video and audio within the room. These functions allow us to dynamically control the starting and stopping of the local peer's video and audio streams during the video call session.

By invoking toggleVideo(), we can initiate or pause the local peer's video transmission, while toggleAudio() enables us to start or mute the local peer's audio feed. These functionalities provide flexibility and control to the user, allowing them to manage their video and audio presence according to their preferences and requirements within the room.

toogleVideo() {
     if (isVideoOn.value) {
 dyteClient.value.localUser.disableVideo();
     } else {
       dyteClient.value.localUser.enableVideo();
     }
       isVideoOn.toggle();
}
toogleAudio() {
     if (isAudioOn.value) {
        dyteClient.value.localUser.disableAudio();
     } else {
        dyteClient.value.localUser.enableAudio();
     }
        isAudioOn.toggle();
}

Setting up the chess game

Now, let's navigate to the gameScreen.dart file. This file contains the UI code for our game and the video call controls UI. However, we must add some code to display the video tiles correctly. This code will handle the rendering and layout of the video tiles, ensuring they are displayed correctly on the game screen. By incorporating this code, we can seamlessly integrate the game UI and the video call interface, providing users with a comprehensive and immersive experience.

Search for the comment "// Dyte Meeting Video View" in the gameScreen.dart file. Within the corresponding section, we will proceed to check whether the user has joined the meeting or not by using the following code:

if (roomStateNotifier.roomJoin.value)

By implementing this code snippet, we can differentiate between the scenarios where the user has successfully joined the meeting and where they have not. This conditional logic allows us to handle each situation appropriately, ensuring a smooth and coherent user experience throughout the game.

If the roomJoin variable is true, it indicates that the user has successfully joined the meeting. We can use the VideoView widget the Dyte video calling SDK provides to render the user's video tile.

This widget lets us display the video feed for local and remote peers within the game interface. By incorporating the VideoView widget, we can coherently integrate the video streams from the local and remote participants, enriching the interactive experience and fostering real-time communication between the players.

// Local user video tile and name
Row(
 children: [
  const Spacer(),
  Obx(
  () => Text(
  roomStateNotifier.roomJoin.value == false
  ? ""
  : roomStateNotifier.username.value,
  style: const TextStyle(
  color: Colors.black, fontWeight: FontWeight.bold),
  ),
 ),
 const SizedBox(
  width: 10,
 ),
 Obx(
 () => Container(
  height: 100,
  width: 100,
  decoration: BoxDecoration(
  borderRadius: BorderRadius.circular(17),
  boxShadow: [
   if (roomStateNotifier.localUserTurn.value) ...[
   const BoxShadow(
    color: Colors.red,
    spreadRadius: 4,
    blurRadius: 10,
   ),
  const BoxShadow(
   color: Colors.red,
   spreadRadius: -4,
   blurRadius: 5,
  )]
  ],
 ),
 margin: const EdgeInsets.only(right: 10),
 child: ClipRRect(
 borderRadius: BorderRadius.circular(17),
 // Dyte Meeting Video View
 child: roomStateNotifier.roomJoin.value == false
 ? const SizedBox()
 : const VideoView(
  isSelfParticipant: true,
  ),
 ),
 ),
 )

To render the local peer's video feed, we must pass the isSelfParticipant parameter as true, as shown in the code snippet above. This informs the VideoView widget that the video is rendered from the local participant.

On the other hand, for rendering the remote peer's video feed, we can utilize the following code:

roomStateNotifier.remotePeer.value == null
// Room Id
? SizedBox(
 child: Center(
  child: Row(
  mainAxisSize: MainAxisSize.min,
  children: [
   Text("Room Id: ${roomStateNotifier.roomId.value.substring(0,10)}...",),
   IconButton(onPressed: () async{
   //`Clipboard.setData` copy string to clipboard
   await Clipboard.setData(ClipboardData(text: roomStateNotifier.roomId.value));
   }, icon: const Icon(Icons.copy)),
  ]),
 ))
// Remote user video tile and name
: Row(
   mainAxisSize: MainAxisSize.min,
   children: [
    Container(
     height: 100,
     width: 100,
     padding: const EdgeInsets.only(left: 10),
     child: ClipRRect(
      borderRadius: BorderRadius.circular(17),
      child: VideoView(
       meetingParticipant:
       roomStateNotifier.remotePeer.value,
      ),
     ),
    ),
 const SizedBox(
 width: 10,
 ),
 Text(
  roomStateNotifier.remotePeer.value!.name,
  style: const TextStyle(
   color: Colors.black, fontWeight: FontWeight.bold),
  ),
 const Spacer()
 ])

To meet the specific requirement, it's essential to include a meetingParticipant parameter, assigning it the value of remotePeer. This value is stored within the onParticipantJoin override function.

By incorporating the meetingParticipant parameter and accessing the stored remotePeer value, we guarantee the accurate rendering of the video stream for the respective participant. This synchronization ensures that the appropriate participant's video feed is correctly displayed within the VideoView widget.

Conclusion

Congratulations! If you've reached here, it's checkmate for this blog. You've successfully added live video calling to chess and made it multiplayer using Flutter, Dyte video calling SDK, and GetX for expert state management. By integrating these tools, you're empowered to construct a gaming encounter that's not only immersive but also teeming with interactivity.

Here's the source code for the full project.

By harnessing the versatile capabilities of Flutter, you've fashioned a visually captivating and responsive UI across different platforms. The Dyte video calling plugin has bridged geographical gaps, enabling players to engage in real-time bouts of strategic prowess, regardless of location. Lastly, the seamless integration of GetX ensures the game's state remains effortlessly manageable, culminating in an immersive gaming escapade for your users.

Yet, this journey into game development is but the opening chapter. The canvas for expanding and enriching your chess prowess is boundless. Think about infusing features like game statistics to track progress, player rankings to heighten competition, or even exploring novel game modes to keep users enthralled and amused. You're bound to gain tempo.

It's time we stop the clock and offer you a handshake. Until next time.

Get better insights on leveraging Dyte’s technology and discover how it can revolutionize your app’s communication capabilities with its SDKs.

Building a Live Proctoring System to Detect Participant Switching

Ravindra Singh Rathor — Thu, 28 Dec 2023 20:24:50 +0000

Introduction

To stop students from cheating during tests, we often assign a proctor to observe them. But we can't have someone in-person watching each student for online tests. This is where live automatic proctoring helps.

With a webcam and microphone, this method monitors students while they take online tests. It uses computer tools to spot any cheating. 👀

In the previous blog, we built a live proctoring system to detect whether there are multiple people in the candidate's frame.

In this tutorial, we will build a live proctoring system using Dyte APIs that will allow an admin to monitor if the candidate who was supposed to be in the video is still the same candidate during the whole duration in real-time. ✨

High-Level Design of the application

We aim to notify the proctor if the candidate changes during the session. 🔍

The proctor would get the candidate's details and a photograph from their webcam as proof in his meeting sidebar. 📸✨

In this project, we will use React with Dyte UI kit and Dyte React Web Core packages for the frontend.
For the backend, we will use FastApi (Python 3).
We will also use database as a service by ElephantSQL (PostgreSQL).
Lastly, we will use Imgur for storing screenshots. And the image metadata will then be saved to the same database.

Folder Structure

After completing the tutorial, the folder structure will look like this. 👇

dyte-proctoring/
├── frontend
│   ├── README.md
│   ├── package.json
│   ├── public
│   ├── src
│   │   ├── App.css
│   │   ├── App.jsx
│   │   ├── Heading.jsx
│   │   ├── ImageInput.jsx
│   │   ├── Meet.jsx
│   │   ├── Proctor.jsx
│   │   ├── index.css
│   │   ├── index.jsx
│   │   ├── logo.svg
│   │   ├── react-app-env.d.ts
│   │   ├── reportWebVitals.ts
│   │   ├── setupTests.ts
│   │   ├── stage.jsx
│   │   └── utils.js
│   ├── tsconfig.json
├── venv
├── app.py
├── imgur.py
└── requirements.txt

Step 0: Configurations and Setup

🧑‍💻 Before building our live proctoring system, we must set up a Dyte account.

We can create a free account by clicking the "Start Building" button on Dyte.io and signing up using Google or GitHub 🎉.

Once signed up, we can access our Dyte API keys from the "API Keys" tab in the left sidebar. We will keep these keys secure as we will use them later.🔑🤫

For our live proctoring system, we will use React for the frontend and FastAPI for building the Backend and APIs.

We will begin by creating a new directory for our project, called dyte-proctoring and navigating into it using the following commands:

mkdir dyte-proctoring
cd dyte-proctoring

Please note:

We will also require accounts on the following platforms:

Imgur: Create an account on Imgur and create an API key. Here is a step-by-step guide
ElephantSQL: Here is a step-by-step guide to creating a db on ElephantSQL.

Now back to the tutorial.

Step 1: Setting up the frontend

Let's start setting up our frontend project using React and Dyte! ✨

We will create a boilerplate React app using create-react-app. We can do this with the following command:

yarn create react-app frontend

This will initialize a new React app in the frontend directory. 📁

Then, we will go ahead and install the dyte react-web-core, dyte react-ui-kit and react-router packages in this project using the following command 👇

yarn add @dytesdk/react-web-core @dytesdk/react-ui-kit react-router react-router-dom

Step 2: Setting up the backend

Let's get started with setting up our FastAPI backend now. 🙌

We will go back to the root directory of our project and initiate our project here itself for the ease of hosting:

cd ..

First of all, we will go ahead and create our requirements.txt file in the root directory itself with the following content 👇

requirements.txt

cmake
fastapi
uvicorn
face_recognition
numpy
python-multipart
psycopg2-binary
httpx
python-dotenv
pydantic
requests

After this, we will go ahead and create our virtual environment with venv and install the dependencies.

python -m venv venv
source venv/bin/activate # for linux/mac
venv\Scripts\activate.bat # for windows
pip install -r requirements.txt

We will also create an environment variable file .env, for storing our credentials.

.env

DYTE_ORG_ID=<ID>
DYTE_API_KEY=<KEY>
IMGUR_CLIENT_ID=<ID>
DB_USER=<ID>
DB_PASSWORD=<PASSWORD>
DB_HOST=<HOST>

Step 3: Setting up image upload with Imgur

Let's create a new file named imgur.py and add the following code. This will help us upload our screenshots to Imgur. 🌅

Here we are connecting the Imgur API using the CLIENT_ID that we can get from our Imgur API Dashboard and using it to upload the suspicious candidate's image and get back the link to it. 🔗👇

imgur.py

import base64
from fastapi import FastAPI, UploadFile, HTTPException
from httpx import AsyncClient
from dotenv import load_dotenv

load_dotenv()

app = FastAPI()
IMGUR_CLIENT_ID = os.getenv("IMGUR_CLIENT_ID")

async def upload_image(img_data):
    headers = {
        "Authorization": f"Client-ID {IMGUR_CLIENT_ID}"
    }
    data = {
        "image": img_data
    }

    async with AsyncClient() as client:
        response = await client.post("https://api.imgur.com/3/image", headers=headers, data=data)

    if response.status_code != 200:
        raise HTTPException(status_code=500, detail="Could not upload image.")

    print(response.json())
    return response.json()["data"]["link"]

Step 4: Setting up our backend application

Now, we will create a new file named app.py and add our 🐘 ElephantSQL PostgreSQL database connection and code for our APIs, including face detection logic.

In this file, we would need to create the following routes:

GET / - Root route

POST /is_admin/ - Check if the user is an admin

POST /suspicious_list/ - This route retrieves a list of participants who are detected as suspicious

POST /same_faces/ - Detect if the face in the screenshot is the same as the face in the reference image of the candidate.

POST /upload_photo/ - Upload a reference photo of the candidate

POST /meetings - Create a new meeting

POST /meetings/{meetingId}/participants - This route is responsible for adding a participant to a specific meeting identified by meetingId

So let's get started 👇

app.py

import base64
import io
import logging
import random
import requests

import uvicorn
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from imgur import upload_image
import face_recognition
import psycopg2

import os
import base64
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from dotenv import load_dotenv
from httpx import AsyncClient
import uuid

load_dotenv()

DYTE_API_KEY = os.getenv("DYTE_API_KEY")
DYTE_ORG_ID = os.getenv("DYTE_ORG_ID")

API_HASH = base64.b64encode(f"{DYTE_ORG_ID}:{DYTE_API_KEY}".encode('utf-8')).decode('utf-8')

DYTE_API = AsyncClient(base_url='https://api.cluster.dyte.in/v2', headers={'Authorization': f"Basic {API_HASH}"})

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

fh = logging.FileHandler("app.log")
fh.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
fh.setFormatter(formatter)
logger.addHandler(fh)


class ParticipantScreen(BaseModel):
    base64_img: str
    participant_id: str
    meeting_id: str
    ref_image_url: str
    participant_name: str

class ProctorPayload(BaseModel):
    meeting_id: str
    admin_id: str

class AdminProp(BaseModel):
    meeting_id: str
    admin_id: str

class Meeting(BaseModel):
    title: str

class Participant(BaseModel):
    name: str
    preset_name: str
    meeting_id: str

origins = [
    # allow all
    "*",
]

app = FastAPI()

# enable cors
app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],  # allow all
    allow_headers=["*"],  # allow all
)

def connect_to_db():
    conn = psycopg2.connect(
            dbname=os.getenv('DB_USER'),
            user=os.getenv('DB_USER'),
            password=os.getenv('DB_PASSWORD'),
            host=os.getenv('DB_HOST'),
            port=5432
    )
    return conn

@app.get("/")
async def root():
    return {"message": "Hello World"}

@app.post("/is_admin/")
async def is_admin(admin: AdminProp):
    conn = connect_to_db()
    cur = conn.cursor()
    cur.execute("SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s", (admin.meeting_id, admin.admin_id,))

    count = cur.fetchone()[0]

    if(count > 0):
        return { "admin": True }
    else:
        return { "admin": False }

@app.post("/suspicious_list/")
async def suspicious_list(meeting: ProctorPayload):
    conn = connect_to_db()
    cur = conn.cursor()
    cur.execute("SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s", (meeting.meeting_id, meeting.admin_id,))

    count = cur.fetchone()[0]

    if(count > 0):
        cur.execute("SELECT * FROM meeting_proc_details WHERE meeting_id = %s ORDER BY ts DESC", (meeting.meeting_id,))
        rows = cur.fetchall()
        conn.commit()
        cur.close()
        conn.close()
        return rows
    else:
        conn.commit()
        cur.close()
        conn.close()
        raise HTTPException(status_code=401, detail="Participant dose not has admin role")

@app.post("/same_faces/")
async def same_faces(participant: ParticipantScreen):
    conn = connect_to_db()
    cur = conn.cursor()

    cur.execute("CREATE TABLE IF NOT EXISTS meeting_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), img_url VARCHAR(255), verdict VARCHAR(255))")

    img_url = participant.ref_image_url

    response = requests.get(img_url)
    ref_img_data = io.BytesIO(response.content)

    img_data = participant.base64_img.split(",")[1]
    img_data_dc = base64.b64decode(participant.base64_img.split(",")[1])

    file_obj = io.BytesIO(img_data_dc)
    unknown_image = face_recognition.load_image_file(file_obj)
    known_image = face_recognition.load_image_file(ref_img_data)

    biden_encoding = face_recognition.face_encodings(known_image)[0]
    unknown_encoding = face_recognition.face_encodings(unknown_image)[0]

    results = face_recognition.compare_faces([biden_encoding], unknown_encoding)
    is_same_face = True
    for res in results:
        is_same_face = is_same_face and res


    if not is_same_face:
        logger.info(
            f"Detected different faces for participant {participant.participant_id}"
        )

        verdict = f"Participant Name: {participant.participant_name} <> Anomaly: Different Faces Detected <> Participant ID: {participant.participant_id}"
        cur.execute("SELECT count(1) FROM meeting_proc_details WHERE meeting_id=%s AND participant_id=%s AND ts >= (current_timestamp - INTERVAL '10 minutes')", (participant.meeting_id, participant.participant_id))
        count = cur.fetchone()[0]

        if count == 0:
            upload_resp = await upload_image(img_data)
            cur.execute("INSERT INTO meeting_proc_details (ts, meeting_id, participant_id, img_url, verdict) VALUES (current_timestamp, %s, %s, %s, %s)",
                (participant.meeting_id, participant.participant_id, upload_resp, verdict)
            )

        conn.commit()
        cur.close()
        conn.close()

        if count == 0:
            return { "id": participant.participant_id, "different_faces_detected": True, "url": upload_resp }
        return { "id": participant.participant_id, "different_faces_detected": True, "url": "not uploaded" }

    return {"id": participant.participant_id, "different_faces_detected": False}

@app.post("/upload_photo/")
async def upload_photo(participant: ParticipantScreen):
    img_data = participant.base64_img.split(",")[1]

    conn = connect_to_db()
    cur = conn.cursor()

    cur.execute("CREATE TABLE IF NOT EXISTS meeting_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), img_url VARCHAR(255), verdict VARCHAR(255))")

    verdict = f"REF_IMAGE"

    upload_resp = await upload_image(img_data)
    cur.execute("INSERT INTO meeting_proc_details (ts, meeting_id, participant_id, img_url, verdict) VALUES (current_timestamp, %s, %s, %s, %s)",
        (participant.meeting_id, participant.participant_id, upload_resp, verdict)
    )

    conn.commit()
    cur.close()
    conn.close()

    return {"id": participant.participant_id, "img_url": upload_resp, "reference_image_uploaded": True}

@app.post("/meetings")
async def create_meeting(meeting: Meeting):
    response = await DYTE_API.post('/meetings', json=meeting.dict())
    if response.status_code >= 300:
        raise HTTPException(status_code=response.status_code, detail=response.text)
    admin_id = ''.join(random.choices('abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789', k=32))
    resp_json = response.json()
    resp_json['admin_id'] = admin_id
    meeting_id = resp_json['data']['id']

    conn = connect_to_db()
    cur = conn.cursor()
    cur.execute("INSERT INTO meeting_host_info (ts, meeting_id, admin_id) VALUES (CURRENT_TIMESTAMP, %s, %s)", (meeting_id, admin_id))
    conn.commit()
    cur.close()
    conn.close()

    return resp_json


@app.post("/meetings/{meetingId}/participants")
async def add_participant(meetingId: str, participant: Participant):
    client_specific_id = f"react-samples::{participant.name.replace(' ', '-')}-{str(uuid.uuid4())[0:7]}"
    payload = participant.dict()
    payload.update({"client_specific_id": client_specific_id})
    del payload['meeting_id']
    resp = await DYTE_API.post(f'/meetings/{meetingId}/participants', json=payload)
    if resp.status_code > 200:
        raise HTTPException(status_code=resp.status_code, detail=resp.text)
    return resp.text

if __name__ == "__main__":
    uvicorn.run("app:app", host="localhost", port=8000, log_level="debug", reload=True)

This code defines a ⚡️ FastAPI application with an endpoint /detect_faces, which takes in a base64 encoded image and returns a boolean value indicating if the face matches the reference image of the candidate.

It uses the face recognition library to detect faces in the image received.

The code also makes use of the Dyte API for meeting-related operations. 📹

We can start the backend server simply by using the following command 🧑‍💻:

python app.py

This Python server helps us create and join meetings, detect different faces, and get the list of suspicious candidates. 🕵️

Here, when we hit the /same_faces endpoint with an image file encoded as a base64 string, the different_faces_detected key of the response would be set to True if the face of the candidate from the reference image doesn't match the face in the screenshot, else it will be set to False.

We can call this from our frontend with the participant's webcam feed.

With this sorted, let's return to our React application and create our UI. ✨

Step 5: Setting up the Meeting UI

First, let us add our CSS file, create a new file, frontend/src/App.css and paste the following code.

.App {
    text-align: center;
}

.App-logo {
    height: 40vmin;
    pointer-events: none;
}

@media (prefers-reduced-motion: no-preference) {
    .App-logo {
        animation: App-logo-spin infinite 20s linear;
    }
}

.App-header {
    background-color: #282c34;
    min-height: 100vh;
    display: flex;
    flex-direction: column;
    align-items: center;
    justify-content: center;
    font-size: calc(10px + 2vmin);
    color: white;
}

.App-link {
    color: #61dafb;
}

.heading-proctor {
    font-size: x-large;
    font-weight: bolder;
    color: #fff;
}

@keyframes App-logo-spin {
    from {
        transform: rotate(0deg);
    }
    to {
        transform: rotate(360deg);
    }
}

.file-input label {
    margin-top: 20px;
    display: block;
    position: relative;
    width: 200px;
    height: 50px;
    border-radius: 10px;
    background-color: #2160fd;
    display: flex;
    align-items: center;
    justify-content: center;
    flex-direction: row;
    color: #fff;
    font-weight: bold;
    cursor: pointer;
    transition: transform 0.2s ease-out;
}

.file {
    opacity: 0;
    width: 0.1px;
    height: 0.1px;
    position: absolute;
}

Next, we will add the initial Dyte Meeting component to our app. We can do this by replacing the contents of frontend/src/App.jsx with the following code:

import { useEffect, useState } from "react";
import Home from "./Home";
import { BrowserRouter, Routes, Route, Link } from "react-router-dom";
import "./App.css";
import Stage from "./Stage";

const SERVER_URL = process.env.SERVER_URL || "http://localhost:8000";

function App() {
    const [meetingId, setMeetingId] = useState();

    const createMeeting = async () => {
        const res = await fetch(`${SERVER_URL}/meetings`, {
            method: "POST",
            body: JSON.stringify({ title: "Joint Entrance Examination" }),
            headers: { "Content-Type": "application/json" },
        });
        const resJson = await res.json();
        window.localStorage.setItem("adminId", resJson.admin_id);
        setMeetingId(resJson.data.id);
    };

    useEffect(() => {
        window.localStorage.removeItem("refImgUrl");
        const id = window.location.pathname.split("/")[2];
        if (!!!id) {
            createMeeting();
        }
    }, []);

    return (
        <BrowserRouter>
            <Routes>
                <Route path="/" element={<Home meetingId={meetingId} />}></Route>
                <Route path="/meeting/:meetingId" element={<Stage />}></Route>
            </Routes>
        </BrowserRouter>
    );
}

export default App;

This component will create a Dyte meeting link and an adminId for the admin. We will store the adminId secretly in localstorage. The adminId will be used later for accessing any sensitive data.

Home component

The home component renders the / route. Create a file as frontend/src/Home.jsx.

import { Link } from "react-router-dom";
function Home({ meetingId }) {
    return (
        <div
            style={{
                height: "100vh",
                width: "100vw",
                fontSize: "x-large",
                display: "flex",
                justifyContent: "center",
                alignItems: "center",
            }}
        >
            {meetingId && !window.location.pathname.split("/")[2] && (
                <Link to={`/meeting/${meetingId}`}>Create and Join Meeting</Link>
            )}
        </div>
    );
}

export default Home;

Heading component

Now we will create a file as frontend/src/Heading.jsx.

const Heading = ({ text }) => {
    return (
        <div
            className="heading-proctor"
            style={{
                padding: "10px",
                textAlign: "center",
                backgroundColor: "#000",
                borderBottom: "solid 0.5px gray",
            }}
        >
            {text}
        </div>
    );
};

export default Heading;

Let us now build the component to upload a reference image of the candidate.

Create a file frontend/src/ImageInput.jsx

import { useState } from "react";

function ImageUploader({ setImgUploadedStatus }) {
    const [selectedFile, setSelectedFile] = useState(null);
    const [uploading, setUploading] = useState(false);
    const [uploadError, setUploadError] = useState(null);
    const [uploadSuccess, setUploadSuccess] = useState(false);

    const handleFileChange = (event) => {
        setSelectedFile(event.target.files[0]);
        setUploadSuccess(false);
        setUploadError(null);
    };

    const handleSubmit = async (event) => {
        event.preventDefault();
        setUploading(true);
        setUploadError(null);

        try {
            const formData = new FormData();
            formData.append("image", selectedFile);

            const response = await fetch("https://api.imgur.com/3/image", {
                method: "POST",
                headers: {
                    Authorization: "Client-ID 48f0caef7256b40",
                },
                body: formData,
            });

            if (!response.ok) {
                throw new Error("Failed to upload image");
            }

            const data = await response.json();
            window.localStorage.setItem("refImgUrl", data.data.link);
            setUploadSuccess(data.data.link);
            setImgUploadedStatus(true);
        } catch (error) {
            setUploadError(error.message);
        } finally {
            setUploading(false);
        }
    };

    return (
        <div
            style={{
                display: "flex",
                justifyContent: "center",
                flexDirection: "column",
                alignItems: "center",
                fontSize: "x-large",
            }}
        >
            <div>Hey Candidate 👋</div>
            <div style={{ marginBottom: "30px" }}>
                Please upload a image of yours to proceed
            </div>
            <input
                type="file"
                id="file"
                style={{ border: "0.5px white solid" }}
                onChange={handleFileChange}
            />
            <form
                onSubmit={handleSubmit}
                style={{
                    display: "flex",
                    flexDirection: "column",
                    alignItems: "center",
                    marginTop: "1rem",
                }}
            >
                <button
                    type="submit"
                    disabled={!selectedFile || uploading || uploadSuccess}
                    style={{
                        marginTop: "10px",
                        padding: "0.5rem 1rem",
                        backgroundColor: "#3460f3",
                        color: "#fff",
                        border: "none",
                        borderRadius: "0.25rem",
                        cursor: "pointer",
                    }}
                >
                    {uploading ? "Uploading..." : "Upload"}
                </button>
            </form>
            {uploadError && (
                <div style={{ marginTop: "10px", color: "red" }}>{uploadError}</div>
            )}
        </div>
    );
}

export default ImageUploader;

It will look like this 👇

Let's create a staging area for participants joining the meeting. Admin bypasses the staging area, but the candidate will be asked to upload a reference image of himself on this page.

For this, we will create another file, frontend/src/Stage.jsx

import { useState, useEffect } from "react";
import ImageUploader from "./ImageInput";
import Meet from "./Meet";

const SERVER_URL = process.env.REACT_APP_SERVER_URL || "http://localhost:8000";

const Stage = () => {
    const [isAdminBool, setAdminBool] = useState(null);
    const [isRefImgUploaded, setImgUploadedStatus] = useState(null);
    const meetingId = window.location.pathname.split("/")[2];

    const isAdmin = async (id) => {
        const res = await fetch(`${SERVER_URL}/is_admin`, {
            method: "POST",
            body: JSON.stringify({
                admin_id: window.localStorage.getItem("adminId") || "",
                meeting_id: meetingId || "",
            }),
            headers: { "Content-Type": "application/json" },
        });
        const resJson = await res.json();
        setAdminBool(resJson.admin);
    };

    useEffect(() => {
        isAdmin();
    }, []);

    return (
        <div
            style={{
                height: "100vh",
                width: "100vw",
                display: "flex",
                justifyContent: "center",
                alignItems: "center",
                color: "white",
            }}
        >
            {isAdminBool == null ? (
                <>Loading...</>
            ) : (
                <>
                    {isAdminBool || (isAdminBool === false && isRefImgUploaded) ? (
                        <Meet isAdminBool={isAdminBool} />
                    ) : (
                        <ImageUploader setImgUploadedStatus={setImgUploadedStatus} />
                    )}
                </>
            )}
        </div>
    );
};

export default Stage;

Now, let's delve into the Meet component that renders on route /meeting/:meetingId.

When the admin clicks on the link provided on the / route, he gets redirected to the meeting page, where we add the user to the meeting as a participant with group_call_host preset. 🤙

Since this user created the meeting and was redirected to the meet page, we will assign him the admin role. Now the link from the address bar can be shared with the candidates.

When a candidate opens the shared link, they become a regular user. And for every regular user, the component emits screenshots of the users' videos to our directed to our Python server. 🐍

/* eslint-disable */

import { useState, useEffect, useRef } from "react";
import { DyteMeeting, provideDyteDesignSystem } from "@dytesdk/react-ui-kit";
import { useDyteClient } from "@dytesdk/react-web-core";
import Proctor from "./Proctor";
import Heading from "./Heading";
import { SendImageToBackendMiddleware, joinMeeting } from "./utils";
// Constants
const SERVER_URL = process.env.SERVER_URL || "http://localhost:8000";
let LAST_BACKEND_PING_TIME = 0;
const DETECT_FACES_ENDPOINT = `${SERVER_URL}/same_faces`;
const TIME_BETWEEN_BACKEND_PINGS = 10000;

const Meet = () => {
    const meetingEl = useRef();
    const [meeting, initMeeting] = useDyteClient();
    const [userToken, setUserToken] = useState();
    const [isAdminBool, setAdminBool] = useState(null);
    const meetingId = window.location.pathname.split("/")[2];

    function SendImageToBackendMiddleware() {
        return async (canvas, ctx) => {
            const currentTime = Date.now();
            if (currentTime - LAST_BACKEND_PING_TIME > TIME_BETWEEN_BACKEND_PINGS) {
                LAST_BACKEND_PING_TIME = currentTime;
                const imgBase64String = canvas.toDataURL("image/png");
                const response = await fetch(DETECT_FACES_ENDPOINT, {
                    method: "POST",
                    headers: {
                        "Content-Type": "application/json",
                    },
                    body: JSON.stringify({
                        base64_img: imgBase64String,
                        participant_id: meeting?.self.id,
                        participant_name: meeting?.self.name,
                        ref_image_url: window.localStorage.getItem("refImgUrl"),
                        meeting_id: meetingId,
                    }),
                });
            }
        };
    }

    const isAdmin = async (id) => {
        const res = await fetch(`${SERVER_URL}/is_admin`, {
            method: "POST",
            body: JSON.stringify({
                admin_id: window.localStorage.getItem("adminId") || "",
                meeting_id: meetingId || "",
            }),
            headers: { "Content-Type": "application/json" },
        });
        const resJson = await res.json();
        setAdminBool(resJson.admin);
    };

    const joinMeetingId = async () => {
        if (meetingId) {
            const authToken = await joinMeeting(meetingId);
            await initMeeting({
                authToken,
            });
            setUserToken(authToken);
        }
    };

    useEffect(() => {
        if (meetingId && !userToken) joinMeetingId();
        isAdmin();
    }, []);

    useEffect(() => {
        if (userToken) {
            provideDyteDesignSystem(meetingEl.current, {
                theme: "dark",
            });
        }
    }, [userToken]);

    useEffect(() => {
        if (isAdminBool === false && meeting?.self) {
            meeting.self.addVideoMiddleware(SendImageToBackendMiddleware);
        }
    }, [meeting?.self]);

    return (
        <div style={{ height: "96vh", width: "100vw", display: "flex" }}>
            {userToken && (
                <>
                    {isAdminBool && (
                        <div
                            style={{
                                width: "40vw",
                                height: "100vh",
                                overflowY: "scroll",
                                backgroundColor: "black",
                                borderRight: "solid 0.5px gray",
                            }}
                        >
                            <Heading text={"Proctoring Information"} />
                            <Proctor meeting={meeting} />
                        </div>
                    )}
                    {isAdminBool ? (
                        <div style={{ width: "60vw", height: "96vh" }}>
                            <Heading text={"Proctoring Admin Interface"} />
                            <DyteMeeting mode="fill" meeting={meeting} ref={meetingEl} />
                        </div>
                    ) : (
                        <div style={{ width: "100vw", height: "96vh" }}>
                            <Heading text={"Proctoring Candidate Interface"} />
                            <DyteMeeting mode="fill" meeting={meeting} ref={meetingEl} />
                        </div>
                    )}
                </>
            )}
        </div>
    );
};

export default Meet;

Let's briefly go through some of the functions:

isAdmin talks to the Python server to identify whether the current client is the admin.
joinMeeting adds the current client to the meeting.
SendImageToBackendMiddleware sends screenshots of candidates' videos to the Python server.

Proctor component

The proctor component gets activated only for admins. The proctor component, with the help of adminId fetches the list of suspicious candidates and renders it in a chat-like format. Create a file frontend/src/Proctor.jsx

import { useEffect, useState } from "react";
import { getCandidateStatus } from "./utils";

const Proctor = () => {
    const [candidateStatuses, updateCandidateStatusState] = useState([]);
    const [error, setError] = useState("");

    const updateCandidateStatus = async () => {
        try {
            const res = await getCandidateStatus();
            updateCandidateStatusState(res);
        } catch (e) {
            setError("User don't have admin privileges.");
        }
    };

    useEffect(() => {
        updateCandidateStatus();
    }, []);

    useEffect(() => {
        if (candidateStatuses?.map) {
            const id = setInterval(() => {
                updateCandidateStatus();
            }, 30000);
            return () => {
                clearInterval(id);
            };
        }
    }, [candidateStatuses]);

    return (
        <>
            <div style={{ padding: "0px 20px" }}>
                {candidateStatuses?.map && candidateStatuses ? (
                    candidateStatuses.map((status) => (
                        <div
                            style={{
                                display: "flex",
                                justifyContent: "start",
                                margin: "50px 20px",
                            }}
                        >
                            <div style={{ marginRight: "20px" }}>
                                <img
                                    src="https://images.yourstory.com/cs/images/companies/Dyte-1608553297314.jpg"
                                    style={{
                                        borderRadius: "50px",
                                        height: "60px",
                                        border: "1px double lightblue",
                                    }}
                                />
                            </div>
                            <div
                                style={{
                                    textAlign: "center",
                                    padding: "20px",
                                    backgroundColor: "#2160fd",
                                    fontSize: "large",
                                    fontWeight: "400",
                                    borderRadius: "10px 10px 10px 10px",
                                    width: "80%",
                                }}
                            >
                                <div
                                    style={{
                                        color: "white",
                                        padding: "20px 0px",
                                        textAlign: "left",
                                    }}
                                >
                                    {status[4].split("<>").map((text) => (
                                        <div>{text}</div>
                                    ))}
                                    <div>Timestamp: {new Date(status[0]).toLocaleString()}</div>
                                </div>
                                <img
                                    style={{ borderRadius: "10px", width: "100%" }}
                                    src={status[3]}
                                />
                            </div>
                        </div>
                    ))
                ) : (
                    <div style={{ color: "white" }}>
                        Wait or check if you have admin privileges to access the proctoring
                        dashboard.
                    </div>
                )}
            </div>
        </>
    );
};

export default Proctor;

Utility functions

Now we will add utility functions.

Create a file frontend/src/utils.js.

const joinMeeting = async (id) => {
    const res = await fetch(`http://localhost:8000/meetings/${id}/participants`, {
        method: "POST",
        body: JSON.stringify({
            name: "new user",
            preset_name: "group_call_host",
            meeting_id: meetingId,
        }),
        headers: { "Content-Type": "application/json" },
    });
    const resJson = await res.json();
    console.log(resJson.detail);
    const data = JSON.parse(resJson.detail);
    return data.data.token;
};

const getCandidateStatus = async () => {
    const response = await fetch("http://localhost:8000/suspicious_list", {
        method: "POST",
        headers: {
            "Content-Type": "application/json",
        },
        body: JSON.stringify({
            meeting_id: window.location.pathname.split("/")[2],
            admin_id: window.localStorage.getItem("adminId") || "undefined",
        }),
    });
    const res = await response.json();
    if (res.details) return undefined;
    console.log(res);
    return res;
};

export { joinMeeting, getCandidateStatus };

To start the React app on the local server, we can run the following command:

yarn start

Now, upon visiting http://localhost:3000/, we should be able to see the Dyte meeting in our browser.

Step 6: Adding the face detection logic to the frontend

Since now we have a nice backend server to detect faces and a great UI 🎉, we can add the face detection logic to our frontend. For this, we will first add some constants to our previously edited frontend/src/App.jsx file:

We will be using the above constants in the SendImageToBackendMiddleware function, which we will add to our App component just after the useDyteClient hook. 🪝

The SendImageToBackendMiddleware is a Dyte Video Middleware. Middlewares are add-ons that we can use to easily add effects and filters to your audio and video streams.

Here, we use the middleware functionality to get the canvas object of the participant's webcam feed, convert it to a base64 encoded image, and send it to our backend server.

That was all the code we needed to add basic proctoring functionality to our Dyte meeting. 👍

The app sends a screenshot of the participant's webcam feed to the backend server every 30 seconds. If the backend detects the face isn't similar to the reference image, it sends a warning notification to the proctor. ⚠️

The backend also logs the participant's ID and the time of the detection in the terminal. This can be used to keep track of the participants who may have cheated during the meeting for later review.

Step 7: Trying out our live proctoring system

Ta-da! 🎩✨ It's time to put our live proctoring system to the test and see it in action!

First, let's look at the candidate's view after uploading the reference image. The candidate can see that the proctor is in the meeting but cannot see the Proctoring Panel. 🧑‍💻

Now, let's swap the candidates.

In the proctor's view, we can see the details (proctoring information) along with proof when the candidate got swapped in the candidate's webcam view. 🙌

🧑‍💻 You can try out the live proctoring system here. And here's the link to the repository for you to take a look at the whole codebase.

Conclusion

Celebrate! 🎉✨ We've built a powerful live proctoring system with Dyte, ensuring integrity and fairness in online exams and interviews. But that's not all! We can now create our own customized online classroom or meeting platform.

We can now use this live proctoring system to proctor our online exams and interviews. ✍️

The possibilities are endless with Dyte, go ahead and try bringing your ideas to life by visiting dyte.io! 🚀

Building a Live Proctoring System to Detect Multiple Speakers

Ravindra Singh Rathor — Thu, 28 Dec 2023 19:44:13 +0000

Introduction

In the early days of online education, as the need arose to maintain the integrity of exams, proctoring emerged as a trusted monitoring method to prevent students from engaging in dishonest behavior. As the digital landscape expanded and online courses became more prevalent, the traditional ways of proctoring started to show their limitations.

Consider online exams, where having a physical proctor for every student becomes impractical. This gave rise to the adoption of live automatic proctoring.

This modern approach harnesses the power of a student’s webcam and microphone to oversee the exam process. By incorporating computer vision and machine learning, it efficiently detects cheating attempts.

In this tutorial, we’ll create a live proctoring system using Dyte APIs, enabling an admin to detect the presence of multiple speakers in a candidate’s background throughout the exam in real-time. ✨

High-Level Design of the application

Our aim is to notify the proctor if we hear multiple voices from the candidate’s background.

The proctor would get the candidate's details right in his meeting sidebar. 🔊✨

In this project, we will use React with Dyte UI kit and Dyte React Web Core packages for the frontend.
For the backend, we will use FastApi (Python 3).
We will also use database as a service by ElephantSQL (PostgreSQL).
Lastly, we will be using Deepgram Audio Diarization APIs to get speaker details from a sample audio clip.

Folder Structure

After completing the tutorial, the folder structure will look like this. 👇

dyte-proctoring/
├── frontend
│   ├── README.md
│   ├── package.json
│   ├── public
│   ├── src
│   │   ├── App.css
│   │   ├── App.jsx
│   │   ├── Heading.jsx
│   │   ├── ImageInput.jsx
│   │   ├── Meet.jsx
│   │   ├── Proctor.jsx
│   │   ├── index.css
│   │   ├── index.jsx
│   │   ├── logo.svg
│   │   ├── react-app-env.d.ts
│   │   ├── reportWebVitals.ts
│   │   ├── setupTests.ts
│   │   ├── stage.jsx
│   │   └── utils.js
│   ├── tsconfig.json
├── venv
├── app.py
├── audio_diarization.py
└── requirements.txt

Step 0: Configurations and Setup

🧑‍💻 Before building our live proctoring system, we must set up a Dyte account.

We can create a free account by clicking the "Start Building" button on Dyte.io and signing up using Google or GitHub. 🎉

Once signed up, we can access our Dyte API keys from the "API Keys" tab in the left sidebar. We will keep these keys secure as we will use them later. 🔑🤫

For our live proctoring system, we will use React for the frontend and FastAPI for building the Backend and APIs.

We will begin by creating a new directory for our project, called dyte-proctoring, and navigating into it using the following commands:

mkdir dyte-proctoring
cd dyte-proctoring

NOTE

We will require to create accounts on the following platforms:

ElephantSQL: Here is a step-by-step guide to create a db on ElephantSQL.
Deepgram: Here is a step-by-step guide to start using Deepgram.

Now back to the tutorial.

Step 1: Setting up the Frontend

Let's start setting up our frontend project using React and Dyte! ✨

We will create a boilerplate React app using create-react-app. We can do this with the following command:

yarn create react-app frontend

This will initialize a new React app in the frontend directory. 📁

Then, we will go ahead and install the dyte react-web-core, dyte react-ui-kit and react-router packages in this project using the following command 👇

yarn add @dytesdk/react-web-core @dytesdk/react-ui-kit react-router react-router-dom

Step 2: Setting up the backend

Let's get started with setting up our FastAPI backend now. 🙌

We will go back to the root directory of our project and initiate our project here itself for the ease of hosting:

cd ..

First of all, we will go ahead and create our requirements.txt file in the root directory itself with the following content 👇

requirements.txt

cmake
fastapi
uvicorn
face_recognition
numpy
python-multipart
psycopg2-binary
httpx
python-dotenv
pydantic
requests

After this, we will go ahead and create our virtual environment with venv and install the dependencies.

python -m venv venv
source venv/bin/activate # for linux/mac
venv\Scripts\activate.bat # for windows
pip install -r requirements.txt

We will also create an environment variable file .env, for storing our credentials.

.env

DYTE_ORG_ID=<ID>
DYTE_API_KEY=<KEY>
IMGUR_CLIENT_ID=<ID>
DB_USER=<ID>
DB_PASSWORD=<PASSWORD>
DB_HOST=<HOST>
DEEPGRAM_API_KEY=<API_KEY>

Step 3: Setting up our backend application

Now, we will create a new file named app.py and add our 🐘 ElephantSQL PostgreSQL database connection and code for our APIs, including face detection logic.

In this file, we would need to create the following routes:

GET / - Root route

POST /is_admin/ - Check if the user is an admin

POST /multiple_voices_list/ - This route retrieves a list of participants who are detected as suspicious

POST /multiple_voices/ - Detect if there are multiple voices in the audio stream

POST /meetings - Create a new meeting

POST /meetings/{meetingId}/participants - This route is responsible for adding a participant to a specific meeting identified by meetingId

So let’s get started 👇

app.py

import base64
import io
import logging
import random
import requests

import uvicorn
from fastapi import FastAPI, Form, UploadFile, File
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
import psycopg2
from utils import upload_audio

import os
import base64
from fastapi import FastAPI, HTTPException
from dotenv import load_dotenv
from httpx import AsyncClient
import httpx
import uuid

from audio_diarization import speakers_count

load_dotenv()

DYTE_API_KEY = os.getenv("DYTE_API_KEY")
DYTE_ORG_ID = os.getenv("DYTE_ORG_ID")

API_HASH = base64.b64encode(f"{DYTE_ORG_ID}:{DYTE_API_KEY}".encode('utf-8')).decode('utf-8')

timeout = httpx.Timeout(10.0, read=None)
DYTE_API = AsyncClient(base_url='https://api.cluster.dyte.in/v2', headers={'Authorization': f"Basic {API_HASH}"}, timeout=timeout)

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

fh = logging.FileHandler("app.log")
fh.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
fh.setFormatter(formatter)
logger.addHandler(fh)


class ProctorPayload(BaseModel):
    meeting_id: str
    admin_id: str

class AdminProp(BaseModel):
    meeting_id: str
    admin_id: str

class Meeting(BaseModel):
    title: str

class Participant(BaseModel):
    name: str
    preset_name: str
    meeting_id: str

origins = [
    # allow all
    "*",
]

app = FastAPI()

# enable cors
app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],  # allow all
    allow_headers=["*"],  # allow all
)

def connect_to_db():
    conn = psycopg2.connect(
            dbname=os.getenv('DB_USER'),
            user=os.getenv('DB_USER'),
            password=os.getenv('DB_PASSWORD'),
            host=os.getenv('DB_HOST'),
            port=5432
    )
    return conn

@app.get("/")
async def root():
    return {"message": "Hello World"}

@app.post("/is_admin/")
async def multiple_faces_list(admin: AdminProp):
    conn = connect_to_db()
    cur = conn.cursor()
    cur.execute("SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s", (admin.meeting_id, admin.admin_id,))

    count = cur.fetchone()[0]

    if(count > 0):
        return { "admin": True }
    else:
        return { "admin": False }

@app.post("/multiple_voices_list/")
async def multiple_faces_list(meeting: ProctorPayload):
    conn = connect_to_db()
    cur = conn.cursor()
    cur.execute("SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s", (meeting.meeting_id, meeting.admin_id,))

    count = cur.fetchone()[0]

    if(count > 0):
        cur.execute("CREATE TABLE IF NOT EXISTS meeting_audio_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), audio_url VARCHAR(255), verdict VARCHAR(255))")
        cur.execute("SELECT * FROM meeting_audio_proc_details WHERE meeting_id = %s ORDER BY ts DESC", (meeting.meeting_id,))
        rows = cur.fetchall()
        conn.commit()
        cur.close()
        conn.close()
        return rows
    else:
        conn.commit()
        cur.close()
        conn.close()
        raise HTTPException(status_code=401, detail="Participant dose not has admin role")

@app.post("/multiple_voices/")
async def multiple_voices(file: UploadFile = File(...), meeting_id: str = Form(...), participant_id: str = Form(...), participant_name: str = Form(...)):
    contents = file.file.read()
    filename = file.filename.split('.')[0] + str(random.randint(1, 100)) + '.mp3'
    with open(filename, 'wb') as f:
        f.write(contents)
    file.file.close()

    conn = connect_to_db()
    cur = conn.cursor()

    cur.execute("CREATE TABLE IF NOT EXISTS meeting_audio_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), audio_url VARCHAR(255), verdict VARCHAR(255))")

    try:
        count = speakers_count(filename)
    except Exception as e:
        logger.error(e)

    if count > 1:
        logger.info(
            f"Detected different voices for participant {participant_id}"
        )

        verdict = f"Participant Name: {participant_name} <> Anomaly: Different Voices Detected <> Participant ID: {participant_id}"
        cur.execute("SELECT count(1) FROM meeting_audio_proc_details WHERE meeting_id=%s AND participant_id=%s AND ts >= (current_timestamp - INTERVAL '10 minutes')", (meeting_id, participant_id))
        count = cur.fetchone()[0]

        if count == 0:
            # if there is a requirement of storing audio files
            upload_resp = await upload_audio(img_data)
            cur.execute("INSERT INTO meeting_audio_proc_details (ts, meeting_id, participant_id, audio_url, verdict) VALUES (current_timestamp, %s, %s, %s, %s)",
                (meeting_id, participant_id, upload_resp, verdict)
            )

        conn.commit()
        cur.close()
        conn.close()

        if count == 0:
            return { "id": participant_id, "different_voices_detected": True, "url": upload_resp }
        return { "id": participant_id, "different_voices_detected": True, "url": "not uploaded" }

    return {"id": participant_id, "different_voices_detected": False}

@app.post("/meetings")
async def create_meeting(meeting: Meeting):
    response = await DYTE_API.post('/meetings', json=meeting.dict())
    if response.status_code >= 300:
        raise HTTPException(status_code=response.status_code, detail=response.text)
    admin_id = ''.join(random.choices('abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789', k=32))
    resp_json = response.json()
    resp_json['admin_id'] = admin_id
    meeting_id = resp_json['data']['id']

    conn = connect_to_db()
    cur = conn.cursor()
    cur.execute("INSERT INTO meeting_host_info (ts, meeting_id, admin_id) VALUES (CURRENT_TIMESTAMP, %s, %s)", (meeting_id, admin_id))
    conn.commit()
    cur.close()
    conn.close()

    return resp_json


@app.post("/meetings/{meetingId}/participants")
async def add_participant(meetingId: str, participant: Participant):
    client_specific_id = f"react-samples::{participant.name.replace(' ', '-')}-{str(uuid.uuid4())[0:7]}"
    payload = participant.dict()
    payload.update({"client_specific_id": client_specific_id})
    del payload['meeting_id']
    resp = await DYTE_API.post(f'/meetings/{meetingId}/participants', json=payload)
    if resp.status_code > 200:
        raise HTTPException(status_code=resp.status_code, detail=resp.text)
    return resp.text

if __name__ == "__main__":
    uvicorn.run("app:app", host="localhost", port=8000, log_level="debug", reload=True)

This code defines a ⚡️ FastAPI application with an endpoint /multiple_voices, which takes in an audio file in binary format and returns a boolean value indicating if there are multiple voices in the audio submitted.

It uses the audio_diarization api by Deepgram API service to detect multiple voices in the audio received.

audio_diarization.py

import os
from httpx import AsyncClient
import requests
from dotenv import load_dotenv

load_dotenv()


DEEPGRAM_API_KEY = os.getenv("DEEPGRAM_API_KEY")
DEEPGRAM_API = AsyncClient(base_url='https://api.deepgram.com/v1', headers={'Authorization': f"Token {DEEPGRAM_API_KEY}", 'Content-Type': "audio/mp3"})

def speakers_count(audio_file_path):
    speaker_set = set()
    audio = open(audio_file_path, 'rb')
    response = requests.post('https://api.deepgram.com/v1/listen?diarize=true', data=audio, headers={'Authorization': f"Token {DEEPGRAM_API_KEY}", 'Content-Type': "audio/mp3"})
    resp_json = response.json()
    statements = resp_json['results']['channels'][0]['alternatives']
    for statement in statements:
        words = statement['words']
        for word in words:
            speaker_set.add(word['speaker'])
    return len(speaker_set)

The code also makes use of the Dyte API for meeting-related operations. 📹

We can start the backend server simply by using the following command:

python app.py

This Python server helps us create and join meetings, detect different faces, and get the list of suspicious candidates. 🕵️

Here, when we hit the /multiple_voices endpoint with an audio file, the multiple_voices_detected key of the response would be set to True if there are multiple voices in the audio file received, else it will be set to False.

We can call this from our frontend with the participant’s mic.

With this sorted, let's return to our React application and create our UI. ✨

Step 4: Setting up the Meeting UI

First, let us add our CSS file, create a new file frontend/src/App.css and paste the following code.

.App {
    text-align: center;
}

.App-logo {
    height: 40vmin;
    pointer-events: none;
}

@media (prefers-reduced-motion: no-preference) {
    .App-logo {
        animation: App-logo-spin infinite 20s linear;
    }
}

.App-header {
    background-color: #282c34;
    min-height: 100vh;
    display: flex;
    flex-direction: column;
    align-items: center;
    justify-content: center;
    font-size: calc(10px + 2vmin);
    color: white;
}

.App-link {
    color: #61dafb;
}

.heading-proctor {
    font-size: x-large;
    font-weight: bolder;
    color: #fff;
}

@keyframes App-logo-spin {
    from {
        transform: rotate(0deg);
    }
    to {
        transform: rotate(360deg);
    }
}

.file-input label {
    margin-top: 20px;
    display: block;
    position: relative;
    width: 200px;
    height: 50px;
    border-radius: 10px;
    background-color: #2160fd;
    display: flex;
    align-items: center;
    justify-content: center;
    flex-direction: row;
    color: #fff;
    font-weight: bold;
    cursor: pointer;
    transition: transform 0.2s ease-out;
}

.file {
    opacity: 0;
    width: 0.1px;
    height: 0.1px;
    position: absolute;
}

Next, we will add the initial Dyte Meeting component to our app. We can do this by replacing the contents of frontend/src/App.jsx with the following code:

import { useEffect, useState } from "react";
import Home from "./Home";
import { BrowserRouter, Routes, Route, Link } from "react-router-dom";
import "./App.css";
import Stage from "./Stage";

const SERVER_URL = process.env.REACT_APP_SERVER_URL || "http://localhost:8000";

function App() {
    const [meetingId, setMeetingId] = useState();

    const createMeeting = async () => {
        const res = await fetch(`${SERVER_URL}/meetings`, {
            method: "POST",
            body: JSON.stringify({ title: "Joint Entrance Examination" }),
            headers: { "Content-Type": "application/json" },
        });
        const resJson = await res.json();
        window.localStorage.setItem("adminId", resJson.admin_id);
        setMeetingId(resJson.data.id);
    };

    useEffect(() => {
        window.localStorage.removeItem("refImgUrl");
        const id = window.location.pathname.split("/")[2];
        if (!!!id) {
            createMeeting();
        }
    }, []);

    return (
        <BrowserRouter>
            <Routes>
                <Route path="/" element={<Home meetingId={meetingId} />}></Route>
                <Route path="/meeting/:meetingId" element={<Stage />}></Route>
            </Routes>
        </BrowserRouter>
    );
}

export default App;

This component will create a Dyte meeting link and an adminId for the admin. We will store the adminId secretly in localstorage. The adminId will be used later for accessing any sensitive data.

Home component

The home component renders the / route. Create a file as frontend/src/Home.jsx.

import { Link } from "react-router-dom";
function Home({ meetingId }) {
    return (
        <div
            style={{
                height: "100vh",
                width: "100vw",
                fontSize: "x-large",
                display: "flex",
                justifyContent: "center",
                alignItems: "center",
            }}
        >
            {meetingId && !window.location.pathname.split("/")[2] && (
                <Link to={`/meeting/${meetingId}`}>Create and Join Meeting</Link>
            )}
        </div>
    );
}

export default Home;

Heading component

Now we will create a file as frontend/src/Heading.jsx.

const Heading = ({ text }) => {
    return (
        <div
            className="heading-proctor"
            style={{
                padding: "10px",
                textAlign: "center",
                backgroundColor: "#000",
                borderBottom: "solid 0.5px gray",
            }}
        >
            {text}
        </div>
    );
};

export default Heading;

Let’s create a staging area for participants joining the meeting. Admin bypasses the staging area, but the candidate will be asked to upload a reference image of himself on this page.

For this, we will create another file, frontend/src/Stage.jsx

import { useState, useEffect } from "react";
import Meet from "./Meet";

const SERVER_URL = process.env.REACT_APP_SERVER_URL || "http://localhost:8000";

const Stage = () => {
    const [isAdminBool, setAdminBool] = useState(null);
    const meetingId = window.location.pathname.split("/")[2];

    const isAdmin = async (id) => {
        const res = await fetch(`${SERVER_URL}/is_admin`, {
            method: "POST",
            body: JSON.stringify({
                admin_id: window.localStorage.getItem("adminId") || "",
                meeting_id: meetingId || "",
            }),
            headers: { "Content-Type": "application/json" },
        });
        const resJson = await res.json();
        setAdminBool(resJson.admin);
    };

    useEffect(() => {
        isAdmin();
    }, []);

    return (
        <div
            style={{
                height: "100vh",
                width: "100vw",
                display: "flex",
                justifyContent: "center",
                alignItems: "center",
                color: "white",
            }}
        >
            {isAdminBool == null ? (
                <>Loading...</>
            ) : (
                <>
                    <Meet isAdminBool={isAdminBool} />
                </>
            )}
        </div>
    );
};

export default Stage;

Now, let's delve into the Meet component that renders on route /meeting/:meetingId.

When the admin clicks on the link provided on the / route, he gets redirected to the meeting page, where we add the user to the meeting as a participant with audio_proc_preset preset. 🤙

Since this user created the meeting and was redirected to the meet page, we will assign him the admin role. Now the link from the address bar can be shared with the candidates.

When a candidate opens the shared link, they become a regular user. And for every regular user, the component emits screenshots of the users' videos to our directed to our Python server. 🐍

/* eslint-disable */

import { useState, useEffect, useRef } from "react";
import { DyteMeeting, provideDyteDesignSystem } from "@dytesdk/react-ui-kit";
import { useDyteClient } from "@dytesdk/react-web-core";
import Proctor from "./Proctor";
import Heading from "./Heading";
import { joinMeeting } from "./utils";
import lamejs from "lamejstmp";

// Constants
const SERVER_URL = process.env.REACT_APP_SERVER_URL || "http://localhost:8000";
let LAST_BACKEND_PING_TIME = 0;
const TIME_BETWEEN_BACKEND_PINGS = 60000;

function convertFloat32ToInt16(buffer) {
    var l = buffer.length;
    var buf = new Int16Array(l);
    while (l--) {
        buf[l] = Math.min(1, buffer[l]) * 0x7fff;
    }
    return buf;
}

const Meet = () => {
    const meetingEl = useRef();
    const [meeting, initMeeting] = useDyteClient();
    const [userToken, setUserToken] = useState();
    const [isAdminBool, setAdminBool] = useState(null);
    const meetingId = window.location.pathname.split("/")[2];

    async function audioToMp3Middleware(audioContext) {
        const processor = audioContext.createScriptProcessor(1024, 1, 1);
        const encoder = new lamejs.Mp3Encoder(1, 44100, 128);
        const bufferSize = 512;
        let mp3Data = [];

        processor.onaudioprocess = (e) => {
            const inputData = e.inputBuffer.getChannelData(0);
            const mp3buf = encoder.encodeBuffer(convertFloat32ToInt16(inputData));
            if (mp3buf.length > 0) {
                mp3Data.push(mp3buf);
            }
            const currentTime = Date.now();
            if (
                mp3Data.length >= bufferSize &&
                currentTime - LAST_BACKEND_PING_TIME > TIME_BETWEEN_BACKEND_PINGS
            ) {
                LAST_BACKEND_PING_TIME = currentTime;
                const file = new File(mp3Data, "audio.mp3");
                const form = new FormData();
                form.append("file", file);
                form.append("meeting_id", meetingId);
                form.append("participant_id", meeting?.self.id);
                form.append("participant_name", meeting?.self.name);

                fetch(`${SERVER_URL}/multiple_voices`, {
                    method: "POST",
                    body: form,
                });

                mp3Data = [];
            }
        };

        return processor;
    }

    const isAdmin = async (id) => {
        const res = await fetch(`${SERVER_URL}/is_admin`, {
            method: "POST",
            body: JSON.stringify({
                admin_id: window.localStorage.getItem("adminId") || "",
                meeting_id: meetingId || "",
            }),
            headers: { "Content-Type": "application/json" },
        });
        const resJson = await res.json();
        setAdminBool(resJson.admin);
    };

    const joinMeetingId = async () => {
        if (meetingId) {
            const authToken = await joinMeeting(meetingId);
            await initMeeting({
                authToken,
            });
            setUserToken(authToken);
        }
    };

    useEffect(() => {
        if (meetingId && !userToken) joinMeetingId();
        isAdmin();
    }, []);

    useEffect(() => {
        if (userToken) {
            provideDyteDesignSystem(meetingEl.current, {
                theme: "dark",
            });
        }
    }, [userToken]);

    useEffect(() => {
        if (isAdminBool === false && meeting?.self) {
            meeting.self.addAudioMiddleware(audioToMp3Middleware);
        }

        return () => {
            if (isAdminBool === false && meeting?.self) {
                meeting.self.removeAudioMiddleware(audioToMp3Middleware);
            }
        };
    }, [meeting?.self, isAdminBool]);

    return (
        <div style={{ height: "96vh", width: "100vw", display: "flex" }}>
            {userToken && (
                <>
                    {isAdminBool && (
                        <div
                            style={{
                                width: "40vw",
                                height: "100vh",
                                overflowY: "scroll",
                                backgroundColor: "black",
                                borderRight: "solid 0.5px gray",
                            }}
                        >
                            <Heading text={"Proctoring Information"} />
                            <Proctor />
                        </div>
                    )}
                    {isAdminBool ? (
                        <div style={{ width: "60vw", height: "96vh" }}>
                            <Heading text={"Proctoring Admin Interface"} />
                            <DyteMeeting mode="fill" meeting={meeting} ref={meetingEl} />
                        </div>
                    ) : (
                        <div style={{ width: "100vw", height: "96vh" }}>
                            <Heading text={"Proctoring Candidate Interface"} />
                            <DyteMeeting mode="fill" meeting={meeting} ref={meetingEl} />
                        </div>
                    )}
                </>
            )}
        </div>
    );
};

export default Meet;

Let’s briefly go through some of the functions:

isAdmin talks to the Python server to identify whether the current client is the admin.
joinMeeting adds the current client to the meeting.
audioToMp3Middleware sends audio samples of candidates to the Python server.

Proctor component

The proctor component gets activated only for admins. The proctor component, with the help of adminId fetches the list of suspicious candidates and renders it in a chat-like format.

Create a file frontend/src/Proctor.jsx.

import { useEffect, useState } from "react";
import { getCandidateStatus } from "./utils";

const Proctor = () => {
    const [candidateStatuses, updateCandidateStatusState] = useState([]);
    const [error, setError] = useState("");

    const updateCandidateStatus = async () => {
        try {
            const res = await getCandidateStatus();
            updateCandidateStatusState(res);
        } catch (e) {
            setError("User don't have admin privileges.");
        }
    };

    useEffect(() => {
        if (candidateStatuses?.map) {
            const id = setInterval(() => {
                updateCandidateStatus();
            }, 30000);
            return () => {
                clearInterval(id);
            };
        }
    }, [candidateStatuses]);

    return (
        <>
            <div style={{ padding: "0px 20px" }}>
                {candidateStatuses?.map && candidateStatuses ? (
                    candidateStatuses.map((status) => (
                        <div
                            style={{
                                display: "flex",
                                justifyContent: "start",
                                margin: "50px 20px",
                            }}
                        >
                            <div style={{ marginRight: "20px" }}>
                                <img
                                    src="https://images.yourstory.com/cs/images/companies/Dyte-1608553297314.jpg"
                                    style={{
                                        borderRadius: "50px",
                                        height: "60px",
                                        border: "1px double lightblue",
                                    }}
                                />
                            </div>
                            <div
                                style={{
                                    textAlign: "center",
                                    padding: "20px",
                                    backgroundColor: "#2160fd",
                                    fontSize: "large",
                                    fontWeight: "400",
                                    borderRadius: "10px 10px 10px 10px",
                                    width: "80%",
                                }}
                            >
                                <div
                                    style={{
                                        color: "white",
                                        padding: "20px 0px",
                                        textAlign: "left",
                                    }}
                                >
                                    {status[4].split("<>").map((text) => (
                                        <div>{text}</div>
                                    ))}
                                    <div>Timestamp: {new Date(status[0]).toLocaleString()}</div>
                                </div>
                            </div>
                        </div>
                    ))
                ) : (
                    <div style={{ color: "white" }}>
                        Wait or check if you have admin privileges to access the proctoring
                        dashboard.
                    </div>
                )}
            </div>
        </>
    );
};

export default Proctor;

Utility functions

Now we will add utility functions.

Create a file frontend/src/utils.js.

const SERVER_URL = process.env.REACT_APP_SERVER_URL || "http://localhost:8000";

const joinMeeting = async (id) => {
    const res = await fetch(`${SERVER_URL}/meetings/${id}/participants`, {
        method: "POST",
        body: JSON.stringify({
            name: "new user",
            preset_name: "audio_proc_preset",
            meeting_id: id,
        }),
        headers: { "Content-Type": "application/json" },
    });
    const resJson = await res.json();
    const data = JSON.parse(resJson.detail);
    return data.data.token;
};

const getCandidateStatus = async () => {
    const response = await fetch(`${SERVER_URL}/multiple_voices_list`, {
        method: "POST",
        headers: {
            "Content-Type": "application/json",
        },
        body: JSON.stringify({
            meeting_id: window.location.pathname.split("/")[2],
            admin_id: window.localStorage.getItem("adminId") || "undefined",
        }),
    });
    const res = await response.json();
    if (res.details) return undefined;
    return res;
};

const FloatArray2Int16 = (floatbuffer) => {
    var int16Buffer = new Int16Array(floatbuffer.length);
    for (var i = 0, len = floatbuffer.length; i < len; i++) {
        if (floatbuffer[i] < 0) {
            int16Buffer[i] = 0x8000 * floatbuffer[i];
        } else {
            int16Buffer[i] = 0x7fff * floatbuffer[i];
        }
    }
    return int16Buffer;
};

export { joinMeeting, getCandidateStatus, FloatArray2Int16 };

To start the React app on the local server, we can run the following command:

yarn start

Now, upon visiting http://localhost:3000/, we should be able to see the Dyte meeting in our browser.

Step 5: Adding multiple voice detection logic to the Frontend

Since now we have a nice backend server to detect multiple voices and a great UI, we can add the multiple voice detection logic to our frontend. For this, we will first add some constants to our previously edited frontend/src/App.jsx file:

We will be using the above constants in the audioToMp3Middleware function, which we will add to our Meet component.

The audioToMp3Middleware is a Dyte Audio Middleware. Middlewares are add-ons that we can use to add effects and filters to your audio and video streams with ease.

Here, we are using the middleware functionality to get the audio sample from the participant’s mic and send it to our backend server. We are also ensuring that the backend is pinged only once every 60 seconds to avoid unnecessary load on the server.

That was all the code we needed to add basic proctoring functionality to our Dyte meeting. 👍

The app sends an audio sample from the participant’s mic to the backend server every 60 seconds, and if the backend detects multiple voices in the audio sample, it sends a warning notification to the proctor. ⚠️

The backend also logs the participant's ID and the time of the detection in the terminal. This can be used to keep track of the participants who may have cheated during the meeting for later review.

Step 6: Trying out our project

Ta-da! 🎩✨ It’s time to put our proctoring system to the test and see it in action!

First, let's look at the candidate's view after uploading the reference image. The candidate can see that the proctor is in the meeting but cannot see the Proctoring Panel. 🧑‍💻

In the proctor’s view, we can see the details (proctoring information) along with proof when two or more people can be heard talking in the candidate’s background. 🙌

Here's the link to the repository for you to take a look at the whole codebase. You can also try out the project here. 🔗

Conclusion

We can now use this live proctoring system to proctor our online exams and interviews. ✍️

The possibilities are endless with Dyte; go ahead and try to bring your ideas to life by visiting dyte.io! 🚀

Render Video Tracks From WebRTC Using Flutter PlatformViews

Aman Kumar — Mon, 16 Oct 2023 11:24:31 +0000

Flutter, unlike native Android, iOS, or even React Native apps, does not use system drawing primitives for rendering your application. This blog will give you all the theory of how, what, and why of using Flutter PlatformViews and how WebRTC(lib) VideoTracks can be rendered in your Flutter applications using PlatformViews.

Background

Before we start our discussion on how PlatformViews are rendered, let's discuss how Flutter generally draws its UI.

Flutter paints its UI from scratch every time using its graphics engine, Impeller. Flutter draws every pixel on the screen, giving developers a high degree of control over the UI.

Flutter uses 3 threads to render its UIs.

UI thread
Platform thread
Raster thread

The UI thread is where your Dart code runs. The Platform thread is responsible for creating widgets, calculating the layout, and other layout-related tasks. Once this is done, the layout tree is delegated to the Raster thread, which converts the tree into actual pixels on the screen.

// Dart code runs on the UI thread
void main() {
  runApp(MyApp());
}

Let’s see how this fits with our use case of making video available from the native side to Flutter.

WebRTC connection

There are multiple ways through which you can render your libwebrtc video tracks on Flutter, like for Android, you can

Use SurfaceTextureRenderer to render the VideoTrack, make the Texture available to Flutter using TextureRegistry API, and render the Texture in Flutter.
Use SurfaceViewRenderer to render the VideoTrack as an AndroidView, and use PlatformViews to use the native view in Flutter.

In this blog are going to talk about how the 2nd approach works under the hood. Specifically, what and why of PlatformViews.

What are Flutter PlatformViews?

PlatformViews are used when we want to use native views as a Flutter widget. There are two ways we can create PlatformView in Flutter.

1. Hybrid composition

In Hybrid composition, Flutter creates a special type of view and asks Android/iOS to create a corresponding view and embeds the native view into its own widget tree.

Below is how you can implement PlatformViews by using the Hybrid composition method -

Widget build(BuildContext context) {
  // This is used in the platform side to register the view.
  const String viewType = '<platform-view-type>';
  // Pass parameters to the platform side.
  const Map<String, dynamic> creationParams = <String, dynamic>{};

  return PlatformViewLink(
    viewType: viewType,
    surfaceFactory:
        (context, controller) {
      return AndroidViewSurface(
        controller: controller as AndroidViewController,
        gestureRecognizers: const <Factory<OneSequenceGestureRecognizer>>{},
        hitTestBehavior: PlatformViewHitTestBehavior.opaque,
      );
    },
    onCreatePlatformView: (params) {
      return PlatformViewsService.initSurfaceAndroidView(
        id: params.id,
        viewType: viewType,
        layoutDirection: TextDirection.ltr,
        creationParams: creationParams,
        creationParamsCodec: const StandardMessageCodec(),
        onFocus: () {
          params.onFocusChanged(true);
        },
      )
        ..addOnPlatformViewCreatedListener(params.onPlatformViewCreated)
        ..create();
    },
  );
}

Here's a breakdown of what each part does:

1. Variables initialization:

viewType is a unique identifier for the native view. This should match the identifier used in the native Android code.
creationParams is a map that can hold any parameters you want to pass to the native view for its initialization.

2. PlatformViewLink widget:

This widget serves as a bridge between the Flutter framework and the native view. It takes in the viewType and two factory functions: surfaceFactory and onCreatePlatformView.

3. surfaceFactory function:

This function returns an AndroidViewSurface widget, which is responsible for displaying the native Android view.
It takes a controller argument, an instance of AndroidViewController, which is used to control the native Android view.
gestureRecognizers specifies which gestures the native view should consume. In this example, it's set to an empty set, meaning the native view won't consume any gestures.
hitTestBehavior is set to PlatformViewHitTestBehavior.opaque, which means the native view will block touches to underlying Flutter widgets.

4. onCreatePlatformView function:

This function initializes the native Android view and returns an instance of it.
It uses PlatformViewsService.initSurfaceAndroidView to initialize the native view, passing in various parameters like id, viewType, and creationParams.
An onFocus callback is also defined, which is triggered when the native view gains focus.
A listener is added to notify when the Android view is created successfully.

By using this build method in your Flutter app, you can seamlessly integrate a native Android view into your Flutter widget tree.

2. Virtual display mode

In virtual display mode, Flutter creates an offscreen android.view.Surface, which is a drawing surface to render graphics. It’s like a blank canvas. Then the native view renders its content onto this surface. Flutter then takes the content of that surface and uses it as a texture within its rendering pipeline. This texture is drawn where the platform view should appear.

iOS only supports Hybrid composition.

Below is how you can implement virtual display mode.

Widget build(BuildContext context) {
  // This is used in the platform side to register the view.
  const String viewType = '<platform-view-type>';
  // Pass parameters to the platform side.
  final Map<String, dynamic> creationParams = <String, dynamic>{};

  return AndroidView(
    viewType: viewType,
    layoutDirection: TextDirection.ltr,
    creationParams: creationParams,
    creationParamsCodec: const StandardMessageCodec(),
  );
}

AndroidView widget:

This widget is used to display the native Android view within the Flutter app.
It takes several parameters:
viewType specifies the type of Android view to create.
layoutDirection sets the text direction, which is left-to-right (TextDirection.ltr) in this example.
creationParams are the initial parameters to pass to the Android view.
creationParamsCodec specifies how to encode creationParams. The StandardMessageCodec is used for encoding basic types like strings, numbers, and collections.

Native implementation

On the native side, we need to create the view that we need to serve, create a factory, and register it so that the native platform can create views whenever it is asked by Flutter.

To do this in Android we do -

class NativeViewFactory : PlatformViewFactory(StandardMessageCodec.INSTANCE) {
    override fun create(context: Context, viewId: Int, args: Any?): PlatformView {
        val creationParams = args as Map<String?, Any?>?
        return NativeView(context, viewId, creationParams)
    }
}

Next, we have to register this factory in the flutter engine -

binding.platformViewRegistry
                .registerViewFactory("<platform-view-type>", NativeViewFactory())

Performance Impact

Before Android 10, hybrid composition required a lot of to and fro between main memory and GPU to compose native views with flutter widgets, after Android 10, copying is done only once, which makes it very performant.

Before Android 10

After Android 10:

Here is a small table that can help you decide which method to choose:

What did we choose at Dyte?

At Dyte, our Flutter SDK is a wrapper over the Android and iOS SDKs, and we don't load any WebRTC video/audio streams directly. We use VideoView from our Android SDK to display the video, which is then served as a PlatformView. We prefer a hybrid composition due to its performance benefits.

Challenges

We have two Flutter SDKs: dyte core and dyte uikit. Dyte core has no knowledge of where the VideoView (which is a PlatformView) is going to be used. It might be possible that the VideoView for a single participant is used on multiple screens. Let’s call this VideoView as FlutterVideoView since VideoView is also present in our Android SDK and is a crucial part of this discussion.

We wrote a PlatformView, which serves VideoView. Let’s call it AndroidVideoView. Dyte core is dependent on our core mobile SDKs and VideoView is a View in Android.

We know that one View can be part of only one ViewGroup at any moment of time.

Let’s keep this in the back of the mind that Flutter has no direct support to detect the visibility of a widget.

The problem we were facing was when we used FlutterVideoView for the same participant on two different screens. For the sake of simplicity, let’s call it screen A and screen B. Now when the user navigates from screen A to screen B everything was working smoothly as screen B was creating a fresh FlutterVideoView, which internally created a new instance of AndroidVideoView, so the VideoView is now attached to a new ViewGroup.

The problem was when the user navigated back from screen B to screen A, there was no way to detect if FlutterVideoView widget had gone from background to foreground so we could call render() on the AndroidVideoView, which is a PlatformView in our Flutter SDK. render() method calls the render() of VideoView, which internally takes care of removing the view from the old ViewGroup and renders the video track.

On the Dyte UI kit, we can get the lifecycle methods if the screen has been changed, but that would not solve the actual problem of the view not getting refreshed on its own. To solve this, we used a community plugin, [visibility_detector](https://pub.dev/packages/visibility_detector) in Dyte core flutter SDK. It gave us a widget that had callbacks that get triggered when its child widget’s visibility is changed.

Now the second part of the problem was to find out the PlatformView, which was associated with the widget and call render() on it. To tackle this we had to retrieve the View from the Flutter engine since all the PlatformViews are created by the Flutter engine itself.

To tackle this, we cached the FlutterEngine using FlutterEngineCache. This allowed us to access the PlatformView by its viewId.

FlutterEngineCache.getInstance().put("DyteFlutterEngine", flutterPluginBinding.flutterEngine)

Through the engine, we can access the PlatformView with the help of viewId, which is assigned to every PlatformView when it is created.

onCreatePlatformView: (params) {
        return PlatformViewsService.initSurfaceAndroidView(
         id: params.id,
         viewType: viewType,
         layoutDirection: TextDirection.ltr,
         creationParams: creationParams,
         reationParamsCodec: const StandardMessageCodec(),
         onFocus: () {
             params.onFocusChanged(true);
          },
        )
                // Here we assign the view id.
          ..addOnPlatformViewCreatedListener(setNativeViewId)
          ..create();
      },

To access the native view, we did

val targetView = FlutterEngineCache.getInstance()["DyteFlutterEngine"]!!
                .platformViewsController.getPlatformViewById(viewId) as AndroidVideoView?

Finally, we call the render() method on this view.

targetView?.render()

After all these steps, we clear the cached FlutterEngine to free up the memory. This helped us solve a critical issue in our Flutter video SDK.

Conclusion

This concludes our take on Flutter PlatformViews and how we use it at Dyte. Video is the most essential part of our Flutter SDK, and the Flutter ecosystem has provided us with great utilities to deal with it.

On top of it, we have made it seamless for you to have feature-rich audio video conferencing in your app with the least hassle. Stay tuned for more engineering blogs, we talk about behind the scenes, WebRTC, and cool things that can be built upon Dyte. Check out our blog here.

Building an Async Survey Tool With Dyte and React

Vaibhav Shinde — Thu, 12 Oct 2023 21:04:59 +0000

In today's fast-paced digital landscape, gathering valuable insights from your audience is crucial for making informed decisions. Traditional surveys have limitations, often causing respondents to drop out due to lengthy forms or slow-loading pages. This is where async survey tools come to the rescue. These innovative tools leverage asynchronous communication to create a more engaging and efficient survey-taking experience. In this blog post, we'll explore the world of async survey tools, shedding light on what they are and why you should consider building one.

Why build an async survey tool?

Asynchronous survey tools offer a refreshing data collection approach, revolutionizing how we interact with respondents. Here are some compelling reasons why you should consider embarking on the journey of building your very own async survey tool:

Enhanced user experience: Traditional surveys can be tedious, leading to survey fatigue and dropouts. Async survey tools allow respondents to answer questions independently, reducing survey abandonment rates and ensuring a positive user experience.
Increased response rates: By accommodating respondents' schedules and allowing them to participate when convenient, async surveys are more likely to garner higher response rates than their synchronous counterparts.
Global reach: Async surveys transcend time zones and geographical boundaries. They empower you to collect data from a diverse global audience, making it ideal for research with broader demographics.
Data quality: Respondents can provide well-thought-out responses with asynchronous surveys, leading to higher-quality data. This is especially valuable when conducting in-depth research or gathering detailed feedback.
Flexibility and customization: Building your async survey tool gives you complete control over the design, structure, and features. You can tailor it to meet the specific needs of your research, ensuring it aligns perfectly with your objectives.
Seamless integration: Integrating your async survey tool with other data collection and analysis tools is more straightforward than ever, enabling you to streamline your research process.
Competitive advantage: In a world where innovation is vital, having your async survey tool sets you apart from competitors. It demonstrates your commitment to delivering a superior user experience and gathering accurate data.

In the following sections of this blog post, we'll delve deeper into the technical aspects of building an asynchronous survey tool, exploring key features, design considerations, and best practices to ensure success in your survey endeavors.

Before you start

This guide is written assuming you have basic HTML, CSS, and JS knowledge, along with React.
You will need an API Key and Organisation ID from dev.dyte.io to perform operations with Dyte's REST APIs. Check out its documentation at docs.dyte.io/api. To know more about getting started and learning basic concepts about Dyte, read this guide: docs.dyte.io/getting-started.

Application flow

User is taken to a screen where they are asked a question. To answer, they have to start the recording. A minimum duration of 15 seconds would be required for the recording and a maximum of 60 seconds, after which the recording will stop automatically, and the survey will be completed. The user will be shown visual guidelines to center their face in the video and told to speak up if their volume is low.

Understanding the code

Follow along with the code walkthrough as you read through the source here.

In the source, we've set up a React app with Vite, and this is the main <App /> component that automatically initializes and joins a Dyte meeting.

export default function Meeting() {
  const [meeting, initMeeting] = useDyteClient();

  useEffect(() => {
      // use "light" theme
    provideDyteDesignSystem(document.body, {
      theme: 'light',
    });

    const searchParams = new URLSearchParams(window.location.search);
      // for testing we get authToken from URL search params
      // but in prod, you would get this as a response
      // from one of your backend APIs
    const authToken = searchParams.get('authToken');

    if (!authToken) {
      alert(
        "An authToken wasn't passed, please pass an authToken in the URL query to join a meeting.",
      );
      return;
    }

    initClient({
      authToken,
    }).then((meeting) => {
      // automatically join room
      meeting?.joinRoom();
    });
  }, []);

  return (
    <DyteProvider
      value={meeting}
      fallback={
        <div className="w-full h-full flex flex-col gap-4 place-items-center justify-center">
          <DyteSpinner className="w-14 h-14 text-blue-500" />
          <p className="text-xl font-semibold">Starting Dyte Video Survey</p>
        </div>
      }
    >
      <Recorder />
    </DyteProvider>
  );
}

We're using the useDyteClient() hook to initialize a meeting object via the initClient() method. It accepts the authToken value returned by the Add Participant API.

We then use the <DyteProvider /> component and pass the meeting object to it that can be consumed by custom hooks. We also provided a fallback UI to render until the meeting object is initialized.

If you are unfamiliar with generating the authToken, check out our 'Getting Started' guide.

Now, let's move on to the main logic in the <Recorder /> component.

function Recorder() {
  const { meeting } = useDyteMeeting();

  // timestamp value from `meeting` object is stored
  const [timestamp, setTimestamp] = useState<Date>();
  // global states values emitted from components is stored
  const [UIStates, setUIStates] = useState<any>({});

  // derived from `timestamp` value
  const [duration, setDuration] = useState(0);
  // state to disable recording button as per logic
  const [recordingDisabled, setRecordingDisabled] = useState(false);

  // error feedback states to be shown to the user
  const [errors, dispatchError] = useReducer(errorReducer, []);
  // custom hook to calculate brightness and silence from
  // video and audio tracks
  useBrightnessAndSilenceDetector(dispatchError);

  useEffect(() => {
    // calculate duration from recording timestamp
    if (timestamp) {
      const interval = setInterval(() => {
        const duration = getElapsedDuration(timestamp);
        setDuration(duration);
      }, 500);
      return () => {
        clearInterval(interval);
      };
    }
  }, [timestamp]);

  useEffect(() => {
    const onRecordingUpdate = (state: string) => {
      switch (state) {
        case 'RECORDING':
          setTimestamp(new Date());
          break;
        case 'STOPPING':
          setTimestamp(undefined);
          break;
      }
    };

    meeting.recording.addListener('recordingUpdate', onRecordingUpdate);
    return () => {
      meeting.recording.removeListener('recordingUpdate', onRecordingUpdate);
    };
  }, []);

  useEffect(() => {
    // stop recording when you reach max duration of 60 seconds
    if (duration >= 60) {
      meeting.recording.stop();
      setRecordingDisabled(false);
    }
  }, [duration]);

  return (
    <div className="w-full h-full flex place-items-center justify-center p-4 flex-col">
      <div className="max-w-4xl pb-8">
        <h3 className="text-xl font-bold mb-4">
          Have you worked with any of the following technologies: JavaScript
          Core, Web Assembly, Protobufs?{' '}
        </h3>
        <div className="mb-2">
          List out the ones you have experience in and pick 1 to elaborate. If
          you haven't worked with any of these technologies, pick 2-3 skills
          mentioned in the job description to describe instead. Here are some
          tips to help you record a great video:
        </div>
        <li>Please provide as much detail as you can</li>
        <li>Use your webcam or mobile camera to record your video response</li>
        <li>Make sure you have plenty of light so we can clearly see you</li>
        <li>
          Avoid places with lots of background noise so we can clearly hear you
        </li>
      </div>
      <div className="flex flex-col w-full max-w-lg border rounded-xl overflow-clip">
        <div className="relative">
          <DyteParticipantTile
            participant={meeting.self}
            className="w-full h-auto rounded-none aspect-[3/2] bg-zinc-300"
          />
          <p className="text-white bg-purple-950 p-3 text-xs text-center">
            {/* Show okay message, or last error message */}
            {errors.length === 0
              ? messages['ok']
              : messages[errors[errors.length - 1]]}
          </p>
          {/* Show placement container only when recording hasn't started */}
          {!timestamp && (
            <div className="absolute w-44 z-50 left-1/2 -translate-x-1/2 top-1/2 -translate-y-28 aspect-square border-2 border-dashed border-pink-50 rounded-lg" />
          )}
        </div>
        {/* Duration indicator */}
        <Duration duration={duration} />

        <div className="flex items-center justify-center p-2">
          <DyteRecordingToggle
            meeting={meeting}
            disabled={(timestamp && duration <= 15) || recordingDisabled}
          />
          <DyteSettingsToggle
            onDyteStateUpdate={(e) => setUIStates(e.detail)}
          />
        </div>
      </div>

      <DyteDialogManager
        states={UIStates}
        meeting={meeting}
        onDyteStateUpdate={(e) => setUIStates(e.detail)}
      />
    </div>
  );
}

Breaking down the code:

The timestamp state value is set from the meeting recording API in a useEffect containing the recordingUpdate event listeners.
UIStates state value gets updated whenever components like <DyteDialogManager /> or <DyteSettingsToggle /> emit global UI state updates.
We derive the duration from the timestamp value in a useEffect.
The state recordingDisabled is set as per the logic we defined before.
To render the UI, we use the DyteParticipantTile, DyteSettingsToggle, DyteRecordingToggle and DyteDialogManager components from Dyte's React UI Kit. Note that DyteSettingsToggle emits a state whenever you toggle it, and the DyteDialogManager shows the DyteSettings component accordingly.
Note how we disable the DyteRecordingToggle button as per the duration state.
We also render a custom Duration component showing the progress of the recording duration along with our constraints.
The useBrightnessAndSilenceDetector() hook updates the errors state that gets shown to the user.

The messages object contains a pre-defined list of strings for each error state:

const messages = {
  ok: 'Ensure your head and shoulders are in shot. Hit record when you are ready.',
  not_bright:
    'You seem to be in a dark room, please try turning on the lights.',
  not_loud: 'Your voice is not loud enough. Please speak loud and clearly.',
};

Coming to the brightness and silence detector hook useBrightnessAndSilenceDetector(). Its code looks like this:

function useBrightnessAndSilenceDetector(
  dispatchError: Dispatch<Parameters<typeof errorReducer>[1]>,
) {
  const { meeting } = useDyteMeeting();
  const videoEnabled = useDyteSelector((m) => m.self.videoEnabled);
  const audioEnabled = useDyteSelector((m) => m.self.audioEnabled);

  useEffect(() => {
    const { audioTrack } = meeting.self;
    if (!audioTrack || !audioEnabled) return;

    const stream = new MediaStream();
    stream.addTrack(audioTrack);
    const audioContext = new AudioContext();
    audioContext.resume();
    const analyserNode = audioContext.createAnalyser();
    analyserNode.fftSize = 2048;
    const micSource = audioContext.createMediaStreamSource(stream);
    micSource.connect(analyserNode);
    const bufferLength = 2048;
    const dataArray = new Float32Array(bufferLength);
    const silenceThreshold = 0.05;
    const segmentLength = 1024;

    function getRMS(
      dataArray: Float32Array,
      startIndex: number,
      endIndex: number,
    ) {
      let sum = 0;
      for (let i = startIndex; i < endIndex; i++) {
        sum += dataArray[i] * dataArray[i];
      }
      const mean = sum / (endIndex - startIndex);
      const rms = Math.sqrt(mean);
      return rms;
    }

    function detectSilence() {
      analyserNode.getFloatTimeDomainData(dataArray);
      const numSegments = Math.floor(bufferLength / segmentLength);
      for (let i = 0; i < numSegments; i++) {
        const startIndex = i * segmentLength;
        const endIndex = (i + 1) * segmentLength;
        const rms = getRMS(dataArray, startIndex, endIndex);
        if (rms > silenceThreshold) {
          // Detected non-silence in this segment
          return false;
        }
      }
      // Detected silence
      return true;
    }

    const interval = setInterval(() => {
      const isSilent = detectSilence();

      if (isSilent) {
        dispatchError({ type: 'add', error: 'not_loud' });
      } else {
        dispatchError({ type: 'remove', error: 'not_loud' });
      }
    }, 1000);

    return () => {
      clearInterval(interval);
      dispatchError({ type: 'remove', error: 'not_loud' });
    };
  }, [audioEnabled]);

  useEffect(() => {
    if (!videoEnabled) return;

    const { videoTrack } = meeting.self;
    if (!videoTrack) return;
    const videoStream = new MediaStream();
    videoStream.addTrack(videoTrack);
    const video = document.createElement('video');
    video.style.width = '240px';
    video.style.height = '180px';
    video.muted = true;
    video.srcObject = videoStream;
    video.play();
    const canvas = document.createElement('canvas');
    canvas.width = 240;
    canvas.height = 180;
    const ctx = canvas.getContext('2d', { willReadFrequently: true })!;

    const interval = setInterval(() => {
      const brightness = getBrightness(video, canvas, ctx);
      if (brightness < 0.4) {
        dispatchError({ type: 'add', error: 'not_bright' });
      } else {
        dispatchError({ type: 'remove', error: 'not_bright' });
      }
    }, 1000);

    return () => {
      clearInterval(interval);
      dispatchError({ type: 'remove', error: 'not_bright' });
    };
  }, [videoEnabled]);

  return null;
}

It takes the audioTrack and videoTrack objects from the meeting object and then processes it to detect silence/low volume and low brightness, respectively, and dispatch the errors for our UI to show.

Finally, let's take a look at the Duration component and the ProgressBar component.

function ProgressBar({
  value,
  max,
}) {
  return (
    <div className="h-1 w-full bg-zinc-200 relative">
      <div
        className="max-w-[25%] h-full bg-red-500 z-10 absolute left-0 top-0"
        style={{
          width: Math.min((value / max) * 100, 25) + '%',
          transition: 'all 0.3s',
        }}
      ></div>
      <div
        className="max-w-full h-full z-0 bg-black absolute left-0 top-0"
        style={{
          width: Math.min((value / max) * 100, 100) + '%',
          transition: 'all 0.3s',
        }}
      ></div>
    </div>
  );
}

function Duration({ duration }) {
  return (
    <div className="flex flex-col gap-1 bg-zinc-50 p-2 text-center">
      <div className="relative">
        <ProgressBar value={duration} max={60} />
        <div className="z-10 bg-red-500 absolute left-1/4 h-full w-1 top-0"></div>
        <div className="z-10 bg-black absolute right-0 h-full w-1 top-0"></div>
      </div>
      <div className="text-xs flex items-center justify-between">
        <div>
          Min recording time{' '}
          <span className="font-semibold text-red-500">15s</span>
        </div>
        <div>
          Max recording time <span className="font-semibold">60s</span>
        </div>
      </div>
    </div>
  );
}

We wrote a custom UI with the help of TailwindCSS to render the Duration and ProgressBar UI that looks like this.

Now, the user can start answering the question after hitting the record.

Conclusion

With this, we have built an async survey tool, changing your data collection approach and modernizing your interactions with respondents. We have broken down the code, rigorously tested it, and fine-tuned the interface to ensure user-friendliness, making it prime and polished for deployment. So, now you can get on and elevate your survey-taking experience.

While on the subject of async tools, check out this blog we wrote on creating an async interview platform!

If you have any thoughts or feedback, please get in touch with me on LinkedIn and Twitter. Stay tuned for more related blog posts in the future!

Get better insights on leveraging Dyte's technology and discover how it can revolutionize your app's communication capabilities with its__ SDKs__. Head over to dyte.io to learn how to start quickly on your 10,000 free minutes, which renew every month. You can reach us at support@dyte.io or ask our developer community if you have any questions.

Migrate From Twilio Video Using Dyte's Twilio Shim

Rohan Mukherjee — Thu, 12 Oct 2023 15:34:24 +0000

Our business has been fortunate enough to have garnered the interest of several customers. These customers have expressed their satisfaction with our products and services, which we take great pride in providing.

Dyte has been gaining popularity among customers who require reliable cloud communication services. Many of these customers, who were initially using Twilio Video, have been making the switch to Dyte, possibly owing to our call quality, reliability, expeditious customer support, and pricing.

This guide presents an approach to help users smoothly migrate from Twilio Video to Dyte with the help of Shims.

Thinking of migrating from Twilio Video?

Since you are reading this page, I understand that you are probably considering a switch from Twilio Video to another video provider. In recent times, a lot of our customers migrated from Twilio Video to Dyte. They made this choice due to Dyte's compelling advantages, which include:

Easy to use APIs with high-level abstractions and inbuilt access control - fewer bugs and less maintenance required
UI components library - responsive components for React, Angular, WebComponents, React Native, Flutter, Native Android, and iOS. Build real-time applications faster.
Better media performance - with Dyte's global mesh network of media servers, calls spanning multiple geographies see a significantly better experience.
Better support - dedicated Slack support, code reviews, and technical assistance 24/7.

You can view a comprehensive feature comparison between Dyte and Twilio Video.

Once you've reviewed the reasons behind migrating from Twilio Video, it becomes evident that Dyte will likely meet all your requirements!

However, transitioning from one SDK to another can be a demanding process, considering the time and effort involved. Will migrating from Twilio Video to Dyte entail a similar level of effort? The answer is - No.

Enter Dyte's Twilio Shim, which simplifies the migration.

Introducing our Shim library

Even though Dyte can be easily integrated (don't believe me? Check out our docs) into your codebase, some developers may find the process cumbersome and time-consuming as they have to get rid of their old code and refactor it to fit a new SDK.

To make the lives of these developers easier, we created a shim library - @dytesdk/twilio-shim.

You may be wondering, what is a shim? According to Wikipedia:

In computer programming, a shim is a library that transparently intercepts API calls and changes the arguments passed, handles the operation itself or redirects the operation elsewhere. Shims can be used to support an old API in a newer environment, or a new API in an older environment.

Basically, we created a wrapper around Dyte's SDK with a Twilio Video-compatible external API as a drop-in replacement for the Twilio Video SDK. You can use the same classes, methods, and other APIs that you used with Twilio Video, but behind the scenes, the shim will perform some magic and translate it to Dyte API calls.

For instance, if you were to connect to a room using Twilio Video and login to the console whenever a new participant joined the room, this is what your code would look like:

const { connect } = require('twilio-video');

const authToken = '<AUTH_TOKEN_FROM_TWILIO_API>';

const room = await connect(authToken, {audio: false, video: true});
console.log(`Successfully joined a Room: ${room.name}`);
console.log(room);
room.on('participantConnected', participant => {
    console.log(`A remote Participant connected: ${participant}`);
});

The following code snippet demonstrates how you would do it with the shim:

const { connect } = require('@dytesdk/twilio-shim');

const authToken = '<AUTH_TOKEN_FROM_DYTE_API>';

const room = await connect(authToken, {audio: false, video: true});
console.log(`Successfully joined a Room: ${room.name}`);
console.log(room);
room.on('participantConnected', participant => {
    console.log(`A remote Participant connected: ${participant}`);
});

Can you spot the difference?

As mentioned previously, the external API is almost identical to Twilio Video. However, intended actions, such as joining a room, are performed internally using Dyte's web-core SDK.

You can check out more examples of using our Twilio shim here.

Here is the complete API compatibility list of methods and classes we currently translate

https://docs.dyte.io/guides/migration/twilio/compatibility-shim-twilio

Also, another important note is that this library should just be used to test out the integration, and clients should plan to eventually move to our SDK natively to utilize Dyte's capabilities fully

That's all, folks!

That was a quick and easy way to switch from Twilio Video - just like integrating Dyte into your project. We are working on creating migration guides for other commonly used real-time communication APIs, such as live streaming and AI-powered chat, among others. If you would like us to create a migration guide from your current video provider, you can contact us on our community Discord.

If you haven't heard about Dyte yet, head over to dyte.io to learn how we are revolutionizing communication through our SDKs and libraries and how you can get started quickly on your 10,000 free minutes, which renew every month. You can reach us at support@dyte.io or ask our developer community if you have any questions.

Add Transcription to Your Video Calls With AWS Transcribe

Ravindra Singh Rathor — Tue, 10 Oct 2023 06:21:08 +0000

With the rise of remote job opportunities and geographically dispersed teams, effective communication is more crucial than ever. It is essential that there are no gaps in understanding among team members from diverse locations, ensuring everyone feels included and valued.

In a previous blog, we explored the seamless integration of an audio transcriptions and translation solution using Google STT (Speech to Text) in conjunction with Dyte. This powerful combination showcased its potential to foster improved communication among teams that are geographically dispersed and culturally diverse in modern settings, catering to various races, ethnicities, and cultural backgrounds.

Now, we have devised an AWS (Amazon Web Services) based solution that overcomes not only the barriers between people but also those between cloud providers. This new solution offers efficient transcriptions and translations, further enhancing cross-cultural collaboration and communication. Azure, you're up next for our exploration!

In this blog, we will discuss the integration of a third-party service from Amazon Web Services named AWS Transcribe with Dyte. As the participants speak, it will add transcription to audio/video meetings. These transcriptions, with the help of AWS translate, another service from AWS, can then be converted to the desired language of individual participants, as needed.

As usual, there are prerequisites.

Prerequisites

Before we dive into the process, it's important to note that you should have a Dyte meeting already created to add a transcription of the audio of the meeting's participants.

Here are some useful resources for your quick navigation:

Make sure you've read the Getting Started with Dyte topic and completed the following steps:

Create a Dyte Developer Account
Create Presets
Create a Dyte Meeting
Add Participant to the meeting
Integrate with the SDK of your choice

Assuming you have successfully integrated Dyte into your website or application, let's dive into the process of adding transcriptions in the Dyte meetings.

Adding transcription components

To achieve the audio transcriptions, we need to call AWS (Amazon Web Services) Services: AWS Transcribe & AWS Translate.

These AWS services are paid. Please note that Dyte does NOT act as a broker between AWS and any of our clients that wants to integrate AWS services with Dyte SDK. We recommend that our clients should directly connect with the AWS team.

However, we at Dyte, provide free of cost code samples so that our clients can seamlessly integrate with such services.

Integration Steps

1. Setup AWS IAM account, get credentials

To use AWS services, you should either be an IAM user, or your backend services should be deployed using roles & policies.

For ease of use, we are going ahead with IAM user credentials. Please ensure that the IAM user can actually use AWS transcribe & AWS translate services.

To proceed further, we need accessKeyId, secretAccessKey & the preferred region where you would want your services to be deployed in the future.

Once you have these, let’s proceed to the next step.

2. Setup a backend server to call these APIs securely

If you have used AWS services before, you would know that these services are costly, they can charge you a lot. Obviously, you would not want to expose your credentials for miscreants to exploit.

If you put these credentials by any chance in your frontend code, anyone would be able to get a hold of this and land you with a huge sum of charge. Also, if you expose an endpoint that is not performing checks to validate users and blindly providing these services to anyone with an API endpoint link, it would be a potential security risk.

Therefore we will not expose these credentials. We will put them in your backend services for safer usage. This would give you better control to ensure that no unauthorized personnel can use your APIs & secrets.

For this, we have provided the BE sample in NodeJS for you to see. Please find it here. Currently, we only have an ExpressJS sample, if you working on a different backend, feel free to port this code or connect with us so that we can help you out in porting it.

To use this sample, Please clone this using the following command.

git clone git@github.com:dyte-in/aws-transcribe.git

Or simply download it as a Zip. Click on Code (Green Icon → Download ZIP) and extract once downloaded.

Now that you have downloaded the repository. Please proceed with the following steps to set up your backend endpoint.

2.1 Go to the server folder

cd server

2.2 Replicate .env.example as .env

cp .env.example .env

Open the .env in your choice of Text File Editor. Edit it as per your AWS service account credentials and Save it.

2.3 Install NPM packages

This would install @aws-sdk/client-translate automatically. Instead of @aws-sdk/client-transcribe-streaming, we will be using WebSocket with presigned urls.

npm install

2.4 Run the server

npm run dev

If successful, you would see the confirmation in Terminal that it is running on localhost:3001 or the PORT specified in the .env file.

This endpoint (https://localhost:3001) will be called the backend_url from now onwards. We advise you to copy the code from https://github.com/dyte-in/aws-transcribe/blob/main/server/src/index.ts and alter it as per your need and security practices.

We strongly advise you not to use this code as is, without a proper security mechanism, for production.

3. Integrate the SDK in the Front end project

Now that the backend setup is ready let’s start with the frontend setup.

3.1 Install the FE package in your FE codebase

The first action would be to install Dyte’s wrapper package to take out complexity from you in front-end integration using:

npm install @dytesdk/aws-transcribe

if you are interested in knowing what this package does behind the scene, and this is something we recommend you to check out, refer to https://github.com/dyte-in/aws-transcribe/tree/main/client so that you can further customize the code to your liking. This is necessary as well so that you can put proper security measures. It is advised to copy-paste the files of the client folder and alter them instead of using this package directly in production.

3.2. Integrate the AWS Transcribe SDK with Dyte Meeting

The second step is to look for the place in your codebase where you are initiating a Dyte meeting.

Once you have found the place and got a hold of the meeting object. Add the following code on top of the file to import the FE SDK.

import DyteAWSTranscribe from '@dytesdk/aws-transcribe';

Add the following code just after the point where you have access to the meeting object.

const awsTranscribe = new DyteAWSTranscribe({
        meeting,
    target: 'hi', // Optional if translate is false, Supported languages: https://docs.aws.amazon.com/translate/latest/dg/what-is-languages.html
    translate: true, // Control whether to translate the source language to target language or just transcribe
    source: 'en-US', // Supported languages: https://docs.aws.amazon.com/translate/latest/dg/what-is-languages.html
    preSignedUrlEndpoint: 'http://localhost:3001/aws-transcribe-presigned-url',
    translationEndpoint: 'http://localhost:3001/translate', // ${backend_url}/translate. backend_url is from step 2.4
});

awsTranscribe.on('transcription', async (data) => {
    // ... do something with transcription
});

awsTranscribe.transcribe();

Here you are setting up the DyteAWSTranscribe with the values that the current user would prefer and activating the recognition just afterward using awsTranscribe.transcribe(). Then we are listening to every new transcription using awsTranscribe.on(’transcription’, aJsCallbackFunction)

To see the support languages, please refer to https://docs.aws.amazon.com/transcribe/latest/dg/supported-languages.html and https://docs.aws.amazon.com/translate/latest/dg/what-is-languages.html.

With this, now you would be able to receive the live transcriptions. Feel free to put them in UI as per your need.

If you need a sample of how to do it, please refer to https://github.com/dyte-in/aws-transcribe/blob/main/client/demo/index.ts.

<!DOCTYPE html>
<html lang="en">
   <head>
      <meta charset="UTF-8" />
      <meta http-equiv="X-UA-Compatible" content="IE=edge" />
      <meta name="viewport" content="width=device-width, initial-scale=1.0" />
      <title>Google Transcriptions Demo </title>
      <!-- UI Kit -->
      <script type="module">
         import { defineCustomElements } from 'https://cdn.jsdelivr.net/npm/@dytesdk/ui-kit/loader/index.es2017.js';
         defineCustomElements();
      </script>
      <!-- Web Core -->
      <script src="https://cdn.dyte.in/core/dyte.js"></script>
      <script src="https://cdn.jsdelivr.net/npm/@dytesdk/aws-transcribe@0.0.1/dist/index.umd.js"></script>
      <style>
         body {
         height: 100vh;
         width: 100vw;
         }
         #dyte-transcriptions{
         position: absolute;
         z-index: 99999;
         bottom: 15%;
         width: 100%;
         display:flex;
         flex-direction: column;
         justify-content: center;
         align-items:center;
         }
         .dyte-transcription-line{
         display: block;
         max-width: 80%;
         text-align: center !important;
         }
         .dyte-transcription-speaker{
         font-weight: 500;
         color: orange;
         }
         .dyte-transcription-text{
         color: white;
         }
      </style>
   </head>
   <body>
      <dyte-meeting id="my-meeting"></dyte-meeting>
      <div id="dyte-transcriptions"></div>

      <script>
         const roomName = ''; // Put roomName if using v1 APIs. Leave Blank for v2 APIs
         const authToken = ''; // Put v1 or v2 participant auth token

         async function initMeeting() {
           const meetingEl = document.getElementById('my-meeting');

           const meeting = await DyteClient.init({
             authToken,
             roomName,
             defaults: {
               audio: false,
               video: false,
             },
           });

           meetingEl.meeting = meeting;

           const awsTranscribe = new DyteAWSTranscribe({
               target: 'hi', // Optional if translate is false, Supported languages: Supported languages: https://docs.aws.amazon.com/translate/latest/dg/what-is-languages.html
                           translate: true, // Control whether to translate the source language to target language or just transcribe
                         source: 'en-US', // Supported languages: https://docs.aws.amazon.com/transcribe/latest/dg/supported-languages.html
                         preSignedUrlEndpoint: 'http://localhost:3001/aws-transcribe-presigned-url',
                         translationEndpoint: 'http://localhost:3001/translate',
           });


           awsTranscribe.on('transcription', async () => {
               const transcription = document.getElementById('dyte-transcriptions');
               const list = awsTranscribe.transcriptions.slice(-3);
               transcription.innerHTML = '';
               list.forEach((item) => {
                   const speaker = document.createElement('span');
                   speaker.classList.add('dyte-transcription-speaker');
                   speaker.innerText = `${item.name}: `;

                   const text = document.createElement('span');
                   text.classList.add('dyte-transcription-text');
                   text.innerText = item.transcript.trim() !== '' ? item.transcript : '...';

                   const container = document.createElement('span');
                   container.classList.add('dyte-transcription-line');
                   container.appendChild(speaker);
                   container.appendChild(text);

                   transcription.appendChild(container);
               });
           });

           awsTranscribe.transcribe();
         }

         window.onload = async function () {
           initMeeting();
         };
      </script>
   </body>
</html>

That’s it. You have successfully added transcriptions and custom language translations to your video calling app using AWS Transcribe.

I hope you found this post informative and engaging. If you have any thoughts or feedback, please get in touch with me on Twitter or LinkedIn. Stay tuned for more related blog posts in the future!

If you haven't heard about Dyte yet, head over to dyte.io to learn how we are revolutionizing communication through our SDKs and libraries and how you can get started quickly on your 10,000 free minutes, which renew every month. If you have any questions, you can reach us at support@dyte.io or ask our developer community.

Packaging Libraries in iOS: A Comprehensive Guide v2

Shaunak Jagtap — Mon, 09 Oct 2023 07:05:45 +0000

Understanding the premise

Software Development Kits (SDKs) are the lifeblood of development, offering a treasure of pre-packaged libraries, tools, and resources that empower developers to craft rich and feature-packed applications. However, for those new to distributing SDKs in iOS, the path can be filled with uncertainty.

We have shipped 100s of iOS SDK builds in the form of our WebRTC clients for customers looking to integrate real time communications within their applications. With that experience, this technical blog post is tailored for iOS developers and aims to illuminate the nuances of SDK distribution, including key concepts, best practices, and tools. With this knowledge, developers can unlock the full potential of their SDKs and ensure seamless integration experiences.

Static vs. dynamic libraries and frameworks in iOS

Seasoned iOS developers often grapple with crucial decisions regarding system frameworks, packaging their code, and integrating third-party components. Among these decisions, choosing between static and dynamic libraries or frameworks is pivotal, making profound implications for application performance and resource management.

Implications on app size and launch time

The choice between static and dynamic libraries or frameworks carries significant weight regarding your app's binary size and launch time.

Summary of static Vs. dynamic linking

Here's a concise summary of how static and dynamic linking impacts various facets of your application:

tr td:first-child { min-width: 8em; width: 8em; }

Facets	Static Linking	Dynamic Linking
App Size	Large app bundle size	Smaller app bundle size
App Launch Time	Faster load time	Slower load time
Safety	Scrutinised and copied at build time	Risk of runtime glitches, potential for runtime crashes
Deployment	In static linking, the SDK or library is bundled within the single app binary, and the entire library's code becomes an integral part of the app's binary executable.	App references the library at runtime, and the operating system is responsible for loading the required library when the app is launched or when the specific library functions are first called
Debugging	Easier to debug as all code is available	Harder to debug as code may not be available at runtime
Memory Usage	apps that are statically linked tend to have less memory usage during runtime.	may have slightly more memory usage during runtime
Flexibility	Updates to the SDK or library require app recompilation and release because the entire library is bundled with the app's binary during compilation, preventing separate updates.	Library updates can be made independently of the app, offering flexibility and allowing users to benefit from updates without app recompilation.

When to use dynamic linking

While statically linked modules proffer a smaller app size and accelerated loading times, dynamic linking has its own set of compelling use cases:

Multiple static modules depending on the same module: When your app comprises multiple static modules that lean on a common module, you might encounter warnings about duplicate symbols at runtime. Transitioning the shared module to a dynamic one can effectively mitigate this issue.
iOS increased app launch time with many dynamic libraries/frameworks: Loading numerous third-party dynamic libraries or frameworks on iOS can lead to prolonged app launch times. Vigilant monitoring and optimisation efforts are essential to upholding app launch performance.

Static/dynamic with different integration techniques

To make well-informed decisions about linking, a profound understanding of how various dependency managers handle linking behaviour is imperative:

Own targets or projects linked directly: Exert precise control over the linking behaviour of targets within your repository or external repositories by fine-tuning Build Settings. A simple adjustment to the MACH_O_TYPE Build Setting allows you to toggle between static library and dynamic library.
CocoaPods: By default, CocoaPods constructs and links dependencies as static libraries. The introduction of use_frameworks! in your Podfile enables the construction of dynamic frameworks. Moreover, :linkage => :static can be employed to shape dependencies as static frameworks.
Swift Package Manager (SPM): SPM, by default, fabricates dependencies as static libraries, offering minimal control over linking behaviour. However, if you are the custodian of a package, you can specify type: .dynamic within your Package.swift file to fashion a dynamic package.
Carthage: In its default configuration, Carthage leans towards using dynamic frameworks for dependencies. Nevertheless, you retain the flexibility to configure it to construct and link them statically when circumstances dictate such an arrangement.

Understanding different formats

In the realm of iOS development, many formats for packaging libraries and resources exist. Each format serves distinct purposes, and as an expert iOS developer, comprehending their nuances is indispensable. Here's an exploration of some essential formats:

xcframework

XCFramework is a relatively recent addition to Apple's arsenal of formats. It is a versatile container meticulously designed for packaging frameworks for diverse platforms and architectures into a single, harmonious bundle. Embracing XCFrameworks streamlines the distribution of binary frameworks, ensuring harmonious coexistence across various Apple devices and processor architectures. Adopting XCFrameworks bestows developers with the gift of streamlined development, reduced integration complexities, and enhanced application performance.

Framework

The classic framework format remains a stalwart choice for packaging code and resources in iOS development. Frameworks offer a structured habitat for your codebase, gracefully accommodating header files, binaries, and resources. They advocate the virtues of encapsulation and modularity, simplifying the process of integration and maintenance of your code. Frameworks wear the dual hats of either static or dynamic, contingent upon whether the code binds at compile time (static) or runtime (dynamic).

.a (Static Library) and .o (Object File)

.a files, heralded as static libraries, house compiled code snugly woven into the fabric of your application at compile time. These libraries accompany your app's binary, bestowing it with a petite binary size and promising accelerated startup times. .o files, or object files, inhabit the realm of intermediate compilation units, wielding the potential to unite and give birth to static libraries or dynamically linked frameworks. Distinguishing between the use cases of static libraries and object files is pivotal for optimising your app's performance and memory footprint.

.dylib (Dynamic Library)

The world of .dylib files beckons dynamic libraries, extending an invitation for their ad hoc arrival at runtime as your app springs into existence. Dynamic libraries usher in a measure of flexibility in code sharing but might nudge your app's binary size northward. Dynamic libraries traditionally find their calling in system frameworks and shared system components. Handling them with care and vigilance is a requisite, for any slip-up in configuration or inclusion can usher in the Spector of runtime crashes.

Universal Framework

Universal frameworks, the "Jack of all trades" among formats, don the cloak of versatility. They are a specialised framework format engineered to accommodate multiple architectures and platforms under one sprawling roof. This format performs an invaluable service, simplifying the distribution of cross-platform libraries. Developers can present a single binary, an embodiment of unity, capable of functioning seamlessly across an array of iOS devices and processor architectures.

Swift Package Manager

As a seasoned iOS developer, you're no stranger to the capabilities wielded by Swift Package Manager (SPM) when it comes to dependency management. Within the labyrinthine corridors of SPM, two pivotal concepts demand your attention: .binaryTarget, .target, and the strategic utilization of linkerSettings.

`.binaryTarget`

Introduced as a breath of fresh air, .binaryTarget assumes the mantle of a feature within Swift Package Manager designed explicitly to streamline the integration of binary dependencies. Binary dependencies are pre-compiled libraries or frameworks presented by third-party sources, emerging as swift and efficient companions for integration. Here's the lowdown:

Efficient integration: With the declaration of a **.binaryTarget**, Swift Package Manager orchestrates retrieving a pre-compiled binary from a designated source, whether a Git repository or a URL. This streamlined approach expedites integration, ushering inconvenience.
Platform-agnostic brilliance: Binary targets shine as platform-agnostic stars, casting their glow across diverse platforms and architectures, including iOS, macOS, and beyond. This trait is especially beneficial for those engaged in cross-platform development endeavors.
Version control vigilance: Binary targets, operating in the realm of pre-compiled artifacts, dwell outside the confines of direct version control within your package. Instead, the version or tag of the binary dependency takes center stage within your Package.swift file.
Swift harmony: Ensure the selected binary target aligns with your project's Swift version for harmony and compatibility. A mismatch in Swift versions can sow the seeds of discord.

`.target`

In stark contrast to the pragmatic elegance of .binaryTarget, .target emerges as the go-to directive for source-based dependencies. Source-based packages carry their source code, ready to undergo the rites of compilation upon integration into your project. Vital insights into this concept include:

Source code integration: Invocation of a .target commands Swift Package Manager to embark on a journey of source code retrieval, cloning the package's source code and forging it into your project. This process offers the boon of customization, enabling you to mold or modify the package as per your requirements.
Version control ascendancy: Source-based packages ascend to prominence as the champions of direct version control within your project's repository. This translates into the power to wield influence over the package's code, affording the liberty to make modifications as needed.
Swift compatibility: As with binary targets, extending a cordial handshake of compatibility between your project's Swift version and the chosen source-based package is paramount. Avoidance of mismatched Swift versions can be your shield against compatibility conundrums.
The web of dependency: Source-based packages often weave a web of dependencies, crafting a sprawling tapestry of interconnected packages. Swift Package Manager rises to the challenge, orchestrating the management of this intricate dependency graph, ensuring that all required packages partake in the grand symphony of integration.

`linkerSettings`

linkerSettings, an entity of significance within the grand configuration of Swift Package Manager, entrusts you with the reins of control over linker flags and settings that govern the orchestration of your targets. Its importance is not to be underestimated:

Fine-grained command: **linkerSettings** stands as your herald, bearing the mandate of fine-grained control over the linking process. It paves the way for the specification of linker flags, search paths, and sundry settings, exerting a profound influence on how the package melds with your project.
Integration tailoring: Instances may arise where a package exhibits a penchant for specific linker flags or settings. Enter linkerSettings; this savior allows you to fashion an integration that gracefully accommodates the package's unique requirements.
Mitigating linking conflicts: In the integration, where multiple packages jostle for space, conflicts on the frontiers of linking can rear their heads. linkerSettings steps in as the peacemaker, offering the means to navigate these conflicts with poise and elegance, ensuring a harmonious integration experience.
Compatibility crusade: To preserve the sanctity of your project, heed the call of compatibility. Scrutinize the compatibility of linker flags and settings with your project's Swift version and platform. Mismatched configurations have the power to disrupt the tranquil flow of runtime.

The role of checksums in SPM

Checksums, those guardians of integrity and security, play a pivotal role within the domain of Swift Package Manager (SPM), specifically in the realm of package management. They serve as sentinels tasked with verifying the authenticity and integrity of external dependencies before ushering them into your project.

Package resolution: Your project's **Package.swift** file hosts declarations of dependencies. When summoned, the Swift Package Manager embarks on a mission to retrieve the package manifest, an essential dossier housing information about the package and its version.
Download and check: The saga continues with downloading the package's source code or binary artifact. Simultaneously, Swift Package Manager, armed with diligence, fetches the checksum linked to the package's version from a trustworthy source, often the package repository.
Verification: As the download completes, Swift Package Manager commences an expedition into checksum calculation. The calculated checksum stands face-to-face with its repository-forged counterpart, and only harmony, represented by a perfect match, rings true. A matching checksum signifies that the downloaded package mirrors the precise form and content expected by the package repository.

Mergeable libraries (New in Town!)

Apple announced Mergeable libraries in WWDC23, the unsung heroes sometimes cloaked in the monikers of "umbrella frameworks" or "universals," which is the fusion of multiple frameworks or libraries into a singular, cohesive framework. This gives a leaner, meaner, and more efficient application binary. Let's shine a light on the critical facets of mergeable libraries:

1. Reduction in binary size

The most prominent jewel in the crown of mergeable libraries is the substantial reduction in the size of your application binary. By amalgamating multiple frameworks into a singular entity, developers surgically excise redundant code and resources, rendering the binary trim and sleek. This trimness finds its true value in mobile applications, where app size directly influences download times and device storage.

2. Streamlined maintenance

Mergeable libraries don the mantle of the custodian of dependencies, simplifying the labyrinthine maintenance process. Instead of juggling several individual libraries, each with its own versioning and update cycle, developers are entrusted with the guardianship of a solitary, consolidated library. This harmonisation streamlines the update process and curtails the risk of version conflicts and compatibility conundrums.

3. Improved load/launch times

Diminished binary sizes usher in improved app launch times. With fewer resources to load into memory, the app leaps to life more swiftly, embellishing the user experience with the gift of alacrity. Reduced load times are particularly invaluable in scenarios where instant access to the application is not just a luxury but an expectation.

4. Cross-platform compatibility

Mergeable libraries possess the unique capability of accommodating multiple platforms and architectures, transforming them into the darlings of cross-platform development. In the hands of adept developers, a single library can extend its benevolent embrace across iOS, macOS, watchOS, tvOS, and many other platforms, fostering an ecosystem of harmonious coexistence.

Dyld vs. Dyld3: dynamic linker in iOS

In the vast landscape of iOS development, two linkers shine brightly: Dyld and its evolutionary offspring, Dyld3. These dynamic linkers perform the critical role of managing the loading and linking of libraries during an app's launch. For expert iOS developers, a deep understanding of their inner workings is essential.

Dyld

Startup performance: Dyld, a stalwart of iOS, has been meticulously designed to deliver efficient startup performance. It employs an arsenal of optimisations to minimise the time required for loading and linking libraries when an app takes its first breath. This efficiency is paramount in ensuring a seamless user experience.
Lazy binding: Dyld employs the ingenious strategy of lazy binding, which defers symbol resolution until the precise moment when a symbol is first utilized. This mechanism trims the startup overhead by avoiding unnecessary work during the initial stages of the app launch.
Shared caches: In its quest to enhance startup performance, Dyld harnesses the power of shared caches. These caches store pre-processed libraries, allowing multiple applications to share them. This shared resource optimises resource utilisation and further expedites the launch process.

Dyld3: The evolutionary leap

As iOS and macOS continued to evolve, the demands on dynamic linking also grew. This prompted the emergence of Dyld3, representing a substantial leap forward in dynamic linking technology. Dyld3 introduced several key advancements aimed at optimising app performance and resource management.

Key advancements in Dyld3:

Reduced memory overhead: Dyld3 was engineered with a focus on minimising memory overhead. It employs a more efficient data structure for managing loaded libraries, precious in resource-constrained environments such as mobile devices.
Parallel loading: Dyld3 introduces the paradigm of parallel loading, enabling it to load multiple libraries concurrently. This parallelism takes full advantage of multi-core processors, resulting in faster app launch times.
On-demand loading: Dyld3 adopts the strategy of on-demand loading, loading only the portions of libraries that are required at runtime. This "just-in-time" approach conserves memory and accelerates startup times.
Improved symbol binding: Dystrongly emphasizess on enhancing symbol binding performance, ensuring that symbols are resolved efficiently as an app runs. This is crucial for maintaining smooth app performance during execution.

Creating and distributing an iOS SDK demands careful consideration of various elements. To embark on this journey, you must make critical decisions regarding the type of library (static or dynamic) that best suits your needs.

Dependency management tools like CocoaPods and Swift Package Manager (SPM) are crucial in linking and integrating your SDK into other projects. Understanding library formats, such as frameworks and xcframeworks, is essential for packaging your code effectively. Don't forget the importance of checksums in ensuring the security of your SDK.

Additionally, exploring the advantages of mergeable libraries can help reduce app size and simplify maintenance. Lastly, delve into the world of dynamic linkers like Dyld and Dyld3 to optimise app startup performance and memory usage. By mastering these components, you'll be well-prepared to create and distribute iOS SDKs that enhance the efficiency and reliability of your development projects.

Understanding package.json II: Scripts

Rohan Mukherjee — Wed, 05 Jul 2023 08:50:00 +0000

Introduction to package.json scripts

Welcome to the world of Javascript development, where building, testing, and deploying applications can be a complex and time-consuming. As developers, we strive to automate these tasks as much as possible, and that's where [npm scripts](https://docs.npmjs.com/cli/v9/using-npm/scripts) come in. npm scripts are powerful and flexible tools that allow us to define custom commands and automate repetitive tasks with ease. They can help us save time, reduce errors, and improve collaboration within our teams.

In this blog post, we'll explore the basics of npm scripts and how they can simplify your Node.js development workflow. We'll cover the syntax and structure of npm scripts, how to run them, and some common use cases. Whether you're a seasoned Node.js developer or just getting started, this guide will help you unlock the full potential of npm scripts and streamline your development process.

This article is a continuation of another article I recently published - Discovering package.json, make sure to check it out as well!

Why use `npm` scripts?

npm scripts are used in every Node.js project. Certain workflows and commands are defined in the scripts section of the package.json file. npm (and other package managers) provide commands that can run these scripts without any additional dependencies or build requirements.

Simplify build and development workflows: npm scripts allow developers to define custom commands for tasks like building, testing, and deploying applications. This makes it easy to automate repetitive tasks and simplify complex workflows, improving overall efficiency and productivity.
No need for additional build tools or dependencies: npm scripts are built into Node.js, so there's no need for additional build tools or dependencies. This reduces the complexity of the development environment and makes it easier to share code across different platforms.
Easy to use and customize: npm scripts are easy to use and can be customized to suit the specific needs of each project. They provide a simple and flexible way to define custom commands and automate tasks without requiring extensive knowledge of build tools or complicated configuration files.
Document and share workflows: npm scripts can be used to document and share development workflows within a team. This helps ensure that everyone is on the same page and that the development process is consistent and reproducible.
Provide pre-and post-scripts: npm scripts can define pre- and post-scripts that run before or after a specific command. This allows developers to run any necessary setup or cleanup tasks before or after a command is executed, improving the overall reliability of the development process.

Basic `npm` scripts

In your package.json file, you can add a scripts field to define your scripts. Inside this object, you can enter key-value pairs for your npm scripts, where the key is the script's name and the value is the shell command. The commands are typically passed to [sh](https://linux.die.net/man/1/sh), so you can run commands like ls too in your npm scripts.

The basic syntax for defining a script in the scripts field is:

"script-name": "command"

Here, script-name is the name you want to give your script, and command is the shell command you want to execute when the script is run. For example, to define a script that runs tests using the mocha testing framework, you could add the following to your scripts field:

"test": "mocha"

To run this script, you would execute the following command in your terminal:

npm run test

You can also define scripts that run multiple commands or use variables. For example, you could define a script that builds your application and then starts a development server like this:

"start": "npm run build && node server.js"

In this case, the start script runs two commands: npm run build (which could be defined to compile your source code) and node server.js (which starts the development server).

The start script is a special script, like test, etc. You can run npm start to run the start script instead of npm run start. Similarly, for running a script called test, you can just run npm test or even npm t instead of npm run test.

Tricks and tips in scripts

Let's say you have the following package.json. You might not be aware of some neat tips and tricks in using these scripts.

{
  "name": "dyte",
  "version": "1.0.0",
  "description":"  ",
  "main": "index.js",
  "scripts": {
    "build": "echo VERSION=$npm_package_version",
    "test": "echo 'Running tests'",
    "build-and-test": "npm run build && npm run test",
    "build-and-test-parallel": "npm run build & npm run test"
  },
  "author": "Rohan Mukherjee <rohan@dyte.io> (https://dyte.io)",
  "license": "ISC"
}

Using features provided

When you run npm run build, you can see that the $npm_package_version variable is replaced with the version of the package specified in the package.json.

Similarly, you can use other variables like $npm_package_name in your script to extract values from the package.json file.

If you want to execute a script you're unsure about the existence of, you can use the -if-present flag so that the command doesn't exit with a non-zero status code.

npm run build --if-present

If you're using workspaces and want to run a script in a different workspace, you can specify the workspace name using --workspace or -w.

Let's say that you've already defined a lint script as follows:

"lint": "eslint ./src --ext .ts --ext .js"

Upon running the lint script, eslint will list all the linting issues from the src directory of your project. You want to define a script to fix all auto-fixable linting issues. You'd typically have to define a command like

"lint:fix": "eslint ./src --ext .ts --ext .js --fix"

Now, if you want to include the extension for .jsx files also in your lint script, you'd have to make the change in both places, and it's easy to miss changing one of these places.

Since the lint:fix command is very similar to the lint command, and only requires an additional flag, you can reference the lint command in the lint:fix command to reuse it. For instance, you can run

"lint:fix": "npm run lint -- --fix"

This will run the lint command and pass an additional argument --fix to it. Anything you add after -- will be passed as a string to the npm command specified before.

Using `sh` tricks in scripts

Since the scripts are run using sh (on all Unix-based systems), you can use some neat sh tricks to run multiple scripts together. On Windows systems, the scripts are run using the default shell - cmd (Command Prompt). However, you can change the shell used on Windows to something like git bash using the following command.

npm config set script-shell "C:\\Program Files\\git\\bin\\bash.exe"

You can also run 2 scripts using the && operator in a script. For instance, if you run the build-and-test script, it will execute the test npm command only after the build script completes successfully (exits with status code 0).

You can also run these scripts in parallel, using a since &. The build-and-test-parallel command runs the scripts together and is not guaranteed which process will complete first. In this example, the test script finished before the build script.

These operators are however not specific to npm scripts. You can run them in sh or bash directly as well. It's also possible to run a command upon the failure of another. For instance, if you want to run a command upon the failure of another, you can use the || operator.

In the above screenshot, one command exits with code 1 (which indicates that the command failed), so the Goodbye script runs post that.

In fact, you can chain commands using && and || to decide what to execute upon success and what to execute upon failure.

When the first command succeeds.

When the first command fails.

Lifecycle Scripts

lifecycle scripts are a set of predefined script names that are executed automatically by npm at specific stages during the lifecycle of a package. These scripts are defined in the package.json file of a Node.js project and can be used to perform various tasks such as installing dependencies, running tests, building the project, and deploying the application.

The npm lifecycle consists of several stages that are executed in a specific order:

"prepublish": This script runs before the package is packed and published, and is used to prepare the package for distribution.
"prepare": This script runs both during local development and when the package is installed as a dependency of another package. It is used to prepare the package for use, such as by building or compiling the code.
"preinstall": This script runs before the package is installed and is used to perform any necessary setup tasks before dependencies are installed.
"postinstall": This script runs after the package is installed and is used to perform any necessary setup tasks after dependencies are installed.
"preuninstall": This script runs before the package is uninstalled and is used to perform any necessary cleanup tasks before dependencies are removed.
"postuninstall": This script runs after the package is uninstalled and is used to perform any necessary cleanup tasks after dependencies are removed.
"preversion": This script runs before the version of the package is updated and is used to perform any necessary tasks before the version is changed.
"postversion": This script runs after the version of the package is updated and is used to perform any necessary tasks after the version is changed.

To define a custom pre- or post-script, you need to add a property to the scripts object in your package.json file. The property name should be in the format of "pre" or "post" followed by the name of the script you want to run. For example, to define a preinstall script, you can add the following to your package.json file:

{
  "name": "my-package",
  "version": "1.0.0",
  "scripts": {
    "preinstall": "echo Running pre-install script"
  }
}

In this example, the pre-install script will run before the npm install command is executed. You can replace the echo command with any shell command or node script you want to run.

Similarly, you can define a post-install script by adding a property named "post-install" to the scripts object:

{
  "name": "my-package",
  "version": "1.0.0",
  "scripts": {
    "postinstall": "echo Running post-install script"
  }
}

This script will run after the npm install command is executed.

You can define pre- and post-scripts for other npm commands as well, such as "pretest", "posttest", "prepublish", "postpublish", etc. Just prefix the script name with "pre" or "post" accordingly.

Note that the pre and post-scripts are executed in the order they are defined in the package.json file, and you can define multiple pre and post-scripts for the same command.

Disclaimer: Use “postinstall” scripts carefully, as supply chain security software will fire up alerts upon detecting postinstall scripts. In npm packages, postinstall scripts can be used as a means to install malware into the installer's system, hence it's advisable not to use it in an npm package. However, it's okay to use it as long as you remove it from your package.json when publishing the package.

Best Practices

Here are some best practices for writing npm scripts:

Keep scripts simple and readable: It is important to keep your scripts simple and easy to read. Use meaningful script names and avoid writing long scripts with many commands. Break down complex tasks into smaller sub-tasks and write separate scripts for them.
Use variables: Use environment variables to store commonly used values, such as directories or file paths, and use them in your scripts to avoid hardcoding.
Use package.json to declare dependencies: Declare dependencies in your package.json file instead of installing them inside your scripts. This will ensure that your dependencies are installed before your script runs.
Avoid platform-specific commands: Avoid using platform-specific commands in your scripts. Use cross-platform tools like Node.js or Bash to ensure that your scripts work on different platforms. For instance, if you want your npm script to remove a certain directory using the rm -rf command, this would work perfectly on a Linux or Mac machine but would error out on Windows. To avoid this, you can use a cross-platform package such as [rimraf](https://www.npmjs.com/package/rimraf).
Keep scripts independent: Keep your scripts independent of each other to avoid dependency issues. If you need to run one script after another, use a task runner like Gulp or Grunt to define tasks and their dependencies.
Use exit codes: Use exit codes to indicate the success or failure of your scripts. Return a non-zero exit code to indicate that the script failed and zero to indicate that the script succeeded.
Use pre- and post-scripts: Use pre- and post-scripts to execute common tasks before and after your scripts run. For example, use pre-scripts to ensure that your dependencies are up-to-date and post-scripts to clean up any temporary files.

Take a look at this package.json file in our docs repository to get a good idea about how npm scripts are used in repositories. You can also check out how lifecycle scripts can be used efficiently in this example: react-vnc.

Summary

In conclusion, npm scripts are a powerful tool for automating tasks and simplifying the development workflow in modern web development. With just a few lines of code, you can define scripts to run tests, build your project, start your development server, and much more.

To make the most of npm scripts, it is essential to follow some best practices such as keeping scripts simple and readable, using variables, avoiding platform-specific commands, documenting your scripts, and using pre- and post-scripts.

By using package.json npm scripts, you can streamline your development workflow and save time and effort. You can also make your code more reliable and maintainable by automating repetitive tasks and ensuring that your code is error-free.

Overall, npm scripts are a powerful and essential tool for any modern Node.js developer. With the right knowledge and best practices, you can harness the full potential of npm scripts to create more efficient, reliable, and scalable projects.

I hope you found this post informative and engaging. If you have any thoughts or feedback, feel free to reach out to me on Twitter or LinkedIn 😄. Stay tuned for more related blog posts in the future!

Adding AR Filters to Video Calls Using DeepAR and Dyte

Ishita Kabra — Wed, 05 Jul 2023 08:12:58 +0000

Unless you live under a rock, you would have used social media platforms such as TikTok, Instagram, Facebook, or Snapchat. When posting a photo or video on these platforms, you usually have the option to enhance the photo or video using various filters.

You can transform yourself in an image or video with a different hair color or wear a cap even though you didn't take the photo or record a video wearing one.

Social media platforms use Augmented Reality (AR) filters to make all these photo or video enhancements possible. AR filters are digital effects that can be applied to images or videos in real time, using computer vision technology to track the movements and facial features of someone in an image or video.

The AR filters are mainly used for entertainment, branding, and marketing because they offer a fun and engaging way to enhance visual content. These filters can also create immersive experiences, such as virtual try-ons for clothing, and showcase products uniquely and interactively.

In this article, you will learn how to build a video calling app with AR filters using Dyte video SDK, DeepAR, and web components UI kit. Dyte is a developer-friendly platform that offers robust SDKs to build live experiences within an application.

Prerequisites

To follow through with this article, you will need the following software installed on your computer:

A text editor: I will use Visual Studio Code as a text editor for this article. To install VS Code, navigate to the VS Code download page and download it for your operating system.

Create and set up a Dyte account

Step 1: Navigate to dyte.io and click the ‘Start building’ button.

Step 2: Sign up for a free Dyte account on the next page using your Google or GitHub account.

Step 3: After creating a Dyte account, you will be redirected to the ‘API Keys’ page, with an Organization ID and API key provided. Keep your API key secure, and don’t share it with anyone.

Create Presets

Dyte includes a bunch of pre-configured presets. You can simply use the default preset if you don't wish to create one.

Create a Dyte Meeting

Step 1: Navigate to the Create a Meeting page.

Step 2: On the right side of the page, fill the Username input field with Organization ID and the Password input field with your API Key provided on the API Keys page.

Step 3: In the Body section, give the meeting a title of “AR Filters.” Then click the Send API Request button.

Step 4: Navigate to the Meetings page in the Dyte Developer Portal, and you should see that a meeting titled AR Filters was created.

Add a participant to the meeting

Step 1: Navigate to the Add a Participant page and fill out the Organization ID and API Key (you can get these from the Dyte Developer Portal) input fields as shown in the step above.

Step 2: In the Parameters section, add the Meeting ID of the AR Filters meeting we created earlier.

Step 3: In the Body section, fill out the options as shown below. Then click the 'Send API Request' button.

Step 4: Obtain the token. This token is unique for each participant in the meeting, and you would ideally want to fetch it for each participant by calling Dyte’s APIs in your own backend.

Creating your Dyte Meeting

Step 1: Create a folder called 'ARFilters' on your computer. Then open the folder in a text editor and add to it an index.html file.

Step 2: Install the Dyte SDK.

<head>
  <script type="module">
      import { defineCustomElements } from '<https://cdn.jsdelivr.net/npm/@dytesdk/ui-kit@1.51.1/loader/index.es2017.js>';
      defineCustomElements();
  </script>
  <!-- Import Web Core via CDN too -->
  <script src="<https://cdn.dyte.in/core/dyte-1.14.0.js>"></script>
</head>

Step 3: In your index.html build a basic meeting UI with two custom buttons to control the AR filters.

<body>
    <div id="app">
      <dyte-dialog-manager class="dyte"></dyte-dialog-manager>
      <dyte-participants-audio class="dyte"></dyte-participants-audio>
      <dyte-notifications class="dyte"></dyte-notifications>
      <dyte-header class="dyte"></dyte-header>
      <div class="dyte-grid">
        <dyte-grid class="dyte"></dyte-grid>
        <dyte-sidebar class="dyte" default-section="none">
        </dyte-sidebar>
      </div>
      <div class="dyte-controlbar">
        <div class="controlbar-section">
          <dyte-settings-toggle class="dyte"></dyte-settings-toggle>
        </div>
        <div class="controlbar-section">
          <dyte-mic-toggle class="dyte"></dyte-mic-toggle>
          <dyte-camera-toggle class="dyte"></dyte-camera-toggle>
          <dyte-leave-button class="dyte"></dyte-leave-button>
          <dyte-controlbar-button
            label="Add Filter"
            size="lg"
            id="arFilter"
            style="background-color: black; color: white">
          </dyte-controlbar-button>
          <dyte-controlbar-button
            label="Change Filter"
            size="lg"
            id="switchFilter"
            style="background-color: black; color: white">
          </dyte-controlbar-button>
        </div>
        <div class="controlbar-section">
          <dyte-chat-toggle class="dyte"></dyte-chat-toggle>
          <dyte-participants-toggle class="dyte"></dyte-participants-toggle>
          <dyte-polls-toggle class="dyte"></dyte-polls-toggle>
        </div>
      </div>
    </div>
    <script type="module" src="/main.js"></script>
  </body>

We use custom components offered by Dyte to create a default meeting UI.

Dyte Header: This component loads all the basic meeting details like the title, participant count, clock, and much more.
Dyte Grid: This component renders all the video streams from all the participants in the meeting.
Dyte Sidebar: This component renders chat, polls, participants list, and more.
Dyte Controlbar: This is where we add controls for our meeting. A user can toggle their camera and mic, adjust settings or manage the AR filters from this control bar.

We also insert a script that links to main.js, an external file that will contain all our business logic.

Step 4: Write the business logic

We initialize a basic Dyte Meeting using our authToken and pass the meeting object to all our components.

import icons from './icons.json';

const init = async () => {
  // Initialize Dyte Meeting
  const meeting = await DyteClient.init({
    authToken: import.meta.env.VITE_AUTH_TOKEN,
    defaults: {
      audio: false,
      video: false,
    },
  });

    // Passing the `meeting` prop to all our components
    const els = document.getElementsByClassName('dyte');
  els[0].config = meeting.self.config;
  for (const el of els) {
    el.meeting = meeting;
  }

    // Adding state management for the sidebar component
    document.body.addEventListener('dyteStateUpdate', ({ detail }) => {
      document.querySelector('dyte-sidebar').style.display = detail.activeSidebar ? 'flex' : 'none';
      document.querySelector('dyte-sidebar').states = { sidebar: detail.activeSidebar ? detail.sidebar : 'none' };
    });

    // Adding icons to our custom buttons
  const AddFilter = document.getElementById("arFilter");
  const SwitchFilter = document.getElementById("switchFilter");

  AddFilter.icon = icons.addFilter;
  SwitchFilter.icon = icons.switchFilter;
};

init();

Step 5: Adding CSS to our meeting.

Create a style.css; this file will hold all our styles for the meeting.

body, html {
  height: 100vh;
  width: 100vw;
  margin: 0;
  padding: 0;
  display: flex;
  flex-direction: column;
  background: rgb(8, 8, 8);
}

#app {
  display: flex;
  flex-direction: column;
  font-family: Arial, Helvetica, sans-serif;
  flex-grow: 1;
  color: white;
}

.ended-screen {
  align-items: center;
  justify-content: center;
}

.dyte-grid {
  flex-grow: 1;
  display: flex;
  flex-direction: row;
}

dyte-grid {
 flex-grow: 1;
}

dyte-sidebar {
  margin: 10px 20px;
  border-radius: 10px;
}

.dyte-controlbar {
  display: flex;
  flex-direction: row;
  align-items: center;
  justify-content: space-between;
  min-height: 20px;
  margin:10px 0;
  width: 100%;
}

.controlbar-section {
  display: flex;
  flex-direction: row;
  align-items: center;
}

Step 6: Open the HTML file on a browser, and you will be prompted to allow access to your microphone and camera. You should see yourself on the screen if your camera is enabled.

Setting up a DeepAR account

Step 1: Navigate to the DeepAR developer home page and click the ‘Get Started’ button.

Step 2: Sign up for a free DeepAR account using your Google or GitHub account on the next page.

Create a DeepAR project

Step 1: After you’ve signed up and logged in to your DeepAR account, you will be redirected to the projects page.

Step 2: Click the ‘Create A Project’ button. Name the project AR Filters, agree to the terms and conditions, then click the continue button.

Step 3: After creating the project, you will be redirected to add an app. In this case, go to the Web App section and select Add App option. You will be prompted to add a domain name. Add your domain name and click the continue button.

Step 4: You will receive DeepAR App Key after creating the web app. The key will initialize the DeepAR filter as a license key.

Adding AR Filters to your video calling app

Now that you have obtained your License key and have a basic Dyte Meeting up and running. It is time to add fun AR filters to your meeting.

Step 1: Install DeepAR

You can import DeepAR into your project using CDN.

<script src="<https://cdn.jsdelivr.net/npm/deepar/js/deepar.js>"></script>

Step 2: Initialize the DeepAR module

Add some filters to our project repositories and provide a path to those filters.

// Initialize DeepAR
  const filters = [
    "./effects/lion",
    "./effects/flowers",
    "./effects/dalmatian",
    "./effects/background_segmentation",
    "./effects/background_blur",
    "./effects/aviators",
  ];

Next, we create a canvas element and initialize the DeepAR module.

const deepARCanvas = document.createElement("canvas");
  deepARCanvas.width = 680;
  deepARCanvas.height = 480;
  const deepAR = await deepar.initialize({
    licenseKey: import.meta.env.VITE_DEEP_AR_TOKEN,
    canvas: deepARCanvas,
    effect: filters[0],
    additionalOptions: {
      cameraConfig: {
        disableDefaultCamera: true,
      },
    },
  });

Step 3: Add AR filters to your Dyte video call

Here, we are creating two functions, one to toggle the AR filter on/off and the other to switch between filters.

We add middleware to Dyte using its AddVideoMiddleware API, which takes in a middleware function.

To switch between filters, we use DeepAR’s switchEffect method.

let count = 0;
let filterIndex = 0;

const AddFilter = document.getElementById("arFilter");
const SwitchFilter = document.getElementById("switchFilter");

AddFilter.icon = icons.addFilter;
SwitchFilter.icon = icons.switchFilter;

AddFilter.addEventListener("click", toggleAR);
SwitchFilter.addEventListener("click", filterChangeHandler);

function toggleAR() {
  count++;

  if (count % 2 == 0) {
    AddFilter.label = 'Add Filter'
    meeting.self.removeVideoMiddleware(AddTheme);
  } else {
    AddFilter.label = 'Remove Filter'
    meeting.self.addVideoMiddleware(AddTheme);
  }
}

async function filterChangeHandler() {
  filterIndex = (filterIndex + 1) % filters.length;
  await deepAR.switchEffect(filters[filterIndex]);
}

Now, let’s create our video middleware function using DeepAR.

async function AddTheme() {
  let lastProcessedImage = null;
  const intermediatoryCanvas = document.createElement("canvas");
  intermediatoryCanvas.width = 640;
  intermediatoryCanvas.height = 480;
  const intermediatoryCanvasCtx = intermediatoryCanvas.getContext("2d");

  return async (canvas, ctx) => {
    intermediatoryCanvasCtx.drawImage(canvas, 0, 0);
    if (lastProcessedImage) {
      ctx.drawImage(
        lastProcessedImage, 0, 0,
        lastProcessedImage.width,
        lastProcessedImage.height, 0, 0,
        canvas.width, canvas.height
      );
    }
    await deepAR.processImage(intermediatoryCanvas);
    const image = new Image();
    image.id = "picture";
    image.src = await deepAR.takeScreenshot();
    lastProcessedImage = image;
  };
}

With that done, we have built a video calling app with AR filters using Dyte video SDK and DeepAR! Get your AR filters on to look something like this.

You can try out the live demo of the sample application here, and the entire source code for the project is available here.

Conclusion

In conclusion, we have walked you through how to build a video calling app with AR filters using the Dyte video and DeepAR SDKs. With Dyte’s customizable UI Kit, you can easily manage controls, add or remove buttons, create custom controls, and do much more with just a few lines of code.

If this article tickled the builder in you, then check out Dyte’s live streaming SDK. We provide easy-to-use SDKs to deliver interactive live streams at scale with much lower latency than traditional HLS. Feel at home with the same friendly developer experience and extensibility. Dyte’s live streaming SDK is launching soon on Product Hunt!

Get better insights on leveraging Dyte’s technology and discover how it can revolutionize your app’s communication capabilities with its SDKs. You can reach us at support@dyte.io or ask our developer community if you have any questions.

Building an Async Interview Platform With React and Dyte

Vaibhav Shinde — Fri, 30 Jun 2023 11:25:16 +0000

Introduction

With the emergence of remote work and virtual events, video communication has become an integral part of our daily lives. As a result, video systems have seen substantial growth. Video platforms are helpful for remote teams, online learning, interview processes, and virtual events because they enable synchronous and non-synchronous communication with video, audio, and chat features.

This article will demonstrate how to use React and Dyte to create an async interview platform. As we dive deeper, you will be guided through the steps involved in building the UI elements and how to leverage Dyte’s video SDK.

You'll have the information and resources necessary to build your async platform through Dyte by the end of this article. But first, let’s look at why Async interviews are better than normal synchronous interviews.

Why an async interview?

Asynchronous interviews offer convenience, fairness, and improved efficiency in the hiring process, making them a valuable tool for both candidates and employers alike. Some of the benefits you get with the async interviews

Flexibility - Asynchronous interviews allow both interviewers and candidates to participate at their own convenience, eliminating the need for scheduling conflicts.
Time-saving - Asynchronous platforms streamline the process by allowing candidates to record their responses to interview questions at their own pace, saving time for both parties involved.
Fairness - Asynchronous interviews provide a level playing field for all candidates.
Reduced pressure - Some candidates may feel more comfortable and perform better when not under the immediate pressure of a live interview.
Scalability - Asynchronous interview platforms are highly scalable, making them ideal for organizations with high-volume hiring needs.

Now that we know the importance of Async interviews, let’s look at the prerequisites to build this platform.

Prerequisites

Basic knowledge of React is required, as we will be using it to build the frontend of the application.
Node.js is also required to run the application, so make sure you have it installed on your computer.
Lastly, you will need an API key and organization ID from developer portal to be able to authenticate yourself when you call Dyte's REST APIs.

Let’s understand the application flow to build an async interview app.

Application flow of async interview platform

Users will have to enter their information on the landing page.
After submitting the form, they will be redirected to the interview page, where they can play the intro video.
Once users click on I'm ready, they will be redirected to the first question screen.
After they have listened to the video based question, users can click on Start Recording and start capturing their answer.
Once users submit the answer, they can click “Next” for the next question and finally “Finish” on the last question, and users’ all responses will be recorded.

Now, getting to the key implementation details:

The user will use the default group_call_host preset because this comes default for every organization that is created in Dyte. Any other preset which has the ability to Start and Stop recordings also works.
When the user clicks the “Start Recording My Answer” button, we will call the meeting.recording.start() API in the frontend to start the recording. And when user clicks Next or Finish, we will call meeting.recording.stop() to stop the recording once the user has answered a specific question. Do note that each question answered by the participant will be a separate recorded file.
We will take a look at how you can access the recorded files at the end of this post.

Project Setup

We’re going to use Next.js for our project as we can write frontend and backend code in a single codebase easily with it.

Creating the backend route

First, we’ll just create a simple backend API route at /api/meeting which will create a meeting as well as add the participant to that meeting with the default group_call_host preset. This API returns the participant details with the token that we will use to join to meeting in the frontend.

Check out the code for creating the backend route:

import { NextApiRequest, NextApiResponse } from "next";

const API_KEY = process.env.DYTE_API_KEY;
const ORG_ID = process.env.DYTE_ORG_ID;

const BASIC_TOKEN = Buffer.from(ORG_ID + ":" + API_KEY).toString("base64");

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  if (req.method !== "POST") {
    return res
      .status(405)
      .json({ success: false, message: "Method Not Allowed" });
  }

  const { name, email } = req.body;

  const meetingResponse = await fetch(
    "<https://api.cluster.dyte.in/v2/meetings>",
    {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: "Basic " + BASIC_TOKEN,
      },
      body: JSON.stringify({ title: "Interview with " + name }),
    }
  );

  const meetingData = await meetingResponse.json();

  if (!meetingResponse.ok) {
    return res.status(meetingResponse.status).json(meetingData);
  }

  const { id } = meetingData.data;

  const participantResponse = await fetch(
    `https://api.cluster.dyte.in/v2/meetings/${id}/participants`,
    {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: "Basic " + BASIC_TOKEN,
      },
      body: JSON.stringify({
        name,
        preset_name: "group_call_host",
        custom_participant_id: email,
      }),
    }
  );

  res.status(participantResponse.status).json(await participantResponse.json());
}

Creating the landing page

This will have a simple form that accepts the user’s name and email address, and this data is then passed to the /api/meeting API, and we use the retrieved token to redirect the user to the interview page: /interview?token=<user-token>.

Here is the source for this page which is written in React:

import Head from "next/head";
import { useRouter } from "next/router";
import { FormEvent, useCallback, useReducer, useState } from "react";

interface Form {
  name: string;
  email: string;
}

export default function Home() {
  const [submitting, setSubmitting] = useState(false);
  const [form, setForm] = useReducer(
    (state: Form, payload: Partial<Form>) => {
      return { ...state, ...payload };
    },
    { name: "", email: "" }
  );

  const router = useRouter();

  const handleSubmit = useCallback(
    async (e: FormEvent) => {
      e.preventDefault();

      if (form.email.trim() === "" || form.name.trim() === "") {
        return;
      }

      setSubmitting(true);

      try {
        const res = await fetch("/api/meeting", {
          method: "POST",
          headers: {
            "Content-Type": "application/json",
          },
          body: JSON.stringify(form),
        });

        const data = await res.json();

        if (!res.ok) {
          alert(
            "There was an error when starting the interview, check console for error"
          );
          console.log(res.status, data);
          return;
        }

        const { token } = data.data;

        router.push({
          pathname: "/interview",
          query: {
            token,
          },
        });
      } finally {
        setSubmitting(false);
      }
    },
    [form, router]
  );

  return (
    <div className="flex h-full w-full flex-col items-center justify-center p-6">
      <Head>
        <title>Async Interview Sample - Dyte</title>
      </Head>
      <div className="flex w-full max-w-screen-md flex-col items-center justify-center gap-3 text-center">
        <h1 className="text-3xl font-bold text-blue-600">
          Async Video Interview
        </h1>

        <form
          className="mt-4 flex w-full max-w-sm flex-col gap-3"
          onSubmit={handleSubmit}
        >
          <input
            type="text"
            placeholder="Name"
            value={form.name}
            onInput={(e) => setForm({ name: e.currentTarget.value })}
            required
          />

          <input
            type="email"
            placeholder="Email"
            value={form.email}
            onInput={(e) => setForm({ email: e.currentTarget.value })}
            required
          />

          <button
            className="h-10 rounded-md bg-blue-600 font-semibold text-white"
            disabled={submitting}
          >
            {submitting ? "Starting..." : "Start"}
          </button>
        </form>
      </div>
    </div>
  );
}

Interview page

First, let's write frontend code that actually makes the user join the meeting.

The code would look like this:

export default function InterviewPage() {
  const [meeting, initMeeting] = useDyteClient();

  useEffect(() => {
    const search = new URLSearchParams(window.location.search);

    const authToken = search.get("token");

    if (!authToken) {
      return alert("authToken was not passed");
    }

    initMeeting({
      authToken,
    }).then((m) => m?.joinRoom());
  }, []);

  return (
    <DyteProvider value={meeting} fallback={<div>Loading...</div>}>
      <Interview />
    </DyteProvider>
  );
}

This is a common boilerplate code you have to write when integrating Dyte in your React app.

Let’s break it down:

We get the auth token from the URL query, and we pass it to the initMeeting() method, and once the meeting object is initialized, we will automatically join the user to the meeting via meeting.joinRoom(). However, if you wish to show an interstitial setup page before the user joins the meeting, you can omit this and show a custom setup screen or use our included [DyteSetupScreen](https://docs.dyte.io/react-ui-kit/components/dyte-setup-screen#docusaurus_skipToContent_fallback) component.
We pass the meeting object to the <DyteProvider /> with a fallback UI until the meeting object is initialized.
Then we render the <Interview /> component which renders the actual interview UI.

Now this is where the actual frontend implementation of this project lies, in the Interview component. All the previous steps are the common steps you would have to do when integrating Dyte’s SDKs.

You can take a look at the source code as you read to understand better.

We have stored the introduction and questions in the questions array, with the first one being the introduction which is at the 0 index. The basic logic is that we will start from 0, showing the intro, and then showing the 3 questions and getting responses from the users each time. The stage state will have the current state of the questions answered.

import {
  DyteAudioVisualizer,
  DyteAvatar,
  DyteParticipantTile,
  DyteSpinner,
} from "@dytesdk/react-ui-kit";
import {
  DyteProvider,
  useDyteClient,
  useDyteMeeting,
  useDyteSelector,
} from "@dytesdk/react-web-core";
import Head from "next/head";
import { useEffect, useRef, useState } from "react";

const questions = [
  "https://res.cloudinary.com/doy3ks7ls/video/upload/v1680075287/Intro_xb5ftf.mp4",
  "https://res.cloudinary.com/doy3ks7ls/video/upload/v1680075215/Question_1_rwml32.mp4",
  "https://res.cloudinary.com/doy3ks7ls/video/upload/v1680075219/Question_2_yeddjq.mp4",
  "https://res.cloudinary.com/doy3ks7ls/video/upload/v1680075219/Question_3_kqgbgw.mp4",
];

function Interview() {
  const $video = useRef<HTMLVideoElement>(null);
  const roomJoined = useDyteSelector((m) => m.self.roomJoined);
  const [stage, setStage] = useState(0);
  const { meeting } = useDyteMeeting();

  const recordingState = useDyteSelector((m) => m.recording.recordingState);

  if (!roomJoined) {
    return <div>Loading...</div>;
  }

  return (
    <div className="flex h-full w-full">
      {stage < questions.length && (
        <aside className="relative flex-1">
          <video
            className="h-full w-full object-cover"
            src={questions[stage]}
            ref={$video}
            controls
            autoPlay
          />
        </aside>
      )}
      <main className="flex flex-1 flex-col items-center justify-center gap-12 p-4">
        {stage > 0 && stage < questions.length && (
          <>
            <p className="text-lg font-bold">
              Question {stage} of {questions.length - 1}
            </p>

            <DyteParticipantTile
              participant={meeting.self}
              className="relative aspect-[4/3] h-auto w-full max-w-[540px]"
            >
              <DyteAudioVisualizer
                participant={meeting.self}
                size="lg"
                className="absolute right-3 top-3"
              />
              <DyteAvatar participant={meeting.self} />
            </DyteParticipantTile>
          </>
        )}

        {stage > 0 &&
          stage < questions.length &&
          recordingState === "STARTING" && (
            <p className="info">
              <DyteSpinner />
              Starting the recording...
            </p>
          )}

        {stage > 0 &&
          stage < questions.length &&
          recordingState === "STOPPING" && (
            <p className="info">
              <DyteSpinner />
              Stopping the recording...
            </p>
          )}

        {stage > 0 &&
          stage < questions.length &&
          recordingState === "RECORDING" && (
            <p className="info border-red-500 text-red-700">
              Recording has started, you can start speaking now
            </p>
          )}

        <div className="flex flex-wrap items-center gap-2">
          {stage > 0 &&
            stage < questions.length &&
            recordingState === "IDLE" && (
              <button
                className="btn bg-red-500"
                onClick={() => {
                  $video.current?.pause();
                  meeting.recording.start();
                }}
              >
                Start Recording My Answer
              </button>
            )}

          {stage === 0 && (
            <button
              className="btn"
              onClick={() => {
                setStage(1);
              }}
            >
              Awesome, I&apos;m ready 🚀
            </button>
          )}

          {stage > 0 && stage < questions.length - 1 && (
            <button
              className="btn"
              onClick={async () => {
                if (recordingState === "RECORDING") {
                  await meeting.recording.stop();
                  setStage((stage) => stage + 1);
                }
              }}
              disabled={recordingState !== "RECORDING"}
            >
              Next
            </button>
          )}

          {stage === questions.length - 1 && (
            <button
              className="btn"
              onClick={() => {
                if (recordingState === "RECORDING") {
                  meeting.recording.stop();
                }
                setStage(questions.length);
              }}
            >
              Finish
            </button>
          )}

          {stage === questions.length && (
            <div>
              <p className="max-w-md text-center text-2xl font-semibold">
                Thank you for attending the interview. Your responses were
                recorded.
              </p>
            </div>
          )}
        </div>
      </main>
    </div>
  );
}

Also, we will get the current recording state with the useDyteSelector() like so:

const recordingState = useDyteSelector((m) => m.recording.recordingState);

This is used to show a recording indicator to indicate to the user that they are being recorded and they can start speaking, and also as a check when running start or stop recording APIs so they don’t fail.

Whenever a user clicks “Start Recording My Answer”, we will call meeting.recording.start() API which will start the recording and this indicator is shown:

And once the user finished answering and is willing to move to the next question and click Next or Finish, it will call the meeting.recording.stop() to stop the recording and increment the stage state.

And finally when you’re at the final stage, i.e; you have answered all questions, the user will be shown the final screen with the message:

You can access all recordings of a particular meetingId with the API: https://docs.dyte.io/api#/operations/get_all_recordings.

That is it, we’ve created an Async Interview Platform with Dyte! The live demo of this application is available here. And here’s the source code for the project.

Want to build something similar? Build the future on live video with Dyte’s customizable SDKs, low-code UI builder, and easy to consume documentation. You never know you might create the next big thing 🚀.