DEV Community: Ravindra Singh Rathor

Building a Live Proctoring System to Detect Participant Switching

Ravindra Singh Rathor — Thu, 28 Dec 2023 20:24:50 +0000

Introduction

To stop students from cheating during tests, we often assign a proctor to observe them. But we can't have someone in-person watching each student for online tests. This is where live automatic proctoring helps.

With a webcam and microphone, this method monitors students while they take online tests. It uses computer tools to spot any cheating. 👀

In the previous blog, we built a live proctoring system to detect whether there are multiple people in the candidate's frame.

In this tutorial, we will build a live proctoring system using Dyte APIs that will allow an admin to monitor if the candidate who was supposed to be in the video is still the same candidate during the whole duration in real-time. ✨

High-Level Design of the application

We aim to notify the proctor if the candidate changes during the session. 🔍

The proctor would get the candidate's details and a photograph from their webcam as proof in his meeting sidebar. 📸✨

In this project, we will use React with Dyte UI kit and Dyte React Web Core packages for the frontend.
For the backend, we will use FastApi (Python 3).
We will also use database as a service by ElephantSQL (PostgreSQL).
Lastly, we will use Imgur for storing screenshots. And the image metadata will then be saved to the same database.

Folder Structure

After completing the tutorial, the folder structure will look like this. 👇

dyte-proctoring/
├── frontend
│   ├── README.md
│   ├── package.json
│   ├── public
│   ├── src
│   │   ├── App.css
│   │   ├── App.jsx
│   │   ├── Heading.jsx
│   │   ├── ImageInput.jsx
│   │   ├── Meet.jsx
│   │   ├── Proctor.jsx
│   │   ├── index.css
│   │   ├── index.jsx
│   │   ├── logo.svg
│   │   ├── react-app-env.d.ts
│   │   ├── reportWebVitals.ts
│   │   ├── setupTests.ts
│   │   ├── stage.jsx
│   │   └── utils.js
│   ├── tsconfig.json
├── venv
├── app.py
├── imgur.py
└── requirements.txt

Step 0: Configurations and Setup

🧑‍💻 Before building our live proctoring system, we must set up a Dyte account.

We can create a free account by clicking the "Start Building" button on Dyte.io and signing up using Google or GitHub 🎉.

Once signed up, we can access our Dyte API keys from the "API Keys" tab in the left sidebar. We will keep these keys secure as we will use them later.🔑🤫

For our live proctoring system, we will use React for the frontend and FastAPI for building the Backend and APIs.

We will begin by creating a new directory for our project, called dyte-proctoring and navigating into it using the following commands:

mkdir dyte-proctoring
cd dyte-proctoring

Please note:

We will also require accounts on the following platforms:

Imgur: Create an account on Imgur and create an API key. Here is a step-by-step guide
ElephantSQL: Here is a step-by-step guide to creating a db on ElephantSQL.

Now back to the tutorial.

Step 1: Setting up the frontend

Let's start setting up our frontend project using React and Dyte! ✨

We will create a boilerplate React app using create-react-app. We can do this with the following command:

yarn create react-app frontend

This will initialize a new React app in the frontend directory. 📁

Then, we will go ahead and install the dyte react-web-core, dyte react-ui-kit and react-router packages in this project using the following command 👇

yarn add @dytesdk/react-web-core @dytesdk/react-ui-kit react-router react-router-dom

Step 2: Setting up the backend

Let's get started with setting up our FastAPI backend now. 🙌

We will go back to the root directory of our project and initiate our project here itself for the ease of hosting:

cd ..

First of all, we will go ahead and create our requirements.txt file in the root directory itself with the following content 👇

requirements.txt

cmake
fastapi
uvicorn
face_recognition
numpy
python-multipart
psycopg2-binary
httpx
python-dotenv
pydantic
requests

After this, we will go ahead and create our virtual environment with venv and install the dependencies.

python -m venv venv
source venv/bin/activate # for linux/mac
venv\Scripts\activate.bat # for windows
pip install -r requirements.txt

We will also create an environment variable file .env, for storing our credentials.

.env

DYTE_ORG_ID=<ID>
DYTE_API_KEY=<KEY>
IMGUR_CLIENT_ID=<ID>
DB_USER=<ID>
DB_PASSWORD=<PASSWORD>
DB_HOST=<HOST>

Step 3: Setting up image upload with Imgur

Let's create a new file named imgur.py and add the following code. This will help us upload our screenshots to Imgur. 🌅

Here we are connecting the Imgur API using the CLIENT_ID that we can get from our Imgur API Dashboard and using it to upload the suspicious candidate's image and get back the link to it. 🔗👇

imgur.py

import base64
from fastapi import FastAPI, UploadFile, HTTPException
from httpx import AsyncClient
from dotenv import load_dotenv

load_dotenv()

app = FastAPI()
IMGUR_CLIENT_ID = os.getenv("IMGUR_CLIENT_ID")

async def upload_image(img_data):
    headers = {
        "Authorization": f"Client-ID {IMGUR_CLIENT_ID}"
    }
    data = {
        "image": img_data
    }

    async with AsyncClient() as client:
        response = await client.post("https://api.imgur.com/3/image", headers=headers, data=data)

    if response.status_code != 200:
        raise HTTPException(status_code=500, detail="Could not upload image.")

    print(response.json())
    return response.json()["data"]["link"]

Step 4: Setting up our backend application

Now, we will create a new file named app.py and add our 🐘 ElephantSQL PostgreSQL database connection and code for our APIs, including face detection logic.

In this file, we would need to create the following routes:

GET / - Root route

POST /is_admin/ - Check if the user is an admin

POST /suspicious_list/ - This route retrieves a list of participants who are detected as suspicious

POST /same_faces/ - Detect if the face in the screenshot is the same as the face in the reference image of the candidate.

POST /upload_photo/ - Upload a reference photo of the candidate

POST /meetings - Create a new meeting

POST /meetings/{meetingId}/participants - This route is responsible for adding a participant to a specific meeting identified by meetingId

So let's get started 👇

app.py

import base64
import io
import logging
import random
import requests

import uvicorn
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from imgur import upload_image
import face_recognition
import psycopg2

import os
import base64
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from dotenv import load_dotenv
from httpx import AsyncClient
import uuid

load_dotenv()

DYTE_API_KEY = os.getenv("DYTE_API_KEY")
DYTE_ORG_ID = os.getenv("DYTE_ORG_ID")

API_HASH = base64.b64encode(f"{DYTE_ORG_ID}:{DYTE_API_KEY}".encode('utf-8')).decode('utf-8')

DYTE_API = AsyncClient(base_url='https://api.cluster.dyte.in/v2', headers={'Authorization': f"Basic {API_HASH}"})

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

fh = logging.FileHandler("app.log")
fh.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
fh.setFormatter(formatter)
logger.addHandler(fh)


class ParticipantScreen(BaseModel):
    base64_img: str
    participant_id: str
    meeting_id: str
    ref_image_url: str
    participant_name: str

class ProctorPayload(BaseModel):
    meeting_id: str
    admin_id: str

class AdminProp(BaseModel):
    meeting_id: str
    admin_id: str

class Meeting(BaseModel):
    title: str

class Participant(BaseModel):
    name: str
    preset_name: str
    meeting_id: str

origins = [
    # allow all
    "*",
]

app = FastAPI()

# enable cors
app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],  # allow all
    allow_headers=["*"],  # allow all
)

def connect_to_db():
    conn = psycopg2.connect(
            dbname=os.getenv('DB_USER'),
            user=os.getenv('DB_USER'),
            password=os.getenv('DB_PASSWORD'),
            host=os.getenv('DB_HOST'),
            port=5432
    )
    return conn

@app.get("/")
async def root():
    return {"message": "Hello World"}

@app.post("/is_admin/")
async def is_admin(admin: AdminProp):
    conn = connect_to_db()
    cur = conn.cursor()
    cur.execute("SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s", (admin.meeting_id, admin.admin_id,))

    count = cur.fetchone()[0]

    if(count > 0):
        return { "admin": True }
    else:
        return { "admin": False }

@app.post("/suspicious_list/")
async def suspicious_list(meeting: ProctorPayload):
    conn = connect_to_db()
    cur = conn.cursor()
    cur.execute("SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s", (meeting.meeting_id, meeting.admin_id,))

    count = cur.fetchone()[0]

    if(count > 0):
        cur.execute("SELECT * FROM meeting_proc_details WHERE meeting_id = %s ORDER BY ts DESC", (meeting.meeting_id,))
        rows = cur.fetchall()
        conn.commit()
        cur.close()
        conn.close()
        return rows
    else:
        conn.commit()
        cur.close()
        conn.close()
        raise HTTPException(status_code=401, detail="Participant dose not has admin role")

@app.post("/same_faces/")
async def same_faces(participant: ParticipantScreen):
    conn = connect_to_db()
    cur = conn.cursor()

    cur.execute("CREATE TABLE IF NOT EXISTS meeting_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), img_url VARCHAR(255), verdict VARCHAR(255))")

    img_url = participant.ref_image_url

    response = requests.get(img_url)
    ref_img_data = io.BytesIO(response.content)

    img_data = participant.base64_img.split(",")[1]
    img_data_dc = base64.b64decode(participant.base64_img.split(",")[1])

    file_obj = io.BytesIO(img_data_dc)
    unknown_image = face_recognition.load_image_file(file_obj)
    known_image = face_recognition.load_image_file(ref_img_data)

    biden_encoding = face_recognition.face_encodings(known_image)[0]
    unknown_encoding = face_recognition.face_encodings(unknown_image)[0]

    results = face_recognition.compare_faces([biden_encoding], unknown_encoding)
    is_same_face = True
    for res in results:
        is_same_face = is_same_face and res


    if not is_same_face:
        logger.info(
            f"Detected different faces for participant {participant.participant_id}"
        )

        verdict = f"Participant Name: {participant.participant_name} <> Anomaly: Different Faces Detected <> Participant ID: {participant.participant_id}"
        cur.execute("SELECT count(1) FROM meeting_proc_details WHERE meeting_id=%s AND participant_id=%s AND ts >= (current_timestamp - INTERVAL '10 minutes')", (participant.meeting_id, participant.participant_id))
        count = cur.fetchone()[0]

        if count == 0:
            upload_resp = await upload_image(img_data)
            cur.execute("INSERT INTO meeting_proc_details (ts, meeting_id, participant_id, img_url, verdict) VALUES (current_timestamp, %s, %s, %s, %s)",
                (participant.meeting_id, participant.participant_id, upload_resp, verdict)
            )

        conn.commit()
        cur.close()
        conn.close()

        if count == 0:
            return { "id": participant.participant_id, "different_faces_detected": True, "url": upload_resp }
        return { "id": participant.participant_id, "different_faces_detected": True, "url": "not uploaded" }

    return {"id": participant.participant_id, "different_faces_detected": False}

@app.post("/upload_photo/")
async def upload_photo(participant: ParticipantScreen):
    img_data = participant.base64_img.split(",")[1]

    conn = connect_to_db()
    cur = conn.cursor()

    cur.execute("CREATE TABLE IF NOT EXISTS meeting_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), img_url VARCHAR(255), verdict VARCHAR(255))")

    verdict = f"REF_IMAGE"

    upload_resp = await upload_image(img_data)
    cur.execute("INSERT INTO meeting_proc_details (ts, meeting_id, participant_id, img_url, verdict) VALUES (current_timestamp, %s, %s, %s, %s)",
        (participant.meeting_id, participant.participant_id, upload_resp, verdict)
    )

    conn.commit()
    cur.close()
    conn.close()

    return {"id": participant.participant_id, "img_url": upload_resp, "reference_image_uploaded": True}

@app.post("/meetings")
async def create_meeting(meeting: Meeting):
    response = await DYTE_API.post('/meetings', json=meeting.dict())
    if response.status_code >= 300:
        raise HTTPException(status_code=response.status_code, detail=response.text)
    admin_id = ''.join(random.choices('abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789', k=32))
    resp_json = response.json()
    resp_json['admin_id'] = admin_id
    meeting_id = resp_json['data']['id']

    conn = connect_to_db()
    cur = conn.cursor()
    cur.execute("INSERT INTO meeting_host_info (ts, meeting_id, admin_id) VALUES (CURRENT_TIMESTAMP, %s, %s)", (meeting_id, admin_id))
    conn.commit()
    cur.close()
    conn.close()

    return resp_json


@app.post("/meetings/{meetingId}/participants")
async def add_participant(meetingId: str, participant: Participant):
    client_specific_id = f"react-samples::{participant.name.replace(' ', '-')}-{str(uuid.uuid4())[0:7]}"
    payload = participant.dict()
    payload.update({"client_specific_id": client_specific_id})
    del payload['meeting_id']
    resp = await DYTE_API.post(f'/meetings/{meetingId}/participants', json=payload)
    if resp.status_code > 200:
        raise HTTPException(status_code=resp.status_code, detail=resp.text)
    return resp.text

if __name__ == "__main__":
    uvicorn.run("app:app", host="localhost", port=8000, log_level="debug", reload=True)

This code defines a ⚡️ FastAPI application with an endpoint /detect_faces, which takes in a base64 encoded image and returns a boolean value indicating if the face matches the reference image of the candidate.

It uses the face recognition library to detect faces in the image received.

The code also makes use of the Dyte API for meeting-related operations. 📹

We can start the backend server simply by using the following command 🧑‍💻:

python app.py

This Python server helps us create and join meetings, detect different faces, and get the list of suspicious candidates. 🕵️

Here, when we hit the /same_faces endpoint with an image file encoded as a base64 string, the different_faces_detected key of the response would be set to True if the face of the candidate from the reference image doesn't match the face in the screenshot, else it will be set to False.

We can call this from our frontend with the participant's webcam feed.

With this sorted, let's return to our React application and create our UI. ✨

Step 5: Setting up the Meeting UI

First, let us add our CSS file, create a new file, frontend/src/App.css and paste the following code.

.App {
    text-align: center;
}

.App-logo {
    height: 40vmin;
    pointer-events: none;
}

@media (prefers-reduced-motion: no-preference) {
    .App-logo {
        animation: App-logo-spin infinite 20s linear;
    }
}

.App-header {
    background-color: #282c34;
    min-height: 100vh;
    display: flex;
    flex-direction: column;
    align-items: center;
    justify-content: center;
    font-size: calc(10px + 2vmin);
    color: white;
}

.App-link {
    color: #61dafb;
}

.heading-proctor {
    font-size: x-large;
    font-weight: bolder;
    color: #fff;
}

@keyframes App-logo-spin {
    from {
        transform: rotate(0deg);
    }
    to {
        transform: rotate(360deg);
    }
}

.file-input label {
    margin-top: 20px;
    display: block;
    position: relative;
    width: 200px;
    height: 50px;
    border-radius: 10px;
    background-color: #2160fd;
    display: flex;
    align-items: center;
    justify-content: center;
    flex-direction: row;
    color: #fff;
    font-weight: bold;
    cursor: pointer;
    transition: transform 0.2s ease-out;
}

.file {
    opacity: 0;
    width: 0.1px;
    height: 0.1px;
    position: absolute;
}

Next, we will add the initial Dyte Meeting component to our app. We can do this by replacing the contents of frontend/src/App.jsx with the following code:

import { useEffect, useState } from "react";
import Home from "./Home";
import { BrowserRouter, Routes, Route, Link } from "react-router-dom";
import "./App.css";
import Stage from "./Stage";

const SERVER_URL = process.env.SERVER_URL || "http://localhost:8000";

function App() {
    const [meetingId, setMeetingId] = useState();

    const createMeeting = async () => {
        const res = await fetch(`${SERVER_URL}/meetings`, {
            method: "POST",
            body: JSON.stringify({ title: "Joint Entrance Examination" }),
            headers: { "Content-Type": "application/json" },
        });
        const resJson = await res.json();
        window.localStorage.setItem("adminId", resJson.admin_id);
        setMeetingId(resJson.data.id);
    };

    useEffect(() => {
        window.localStorage.removeItem("refImgUrl");
        const id = window.location.pathname.split("/")[2];
        if (!!!id) {
            createMeeting();
        }
    }, []);

    return (
        <BrowserRouter>
            <Routes>
                <Route path="/" element={<Home meetingId={meetingId} />}></Route>
                <Route path="/meeting/:meetingId" element={<Stage />}></Route>
            </Routes>
        </BrowserRouter>
    );
}

export default App;

This component will create a Dyte meeting link and an adminId for the admin. We will store the adminId secretly in localstorage. The adminId will be used later for accessing any sensitive data.

Home component

The home component renders the / route. Create a file as frontend/src/Home.jsx.

import { Link } from "react-router-dom";
function Home({ meetingId }) {
    return (
        <div
            style={{
                height: "100vh",
                width: "100vw",
                fontSize: "x-large",
                display: "flex",
                justifyContent: "center",
                alignItems: "center",
            }}
        >
            {meetingId && !window.location.pathname.split("/")[2] && (
                <Link to={`/meeting/${meetingId}`}>Create and Join Meeting</Link>
            )}
        </div>
    );
}

export default Home;

Heading component

Now we will create a file as frontend/src/Heading.jsx.

const Heading = ({ text }) => {
    return (
        <div
            className="heading-proctor"
            style={{
                padding: "10px",
                textAlign: "center",
                backgroundColor: "#000",
                borderBottom: "solid 0.5px gray",
            }}
        >
            {text}
        </div>
    );
};

export default Heading;

Let us now build the component to upload a reference image of the candidate.

Create a file frontend/src/ImageInput.jsx

import { useState } from "react";

function ImageUploader({ setImgUploadedStatus }) {
    const [selectedFile, setSelectedFile] = useState(null);
    const [uploading, setUploading] = useState(false);
    const [uploadError, setUploadError] = useState(null);
    const [uploadSuccess, setUploadSuccess] = useState(false);

    const handleFileChange = (event) => {
        setSelectedFile(event.target.files[0]);
        setUploadSuccess(false);
        setUploadError(null);
    };

    const handleSubmit = async (event) => {
        event.preventDefault();
        setUploading(true);
        setUploadError(null);

        try {
            const formData = new FormData();
            formData.append("image", selectedFile);

            const response = await fetch("https://api.imgur.com/3/image", {
                method: "POST",
                headers: {
                    Authorization: "Client-ID 48f0caef7256b40",
                },
                body: formData,
            });

            if (!response.ok) {
                throw new Error("Failed to upload image");
            }

            const data = await response.json();
            window.localStorage.setItem("refImgUrl", data.data.link);
            setUploadSuccess(data.data.link);
            setImgUploadedStatus(true);
        } catch (error) {
            setUploadError(error.message);
        } finally {
            setUploading(false);
        }
    };

    return (
        <div
            style={{
                display: "flex",
                justifyContent: "center",
                flexDirection: "column",
                alignItems: "center",
                fontSize: "x-large",
            }}
        >
            <div>Hey Candidate 👋</div>
            <div style={{ marginBottom: "30px" }}>
                Please upload a image of yours to proceed
            </div>
            <input
                type="file"
                id="file"
                style={{ border: "0.5px white solid" }}
                onChange={handleFileChange}
            />
            <form
                onSubmit={handleSubmit}
                style={{
                    display: "flex",
                    flexDirection: "column",
                    alignItems: "center",
                    marginTop: "1rem",
                }}
            >
                <button
                    type="submit"
                    disabled={!selectedFile || uploading || uploadSuccess}
                    style={{
                        marginTop: "10px",
                        padding: "0.5rem 1rem",
                        backgroundColor: "#3460f3",
                        color: "#fff",
                        border: "none",
                        borderRadius: "0.25rem",
                        cursor: "pointer",
                    }}
                >
                    {uploading ? "Uploading..." : "Upload"}
                </button>
            </form>
            {uploadError && (
                <div style={{ marginTop: "10px", color: "red" }}>{uploadError}</div>
            )}
        </div>
    );
}

export default ImageUploader;

It will look like this 👇

Let's create a staging area for participants joining the meeting. Admin bypasses the staging area, but the candidate will be asked to upload a reference image of himself on this page.

For this, we will create another file, frontend/src/Stage.jsx

import { useState, useEffect } from "react";
import ImageUploader from "./ImageInput";
import Meet from "./Meet";

const SERVER_URL = process.env.REACT_APP_SERVER_URL || "http://localhost:8000";

const Stage = () => {
    const [isAdminBool, setAdminBool] = useState(null);
    const [isRefImgUploaded, setImgUploadedStatus] = useState(null);
    const meetingId = window.location.pathname.split("/")[2];

    const isAdmin = async (id) => {
        const res = await fetch(`${SERVER_URL}/is_admin`, {
            method: "POST",
            body: JSON.stringify({
                admin_id: window.localStorage.getItem("adminId") || "",
                meeting_id: meetingId || "",
            }),
            headers: { "Content-Type": "application/json" },
        });
        const resJson = await res.json();
        setAdminBool(resJson.admin);
    };

    useEffect(() => {
        isAdmin();
    }, []);

    return (
        <div
            style={{
                height: "100vh",
                width: "100vw",
                display: "flex",
                justifyContent: "center",
                alignItems: "center",
                color: "white",
            }}
        >
            {isAdminBool == null ? (
                <>Loading...</>
            ) : (
                <>
                    {isAdminBool || (isAdminBool === false && isRefImgUploaded) ? (
                        <Meet isAdminBool={isAdminBool} />
                    ) : (
                        <ImageUploader setImgUploadedStatus={setImgUploadedStatus} />
                    )}
                </>
            )}
        </div>
    );
};

export default Stage;

Now, let's delve into the Meet component that renders on route /meeting/:meetingId.

When the admin clicks on the link provided on the / route, he gets redirected to the meeting page, where we add the user to the meeting as a participant with group_call_host preset. 🤙

Since this user created the meeting and was redirected to the meet page, we will assign him the admin role. Now the link from the address bar can be shared with the candidates.

When a candidate opens the shared link, they become a regular user. And for every regular user, the component emits screenshots of the users' videos to our directed to our Python server. 🐍

/* eslint-disable */

import { useState, useEffect, useRef } from "react";
import { DyteMeeting, provideDyteDesignSystem } from "@dytesdk/react-ui-kit";
import { useDyteClient } from "@dytesdk/react-web-core";
import Proctor from "./Proctor";
import Heading from "./Heading";
import { SendImageToBackendMiddleware, joinMeeting } from "./utils";
// Constants
const SERVER_URL = process.env.SERVER_URL || "http://localhost:8000";
let LAST_BACKEND_PING_TIME = 0;
const DETECT_FACES_ENDPOINT = `${SERVER_URL}/same_faces`;
const TIME_BETWEEN_BACKEND_PINGS = 10000;

const Meet = () => {
    const meetingEl = useRef();
    const [meeting, initMeeting] = useDyteClient();
    const [userToken, setUserToken] = useState();
    const [isAdminBool, setAdminBool] = useState(null);
    const meetingId = window.location.pathname.split("/")[2];

    function SendImageToBackendMiddleware() {
        return async (canvas, ctx) => {
            const currentTime = Date.now();
            if (currentTime - LAST_BACKEND_PING_TIME > TIME_BETWEEN_BACKEND_PINGS) {
                LAST_BACKEND_PING_TIME = currentTime;
                const imgBase64String = canvas.toDataURL("image/png");
                const response = await fetch(DETECT_FACES_ENDPOINT, {
                    method: "POST",
                    headers: {
                        "Content-Type": "application/json",
                    },
                    body: JSON.stringify({
                        base64_img: imgBase64String,
                        participant_id: meeting?.self.id,
                        participant_name: meeting?.self.name,
                        ref_image_url: window.localStorage.getItem("refImgUrl"),
                        meeting_id: meetingId,
                    }),
                });
            }
        };
    }

    const isAdmin = async (id) => {
        const res = await fetch(`${SERVER_URL}/is_admin`, {
            method: "POST",
            body: JSON.stringify({
                admin_id: window.localStorage.getItem("adminId") || "",
                meeting_id: meetingId || "",
            }),
            headers: { "Content-Type": "application/json" },
        });
        const resJson = await res.json();
        setAdminBool(resJson.admin);
    };

    const joinMeetingId = async () => {
        if (meetingId) {
            const authToken = await joinMeeting(meetingId);
            await initMeeting({
                authToken,
            });
            setUserToken(authToken);
        }
    };

    useEffect(() => {
        if (meetingId && !userToken) joinMeetingId();
        isAdmin();
    }, []);

    useEffect(() => {
        if (userToken) {
            provideDyteDesignSystem(meetingEl.current, {
                theme: "dark",
            });
        }
    }, [userToken]);

    useEffect(() => {
        if (isAdminBool === false && meeting?.self) {
            meeting.self.addVideoMiddleware(SendImageToBackendMiddleware);
        }
    }, [meeting?.self]);

    return (
        <div style={{ height: "96vh", width: "100vw", display: "flex" }}>
            {userToken && (
                <>
                    {isAdminBool && (
                        <div
                            style={{
                                width: "40vw",
                                height: "100vh",
                                overflowY: "scroll",
                                backgroundColor: "black",
                                borderRight: "solid 0.5px gray",
                            }}
                        >
                            <Heading text={"Proctoring Information"} />
                            <Proctor meeting={meeting} />
                        </div>
                    )}
                    {isAdminBool ? (
                        <div style={{ width: "60vw", height: "96vh" }}>
                            <Heading text={"Proctoring Admin Interface"} />
                            <DyteMeeting mode="fill" meeting={meeting} ref={meetingEl} />
                        </div>
                    ) : (
                        <div style={{ width: "100vw", height: "96vh" }}>
                            <Heading text={"Proctoring Candidate Interface"} />
                            <DyteMeeting mode="fill" meeting={meeting} ref={meetingEl} />
                        </div>
                    )}
                </>
            )}
        </div>
    );
};

export default Meet;

Let's briefly go through some of the functions:

isAdmin talks to the Python server to identify whether the current client is the admin.
joinMeeting adds the current client to the meeting.
SendImageToBackendMiddleware sends screenshots of candidates' videos to the Python server.

Proctor component

The proctor component gets activated only for admins. The proctor component, with the help of adminId fetches the list of suspicious candidates and renders it in a chat-like format. Create a file frontend/src/Proctor.jsx

import { useEffect, useState } from "react";
import { getCandidateStatus } from "./utils";

const Proctor = () => {
    const [candidateStatuses, updateCandidateStatusState] = useState([]);
    const [error, setError] = useState("");

    const updateCandidateStatus = async () => {
        try {
            const res = await getCandidateStatus();
            updateCandidateStatusState(res);
        } catch (e) {
            setError("User don't have admin privileges.");
        }
    };

    useEffect(() => {
        updateCandidateStatus();
    }, []);

    useEffect(() => {
        if (candidateStatuses?.map) {
            const id = setInterval(() => {
                updateCandidateStatus();
            }, 30000);
            return () => {
                clearInterval(id);
            };
        }
    }, [candidateStatuses]);

    return (
        <>
            <div style={{ padding: "0px 20px" }}>
                {candidateStatuses?.map && candidateStatuses ? (
                    candidateStatuses.map((status) => (
                        <div
                            style={{
                                display: "flex",
                                justifyContent: "start",
                                margin: "50px 20px",
                            }}
                        >
                            <div style={{ marginRight: "20px" }}>
                                <img
                                    src="https://images.yourstory.com/cs/images/companies/Dyte-1608553297314.jpg"
                                    style={{
                                        borderRadius: "50px",
                                        height: "60px",
                                        border: "1px double lightblue",
                                    }}
                                />
                            </div>
                            <div
                                style={{
                                    textAlign: "center",
                                    padding: "20px",
                                    backgroundColor: "#2160fd",
                                    fontSize: "large",
                                    fontWeight: "400",
                                    borderRadius: "10px 10px 10px 10px",
                                    width: "80%",
                                }}
                            >
                                <div
                                    style={{
                                        color: "white",
                                        padding: "20px 0px",
                                        textAlign: "left",
                                    }}
                                >
                                    {status[4].split("<>").map((text) => (
                                        <div>{text}</div>
                                    ))}
                                    <div>Timestamp: {new Date(status[0]).toLocaleString()}</div>
                                </div>
                                <img
                                    style={{ borderRadius: "10px", width: "100%" }}
                                    src={status[3]}
                                />
                            </div>
                        </div>
                    ))
                ) : (
                    <div style={{ color: "white" }}>
                        Wait or check if you have admin privileges to access the proctoring
                        dashboard.
                    </div>
                )}
            </div>
        </>
    );
};

export default Proctor;

Utility functions

Now we will add utility functions.

Create a file frontend/src/utils.js.

const joinMeeting = async (id) => {
    const res = await fetch(`http://localhost:8000/meetings/${id}/participants`, {
        method: "POST",
        body: JSON.stringify({
            name: "new user",
            preset_name: "group_call_host",
            meeting_id: meetingId,
        }),
        headers: { "Content-Type": "application/json" },
    });
    const resJson = await res.json();
    console.log(resJson.detail);
    const data = JSON.parse(resJson.detail);
    return data.data.token;
};

const getCandidateStatus = async () => {
    const response = await fetch("http://localhost:8000/suspicious_list", {
        method: "POST",
        headers: {
            "Content-Type": "application/json",
        },
        body: JSON.stringify({
            meeting_id: window.location.pathname.split("/")[2],
            admin_id: window.localStorage.getItem("adminId") || "undefined",
        }),
    });
    const res = await response.json();
    if (res.details) return undefined;
    console.log(res);
    return res;
};

export { joinMeeting, getCandidateStatus };

To start the React app on the local server, we can run the following command:

yarn start

Now, upon visiting http://localhost:3000/, we should be able to see the Dyte meeting in our browser.

Step 6: Adding the face detection logic to the frontend

Since now we have a nice backend server to detect faces and a great UI 🎉, we can add the face detection logic to our frontend. For this, we will first add some constants to our previously edited frontend/src/App.jsx file:

We will be using the above constants in the SendImageToBackendMiddleware function, which we will add to our App component just after the useDyteClient hook. 🪝

The SendImageToBackendMiddleware is a Dyte Video Middleware. Middlewares are add-ons that we can use to easily add effects and filters to your audio and video streams.

Here, we use the middleware functionality to get the canvas object of the participant's webcam feed, convert it to a base64 encoded image, and send it to our backend server.

That was all the code we needed to add basic proctoring functionality to our Dyte meeting. 👍

The app sends a screenshot of the participant's webcam feed to the backend server every 30 seconds. If the backend detects the face isn't similar to the reference image, it sends a warning notification to the proctor. ⚠️

The backend also logs the participant's ID and the time of the detection in the terminal. This can be used to keep track of the participants who may have cheated during the meeting for later review.

Step 7: Trying out our live proctoring system

Ta-da! 🎩✨ It's time to put our live proctoring system to the test and see it in action!

First, let's look at the candidate's view after uploading the reference image. The candidate can see that the proctor is in the meeting but cannot see the Proctoring Panel. 🧑‍💻

Now, let's swap the candidates.

In the proctor's view, we can see the details (proctoring information) along with proof when the candidate got swapped in the candidate's webcam view. 🙌

🧑‍💻 You can try out the live proctoring system here. And here's the link to the repository for you to take a look at the whole codebase.

Conclusion

Celebrate! 🎉✨ We've built a powerful live proctoring system with Dyte, ensuring integrity and fairness in online exams and interviews. But that's not all! We can now create our own customized online classroom or meeting platform.

We can now use this live proctoring system to proctor our online exams and interviews. ✍️

The possibilities are endless with Dyte, go ahead and try bringing your ideas to life by visiting dyte.io! 🚀

Building a Live Proctoring System to Detect Multiple Speakers

Ravindra Singh Rathor — Thu, 28 Dec 2023 19:44:13 +0000

Introduction

In the early days of online education, as the need arose to maintain the integrity of exams, proctoring emerged as a trusted monitoring method to prevent students from engaging in dishonest behavior. As the digital landscape expanded and online courses became more prevalent, the traditional ways of proctoring started to show their limitations.

Consider online exams, where having a physical proctor for every student becomes impractical. This gave rise to the adoption of live automatic proctoring.

This modern approach harnesses the power of a student’s webcam and microphone to oversee the exam process. By incorporating computer vision and machine learning, it efficiently detects cheating attempts.

In this tutorial, we’ll create a live proctoring system using Dyte APIs, enabling an admin to detect the presence of multiple speakers in a candidate’s background throughout the exam in real-time. ✨

High-Level Design of the application

Our aim is to notify the proctor if we hear multiple voices from the candidate’s background.

The proctor would get the candidate's details right in his meeting sidebar. 🔊✨

In this project, we will use React with Dyte UI kit and Dyte React Web Core packages for the frontend.
For the backend, we will use FastApi (Python 3).
We will also use database as a service by ElephantSQL (PostgreSQL).
Lastly, we will be using Deepgram Audio Diarization APIs to get speaker details from a sample audio clip.

Folder Structure

After completing the tutorial, the folder structure will look like this. 👇

dyte-proctoring/
├── frontend
│   ├── README.md
│   ├── package.json
│   ├── public
│   ├── src
│   │   ├── App.css
│   │   ├── App.jsx
│   │   ├── Heading.jsx
│   │   ├── ImageInput.jsx
│   │   ├── Meet.jsx
│   │   ├── Proctor.jsx
│   │   ├── index.css
│   │   ├── index.jsx
│   │   ├── logo.svg
│   │   ├── react-app-env.d.ts
│   │   ├── reportWebVitals.ts
│   │   ├── setupTests.ts
│   │   ├── stage.jsx
│   │   └── utils.js
│   ├── tsconfig.json
├── venv
├── app.py
├── audio_diarization.py
└── requirements.txt

Step 0: Configurations and Setup

🧑‍💻 Before building our live proctoring system, we must set up a Dyte account.

We can create a free account by clicking the "Start Building" button on Dyte.io and signing up using Google or GitHub. 🎉

Once signed up, we can access our Dyte API keys from the "API Keys" tab in the left sidebar. We will keep these keys secure as we will use them later. 🔑🤫

For our live proctoring system, we will use React for the frontend and FastAPI for building the Backend and APIs.

We will begin by creating a new directory for our project, called dyte-proctoring, and navigating into it using the following commands:

mkdir dyte-proctoring
cd dyte-proctoring

NOTE

We will require to create accounts on the following platforms:

ElephantSQL: Here is a step-by-step guide to create a db on ElephantSQL.
Deepgram: Here is a step-by-step guide to start using Deepgram.

Now back to the tutorial.

Step 1: Setting up the Frontend

Let's start setting up our frontend project using React and Dyte! ✨

We will create a boilerplate React app using create-react-app. We can do this with the following command:

yarn create react-app frontend

This will initialize a new React app in the frontend directory. 📁

Then, we will go ahead and install the dyte react-web-core, dyte react-ui-kit and react-router packages in this project using the following command 👇

yarn add @dytesdk/react-web-core @dytesdk/react-ui-kit react-router react-router-dom

Step 2: Setting up the backend

Let's get started with setting up our FastAPI backend now. 🙌

We will go back to the root directory of our project and initiate our project here itself for the ease of hosting:

cd ..

First of all, we will go ahead and create our requirements.txt file in the root directory itself with the following content 👇

requirements.txt

cmake
fastapi
uvicorn
face_recognition
numpy
python-multipart
psycopg2-binary
httpx
python-dotenv
pydantic
requests

After this, we will go ahead and create our virtual environment with venv and install the dependencies.

python -m venv venv
source venv/bin/activate # for linux/mac
venv\Scripts\activate.bat # for windows
pip install -r requirements.txt

We will also create an environment variable file .env, for storing our credentials.

.env

DYTE_ORG_ID=<ID>
DYTE_API_KEY=<KEY>
IMGUR_CLIENT_ID=<ID>
DB_USER=<ID>
DB_PASSWORD=<PASSWORD>
DB_HOST=<HOST>
DEEPGRAM_API_KEY=<API_KEY>

Step 3: Setting up our backend application

Now, we will create a new file named app.py and add our 🐘 ElephantSQL PostgreSQL database connection and code for our APIs, including face detection logic.

In this file, we would need to create the following routes:

GET / - Root route

POST /is_admin/ - Check if the user is an admin

POST /multiple_voices_list/ - This route retrieves a list of participants who are detected as suspicious

POST /multiple_voices/ - Detect if there are multiple voices in the audio stream

POST /meetings - Create a new meeting

POST /meetings/{meetingId}/participants - This route is responsible for adding a participant to a specific meeting identified by meetingId

So let’s get started 👇

app.py

import base64
import io
import logging
import random
import requests

import uvicorn
from fastapi import FastAPI, Form, UploadFile, File
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
import psycopg2
from utils import upload_audio

import os
import base64
from fastapi import FastAPI, HTTPException
from dotenv import load_dotenv
from httpx import AsyncClient
import httpx
import uuid

from audio_diarization import speakers_count

load_dotenv()

DYTE_API_KEY = os.getenv("DYTE_API_KEY")
DYTE_ORG_ID = os.getenv("DYTE_ORG_ID")

API_HASH = base64.b64encode(f"{DYTE_ORG_ID}:{DYTE_API_KEY}".encode('utf-8')).decode('utf-8')

timeout = httpx.Timeout(10.0, read=None)
DYTE_API = AsyncClient(base_url='https://api.cluster.dyte.in/v2', headers={'Authorization': f"Basic {API_HASH}"}, timeout=timeout)

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

fh = logging.FileHandler("app.log")
fh.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
fh.setFormatter(formatter)
logger.addHandler(fh)


class ProctorPayload(BaseModel):
    meeting_id: str
    admin_id: str

class AdminProp(BaseModel):
    meeting_id: str
    admin_id: str

class Meeting(BaseModel):
    title: str

class Participant(BaseModel):
    name: str
    preset_name: str
    meeting_id: str

origins = [
    # allow all
    "*",
]

app = FastAPI()

# enable cors
app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],  # allow all
    allow_headers=["*"],  # allow all
)

def connect_to_db():
    conn = psycopg2.connect(
            dbname=os.getenv('DB_USER'),
            user=os.getenv('DB_USER'),
            password=os.getenv('DB_PASSWORD'),
            host=os.getenv('DB_HOST'),
            port=5432
    )
    return conn

@app.get("/")
async def root():
    return {"message": "Hello World"}

@app.post("/is_admin/")
async def multiple_faces_list(admin: AdminProp):
    conn = connect_to_db()
    cur = conn.cursor()
    cur.execute("SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s", (admin.meeting_id, admin.admin_id,))

    count = cur.fetchone()[0]

    if(count > 0):
        return { "admin": True }
    else:
        return { "admin": False }

@app.post("/multiple_voices_list/")
async def multiple_faces_list(meeting: ProctorPayload):
    conn = connect_to_db()
    cur = conn.cursor()
    cur.execute("SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s", (meeting.meeting_id, meeting.admin_id,))

    count = cur.fetchone()[0]

    if(count > 0):
        cur.execute("CREATE TABLE IF NOT EXISTS meeting_audio_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), audio_url VARCHAR(255), verdict VARCHAR(255))")
        cur.execute("SELECT * FROM meeting_audio_proc_details WHERE meeting_id = %s ORDER BY ts DESC", (meeting.meeting_id,))
        rows = cur.fetchall()
        conn.commit()
        cur.close()
        conn.close()
        return rows
    else:
        conn.commit()
        cur.close()
        conn.close()
        raise HTTPException(status_code=401, detail="Participant dose not has admin role")

@app.post("/multiple_voices/")
async def multiple_voices(file: UploadFile = File(...), meeting_id: str = Form(...), participant_id: str = Form(...), participant_name: str = Form(...)):
    contents = file.file.read()
    filename = file.filename.split('.')[0] + str(random.randint(1, 100)) + '.mp3'
    with open(filename, 'wb') as f:
        f.write(contents)
    file.file.close()

    conn = connect_to_db()
    cur = conn.cursor()

    cur.execute("CREATE TABLE IF NOT EXISTS meeting_audio_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), audio_url VARCHAR(255), verdict VARCHAR(255))")

    try:
        count = speakers_count(filename)
    except Exception as e:
        logger.error(e)

    if count > 1:
        logger.info(
            f"Detected different voices for participant {participant_id}"
        )

        verdict = f"Participant Name: {participant_name} <> Anomaly: Different Voices Detected <> Participant ID: {participant_id}"
        cur.execute("SELECT count(1) FROM meeting_audio_proc_details WHERE meeting_id=%s AND participant_id=%s AND ts >= (current_timestamp - INTERVAL '10 minutes')", (meeting_id, participant_id))
        count = cur.fetchone()[0]

        if count == 0:
            # if there is a requirement of storing audio files
            upload_resp = await upload_audio(img_data)
            cur.execute("INSERT INTO meeting_audio_proc_details (ts, meeting_id, participant_id, audio_url, verdict) VALUES (current_timestamp, %s, %s, %s, %s)",
                (meeting_id, participant_id, upload_resp, verdict)
            )

        conn.commit()
        cur.close()
        conn.close()

        if count == 0:
            return { "id": participant_id, "different_voices_detected": True, "url": upload_resp }
        return { "id": participant_id, "different_voices_detected": True, "url": "not uploaded" }

    return {"id": participant_id, "different_voices_detected": False}

@app.post("/meetings")
async def create_meeting(meeting: Meeting):
    response = await DYTE_API.post('/meetings', json=meeting.dict())
    if response.status_code >= 300:
        raise HTTPException(status_code=response.status_code, detail=response.text)
    admin_id = ''.join(random.choices('abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789', k=32))
    resp_json = response.json()
    resp_json['admin_id'] = admin_id
    meeting_id = resp_json['data']['id']

    conn = connect_to_db()
    cur = conn.cursor()
    cur.execute("INSERT INTO meeting_host_info (ts, meeting_id, admin_id) VALUES (CURRENT_TIMESTAMP, %s, %s)", (meeting_id, admin_id))
    conn.commit()
    cur.close()
    conn.close()

    return resp_json


@app.post("/meetings/{meetingId}/participants")
async def add_participant(meetingId: str, participant: Participant):
    client_specific_id = f"react-samples::{participant.name.replace(' ', '-')}-{str(uuid.uuid4())[0:7]}"
    payload = participant.dict()
    payload.update({"client_specific_id": client_specific_id})
    del payload['meeting_id']
    resp = await DYTE_API.post(f'/meetings/{meetingId}/participants', json=payload)
    if resp.status_code > 200:
        raise HTTPException(status_code=resp.status_code, detail=resp.text)
    return resp.text

if __name__ == "__main__":
    uvicorn.run("app:app", host="localhost", port=8000, log_level="debug", reload=True)

This code defines a ⚡️ FastAPI application with an endpoint /multiple_voices, which takes in an audio file in binary format and returns a boolean value indicating if there are multiple voices in the audio submitted.

It uses the audio_diarization api by Deepgram API service to detect multiple voices in the audio received.

audio_diarization.py

import os
from httpx import AsyncClient
import requests
from dotenv import load_dotenv

load_dotenv()


DEEPGRAM_API_KEY = os.getenv("DEEPGRAM_API_KEY")
DEEPGRAM_API = AsyncClient(base_url='https://api.deepgram.com/v1', headers={'Authorization': f"Token {DEEPGRAM_API_KEY}", 'Content-Type': "audio/mp3"})

def speakers_count(audio_file_path):
    speaker_set = set()
    audio = open(audio_file_path, 'rb')
    response = requests.post('https://api.deepgram.com/v1/listen?diarize=true', data=audio, headers={'Authorization': f"Token {DEEPGRAM_API_KEY}", 'Content-Type': "audio/mp3"})
    resp_json = response.json()
    statements = resp_json['results']['channels'][0]['alternatives']
    for statement in statements:
        words = statement['words']
        for word in words:
            speaker_set.add(word['speaker'])
    return len(speaker_set)

The code also makes use of the Dyte API for meeting-related operations. 📹

We can start the backend server simply by using the following command:

python app.py

This Python server helps us create and join meetings, detect different faces, and get the list of suspicious candidates. 🕵️

Here, when we hit the /multiple_voices endpoint with an audio file, the multiple_voices_detected key of the response would be set to True if there are multiple voices in the audio file received, else it will be set to False.

We can call this from our frontend with the participant’s mic.

With this sorted, let's return to our React application and create our UI. ✨

Step 4: Setting up the Meeting UI

First, let us add our CSS file, create a new file frontend/src/App.css and paste the following code.

.App {
    text-align: center;
}

.App-logo {
    height: 40vmin;
    pointer-events: none;
}

@media (prefers-reduced-motion: no-preference) {
    .App-logo {
        animation: App-logo-spin infinite 20s linear;
    }
}

.App-header {
    background-color: #282c34;
    min-height: 100vh;
    display: flex;
    flex-direction: column;
    align-items: center;
    justify-content: center;
    font-size: calc(10px + 2vmin);
    color: white;
}

.App-link {
    color: #61dafb;
}

.heading-proctor {
    font-size: x-large;
    font-weight: bolder;
    color: #fff;
}

@keyframes App-logo-spin {
    from {
        transform: rotate(0deg);
    }
    to {
        transform: rotate(360deg);
    }
}

.file-input label {
    margin-top: 20px;
    display: block;
    position: relative;
    width: 200px;
    height: 50px;
    border-radius: 10px;
    background-color: #2160fd;
    display: flex;
    align-items: center;
    justify-content: center;
    flex-direction: row;
    color: #fff;
    font-weight: bold;
    cursor: pointer;
    transition: transform 0.2s ease-out;
}

.file {
    opacity: 0;
    width: 0.1px;
    height: 0.1px;
    position: absolute;
}

Next, we will add the initial Dyte Meeting component to our app. We can do this by replacing the contents of frontend/src/App.jsx with the following code:

import { useEffect, useState } from "react";
import Home from "./Home";
import { BrowserRouter, Routes, Route, Link } from "react-router-dom";
import "./App.css";
import Stage from "./Stage";

const SERVER_URL = process.env.REACT_APP_SERVER_URL || "http://localhost:8000";

function App() {
    const [meetingId, setMeetingId] = useState();

    const createMeeting = async () => {
        const res = await fetch(`${SERVER_URL}/meetings`, {
            method: "POST",
            body: JSON.stringify({ title: "Joint Entrance Examination" }),
            headers: { "Content-Type": "application/json" },
        });
        const resJson = await res.json();
        window.localStorage.setItem("adminId", resJson.admin_id);
        setMeetingId(resJson.data.id);
    };

    useEffect(() => {
        window.localStorage.removeItem("refImgUrl");
        const id = window.location.pathname.split("/")[2];
        if (!!!id) {
            createMeeting();
        }
    }, []);

    return (
        <BrowserRouter>
            <Routes>
                <Route path="/" element={<Home meetingId={meetingId} />}></Route>
                <Route path="/meeting/:meetingId" element={<Stage />}></Route>
            </Routes>
        </BrowserRouter>
    );
}

export default App;

This component will create a Dyte meeting link and an adminId for the admin. We will store the adminId secretly in localstorage. The adminId will be used later for accessing any sensitive data.

Home component

The home component renders the / route. Create a file as frontend/src/Home.jsx.

import { Link } from "react-router-dom";
function Home({ meetingId }) {
    return (
        <div
            style={{
                height: "100vh",
                width: "100vw",
                fontSize: "x-large",
                display: "flex",
                justifyContent: "center",
                alignItems: "center",
            }}
        >
            {meetingId && !window.location.pathname.split("/")[2] && (
                <Link to={`/meeting/${meetingId}`}>Create and Join Meeting</Link>
            )}
        </div>
    );
}

export default Home;

Heading component

Now we will create a file as frontend/src/Heading.jsx.

const Heading = ({ text }) => {
    return (
        <div
            className="heading-proctor"
            style={{
                padding: "10px",
                textAlign: "center",
                backgroundColor: "#000",
                borderBottom: "solid 0.5px gray",
            }}
        >
            {text}
        </div>
    );
};

export default Heading;

Let’s create a staging area for participants joining the meeting. Admin bypasses the staging area, but the candidate will be asked to upload a reference image of himself on this page.

For this, we will create another file, frontend/src/Stage.jsx

import { useState, useEffect } from "react";
import Meet from "./Meet";

const SERVER_URL = process.env.REACT_APP_SERVER_URL || "http://localhost:8000";

const Stage = () => {
    const [isAdminBool, setAdminBool] = useState(null);
    const meetingId = window.location.pathname.split("/")[2];

    const isAdmin = async (id) => {
        const res = await fetch(`${SERVER_URL}/is_admin`, {
            method: "POST",
            body: JSON.stringify({
                admin_id: window.localStorage.getItem("adminId") || "",
                meeting_id: meetingId || "",
            }),
            headers: { "Content-Type": "application/json" },
        });
        const resJson = await res.json();
        setAdminBool(resJson.admin);
    };

    useEffect(() => {
        isAdmin();
    }, []);

    return (
        <div
            style={{
                height: "100vh",
                width: "100vw",
                display: "flex",
                justifyContent: "center",
                alignItems: "center",
                color: "white",
            }}
        >
            {isAdminBool == null ? (
                <>Loading...</>
            ) : (
                <>
                    <Meet isAdminBool={isAdminBool} />
                </>
            )}
        </div>
    );
};

export default Stage;

Now, let's delve into the Meet component that renders on route /meeting/:meetingId.

When the admin clicks on the link provided on the / route, he gets redirected to the meeting page, where we add the user to the meeting as a participant with audio_proc_preset preset. 🤙

Since this user created the meeting and was redirected to the meet page, we will assign him the admin role. Now the link from the address bar can be shared with the candidates.

When a candidate opens the shared link, they become a regular user. And for every regular user, the component emits screenshots of the users' videos to our directed to our Python server. 🐍

/* eslint-disable */

import { useState, useEffect, useRef } from "react";
import { DyteMeeting, provideDyteDesignSystem } from "@dytesdk/react-ui-kit";
import { useDyteClient } from "@dytesdk/react-web-core";
import Proctor from "./Proctor";
import Heading from "./Heading";
import { joinMeeting } from "./utils";
import lamejs from "lamejstmp";

// Constants
const SERVER_URL = process.env.REACT_APP_SERVER_URL || "http://localhost:8000";
let LAST_BACKEND_PING_TIME = 0;
const TIME_BETWEEN_BACKEND_PINGS = 60000;

function convertFloat32ToInt16(buffer) {
    var l = buffer.length;
    var buf = new Int16Array(l);
    while (l--) {
        buf[l] = Math.min(1, buffer[l]) * 0x7fff;
    }
    return buf;
}

const Meet = () => {
    const meetingEl = useRef();
    const [meeting, initMeeting] = useDyteClient();
    const [userToken, setUserToken] = useState();
    const [isAdminBool, setAdminBool] = useState(null);
    const meetingId = window.location.pathname.split("/")[2];

    async function audioToMp3Middleware(audioContext) {
        const processor = audioContext.createScriptProcessor(1024, 1, 1);
        const encoder = new lamejs.Mp3Encoder(1, 44100, 128);
        const bufferSize = 512;
        let mp3Data = [];

        processor.onaudioprocess = (e) => {
            const inputData = e.inputBuffer.getChannelData(0);
            const mp3buf = encoder.encodeBuffer(convertFloat32ToInt16(inputData));
            if (mp3buf.length > 0) {
                mp3Data.push(mp3buf);
            }
            const currentTime = Date.now();
            if (
                mp3Data.length >= bufferSize &&
                currentTime - LAST_BACKEND_PING_TIME > TIME_BETWEEN_BACKEND_PINGS
            ) {
                LAST_BACKEND_PING_TIME = currentTime;
                const file = new File(mp3Data, "audio.mp3");
                const form = new FormData();
                form.append("file", file);
                form.append("meeting_id", meetingId);
                form.append("participant_id", meeting?.self.id);
                form.append("participant_name", meeting?.self.name);

                fetch(`${SERVER_URL}/multiple_voices`, {
                    method: "POST",
                    body: form,
                });

                mp3Data = [];
            }
        };

        return processor;
    }

    const isAdmin = async (id) => {
        const res = await fetch(`${SERVER_URL}/is_admin`, {
            method: "POST",
            body: JSON.stringify({
                admin_id: window.localStorage.getItem("adminId") || "",
                meeting_id: meetingId || "",
            }),
            headers: { "Content-Type": "application/json" },
        });
        const resJson = await res.json();
        setAdminBool(resJson.admin);
    };

    const joinMeetingId = async () => {
        if (meetingId) {
            const authToken = await joinMeeting(meetingId);
            await initMeeting({
                authToken,
            });
            setUserToken(authToken);
        }
    };

    useEffect(() => {
        if (meetingId && !userToken) joinMeetingId();
        isAdmin();
    }, []);

    useEffect(() => {
        if (userToken) {
            provideDyteDesignSystem(meetingEl.current, {
                theme: "dark",
            });
        }
    }, [userToken]);

    useEffect(() => {
        if (isAdminBool === false && meeting?.self) {
            meeting.self.addAudioMiddleware(audioToMp3Middleware);
        }

        return () => {
            if (isAdminBool === false && meeting?.self) {
                meeting.self.removeAudioMiddleware(audioToMp3Middleware);
            }
        };
    }, [meeting?.self, isAdminBool]);

    return (
        <div style={{ height: "96vh", width: "100vw", display: "flex" }}>
            {userToken && (
                <>
                    {isAdminBool && (
                        <div
                            style={{
                                width: "40vw",
                                height: "100vh",
                                overflowY: "scroll",
                                backgroundColor: "black",
                                borderRight: "solid 0.5px gray",
                            }}
                        >
                            <Heading text={"Proctoring Information"} />
                            <Proctor />
                        </div>
                    )}
                    {isAdminBool ? (
                        <div style={{ width: "60vw", height: "96vh" }}>
                            <Heading text={"Proctoring Admin Interface"} />
                            <DyteMeeting mode="fill" meeting={meeting} ref={meetingEl} />
                        </div>
                    ) : (
                        <div style={{ width: "100vw", height: "96vh" }}>
                            <Heading text={"Proctoring Candidate Interface"} />
                            <DyteMeeting mode="fill" meeting={meeting} ref={meetingEl} />
                        </div>
                    )}
                </>
            )}
        </div>
    );
};

export default Meet;

Let’s briefly go through some of the functions:

isAdmin talks to the Python server to identify whether the current client is the admin.
joinMeeting adds the current client to the meeting.
audioToMp3Middleware sends audio samples of candidates to the Python server.

Proctor component

The proctor component gets activated only for admins. The proctor component, with the help of adminId fetches the list of suspicious candidates and renders it in a chat-like format.

Create a file frontend/src/Proctor.jsx.

import { useEffect, useState } from "react";
import { getCandidateStatus } from "./utils";

const Proctor = () => {
    const [candidateStatuses, updateCandidateStatusState] = useState([]);
    const [error, setError] = useState("");

    const updateCandidateStatus = async () => {
        try {
            const res = await getCandidateStatus();
            updateCandidateStatusState(res);
        } catch (e) {
            setError("User don't have admin privileges.");
        }
    };

    useEffect(() => {
        if (candidateStatuses?.map) {
            const id = setInterval(() => {
                updateCandidateStatus();
            }, 30000);
            return () => {
                clearInterval(id);
            };
        }
    }, [candidateStatuses]);

    return (
        <>
            <div style={{ padding: "0px 20px" }}>
                {candidateStatuses?.map && candidateStatuses ? (
                    candidateStatuses.map((status) => (
                        <div
                            style={{
                                display: "flex",
                                justifyContent: "start",
                                margin: "50px 20px",
                            }}
                        >
                            <div style={{ marginRight: "20px" }}>
                                <img
                                    src="https://images.yourstory.com/cs/images/companies/Dyte-1608553297314.jpg"
                                    style={{
                                        borderRadius: "50px",
                                        height: "60px",
                                        border: "1px double lightblue",
                                    }}
                                />
                            </div>
                            <div
                                style={{
                                    textAlign: "center",
                                    padding: "20px",
                                    backgroundColor: "#2160fd",
                                    fontSize: "large",
                                    fontWeight: "400",
                                    borderRadius: "10px 10px 10px 10px",
                                    width: "80%",
                                }}
                            >
                                <div
                                    style={{
                                        color: "white",
                                        padding: "20px 0px",
                                        textAlign: "left",
                                    }}
                                >
                                    {status[4].split("<>").map((text) => (
                                        <div>{text}</div>
                                    ))}
                                    <div>Timestamp: {new Date(status[0]).toLocaleString()}</div>
                                </div>
                            </div>
                        </div>
                    ))
                ) : (
                    <div style={{ color: "white" }}>
                        Wait or check if you have admin privileges to access the proctoring
                        dashboard.
                    </div>
                )}
            </div>
        </>
    );
};

export default Proctor;

Utility functions

Now we will add utility functions.

Create a file frontend/src/utils.js.

const SERVER_URL = process.env.REACT_APP_SERVER_URL || "http://localhost:8000";

const joinMeeting = async (id) => {
    const res = await fetch(`${SERVER_URL}/meetings/${id}/participants`, {
        method: "POST",
        body: JSON.stringify({
            name: "new user",
            preset_name: "audio_proc_preset",
            meeting_id: id,
        }),
        headers: { "Content-Type": "application/json" },
    });
    const resJson = await res.json();
    const data = JSON.parse(resJson.detail);
    return data.data.token;
};

const getCandidateStatus = async () => {
    const response = await fetch(`${SERVER_URL}/multiple_voices_list`, {
        method: "POST",
        headers: {
            "Content-Type": "application/json",
        },
        body: JSON.stringify({
            meeting_id: window.location.pathname.split("/")[2],
            admin_id: window.localStorage.getItem("adminId") || "undefined",
        }),
    });
    const res = await response.json();
    if (res.details) return undefined;
    return res;
};

const FloatArray2Int16 = (floatbuffer) => {
    var int16Buffer = new Int16Array(floatbuffer.length);
    for (var i = 0, len = floatbuffer.length; i < len; i++) {
        if (floatbuffer[i] < 0) {
            int16Buffer[i] = 0x8000 * floatbuffer[i];
        } else {
            int16Buffer[i] = 0x7fff * floatbuffer[i];
        }
    }
    return int16Buffer;
};

export { joinMeeting, getCandidateStatus, FloatArray2Int16 };

To start the React app on the local server, we can run the following command:

yarn start

Now, upon visiting http://localhost:3000/, we should be able to see the Dyte meeting in our browser.

Step 5: Adding multiple voice detection logic to the Frontend

Since now we have a nice backend server to detect multiple voices and a great UI, we can add the multiple voice detection logic to our frontend. For this, we will first add some constants to our previously edited frontend/src/App.jsx file:

We will be using the above constants in the audioToMp3Middleware function, which we will add to our Meet component.

The audioToMp3Middleware is a Dyte Audio Middleware. Middlewares are add-ons that we can use to add effects and filters to your audio and video streams with ease.

Here, we are using the middleware functionality to get the audio sample from the participant’s mic and send it to our backend server. We are also ensuring that the backend is pinged only once every 60 seconds to avoid unnecessary load on the server.

That was all the code we needed to add basic proctoring functionality to our Dyte meeting. 👍

The app sends an audio sample from the participant’s mic to the backend server every 60 seconds, and if the backend detects multiple voices in the audio sample, it sends a warning notification to the proctor. ⚠️

The backend also logs the participant's ID and the time of the detection in the terminal. This can be used to keep track of the participants who may have cheated during the meeting for later review.

Step 6: Trying out our project

Ta-da! 🎩✨ It’s time to put our proctoring system to the test and see it in action!

First, let's look at the candidate's view after uploading the reference image. The candidate can see that the proctor is in the meeting but cannot see the Proctoring Panel. 🧑‍💻

In the proctor’s view, we can see the details (proctoring information) along with proof when two or more people can be heard talking in the candidate’s background. 🙌

Here's the link to the repository for you to take a look at the whole codebase. You can also try out the project here. 🔗

Conclusion

We can now use this live proctoring system to proctor our online exams and interviews. ✍️

The possibilities are endless with Dyte; go ahead and try to bring your ideas to life by visiting dyte.io! 🚀

Add Transcription to Your Video Calls With AWS Transcribe

Ravindra Singh Rathor — Tue, 10 Oct 2023 06:21:08 +0000

With the rise of remote job opportunities and geographically dispersed teams, effective communication is more crucial than ever. It is essential that there are no gaps in understanding among team members from diverse locations, ensuring everyone feels included and valued.

In a previous blog, we explored the seamless integration of an audio transcriptions and translation solution using Google STT (Speech to Text) in conjunction with Dyte. This powerful combination showcased its potential to foster improved communication among teams that are geographically dispersed and culturally diverse in modern settings, catering to various races, ethnicities, and cultural backgrounds.

Now, we have devised an AWS (Amazon Web Services) based solution that overcomes not only the barriers between people but also those between cloud providers. This new solution offers efficient transcriptions and translations, further enhancing cross-cultural collaboration and communication. Azure, you're up next for our exploration!

In this blog, we will discuss the integration of a third-party service from Amazon Web Services named AWS Transcribe with Dyte. As the participants speak, it will add transcription to audio/video meetings. These transcriptions, with the help of AWS translate, another service from AWS, can then be converted to the desired language of individual participants, as needed.

As usual, there are prerequisites.

Prerequisites

Before we dive into the process, it's important to note that you should have a Dyte meeting already created to add a transcription of the audio of the meeting's participants.

Here are some useful resources for your quick navigation:

Make sure you've read the Getting Started with Dyte topic and completed the following steps:

Create a Dyte Developer Account
Create Presets
Create a Dyte Meeting
Add Participant to the meeting
Integrate with the SDK of your choice

Assuming you have successfully integrated Dyte into your website or application, let's dive into the process of adding transcriptions in the Dyte meetings.

Adding transcription components

To achieve the audio transcriptions, we need to call AWS (Amazon Web Services) Services: AWS Transcribe & AWS Translate.

These AWS services are paid. Please note that Dyte does NOT act as a broker between AWS and any of our clients that wants to integrate AWS services with Dyte SDK. We recommend that our clients should directly connect with the AWS team.

However, we at Dyte, provide free of cost code samples so that our clients can seamlessly integrate with such services.

Integration Steps

1. Setup AWS IAM account, get credentials

To use AWS services, you should either be an IAM user, or your backend services should be deployed using roles & policies.

For ease of use, we are going ahead with IAM user credentials. Please ensure that the IAM user can actually use AWS transcribe & AWS translate services.

To proceed further, we need accessKeyId, secretAccessKey & the preferred region where you would want your services to be deployed in the future.

Once you have these, let’s proceed to the next step.

2. Setup a backend server to call these APIs securely

If you have used AWS services before, you would know that these services are costly, they can charge you a lot. Obviously, you would not want to expose your credentials for miscreants to exploit.

If you put these credentials by any chance in your frontend code, anyone would be able to get a hold of this and land you with a huge sum of charge. Also, if you expose an endpoint that is not performing checks to validate users and blindly providing these services to anyone with an API endpoint link, it would be a potential security risk.

Therefore we will not expose these credentials. We will put them in your backend services for safer usage. This would give you better control to ensure that no unauthorized personnel can use your APIs & secrets.

For this, we have provided the BE sample in NodeJS for you to see. Please find it here. Currently, we only have an ExpressJS sample, if you working on a different backend, feel free to port this code or connect with us so that we can help you out in porting it.

To use this sample, Please clone this using the following command.

git clone git@github.com:dyte-in/aws-transcribe.git

Or simply download it as a Zip. Click on Code (Green Icon → Download ZIP) and extract once downloaded.

Now that you have downloaded the repository. Please proceed with the following steps to set up your backend endpoint.

2.1 Go to the server folder

cd server

2.2 Replicate .env.example as .env

cp .env.example .env

Open the .env in your choice of Text File Editor. Edit it as per your AWS service account credentials and Save it.

2.3 Install NPM packages

This would install @aws-sdk/client-translate automatically. Instead of @aws-sdk/client-transcribe-streaming, we will be using WebSocket with presigned urls.

npm install

2.4 Run the server

npm run dev

If successful, you would see the confirmation in Terminal that it is running on localhost:3001 or the PORT specified in the .env file.

This endpoint (https://localhost:3001) will be called the backend_url from now onwards. We advise you to copy the code from https://github.com/dyte-in/aws-transcribe/blob/main/server/src/index.ts and alter it as per your need and security practices.

We strongly advise you not to use this code as is, without a proper security mechanism, for production.

3. Integrate the SDK in the Front end project

Now that the backend setup is ready let’s start with the frontend setup.

3.1 Install the FE package in your FE codebase

The first action would be to install Dyte’s wrapper package to take out complexity from you in front-end integration using:

npm install @dytesdk/aws-transcribe

if you are interested in knowing what this package does behind the scene, and this is something we recommend you to check out, refer to https://github.com/dyte-in/aws-transcribe/tree/main/client so that you can further customize the code to your liking. This is necessary as well so that you can put proper security measures. It is advised to copy-paste the files of the client folder and alter them instead of using this package directly in production.

3.2. Integrate the AWS Transcribe SDK with Dyte Meeting

The second step is to look for the place in your codebase where you are initiating a Dyte meeting.

Once you have found the place and got a hold of the meeting object. Add the following code on top of the file to import the FE SDK.

import DyteAWSTranscribe from '@dytesdk/aws-transcribe';

Add the following code just after the point where you have access to the meeting object.

const awsTranscribe = new DyteAWSTranscribe({
        meeting,
    target: 'hi', // Optional if translate is false, Supported languages: https://docs.aws.amazon.com/translate/latest/dg/what-is-languages.html
    translate: true, // Control whether to translate the source language to target language or just transcribe
    source: 'en-US', // Supported languages: https://docs.aws.amazon.com/translate/latest/dg/what-is-languages.html
    preSignedUrlEndpoint: 'http://localhost:3001/aws-transcribe-presigned-url',
    translationEndpoint: 'http://localhost:3001/translate', // ${backend_url}/translate. backend_url is from step 2.4
});

awsTranscribe.on('transcription', async (data) => {
    // ... do something with transcription
});

awsTranscribe.transcribe();

Here you are setting up the DyteAWSTranscribe with the values that the current user would prefer and activating the recognition just afterward using awsTranscribe.transcribe(). Then we are listening to every new transcription using awsTranscribe.on(’transcription’, aJsCallbackFunction)

To see the support languages, please refer to https://docs.aws.amazon.com/transcribe/latest/dg/supported-languages.html and https://docs.aws.amazon.com/translate/latest/dg/what-is-languages.html.

With this, now you would be able to receive the live transcriptions. Feel free to put them in UI as per your need.

If you need a sample of how to do it, please refer to https://github.com/dyte-in/aws-transcribe/blob/main/client/demo/index.ts.

<!DOCTYPE html>
<html lang="en">
   <head>
      <meta charset="UTF-8" />
      <meta http-equiv="X-UA-Compatible" content="IE=edge" />
      <meta name="viewport" content="width=device-width, initial-scale=1.0" />
      <title>Google Transcriptions Demo </title>
      <!-- UI Kit -->
      <script type="module">
         import { defineCustomElements } from 'https://cdn.jsdelivr.net/npm/@dytesdk/ui-kit/loader/index.es2017.js';
         defineCustomElements();
      </script>
      <!-- Web Core -->
      <script src="https://cdn.dyte.in/core/dyte.js"></script>
      <script src="https://cdn.jsdelivr.net/npm/@dytesdk/aws-transcribe@0.0.1/dist/index.umd.js"></script>
      <style>
         body {
         height: 100vh;
         width: 100vw;
         }
         #dyte-transcriptions{
         position: absolute;
         z-index: 99999;
         bottom: 15%;
         width: 100%;
         display:flex;
         flex-direction: column;
         justify-content: center;
         align-items:center;
         }
         .dyte-transcription-line{
         display: block;
         max-width: 80%;
         text-align: center !important;
         }
         .dyte-transcription-speaker{
         font-weight: 500;
         color: orange;
         }
         .dyte-transcription-text{
         color: white;
         }
      </style>
   </head>
   <body>
      <dyte-meeting id="my-meeting"></dyte-meeting>
      <div id="dyte-transcriptions"></div>

      <script>
         const roomName = ''; // Put roomName if using v1 APIs. Leave Blank for v2 APIs
         const authToken = ''; // Put v1 or v2 participant auth token

         async function initMeeting() {
           const meetingEl = document.getElementById('my-meeting');

           const meeting = await DyteClient.init({
             authToken,
             roomName,
             defaults: {
               audio: false,
               video: false,
             },
           });

           meetingEl.meeting = meeting;

           const awsTranscribe = new DyteAWSTranscribe({
               target: 'hi', // Optional if translate is false, Supported languages: Supported languages: https://docs.aws.amazon.com/translate/latest/dg/what-is-languages.html
                           translate: true, // Control whether to translate the source language to target language or just transcribe
                         source: 'en-US', // Supported languages: https://docs.aws.amazon.com/transcribe/latest/dg/supported-languages.html
                         preSignedUrlEndpoint: 'http://localhost:3001/aws-transcribe-presigned-url',
                         translationEndpoint: 'http://localhost:3001/translate',
           });


           awsTranscribe.on('transcription', async () => {
               const transcription = document.getElementById('dyte-transcriptions');
               const list = awsTranscribe.transcriptions.slice(-3);
               transcription.innerHTML = '';
               list.forEach((item) => {
                   const speaker = document.createElement('span');
                   speaker.classList.add('dyte-transcription-speaker');
                   speaker.innerText = `${item.name}: `;

                   const text = document.createElement('span');
                   text.classList.add('dyte-transcription-text');
                   text.innerText = item.transcript.trim() !== '' ? item.transcript : '...';

                   const container = document.createElement('span');
                   container.classList.add('dyte-transcription-line');
                   container.appendChild(speaker);
                   container.appendChild(text);

                   transcription.appendChild(container);
               });
           });

           awsTranscribe.transcribe();
         }

         window.onload = async function () {
           initMeeting();
         };
      </script>
   </body>
</html>

That’s it. You have successfully added transcriptions and custom language translations to your video calling app using AWS Transcribe.

I hope you found this post informative and engaging. If you have any thoughts or feedback, please get in touch with me on Twitter or LinkedIn. Stay tuned for more related blog posts in the future!

If you haven't heard about Dyte yet, head over to dyte.io to learn how we are revolutionizing communication through our SDKs and libraries and how you can get started quickly on your 10,000 free minutes, which renew every month. If you have any questions, you can reach us at support@dyte.io or ask our developer community.

Add Audio Transcriptions to Video Meetings With Google STT

Ravindra Singh Rathor — Wed, 21 Jun 2023 12:34:00 +0000

In this new era of an inclusive world, people from all races, ethnicities, and diverse cultures are collaborating and creating new ways to make this world a better place. More and more differently-abled people can participate in such a feat with pride.

On top of that, with the ever-growing remote job profiles and geographically diversely situated teams, it has become even more crucial that there are no gaps in communication among items and that the members from such diverse locations can better understand each other and feel included.

One such step in this direction is to transcribe the words that other team members are speaking. A native Asian might not understand a British accent on the first go. Similarly, an American might not understand a single French sentence if the sentence is like this. In such a case, it becomes necessary that the words a team member is speaking get transcribed for others to read so they don't feel left out.

Therefore, this blog will discuss how you can add audio transcriptions to your Dyte meetings. Not just that, we will also show you how you can convert audio transcriptions to any language that you desire. A person who understands English can see the audio transcriptions in English in real-time, even when the actual speaker is speaking French.

You can achieve this by integrating a third-party service from Google, Google Speech-to-Text (STT), in Dyte meetings. It would transcribe the audio of meeting participants as they speak. With the help of Google translate, another service from Google, these transcriptions can then be converted to the desired language of individual participants, as needed.

As usual, there are prerequisites.

Prerequisites

Before we dive into the process, it's important to note that you should have a Dyte meeting already created to transcribe the audio of participants of that meeting. Here are some valuable resources for your quick navigation:

Make sure you've read the Getting Started with Dyte topic and completed the following steps:

Create a Dyte Developer Account
Create Presets
Create a Dyte Meeting
Add Participants to the meeting
Integrate with the SDK of your choice

Assuming you have successfully integrated Dyte into your website or application, let's add audio transcriptions to your Dyte meetings.

Integration components

To achieve the audio transcriptions, we need to call GCP (Google Cloud Platform) Services: Google Media Translations and Google Translations API.

These Google services are paid. Please note that Dyte does NOT act as a broker between Google and any our clients that wants to integrate Google's services into Dyte. We recommend that our clients should directly connect with the GCP team.

However, we at Dyte provide free-of-cost code samples so our clients can seamlessly integrate with such services.

Integration steps to add audio transcription

1. Setup Google Cloud, get credentials

You must have a service account with GCP (Google Cloud Platform) to use Google transcriptions. Please create a project in that account that allows Google Media Translations and Google Translations API.

Once done, download the keys for the service account.

It would look like this:

{
  "type": "service_account",
  "project_id": "YOUR_GCP_PROJECT_ID",
  "private_key_id": "YOUR_GCP_PRIVATE_KEY_ID",
  "private_key": "-----BEGIN PRIVATE SOMETHING SOMETHING-----END PRIVATE KEY-----\\n",
  "client_email": "xxxxxx@yyyyyy.iam.gserviceaccount.com",
  "client_id": "YOUR_GCP_CLIENT_ID",
  "auth_uri": "<https://accounts.google.com/o/oauth2/auth>",
  "token_uri": "<https://oauth2.googleapis.com/token>",
  "auth_provider_x509_cert_url": "<https://www.googleapis.com/oauth2/v1/certs>",
  "client_x509_cert_url": "<https://www.googleapis.com/robot/v1/metadata/x509/xxxxx%40yyyyy.iam.gserviceaccount.com>"
}

Once you have these, let's proceed to the next step.

2. Setup a backend Server to call these APIs securely

If you have used Google services before, you would know that these services are costly. They can charge you a lot. You would not want to expose your credentials for miscreants to exploit.

If you put these credentials in your frontend code, anyone could get a hold of this and land you with a considerable charge.

Therefore, we will not expose these credentials. We will put them in your backend services for safer usage. This would give you better control to ensure no unauthorized personnel can use your APIs and access your information.

For this, we have provided the BE sample in NodeJS for you to see. Please find it here. Currently, we only have NodeJS samples; if you're working on a different backend, feel free to port this code or connect with us to help you port it.

To use this sample, Please clone this using the following command.

git clone git@github.com:dyte-in/google-transcription.git

Or simply download it as a Zip. Use Code (Green Icon → Download ZIP) and extract once downloaded.

Now that you have downloaded the repository. Please proceed with the following steps to set up your backend endpoint.

2.1 Go to the server folder

cd server

2.2 Replicate .env.example as .env

cp .env.example .env

Open the .env in your choice of Text File Editor. Edit it as per your GCP service account credentials and Save it.

Note: PRIVATE_KEY should be in a single line. Try picking the value from the service account's key's JSON file as is.

2.3 Install NPM packages

This would automatically install @google-cloud/media-translation, @google-cloud/speech, and @google-cloud/translate.

npm install

2.4 Run the server

npm run dev

If successful, you will see the confirmation in Terminal that it is running on localhost:3001 or the PORT specified in the .env file.

This endpoint (https://localhost:3001) will be called the backend_url from now on. We advise you to copy the code from Dyte Google Transcription and alter it as per your need and security practices.

We strongly advise you not to use this code as is, without a proper security mechanism, for production.

3. Integrate the SDK in the frontend project

Now that the backend setup is ready let's start with the frontend setup.

3.1 Install the FE package in your FE codebase

The first action would be to install Dyte's wrapper package to take out complexity from you in frontend integration using:

npm install @dytesdk/google-transcription

If you are interested in knowing what this package does behind the scenes, and this is something we recommend you to check out, refer to https://github.com/dyte-in/google-transcription/tree/main/client so that you can further customize the code to your liking.

3.2. Integrate the Google Transcription SDK with the Dyte meeting

The second step is to look for the place in your codebase where you are initiating a Dyte meeting.

Once you have found the place and got a hold of the meeting object, add the following code to the file to import the FE SDK.

import DyteGoogleSpeechRecognition from '@dytesdk/google-transcription';

Add the following code just after the point where you have access to the meeting object.

const speech = new DyteGoogleSpeechRecognition({
    meeting, // Dyte meeting object from DyteClient.init
    target: 'hi', // Language that the current user wants to see
    source: 'en-US', // Language that the current user would speak in
    baseUrl: <backend-url>, // Backend URL from step 2.4
});

speech.on('transcription', async (data) => {
    // ... do something with transcription
});

speech.transcribe();

Here you are setting up the GoogleSpeechRecognition with the values that the current user would prefer and activating the recognition just afterward using speech.transcribe(). Then we listen to every new transcription using speech.on(’transcription’, aJsCallbackFunction)

To see the support languages, please refer to

With this, you would now be able to receive the live transcriptions. Feel free to put them in UI as per your need. Please refer to this if you need a sample of how to do it.

<!DOCTYPE html>
<html lang="en">
   <head>
      <meta charset="UTF-8" />
      <meta http-equiv="X-UA-Compatible" content="IE=edge" />
      <meta name="viewport" content="width=device-width, initial-scale=1.0" />
      <title>Google Transcriptions Demo </title>
      <!-- UI Kit -->
      <script type="module">
         import { defineCustomElements } from 'https://cdn.jsdelivr.net/npm/@dytesdk/ui-kit/loader/index.es2017.js';
         defineCustomElements();
      </script>
      <!-- Web Core -->
      <script src="https://cdn.dyte.in/core/dyte.js"></script>
      <script src="https://cdn.jsdelivr.net/npm/@dytesdk/google-transcription@1.0.1/dist/index.umd.min.js"></script>
      <style>
         body {
         height: 100vh;
         width: 100vw;
         }
         #dyte-transcriptions{
         position: absolute;
         z-index: 99999;
         bottom: 15%;
         width: 100%;
         display:flex;
         flex-direction: column;
         justify-content: center;
         align-items:center;
         }
         .dyte-transcription-line{
         display: block;
         max-width: 80%;
         text-align: center !important;
         }
         .dyte-transcription-speaker{
         font-weight: 500;
         color: orange;
         }
         .dyte-transcription-text{
         color: white;
         }
      </style>
   </head>
   <body>
      <dyte-meeting id="my-meeting"></dyte-meeting>
      <div id="dyte-transcriptions"></div>

      <script>
         const roomName = ''; // Put roomName if using v1 APIs. Leave Blank for v2 APIs
         const authToken = ''; // Put v1 or v2 participant auth token

         async function initMeeting() {
           const meetingEl = document.getElementById('my-meeting');

           const meeting = await DyteClient.init({
             authToken,
             roomName,
             defaults: {
               audio: false,
               video: false,
             },
           });

           meetingEl.meeting = meeting;

           const speech = new DyteGoogleSpeechRecognition({
               meeting,
               target: 'hi',
               source: 'en-US',
               baseUrl: 'http://localhost:3001', // Replace with backend_url from step 2.4
           });


           speech.on('transcription', async () => {
               const transcription = document.getElementById('dyte-transcriptions');
               const list = speech.transcriptions.slice(-3);
               transcription.innerHTML = '';
               list.forEach((item) => {
                   const speaker = document.createElement('span');
                   speaker.classList.add('dyte-transcription-speaker');
                   speaker.innerText = `${item.name}: `;

                   const text = document.createElement('span');
                   text.classList.add('dyte-transcription-text');
                   text.innerText = item.transcript.trim() !== '' ? item.transcript : '...';

                   const container = document.createElement('span');
                   container.classList.add('dyte-transcription-line');
                   container.appendChild(speaker);
                   container.appendChild(text);

                   transcription.appendChild(container);
               });
           });

           speech.transcribe();
         }

         window.onload = async function () {
           initMeeting();
         };
      </script>
   </body>
</html>

Check out how the audio transcription will appear in your Dyte Meetings here.

Now you have a full-fledged working Google Speech-to-Text integration with custom language translations for your Dyte Meetings.

I hope you found this post informative and engaging. If you have any thoughts or feedback, please get in touch with me on Twitter or LinkedIn. Stay tuned for more related blog posts in the future!

If you haven't heard about Dyte yet, head over to dyte.io to learn how we are revolutionizing communication through our SDKs and libraries and how you can get started quickly on your 10,000 free minutes, which renew every month. If you have any questions, you can reach us at support@dyte.io ask our developer community.