DEV Community: Arjhun777

Split Images by Face Recognizing using React(FaceApi)

Arjhun777 — Mon, 11 Jan 2021 12:29:17 +0000

For Implementing Face Recognization, there are two major steps to be followed

Provided GitHub link below

Need to provide images of a particular person and name to the train.
Then need to provide all the images which need to split which will be compared against our trained data to split.
1) First Create a react app and then we need to provide the models to faceAPI and these resolved models will be used across the components to recognize the face.
React - ./loadModules.js

import * as faceApi from 'face-api.js';
export const loadModules = () => {
    return Promise.all([
        faceApi.nets.faceRecognitionNet.loadFromUri('/src/models'),
        faceApi.nets.faceLandmark68Net.loadFromUri('/src/models'),
        faceApi.nets.ssdMobilenetv1.loadFromUri('/src/models')
    ])
}
export default faceApi

2) Create a home component, Now add an input field, file upload option, and add a button to add the images of a particular person to get the trained data for that person.
React - ./home.tsx

import React, { useState } from "react";
import { Button, Input } from '@material-ui/core';
import React, { useState } from "react";
import { Button, Input } from '@material-ui/core';
import JSZip from 'jszip';
import { saveAs } from 'file-saver';
import faceApi from "../../../loadModules";
import { matchFacesBy, singleFaceDetectionWithImage } from "../../../utils/helpers";

const Home = (props:HomeProps) => {
    const [formData, setFormData] = useState({
        name: '',
        faceData: [],
        actualImages: []
    });
    const [checkPoint, setCheckPoint] = useState([]);
    const [submitDisable, setSubmitDisable] = useState(true);
    const [trainComplete, setTrainComplete] = useState(false);
    const [trainedSet, setTrainedSet] = useState([]);
    const [finalResult, setFinalResult] = useState([]);
    const [duplicate, setDuplicate] = useState(false);    

    const handleNameChange = (event:any) => {
        const { value } = event.target;
        setFormData({ ...formData, name: value });
    }
    const handleSubmit = (event:any) => {
        event.preventDefault();
        checkPoint.push(formData);
        setCheckPoint(checkPoint);
        setFormData({
            name: '',
            faceData: [],
            actualImages: []
        });
    }
    const handleCompareImage = async (event:any) => {
       ..// wil be implemented and discussed below
    }
    return (
        <React.Fragment>
            <div className="form-container">
                <div className="form-title">Upload Known Faces to split</div>
                <form onSubmit={handleSubmit}>
                    <Input type="text" onChange={handleNameChange} 
                        placeholder="Enter The Name" value={formData.name} />
                    <Button variant="contained" component="label" 
                        onChange={handleCompareImage} >
                        Upload Known Face
                        <input type="file" multiple style={{ display: "none" }}/>
                    </Button>
                    <Button color="primary" type="submit" 
                        disabled={submitDisable}>ADD
                    </Button>
                </form>
         </React.Fragment>
    )
}

Here just providing the person name and images and on uploading images it will be passed to handleCompareImages here the images of the person are detected one by one and will be pushed to our form data.
handleCompareImage - ./home.tsx

    const handleCompareImage = async (event:any) => {
        const { files } = event.target;
        setSubmitDisable(true);
        let actualImages:any = [];
        let faceDetections:any = [];
        for (let index = 0; index < files?.length; index++) {
            const file = files[index];
            const result:any = await singleFaceDetectionWithImage(file)
            if (result.singleFaceDetection) {
                actualImages.push(result.actualImage);
                faceDetections.push(result.singleFaceDetection?.descriptor);
            }
        }
        setFormData({ 
            ...formData, 
            faceData: faceDetections, 
            actualImages: actualImages 
        });
        setSubmitDisable(false);
    }

after executing the handleCompareImage we will again enable the add button. Here we are looping through the person images and detecting the face of the person on each image and getting face details data and storing it in formData.

singleFaceDetectionWithImage function will contain the logic to face detection and returning image and face details data.

React - ./helper.tsx

import faceApi from "../loadModules";

export function singleFaceDetectionWithImage(file:Blob) {
    return new Promise(async (resolve, reject) => {
        const actualImage = await faceApi.bufferToImage(file);
        const singleFaceDetection = await faceApi.detectSingleFace(actualImage)
                                    .withFaceLandmarks().withFaceDescriptor();
        resolve({ actualImage, singleFaceDetection });
    })
}

Here we will first convert the file type to a base64 and then passing it to faceApi to detect the face of the person and then getting the trained data withFaceDescriptor.

3) We have the trained data set of the person (we can have multiple people by adding one by one after each person). we have allowed a duplicates button just to allow the same image with multiple people available on their folder. Now by clicking start will start labeling the fate details data with the name and provides us the trained data for each person.
React - ./home.tsx (inside homeComponent add this)

    const handleTrain = () => {
        setTrainComplete(false);
        new Promise((resolve, reject) => {
            const labeledFaceDescriptors = checkPoint.map((data) => {
                return new faceApi.LabeledFaceDescriptors(data.name, data.faceData);
            });
            resolve(labeledFaceDescriptors);
        }).then((data:any) => {
            setTrainedSet(data);
            setTrainComplete(true);
        }).catch(err => {
            console.error(err);
        })
    }
    return (
      <React.Fragment>
         <div className="form-container">
            <div className="form-title">Upload Known Faces to split</div>
                <form onSubmit={handleSubmit}>
                    <Input type="text" onChange={handleNameChange} 
                    placeholder="Enter The Name" value={formData.name} />
                    <Button variant="contained" component="label" 
                        onChange={handleCompareImage} >
                        Upload Known Face
                        <input type="file" multiple style={{ display: "none" }}/>
                    </Button>
                    <Button color="primary" type="submit" 
                        disabled={submitDisable}>ADD
                    </Button>
                </form>
                <Button color="secondary" type="submit" 
                    onClick={() => setDuplicate(!duplicate)}>Allow Duplicates
                </Button>
                {duplicate ? 
                    <div className="duplicate-warining">
                        Allowing duplicates may increase in size
                     </div> 
                : ''}
             </div>
            {/* Form data display */}
            <div className="check-point-wrapper">
               <div className="form-display-container">
                  {checkPoint.map((imgData, index) => (
                     <div className="image-name-wrapper" key={index}>
                         <img 
                              src={imgData?.actualImages[0]?.src ? 
                              imgData?.actualImages[0].src : null} 
                          />
                             <div>{imgData?.name}</div>
                      </div>
                   ))}
                </div>
                {checkPoint?.length ? 
                   <Button className="start-action" color="primary" 
                    variant="contained" onClick={handleTrain}>START</Button>
                    : ''}
                </div>
            </div>
      <React.Fragment>
    )
}
export default Home;

4) Now we need to upload all the images to split by face recognization, so we need input to upload multiple images and need to process to get the face details in the images and need to compare it with the trained date set to split by face.

React - ./home.tsx (Add this in home component below "Form data display")

{/* Image to split upload */}
    {trainComplete ?
        <div className="image-to-split-wrapper">
            <div>Upload All Your Images That needs to Split</div>
                <Button color="secondary" variant="contained" component="label" 
                onChange={handleImageChange} >
                    Upload File
                    <input type="file" multiple style={{ display: "none" }}></input>
                </Button>
        </div> 
    : ''}

And add this in homeComponent (./home.tsx)

const handleImageChange = (event:any) => {
        const { files } = event.target;
        handleFiles(files);
    }

    const handleFiles = async (files:FileList) => {
        const faceMatcher:any = new faceApi.FaceMatcher(trainedSet, 0.45);
        for (let index = 0; index < files.length; index++) {
            const file = files[index];
            const actualImage = await faceApi.bufferToImage(file);
            const allFaceDetection = await faceApi.detectAllFaces(actualImage)
                    .withFaceLandmarks().withFaceDescriptors();
            const finalDataSet = matchFacesBy(allFaceDetection, 
                                    file, faceMatcher, 
                                    finalResult, duplicate);
            setFinalResult(finalDataSet);
        }
        makeZip();
    }

    const makeZip = () => {
        var zip = new JSZip();
        finalResult.map((result) => {
            Object.keys(result).map((name) => {
                const file = result[name].file;
                if (file) {
                    let imageFolder = zip.folder(name);
                    imageFolder.file(file.name, file);
                }
            })
        })
        zip.generateAsync({type: "blob"}).then((content) => {
            saveAs(content, 'split-images.zip');
        })
    }

here we handle the images upload and the uploaded images are passed to handleFiles. Here we will be providing the trained data set and the percentage to match to face matcher api.
then will be looping through all the images and detecting all the faces in the image and pacing the face details to matchFaceBy function in helper.tsx

5) Now matchFaceBy will match the face by the trained data and all face details from the image
React - ./helper.tsx

export const matchFacesBy = (allFaceDetection:any, file:File, 
    faceMatcher:any, finalResult: any, duplicate:Boolean) => {
    const localFinalResult:any = {};
    for (let index = 0; index < allFaceDetection.length; index++) {
        const faceDetection = allFaceDetection[index];
        const result = faceMatcher.findBestMatch(faceDetection?.descriptor);
        localFinalResult[result.label] = {
            result,
            file
        }
        if (result.label !== 'unknown') {
            localFinalResult['unknown'] = {};
            if (!duplicate) break;
        }

    }
    finalResult.push(localFinalResult);
    // setFinalResult(finalResult);
    console.log(finalResult);
    return finalResult;
}

Here we will be looping through all the faces detected from images and will be taking each face and will find the best matching with the trained data, and will push the data into the particular username object(which is the "result.label" which has been associated when training the data). and unknown faces are pushed into the unknown object and the final result is returned.

Now the final result of each image is stored in finalResult. Now the makeZip is called to create a zip file based on the finalResult and will store each user image separately on folders and will download it.

Thats it!!! Face Recognization will not be 100% accurate in faceApi

Github Link - github.com/Arjhun777
Woking demo - splitbyface.netlify.app
Chech out my blog - arjhun777.blogspot.com

Video Chatting and Screen Sharing with React, Node, WebRTC(peerjs)

Arjhun777 — Sat, 02 Jan 2021 12:23:56 +0000

To create a video chatting and screen sharing application requires three major setup

Basic React setup for handling UI.
Needs Backend (Nodejs) for maintaining socket connection.
Needs a peer server to maintain create peer-to-peer connection and to maintain it.

1) React basic setup with join button which makes an API call to backend and gets a unique id and redirects the user to join the room (React running at the port 3000)

Frontend - ./Home.js

import Axios from 'axios';
import React from 'react';

function Home(props) {
    const handleJoin = () => {
        Axios.get(`http://localhost:5000/join`).then(res => {
            props.history?.push(`/join/${res.data.link}? 
           quality=${quality}`);
        })
    }

    return (
        <React.Fragment>
            <button onClick={handleJoin}>join</button>
        </React.Fragment>
    )
}

export default Home;

Here our backend is running at port localhost 5000, as a response will be getting a unique id that will be used as a room id with upcoming steps.

2) Backend - Node basic setup with a server listening in port 5000 and defining router with "/join" to generate a unique id and return it to frontend

Backend - ./server.js

import express from 'express';
import cors from 'cors';
import server from 'http';
import { v4 as uuidV4 } from 'uuid';

const app = express();
const serve = server.Server(app);
const port = process.env.PORT || 5000;

// Middlewares
app.use(cors());
app.use(express.json());
app.use(express.urlencoded({ extended: true }));

app.get('/join', (req, res) => {
    res.send({ link: uuidV4() });
});

serve.listen(port, () => {
    console.log(`Listening on the port ${port}`);
}).on('error', e => {
    console.error(e);
});

Here using uuid package to generate a unique string.

3) At the frontend creating a new route with the id got in the response(looks something like this "http://localhost:3000/join/a7dc3a79-858b-420b-a9c3-55eec5cf199b"). A new component - RoomComponent is created with the disconnect button and having a div container with id="room-container" to hold our video elements

Frontend - ../RoomComponent.js

const RoomComponent = (props) => {
    const handleDisconnect = () => {
        socketInstance.current?.destoryConnection();
        props.history.push('/');
    }
    return (
        <React.Fragment>
            <div id="room-container"></div>
            <button onClick={handleDisconnect}>Disconnect</button>
        </React.Fragment>
    )
}

export default RoomComponent;

4) Now we need our stream from our device cam and mic, we can use the navigator to get the device stream data. For this, we can use a helper class (Connection) to maintain all the incoming and outgoing stream data and to maintain the socket connection with the backend.

Frontend - ./connection.js

import openSocket from 'socket.io-client';
import Peer from 'peerjs';
const { websocket, peerjsEndpoint } = env_config;
const initializePeerConnection = () => {
    return new Peer('', {
        host: peerjsEndpoint, // need to provide peerjs server endpoint 
                              // (something like localhost:9000)
        secure: true
    });
}
const initializeSocketConnection = () => {
    return openSocket.connect(websocket, {// need to provide backend server endpoint 
                              // (ws://localhost:5000) if ssl provided then
                              // (wss://localhost:5000) 
        secure: true, 
        reconnection: true, 
        rejectUnauthorized: false,
        reconnectionAttempts: 10
    });
}
class Connection {
    videoContainer = {};
    message = [];
    settings;
    streaming = false;
    myPeer;
    socket;
    myID = '';
    constructor(settings) {
        this.settings = settings;
        this.myPeer = initializePeerConnection();
        this.socket = initializeSocketConnection();
        this.initializeSocketEvents();
        this.initializePeersEvents();
    }
    initializeSocketEvents = () => {
        this.socket.on('connect', () => {
            console.log('socket connected');
        });
        this.socket.on('user-disconnected', (userID) => {
            console.log('user disconnected-- closing peers', userID);
            peers[userID] && peers[userID].close();
            this.removeVideo(userID);
        });
        this.socket.on('disconnect', () => {
            console.log('socket disconnected --');
        });
        this.socket.on('error', (err) => {
            console.log('socket error --', err);
        });
    }
    initializePeersEvents = () => {
        this.myPeer.on('open', (id) => {
            this.myID = id;
            const roomID = window.location.pathname.split('/')[2];
            const userData = {
                userID: id, roomID
            }
            console.log('peers established and joined room', userData);
            this.socket.emit('join-room', userData);
            this.setNavigatorToStream();
        });
        this.myPeer.on('error', (err) => {
            console.log('peer connection error', err);
            this.myPeer.reconnect();
        })
    }
    setNavigatorToStream = () => {
        this.getVideoAudioStream().then((stream) => {
            if (stream) {
                this.streaming = true;
                this.createVideo({ id: this.myID, stream });
                this.setPeersListeners(stream);
                this.newUserConnection(stream);
            }
        })
    }
    getVideoAudioStream = (video=true, audio=true) => {
        let quality = this.settings.params?.quality;
        if (quality) quality = parseInt(quality);
        const myNavigator = navigator.mediaDevices.getUserMedia || 
        navigator.mediaDevices.webkitGetUserMedia || 
        navigator.mediaDevices.mozGetUserMedia || 
        navigator.mediaDevices.msGetUserMedia;
        return myNavigator({
            video: video ? {
                frameRate: quality ? quality : 12,
                noiseSuppression: true,
                width: {min: 640, ideal: 1280, max: 1920},
                height: {min: 480, ideal: 720, max: 1080}
            } : false,
            audio: audio,
        });
    }
    createVideo = (createObj) => {
        if (!this.videoContainer[createObj.id]) {
            this.videoContainer[createObj.id] = {
                ...createObj,
            };
            const roomContainer = document.getElementById('room-container');
            const videoContainer = document.createElement('div');
            const video = document.createElement('video');
            video.srcObject = this.videoContainer[createObj.id].stream;
            video.id = createObj.id;
            video.autoplay = true;
            if (this.myID === createObj.id) video.muted = true;
            videoContainer.appendChild(video)
            roomContainer.append(videoContainer);
        } else {
            // @ts-ignore
            document.getElementById(createObj.id)?.srcObject = createObj.stream;
        }
    }
    setPeersListeners = (stream) => {
        this.myPeer.on('call', (call) => {
            call.answer(stream);
            call.on('stream', (userVideoStream) => {console.log('user stream data', 
            userVideoStream)
                this.createVideo({ id: call.metadata.id, stream: userVideoStream });
            });
            call.on('close', () => {
                console.log('closing peers listeners', call.metadata.id);
                this.removeVideo(call.metadata.id);
            });
            call.on('error', () => {
                console.log('peer error ------');
                this.removeVideo(call.metadata.id);
            });
            peers[call.metadata.id] = call;
        });
    }
    newUserConnection = (stream) => {
        this.socket.on('new-user-connect', (userData) => {
            console.log('New User Connected', userData);
            this.connectToNewUser(userData, stream);
        });
    }
    connectToNewUser(userData, stream) {
        const { userID } = userData;
        const call = this.myPeer.call(userID, stream, { metadata: { id: this.myID }});
        call.on('stream', (userVideoStream) => {
            this.createVideo({ id: userID, stream: userVideoStream, userData });
        });
        call.on('close', () => {
            console.log('closing new user', userID);
            this.removeVideo(userID);
        });
        call.on('error', () => {
            console.log('peer error ------')
            this.removeVideo(userID);
        })
        peers[userID] = call;
    }
    removeVideo = (id) => {
        delete this.videoContainer[id];
        const video = document.getElementById(id);
        if (video) video.remove();
    }
    destoryConnection = () => {
        const myMediaTracks = this.videoContainer[this.myID]?.stream.getTracks();
        myMediaTracks?.forEach((track:any) => {
            track.stop();
        })
        socketInstance?.socket.disconnect();
        this.myPeer.destroy();
    }
}

export function createSocketConnectionInstance(settings={}) {
    return socketInstance = new Connection(settings);
}

Here we have created a Connection class to maintain all our socket and peer connection, Don't worry we will walk through all the functions above.

we have a constructor that gets a settings object (optional) that can be used to send some data from our component for setting up our connection class like (sending video frame to be used)
Inside constructor we are invoking two methods initializeSocketEvents() and initializePeersEvents()
- initializeSocketEvents() - Will start socket connection with our backend.
- initializePeersEvents() - Will start peer connection with our peer server.
Then we have setNavigatorToStream() which has getVideoAndAudio() function which will get the audio and video stream from the navigator. We can specify the video frame rate in the navigator.
If the stream is available then we will be resolving in .then(streamObj) and now we can create a video element to display our stream bypassing stream object to createVideo().
Now after getting our own stream it's time to listen to the peer events in function setPeersListeners() where we will be listening for any incoming video stream from another user and will stream our data in peer.answer(ourStream).
And the we will be setting newUserConnection(), where we will be sending our stream, if we are connecting to the existing room and also keeping track of the current peer connection by userID in peers Object.
Finally we have removeVideo to remove the video element from dom when any user dissconnected.

5) Now the backend needs to listen to the socket connection. Using socket "socket.io" to make the socket connection easy.

Backend - ./server.js

import socketIO from 'socket.io';
io.on('connection', socket => {
    console.log('socket established')
    socket.on('join-room', (userData) => {
        const { roomID, userID } = userData;
        socket.join(roomID);
        socket.to(roomID).broadcast.emit('new-user-connect', userData);
        socket.on('disconnect', () => {
            socket.to(roomID).broadcast.emit('user-disconnected', userID);
        });
    });
});

Now we have added socket connection to backend to listen to join room, which will be triggred from frontend with userData containing roomID and userID. The userID is available when creating the peer connection.

Then the socket has now connected a room with the roomID (From unique id got as response in frontend) and now we can dispatch message to all the users in the room.

Now socket.to(roomID).broadcast.emit('new-user-connect', userData); with this we can dispatch message to all the user's connected except us. And this 'new-user-connect is listened at the frontend so all the user's connected in the room will receive the new user data.

6) Now you need create a peerjs server by using following commands

npm i -g peerjs
peerjs --port 9000

7) Now in Room Component we need to invoke the Connection class to start the call. In Room Component add this functionality.

Frontend - ./RoomComponent.js

    let socketInstance = useRef(null);    
    useEffect(() => {
        startConnection();
    }, []);
    const startConnection = () => {
        params = {quality: 12}
        socketInstance.current = createSocketConnectionInstance({
            params
        });
    }

Now you will be able to see that after creating a room when a new user joins the user will be peer-to-peer connected.

8) Now for Screen Sharing, You need to replace the current stream with the new screen sharing stream.

Frontend - ./connection.js

    reInitializeStream = (video, audio, type='userMedia') => {
        const media = type === 'userMedia' ? this.getVideoAudioStream(video, audio) : 
        navigator.mediaDevices.getDisplayMedia();
        return new Promise((resolve) => {
            media.then((stream) => {
                if (type === 'displayMedia') {
                    this.toggleVideoTrack({audio, video});
                }
                this.createVideo({ id: this.myID, stream });
                replaceStream(stream);
                resolve(true);
            });
        });
    }
    toggleVideoTrack = (status) => {
        const myVideo = this.getMyVideo();
        if (myVideo && !status.video) 
            myVideo.srcObject?.getVideoTracks().forEach((track) => {
                if (track.kind === 'video') {
                    !status.video && track.stop();
                }
            });
        else if (myVideo) {
            this.reInitializeStream(status.video, status.audio);
        }
    }
    replaceStream = (mediaStream) => {
        Object.values(peers).map((peer) => {
            peer.peerConnection?.getSenders().map((sender) => {
                if(sender.track.kind == "audio") {
                    if(mediaStream.getAudioTracks().length > 0){
                        sender.replaceTrack(mediaStream.getAudioTracks()[0]);
                    }
                }
                if(sender.track.kind == "video") {
                    if(mediaStream.getVideoTracks().length > 0){
                        sender.replaceTrack(mediaStream.getVideoTracks()[0]);
                    }
                }
            });
        })
    }

Now the current stream needs to reInitializeStream() will be checking the type it needs to replace, if it is userMedia then it will be streaming from cam and mic, if its display media it gets the display stream object from getDisplayMedia() and then it will toggle the track to stop or start the cam or mic.

Then the new stream video element is created based on the userID and then it will place the new stream by replaceStream(). By getting the current call object store previosly will contain the curretn stream data will be replaced with the new stream data in replaceStream().

9) At roomConnection we need to create a button to toggle the video and screen sharing.

Frontend - ./RoomConnection.js

    const [mediaType, setMediaType] = useState(false);    
    const toggleScreenShare = (displayStream ) => {
        const { reInitializeStream, toggleVideoTrack } = socketInstance.current;
        displayStream === 'displayMedia' && toggleVideoTrack({
            video: false, audio: true
        });
        reInitializeStream(false, true, displayStream).then(() => {
            setMediaType(!mediaType)
        });
    }
    return (
        <React.Fragment>
            <div id="room-container"></div>
            <button onClick={handleDisconnect}>Disconnect</button>
            <button 
                onClick={() => reInitializeStream(mediaType ? 
                'userMedia' : 'displayMedia')}
            >
            {mediaType ? 'screen sharing' : 'stop sharing'}</button>
        </React.Fragment>
    )

Thats all you have Create a application with video chatting and screen sharing.

Good Luck !!!
Here's my working demo - vichah
Check out my blog - https://arjhun777.blogspot.com/
Github link
Frontend - https://github.com/Arjhun777/VChat-FrontEnd
Backend - https://github.com/Arjhun777/VChat-BackEnd