Emotion Detection with Python: A Flask API for Static Images and Real-Time Video

As a React Native developer, it's always a great idea to explore backend services and APIs that can integrate seamlessly with your mobile applications. One such fascinating project is the integration of emotion detection using Python, Flask, and OpenCV. In this post, we'll dive into how to set up a simple Flask API that detects emotions from static images and real-time video using the FER (Facial Expression Recognition) library.

Here, I’ll break down the code snippet that allows you to upload images and stream video for emotion detection using the power of OpenCV for face detection and FER for recognizing emotions.

---

Setting up the Environment

Install Python on your system following this:
https://realpython.com/installing-python/

Before jumping into the code, let's ensure we have all the necessary dependencies installed. You will need:

• Flask (for building the API)
• FER (Facial Expression Recognition model)
• OpenCV (for handling image and video processing)
• NumPy (for working with image arrays)
• ffmpeg (to ensure OpenCV works well with video streams)

Use the following command to install the required libraries:

pip install Flask fer opencv-python-headless numpy

OR

Step 1: Make a file expression_detection/requirements.txt.

Step 2: Inside requirements.txt add

Flask
fer
opencv-python-headless
numpy
tensorflow
opencv-contrib-python
ffmpeg
moviepy

Step 3:
a) Create a virtual environment:

python3 -m venv path/to/venv

b) Activate the virtual environment:

source path/to/venv/bin/activate

c) Install the packages in requirements.txt:

pip install -r requirements.txt

---

Code

Create a file server.py:

import os
os.environ["IMAGEIO_FFMPEG_EXE"] = "/usr/bin/ffmpeg"
from flask import Flask, request, jsonify, Response # Import Flask for creating API endpoints
from fer import FER  # Import FER for emotion detection
import cv2  # OpenCV for image and video processing
import numpy as np  # NumPy for handling image arrays

app = Flask(__name__)
detector = FER()  # Initialize the FER emotion detector

# Load OpenCV's pre-trained face detection model (Haar Cascade)
face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')

# API to process and detect emotions in static images
@app.route('/upload', methods=['POST'])
def upload_image():
    # Check if a file is provided in the request
    if 'file' not in request.files:
        return jsonify({'error': 'No file provided'}), 400

    file = request.files['file']  # Get the file from the request
    if file.filename == '':  # Check if a file was actually selected
        return jsonify({'error': 'No selected file'}), 400

    # Read the image bytes and convert it to a NumPy array
    img_bytes = file.read()
    np_img = np.frombuffer(img_bytes, np.uint8)
    image = cv2.imdecode(np_img, cv2.IMREAD_COLOR)  # Decode the image using OpenCV

    if image is None:  # If image decoding fails
        return jsonify({'error': 'Invalid image'}), 400

    # Use the FER detector to detect the top emotion in the image
    emotion, score = detector.top_emotion(image)

    # Return the detected emotion and confidence score as JSON
    return jsonify({'emotion': emotion, 'score': score})

# API to stream video from the default camera and perform real-time emotion detection
@app.route('/video_feed')
def video_feed():
    # Function to generate video frames with emotion detection
    def generate_frames():
        cap = cv2.VideoCapture(0)  # Open the default camera (0)

        while True:
            success, frame = cap.read()  # Read a frame from the camera
            if not success:  # If the camera cannot provide frames, break the loop
                break

            # Convert the frame to grayscale for face detection
            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
            faces = face_cascade.detectMultiScale(
                gray, scaleFactor=1.1, minNeighbors=5, minSize=(30, 30)
            )  # Detect faces in the grayscale image

            # Detect the top emotion for the current frame
            emotion, score = detector.top_emotion(frame)

            # For each detected face, draw a rectangle and annotate the emotion
            for (x, y, w, h) in faces:
                # Draw a rectangle around the face
                cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)
                # Annotate the detected emotion on the frame
                cv2.putText(
                    frame, f'Emotion: {emotion}', (x, y - 10), 
                    cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 255, 0), 2
                )

            # Encode the frame to a JPEG image format
            ret, buffer = cv2.imencode('.jpg', frame)
            frame = buffer.tobytes()

            # Yield the frame in multipart format to display in the browser
            yield (b'--frame\r\n'
                   b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')

        # Release the video capture resource when done
        cap.release()

    # Stream the video frames using HTTP multipart format
    return Response(generate_frames(), mimetype='multipart/x-mixed-replace; boundary=frame')

# Start the Flask app on the specified port and IP address
if __name__ == '__main__':
    app.run(host='0.0.0.0', port=3000)

# ### Summary:

# 1. **Static Image Processing (`/upload` endpoint)**:
#    - Accepts an image file via POST request.
#    - The image is decoded using OpenCV and processed by the FER (Facial Expression Recognition) model to detect the top emotion.
#    - Returns the emotion and confidence score in JSON format.

# 2. **Real-Time Video Stream (`/video_feed` endpoint)**:
#    - Streams video from the default camera in real-time.
#    - Detects faces in each video frame using OpenCV's Haar Cascade classifier.
#    - FER detects the top emotion for each frame.
#    - Draws a rectangle around each detected face and annotates the emotion.
#    - The video stream is sent in HTTP multipart format to be viewed in real-time in a browser.

# Both endpoints showcase emotion detection using FER and OpenCV, with one handling static images and the other real-time video.

---

3. Running the Application

To run the application, use the following command:

python3 server.py

This should come:

Copy the base url

Once the server is running, you can test the APIs attach base url:

• For the image upload endpoint, you can use POST {base_url}/upload with an image file.
• For the video stream, open a browser and navigate to {base_url}/video_feed to see the real-time emotion detection in action.

React Native Usage:

service.js

import axios from "axios";

const UPLOAD_API = "http://xxx.xxx.x.x:3000/upload" 
export const uploadImage = async (local_uri: string) => {
    try {
        const formData = new FormData();
        formData.append('file', {
            uri: local_uri,
            name: 'file',
        });

        const res = await axios.post(UPLOAD_API, formData, {
            headers: {
                'Content-Type': 'multipart/form-data',
            },
        });
        return res.data;
    } catch (error) {
        console.log(error);
        throw error
    }
}

Output:

---

Conclusion:
By above approach we can integrate many type of AI models as per requirement.