DEV Community

Cover image for Deep Dive: Building Real-Time Facial Emotion Detection on Raspberry Pi with YOLOv11
Malar Kondappan
Malar Kondappan

Posted on

Deep Dive: Building Real-Time Facial Emotion Detection on Raspberry Pi with YOLOv11

#machinelearning #raspberrypi #python #opensource

In the previous section, we covered why emotion detection matters and how computers “see” feelings.
Now, let’s explore how to implement this in code, understand the architecture, and make sense of the workflow, with real examples and explanations for each step.

Project Architecture: The Three Pillars

  • Roboflow Dataset Manager:
    Gathers and formats image data for training
    GitHub Repository

  • YOLOv11 Model Training:
    Fine-tunes a neural net to recognize emotions
    GitHub Repository

  • Face Emotion Detection System:
    Runs the model for real-time inference on Raspberry Pi
    GitHub Repository

1. Preparing the Dataset (Data Science Foundation)

Before your AI can recognize emotions, it needs to learn from thousands of labeled examples.
Use Roboflow Universe to find or create emotion datasets.
The dataset manager script automates download and formatting in YOLOv11-compatible folders.

Sample Python code to download with Roboflow:

from roboflow import Roboflow

rf = Roboflow(api_key="YOUR_API_KEY")
project = rf.workspace("your-workspace").project("your-emotion-project")
dataset = project.version(3).download("yolov11") # Public datasets may skip API key

# Output: Folders with images and YOLOv11 labels (train/valid/test subfolders)
# Each image gets a companion .txt file with bounding boxes and class labels for the detected emotion.
Enter fullscreen mode Exit fullscreen mode

2. Training the Model (Machine Learning in Action)

Now, let’s teach YOLOv11 to spot emotions. The magic happens through transfer learning, building on a pre-trained model (Nano version) and specializing on emotion data.

Install dependencies:

pip install ultralytics torch opencv-python
Enter fullscreen mode Exit fullscreen mode

Start training (Python):

from ultralytics import YOLO

# Path to dataset config (created above)
model = YOLO('yolo11n.pt') # Nano base model
results = model.train(data='data.yaml', epochs=200, imgsz=320, batch=10, device='cpu', augment=True)
Enter fullscreen mode Exit fullscreen mode
  • data.yaml describes paths and classes
  • epochs=200: More epochs, better accuracy (but risk overfitting)
  • imgsz=320: Smaller images = faster for Pi, good enough accuracy
  • batch=10: Number of images processed per step
  • device='cpu': Set to '0' for GPU if available

Inspect results:

# After training
model = YOLO('runs/detect/train/weights/best.pt') # Best model selected automatically

img = 'test-image.jpg'
results = model(img)
results.show() # Visualize detections and emotion labels
Enter fullscreen mode Exit fullscreen mode

Best weights (best.pt) are saved for deployment; model size just 6.5 MB!

3. Deploying for Real-Time Raspberry Pi Inference

Now the excitement, deploy your model to Pi and see instant results!

Essential script highlights:

import cv2
from ultralytics import YOLO

model = YOLO('yolo-trained-models/emotionsbest.pt')
cam = cv2.VideoCapture(0) # 0 for Pi Camera; use correct ID for USB webcam

while True:
    ret, frame = cam.read()
    results = model(frame)
    for r in results:
        boxes = r.boxes.xyxy
        labels = r.names[r.probs.argmax()]
        conf = r.probs.max()
        # Draw bounding box and label on frame
        # Color-coding for each emotion: see STANDARDCOLORS dict in app-pt.py

    cv2.imshow('Emotion Detection', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cam.release()
cv2.destroyAllWindows()
Enter fullscreen mode Exit fullscreen mode

Performance tips:

  • Reduce image resolution for faster speed: picam2.preview_configuration.main.size = (1920, 1080)
  • Skip frames if slow (N = 2 means process every 2nd frame)
  • The script auto-detects Pi Camera and USB webcams

Code Architecture Explained

  • Input: Frames from Pi camera
  • Processing Pipeline:
    • Convert color spaces (BGR ↔ RGB as needed)
    • Feed frame into YOLOv11 model
    • Get boxes, class labels, confidence scores
    • Annotate output image with emotion predictions
    • Display live feed (with FPS metrics)
  • Customization:
    • Edit emotion colors (STANDARDCOLORS)
    • Adjust camera resolution and frame skipping (N)
    • Record video or save frames with OpenCV’s VideoWriter

Troubleshooting & Optimization (Tips for Developers)

Camera not detected?

libcamera-hello --list-cameras
sudo raspi-config # Enable Pi Camera module
Enter fullscreen mode Exit fullscreen mode

Model too slow? Lower imgsz, increase N, use GPU if available
Out of memory? Lower batch size, image size
Accuracy not improving? Check results charts, confusion matrix images; retrain with more data or longer epochs

Advanced Techniques

  • Fine-tune with your own emotions: Update the Roboflow script to download/label custom datasets, retrain as above.
  • Export to ONNX: For deploying to other platforms: 
model.export(format='onnx')
Enter fullscreen mode Exit fullscreen mode
  • Integrate with applications: Use inference results to log emotion metrics, trigger events (robotic reactions, game states, feedback systems).

Example: Full Inference Script (Python)

Here’s how a simplified detection loop looks:

from ultralytics import YOLO
import cv2

model = YOLO('emotionsbest.pt')
cam = cv2.VideoCapture(0)

while True:
    ret, frame = cam.read()
    results = model(frame)
    for result in results:
        for box in result.boxes:
            x1, y1, x2, y2 = map(int, box.xyxy)
            emotion = result.names[box.cls]
            conf = box.conf.item()
            # Draw box and label code here
    cv2.imshow('Live Emotion Detection', frame)
    if cv2.waitKey(1) == ord('q'):
        break

cam.release()
cv2.destroyAllWindows()
Enter fullscreen mode Exit fullscreen mode

Add your preferred annotation, color coding, and FPS monitoring!

Takeaway

  • The journey from raw images to real-time emotion detection combines data science, machine learning, and systems engineering.
  • Each code block is an invitation to experiment, learn, and innovate.
  • Good luck building and let’s make edge AI more expressive, accessible, and creative!

Get Hands-On: Ready-to-Experiment Project & Community Invitation

Interested in building a complete, working facial emotion detection system?
This open-source project is designed for you to experiment, learn, and contribute; whether you’re a beginner, teacher, student, or developer.

What you can do with this project:

  • Set up a live emotion recognition pipeline using real Raspberry Pi code and datasets.
  • Tweak and train the model for your unique classroom, hobby, or research needs.
  • Open issues, give feedback, or suggest new features, everyone’s ideas are valued.
  • Fork, modify, or contribute your own improvements. Pull requests welcome!

Want to start right now? Visit the repos for everything needed, data management, training, and deployment:

Every star, fork, suggestion, or classroom trial helps grow this project and makes the technology more accessible for everyone.

Questions, ideas, or feedback? Comment below or reach out via GitHub! Let’s learn and create together.

Top comments (0)