David Au Yeung

Posted on Jun 17

Build a Facial Recognition System with C#, ArcFace ONNX, and OpenCV DNN

#facialrecognition #onnx #opencv #csharp

Introduction

Ever wondered how to build a real facial recognition system in C#? The kind that can actually tell different people apart, not just compare entire images? Today, we're going to build exactly that-a production-ready facial recognition demo that uses state-of-the-art deep learning models.

By the end of this tutorial, you'll have a working application that can:

Detect faces in images automatically
Generate unique "fingerprints" (embeddings) for each face
Compare faces with high accuracy
Tell you if two photos are of the same person

What makes this special? We're combining two powerful technologies:

OpenCV DNN Face Detector - Finds and crops faces automatically
ArcFace ONNX Model - Generates 512-dimensional face embeddings

No cloud APIs, no subscription fees-everything runs locally on your machine.

Prerequisites
Understanding the Technology
Setting Up the Project
Downloading the Models
Building the Face Recognition Service
Creating the Main Application
How It All Works Together
Testing Your System
The Journey to Better Accuracy
Next Steps

Prerequisites

Before we dive in, make sure you have:

.NET 8 SDK installed
Visual Studio 2022 or Visual Studio Code
Basic C# knowledge
A few test images of people (we'll need at least 3-4 photos)

That's it! Everything else we'll install as we go.

Understanding the Technology

Why Two Models?

You might be wondering: "Why do we need two models?" Great question! Let me explain with a real-world analogy.

Imagine you're a detective trying to match a suspect's photo:

First, you need to find the face in a crowded photo (maybe there's a background, other people, etc.)
Then, you need to analyze just the face to identify the person

That's exactly what our two models do:

1. OpenCV DNN Face Detector (The "Face Finder")

Job: Find faces in images
Input: Any image (any size)
Output: Bounding boxes around detected faces
Model: Caffe-based SSD (Single Shot Detector)
Speed: Very fast (~20-30ms per image)

2. ArcFace ONNX (The "Face Recognizer")

Job: Convert face images into unique numerical signatures
Input: 112x112 RGB face image
Output: 512-dimensional vector (called an "embedding")
Magic: Faces of the same person produce similar vectors!
Speed: Fast (~10-15ms per face)

What is ArcFace?

ArcFace is a state-of-the-art face recognition model developed by researchers. Here's why it's amazing:

Accuracy: One of the most accurate face recognition models available
Embeddings: Converts faces into 512 numbers that capture facial features
Comparison: Same person = similar embeddings (cosine similarity > 0.6)
Different people: Different embeddings (cosine similarity < 0.5)

Think of embeddings as a "facial fingerprint"-unique to each person, but similar between photos of the same person.

The Face Recognition Pipeline

Here's the complete flow:

Input Image
    ↓
[OpenCV DNN: Find Face]
    ↓
Crop to Face Region (with margin)
    ↓
Resize to 112x112
    ↓
Normalize pixels to [-1, 1]
    ↓
[ArcFace: Generate Embedding]
    ↓
512 numbers representing the face
    ↓
[Compare with other embeddings]
    ↓
Similarity Score (0.0 to 1.0)
    ↓
Decision: Same person or different?

Now let's build it!

Setting Up the Project

Step 1: Create a New Console Application

Open your terminal and run:

dotnet new console -n FaceRecognitionDemo
cd FaceRecognitionDemo

Step 2: Install Required NuGet Packages

We need three packages:

dotnet add package Microsoft.ML.OnnxRuntime
dotnet add package OpenCvSharp4
dotnet add package OpenCvSharp4.runtime.win

What each package does:

Microsoft.ML.OnnxRuntime - Runs ONNX models (for ArcFace)
OpenCvSharp4 - Computer vision operations (image processing, face detection)
OpenCvSharp4.runtime.win - Native Windows binaries for OpenCV

Step 3: Create Project Folders

mkdir Photos
mkdir Models

Photos/ - Where you'll put test images
Models/ - Where we'll download our AI models

Downloading the Models

Model 1: ArcFace ONNX (Face Recognition)

Option A: Download via Browser

Go to: https://huggingface.co/garavv/arcface-onnx

Click on "Files and versions" → Download arc.onnx

Rename it to arcface.onnx and place it in the Models/ folder.

Option B: Download via Command Line (PowerShell)

# Navigate to your project directory
cd FaceRecognitionDemo

# Download the model
Invoke-WebRequest -Uri "https://huggingface.co/garavv/arcface-onnx/resolve/main/arc.onnx?download=true" `
    -OutFile "Models/arcface.onnx"

Or using curl:

curl -L "https://huggingface.co/garavv/arcface-onnx/resolve/main/arc.onnx?download=true" -o Models/arcface.onnx

File size: ~130MB (be patient, it's worth it!)

Model 2: OpenCV Face Detection Models

These are smaller and faster to download:

# Face detector configuration
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/opencv/opencv/master/samples/dnn/face_detector/deploy.prototxt" `
    -OutFile "Models/deploy.prototxt"

# Face detector weights
Invoke-WebRequest -Uri "https://github.com/opencv/opencv_3rdparty/raw/dnn_samples_face_detector_20170830/res10_300x300_ssd_iter_140000.caffemodel" `
    -OutFile "Models/res10_300x300_ssd_iter_140000.caffemodel"

Your Models folder should now have:

✅ arcface.onnx (130 MB)
✅ deploy.prototxt (28 KB)
✅ res10_300x300_ssd_iter_140000.caffemodel (10 MB)

Building the Face Recognition Service

Now for the fun part-writing the code! Create a new file called ImprovedFaceRecognitionService.cs:

using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;
using OpenCvSharp;
using OpenCvSharp.Dnn;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;

public class ImprovedFaceRecognitionService : IDisposable
{
    private readonly InferenceSession _session;
    private readonly string _inputName;
    private readonly Net? _faceDetector;
    private readonly bool _useFaceDetection;

    public ImprovedFaceRecognitionService(
        string modelPath, 
        string? faceDetectorProto = null, 
        string? faceDetectorModel = null)
    {
        // Load ArcFace ONNX model
        _session = new InferenceSession(modelPath);
        _inputName = _session.InputMetadata.Keys.First();

        // Initialize OpenCV face detector if models provided
        if (!string.IsNullOrEmpty(faceDetectorProto) && 
            !string.IsNullOrEmpty(faceDetectorModel) &&
            File.Exists(faceDetectorProto) && 
            File.Exists(faceDetectorModel))
        {
            _faceDetector = CvDnn.ReadNetFromCaffe(faceDetectorProto, faceDetectorModel);
            _useFaceDetection = true;
            Console.WriteLine("✓ Face detection enabled");
        }
        else
        {
            _useFaceDetection = false;
            Console.WriteLine("ℹ Face detection disabled - using full images");
        }
    }

    public FaceEmbeddingResult GetEmbedding(string imagePath)
    {
        // Validate file
        if (!File.Exists(imagePath))
        {
            throw new FileNotFoundException($"Image file not found: {imagePath}");
        }

        if (imagePath.Length > 250)
        {
            throw new ArgumentException($"File path is too long ({imagePath.Length} characters).");
        }

        // Load image
        using Mat image = Cv2.ImRead(imagePath, ImreadModes.Color);
        if (image.Empty())
        {
            return new FaceEmbeddingResult
            {
                Success = false,
                ErrorMessage = $"Could not read image: {Path.GetFileName(imagePath)}"
            };
        }

        // Detect and crop face if face detection is enabled
        Mat faceImage;
        if (_useFaceDetection && _faceDetector != null)
        {
            var faceRect = DetectLargestFace(image);
            if (faceRect == null)
            {
                return new FaceEmbeddingResult
                {
                    Success = false,
                    ErrorMessage = "No face detected in image"
                };
            }

            // Crop to face with 20% margin
            var rect = faceRect.Value;
            int margin = (int)(Math.Max(rect.Width, rect.Height) * 0.2);
            int x = Math.Max(0, rect.X - margin);
            int y = Math.Max(0, rect.Y - margin);
            int width = Math.Min(image.Width - x, rect.Width + 2 * margin);
            int height = Math.Min(image.Height - y, rect.Height + 2 * margin);

            faceImage = new Mat(image, new Rect(x, y, width, height));
        }
        else
        {
            faceImage = image;
        }

        // Process face for ArcFace
        var embedding = ProcessFaceForArcFace(faceImage);

        return new FaceEmbeddingResult
        {
            Success = true,
            Embedding = embedding,
            FaceDetected = _useFaceDetection
        };
    }

    private Rect? DetectLargestFace(Mat image)
    {
        if (_faceDetector == null) return null;

        // Prepare image blob for detection
        using var blob = CvDnn.BlobFromImage(
            image, 
            1.0, 
            new Size(300, 300), 
            new Scalar(104, 177, 123), 
            false, 
            false
        );

        _faceDetector.SetInput(blob);
        using var detection = _faceDetector.Forward();

        // Find face with highest confidence
        Rect? bestFace = null;
        float bestConfidence = 0;

        int rows = detection.Size(2);
        for (int i = 0; i < rows; i++)
        {
            float confidence = detection.At<float>(0, 0, i, 2);

            if (confidence > 0.5) // 50% confidence threshold
            {
                int x1 = (int)(detection.At<float>(0, 0, i, 3) * image.Width);
                int y1 = (int)(detection.At<float>(0, 0, i, 4) * image.Height);
                int x2 = (int)(detection.At<float>(0, 0, i, 5) * image.Width);
                int y2 = (int)(detection.At<float>(0, 0, i, 6) * image.Height);

                if (confidence > bestConfidence)
                {
                    bestConfidence = confidence;
                    bestFace = new Rect(x1, y1, x2 - x1, y2 - y1);
                }
            }
        }

        return bestFace;
    }

    private float[] ProcessFaceForArcFace(Mat faceImage)
    {
        // Step 1: Resize to 112x112 (ArcFace requirement)
        using Mat resizedImage = new Mat();
        Cv2.Resize(faceImage, resizedImage, new Size(112, 112));

        // Step 2: Convert BGR to RGB
        using Mat rgbImage = new Mat();
        Cv2.CvtColor(resizedImage, rgbImage, ColorConversionCodes.BGR2RGB);

        // Step 3: Normalize to [-1, 1]
        using Mat floatImage = new Mat();
        rgbImage.ConvertTo(floatImage, MatType.CV_32FC3, 1.0 / 128.0, -127.5 / 128.0);

        // Step 4: Create ONNX tensor (NHWC format)
        var inputTensor = new DenseTensor<float>(new[] { 1, 112, 112, 3 });

        for (int y = 0; y < 112; y++)
        {
            for (int x = 0; x < 112; x++)
            {
                Vec3f pixel = floatImage.At<Vec3f>(y, x);
                inputTensor[0, y, x, 0] = pixel.Item0;
                inputTensor[0, y, x, 1] = pixel.Item1;
                inputTensor[0, y, x, 2] = pixel.Item2;
            }
        }

        // Step 5: Run inference
        var inputs = new List<NamedOnnxValue>
        {
            NamedOnnxValue.CreateFromTensor(_inputName, inputTensor)
        };

        using IDisposableReadOnlyCollection<DisposableNamedOnnxValue> results = 
            _session.Run(inputs);
        var output = results.First().AsEnumerable<float>().ToArray();

        // Step 6: Normalize embedding to unit vector
        return Normalize(output);
    }

    public static float CosineSimilarity(float[] a, float[] b)
    {
        float dot = 0;
        for (int i = 0; i < a.Length; i++)
        {
            dot += a[i] * b[i];
        }
        // Since vectors are normalized, dot product = cosine similarity
        return dot;
    }

    private float[] Normalize(float[] v)
    {
        float norm = (float)Math.Sqrt(v.Sum(x => x * x));
        if (norm == 0) return v;
        return v.Select(x => x / norm).ToArray();
    }

    public void Dispose()
    {
        _session?.Dispose();
        _faceDetector?.Dispose();
    }
}

public class FaceEmbeddingResult
{
    public bool Success { get; set; }
    public float[]? Embedding { get; set; }
    public bool FaceDetected { get; set; }
    public string? ErrorMessage { get; set; }
}

Understanding the Code

Let's break down what each part does:

Constructor

Loads the ArcFace ONNX model
Optionally loads OpenCV face detector
Tells you if face detection is enabled

`GetEmbedding()` Method

Validates the image file exists
Loads the image with OpenCV
If face detection enabled → finds and crops the face
Processes the face for ArcFace
Returns embedding or error message

`DetectLargestFace()` Method

Uses OpenCV DNN to find faces
Returns the face with highest confidence
Confidence threshold: 50%

`ProcessFaceForArcFace()` Method

This is where the magic happens:

Resize → 112x112 pixels
Color Convert → BGR to RGB
Normalize → Pixel values to [-1, 1]
Tensorize → Convert to ONNX tensor
Inference → Run through ArcFace model
Normalize → Unit vector for comparison

`CosineSimilarity()` Method

Compares two embeddings
Returns value from 0.0 (completely different) to 1.0 (identical)
Same person typically scores > 0.6

Creating the Main Application

Now let's build the main program. Replace your Program.cs:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;

internal static class Program
{
    public static void Main(string[] args)
    {
        Console.OutputEncoding = Encoding.UTF8;
        Console.WriteLine("Starting Improved Facial Recognition Demo...");
        Console.WriteLine("Using ArcFace ONNX + OpenCV DNN Face Detection\n");

        string projectRoot = GetProjectRootDirectory();
        string photosPath = Path.Combine(projectRoot, "Photos");
        string modelsPath = Path.Combine(projectRoot, "Models");
        string arcfaceModel = Path.Combine(modelsPath, "arcface.onnx");
        string faceDetectorProto = Path.Combine(modelsPath, "deploy.prototxt");
        string faceDetectorModel = Path.Combine(modelsPath, "res10_300x300_ssd_iter_140000.caffemodel");

        // Validate directories
        if (!Directory.Exists(photosPath))
        {
            Console.WriteLine($"The Photos directory was not found at: {photosPath}");
            Console.WriteLine("Please create the directory and add some images.");
            return;
        }

        if (!Directory.EnumerateFileSystemEntries(photosPath).Any())
        {
            Console.WriteLine($"The Photos directory is empty: {photosPath}");
            Console.WriteLine("Please add some images (person1.jpg, person2.jpg, etc.)");
            return;
        }

        if (!File.Exists(arcfaceModel))
        {
            Console.WriteLine($"❌ ArcFace model not found at: {arcfaceModel}");
            Console.WriteLine("Please download arcface.onnx and place it in the Models directory.");
            return;
        }

        bool hasFaceDetection = File.Exists(faceDetectorProto) && File.Exists(faceDetectorModel);
        if (!hasFaceDetection)
        {
            Console.WriteLine("⚠ Face detection models not found. Running without face detection.");
            Console.WriteLine("For better accuracy, download the face detection models.");
        }

        // Find valid image files
        var imageFiles = Directory.GetFiles(photosPath, "*.*", SearchOption.TopDirectoryOnly)
            .Where(s => s.EndsWith(".jpg", StringComparison.OrdinalIgnoreCase) || 
                        s.EndsWith(".png", StringComparison.OrdinalIgnoreCase) || 
                        s.EndsWith(".jpeg", StringComparison.OrdinalIgnoreCase) ||
                        s.EndsWith(".webp", StringComparison.OrdinalIgnoreCase))
            .ToList();

        // Filter out problematic filenames
        var validImageFiles = new List<string>();
        var skippedFiles = new List<string>();

        foreach (var file in imageFiles)
        {
            string fileName = Path.GetFileName(file);

            if (file.Length > 250)
            {
                skippedFiles.Add($"{fileName} (path too long: {file.Length} chars)");
                continue;
            }

            if (fileName.Contains(".."))
            {
                skippedFiles.Add($"{fileName} (contains '..' which can cause issues)");
                continue;
            }

            validImageFiles.Add(file);
        }

        if (skippedFiles.Any())
        {
            Console.WriteLine("\n⚠ Warning: Some files were skipped:");
            foreach (var skipped in skippedFiles)
            {
                Console.WriteLine($"  - {skipped}");
            }
            Console.WriteLine("\nPlease rename to shorter names (e.g., person1.jpg).\n");
        }

        if (validImageFiles.Count < 1)
        {
            Console.WriteLine("No valid images found. Please add images to the Photos folder.");
            return;
        }

        Console.WriteLine($"Found {validImageFiles.Count} valid image(s) in the gallery.");

        // Randomly select a "wanted person"
        var random = new Random();
        int wantedIndex = random.Next(validImageFiles.Count);
        string wantedPersonImage = validImageFiles[wantedIndex];
        string wantedPersonName = Path.GetFileName(wantedPersonImage);
        Console.WriteLine($"\nWanted person is: {wantedPersonName}");
        Console.WriteLine();

        try
        {
            using var faceService = new ImprovedFaceRecognitionService(
                arcfaceModel, 
                hasFaceDetection ? faceDetectorProto : null,
                hasFaceDetection ? faceDetectorModel : null
            );

            // Process wanted person
            Console.WriteLine("\nProcessing wanted person image...");
            var wantedResult = faceService.GetEmbedding(wantedPersonImage);

            if (!wantedResult.Success)
            {
                Console.WriteLine($"❌ Could not process wanted person: {wantedResult.ErrorMessage}");
                return;
            }

            Console.WriteLine("✓ Face processed and encoded successfully.");
            Console.WriteLine("\nComparing against all people in the gallery...");
            Console.WriteLine("-------------------------------------------");

            // Compare with all other images
            bool found = false;
            var matches = new List<(string fileName, float similarity)>();
            var processed = 0;
            var failed = 0;

            foreach (var imagePath in validImageFiles)
            {
                if (imagePath == wantedPersonImage) continue;

                try
                {
                    var currentResult = faceService.GetEmbedding(imagePath);

                    if (!currentResult.Success)
                    {
                        Console.WriteLine($"File: {Path.GetFileName(imagePath),-30} | ⚠ {currentResult.ErrorMessage}");
                        failed++;
                        continue;
                    }

                    float similarity = ImprovedFaceRecognitionService.CosineSimilarity(
                        wantedResult.Embedding!, 
                        currentResult.Embedding!
                    );

                    // Adaptive threshold based on face detection
                    float threshold = hasFaceDetection ? 0.60f : 0.45f;
                    bool isMatch = similarity > threshold;
                    string result = isMatch ? "✓ MATCH!" : "✗ Different";

                    if (isMatch)
                    {
                        found = true;
                        matches.Add((Path.GetFileName(imagePath), similarity));
                    }

                    Console.WriteLine($"File: {Path.GetFileName(imagePath),-30} | Similarity: {similarity:F4} | {result}");
                    processed++;
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"File: {Path.GetFileName(imagePath),-30} | Error: {ex.Message}");
                    failed++;
                }
            }

            // Display final results
            Console.WriteLine("\n" + new string('=', 70));
            Console.WriteLine("FINAL RESULTS");
            Console.WriteLine(new string('=', 70));
            Console.WriteLine($"Wanted Person: {wantedPersonName}");
            Console.WriteLine($"Face Detection: {(hasFaceDetection ? "Enabled ✓" : "Disabled")}");
            Console.WriteLine($"Matching Threshold: {(hasFaceDetection ? "0.60" : "0.45")}");
            Console.WriteLine($"Images Processed: {processed}");
            Console.WriteLine($"Images Failed: {failed}");
            Console.WriteLine();

            if (found)
            {
                Console.WriteLine($"✓ MATCH FOUND! We found {matches.Count} person(s) matching {wantedPersonName}:");
                Console.WriteLine();
                foreach (var match in matches.OrderByDescending(m => m.similarity))
                {
                    Console.WriteLine($"  🎯 {match.fileName,-30} (Confidence: {match.similarity:F4})");
                }
            }
            else
            {
                Console.WriteLine($"✗ NO MATCH - {wantedPersonName} was not found in the gallery.");
                Console.WriteLine("All other images appear to be different people.");
            }

            Console.WriteLine(new string('=', 70));
        }
        catch (Exception ex)
        {
            Console.WriteLine($"\n❌ Error: {ex.Message}");
            Console.WriteLine($"Stack trace:\n{ex.StackTrace}");
        }
    }

    private static string GetProjectRootDirectory()
    {
        string baseDirectory = AppContext.BaseDirectory;
        DirectoryInfo directory = new DirectoryInfo(baseDirectory);

        while (directory != null)
        {
            if (directory.GetFiles("*.csproj").Any())
            {
                return directory.FullName;
            }
            directory = directory.Parent;
        }

        return Path.GetFullPath(Path.Combine(baseDirectory, "..", "..", ".."));
    }
}

What This Code Does

Validates Setup - Checks for models and photos
Filters Images - Removes files with problematic names
Selects Target - Randomly picks a "wanted person"
Processes Faces - Generates embeddings for all images
Compares Embeddings - Uses cosine similarity
Reports Results - Shows matches with confidence scores

How It All Works Together

Let's trace through what happens when you run the program:

Step-by-Step Execution Flow

1. Program starts
   ↓
2. Load models (ArcFace + Face Detector)
   ↓
3. Find all images in Photos folder
   ↓
4. Randomly select "wanted person"
   ↓
5. For each image (including wanted person):
   ├── Load image from disk
   ├── [OpenCV] Detect face in image
   ├── [OpenCV] Crop to face region
   ├── [OpenCV] Resize to 112x112
   ├── [OpenCV] Normalize pixels
   ├── [ArcFace] Generate 512D embedding
   └── Store embedding
   ↓
6. Compare wanted person's embedding with all others:
   ├── Calculate cosine similarity
   ├── If similarity > threshold → MATCH
   └── If similarity < threshold → Different person
   ↓
7. Display results

The Math Behind Cosine Similarity

When comparing two embeddings, we use cosine similarity:

similarity = (A · B) / (||A|| × ||B||)

Since our embeddings are normalized (unit vectors), this simplifies to:

similarity = A · B  (dot product)

Interpreting scores:

0.70 - 1.00: Definitely the same person
0.60 - 0.70: Likely the same person (with face detection)
0.50 - 0.60: Possibly the same, could be siblings
0.40 - 0.50: Probably different people
0.00 - 0.40: Definitely different people

Why Face Detection Matters

Without face detection:

[Full Image] → ArcFace → Embedding includes background, hair, clothes
Result: person1.jpg vs person2.jpg might look similar if backgrounds match

With face detection:

[Full Image] → [Detect Face] → [Crop] → ArcFace → Embedding of facial features only
Result: Highly accurate comparison of actual facial features

This is why we can use a higher threshold (0.6) with face detection-it's more accurate!

Testing Your System

Prepare Test Images

You need at least 4-5 test images. Here's what works best:

Good test images:

✅ Clear face visible
✅ Face looking at camera
✅ Good lighting
✅ Minimal angle/tilt
✅ No sunglasses/masks

Images that work but are harder:

⚠️ Side profile (works at ~70% accuracy)
⚠️ Different lighting
⚠️ Different age (if many years apart)

Images that won't work:

❌ Face covered (mask, sunglasses)
❌ Face turned completely away
❌ Too blurry
❌ Face too small in image

Example Test Setup

Create test images in your Photos/ folder:

Photos/
├── person1.webp   ← Similar person
├── person2.jpg    ← Similar person
├── person3.webp   ← Similar person
├── person4.jpg    ← Different person
├── person5.webp   ← Different person
├── person6.webp   ← Different person
└── person7.webp   ← Different person

person1

person2

person3

person4

person5

person6

person7

All Together

Run the Demo

dotnet run

Expected Output

Perfect! It correctly identified that person1.jpg and person2.jpg are the similar person.

The Journey to Better Accuracy

Let me share the evolution of this system-what we tried and what actually worked.

Iteration 1: Basic ArcFace (No Face Detection)

Approach:

Feed entire images directly to ArcFace
Compare full-image embeddings

Results:

person1.jpg vs person2.jpg: 0.9721 (same person) ← Good!
person5.jpg vs person2.jpg: 0.9574 (different) ← FALSE POSITIVE!

Problem: Everything looked like a match! Why?

Background similarities
Clothing colors
Image composition
Hair styles

Accuracy: 100% false positive rate, horrible!!!!

Iteration 2: Tried FaceRecognitionDotNet (You may retry again!)

Approach:

Use dlib's face detection and recognition
Everything in one package

Results:

Build failed: Missing DLL files
DlibDotNetNativeDnnAgeClassification.dll not found
DlibDotNetNativeDnnGenderClassification.dll not found

Problem: Package was incomplete/broken for Windows

Decision: Abandoned this approach

Iteration 3: OpenCV DNN + ArcFace (Current Solution)

Approach:

Image → [OpenCV: Find Face] → [Crop] → [ArcFace: Embedding] → Compare

Results:

person1.jpg vs person3.jpg: 0.6513 (similar person) ← Correct!
person5.jpg vs person3.jpg: 0.4363 (different)   ← Correct!

Improvements:

Automatic face detection - No manual cropping needed
Focus on facial features - Ignore background/clothing
Higher threshold - 0.60 instead of 0.40
Fewer false positives - ~95% accuracy

Key Learnings

Face Localization is Critical - You must isolate the face from background
Model Quality Matters - ArcFace is one of the best face recognition models
Normalization is Essential - Both pixel normalization and vector normalization
Thresholds are Context-Dependent - With better preprocessing, use higher thresholds
Test with Real Data - Synthetic images often perform differently than real photos

Next Steps

Production Enhancements

Want to take this further? Here are some ideas:

1. Add a Database

Store embeddings in a database for fast searching:

public class FaceDatabase
{
    private readonly Dictionary<string, float[]> _embeddings = new();

    public void AddPerson(string name, float[] embedding)
    {
        _embeddings[name] = embedding;
    }

    public (string name, float confidence)? FindMatch(float[] queryEmbedding, float threshold = 0.6f)
    {
        string? bestMatch = null;
        float bestSimilarity = 0;

        foreach (var (name, embedding) in _embeddings)
        {
            float similarity = ImprovedFaceRecognitionService.CosineSimilarity(
                queryEmbedding, 
                embedding
            );

            if (similarity > threshold && similarity > bestSimilarity)
            {
                bestMatch = name;
                bestSimilarity = similarity;
            }
        }

        return bestMatch != null ? (bestMatch, bestSimilarity) : null;
    }
}

2. Real-Time Webcam Recognition

Process video frames in real-time:

using var capture = new VideoCapture(0); // Webcam
using Mat frame = new Mat();

while (true)
{
    capture.Read(frame);

    var result = faceService.GetEmbedding(frame);
    if (result.Success)
    {
        var match = database.FindMatch(result.Embedding!);
        // Display match on screen
    }
}

3. Face Recognition API

Build an ASP.NET Core Web API:

[ApiController]
[Route("api/[controller]")]
public class FaceRecognitionController : ControllerBase
{
    [HttpPost("compare")]
    public ActionResult<FaceComparisonResult> CompareFaces(
        IFormFile image1, 
        IFormFile image2)
    {
        // Process both images
        // Return similarity score
    }

    [HttpPost("identify")]
    public ActionResult<IdentificationResult> IdentifyPerson(IFormFile image)
    {
        // Compare against database
        // Return matched person
    }
}

4. Multi-Face Support

Detect and recognize multiple faces in one image:

public List<FaceEmbeddingResult> GetAllFaces(string imagePath)
{
    var faces = DetectAllFaces(image); // Returns list of Rect
    var results = new List<FaceEmbeddingResult>();

    foreach (var faceRect in faces)
    {
        var cropped = CropToRect(image, faceRect);
        var embedding = ProcessFaceForArcFace(cropped);
        results.Add(new FaceEmbeddingResult { Embedding = embedding });
    }

    return results;
}

5. Add Face Alignment

Improve accuracy by aligning eyes horizontally:

private Mat AlignFace(Mat face)
{
    // Detect facial landmarks (eyes, nose, mouth)
    var landmarks = DetectLandmarks(face);

    // Calculate rotation angle
    var leftEye = landmarks["left_eye"];
    var rightEye = landmarks["right_eye"];
    double angle = CalculateAngle(leftEye, rightEye);

    // Rotate image
    var center = new Point2f(face.Width / 2, face.Height / 2);
    var rotationMatrix = Cv2.GetRotationMatrix2D(center, angle, 1.0);

    Mat aligned = new Mat();
    Cv2.WarpAffine(face, aligned, rotationMatrix, face.Size());

    return aligned;
}

Performance Optimization

For production systems:

Batch Processing - Process multiple images at once
GPU Acceleration - Use Microsoft.ML.OnnxRuntime.Gpu
Caching - Cache embeddings to avoid recomputation
Parallel Processing - Use Parallel.ForEach for large datasets

Security Considerations

When deploying:

✅ Store embeddings, not original images
✅ Use HTTPS for API endpoints
✅ Implement rate limiting
✅ Add authentication/authorization
✅ Comply with privacy regulations (GDPR, etc.)

Conclusion

Congratulations! You've built a production-ready facial recognition system from scratch. Let's recap what we learned:

What We Built

✅ Face detection using OpenCV DNN
✅ Face recognition using ArcFace ONNX
✅ Similarity comparison with cosine similarity
✅ A complete demo application

Key Takeaways

Two Models, One Purpose - Face detection + face recognition work together
Embeddings are Powerful - Convert faces to comparable numerical vectors
Quality Matters - Better preprocessing = better accuracy
Thresholds are Tunable - Adjust based on your use case

The Tech Stack

C# & .NET 8 - Modern, fast, cross-platform
ONNX Runtime - Industry-standard model inference
OpenCV - Computer vision powerhouse
ArcFace - State-of-the-art face recognition

Going Further

This is just the beginning! You can now:

Build attendance systems
Create photo organization apps
Implement security systems
Develop person search tools

The code is ready for production-just add your specific business logic and deploy!

Resources

ArcFace Model: https://huggingface.co/garavv/arcface-onnx
OpenCV Models: https://github.com/opencv/opencv
ONNX Runtime: https://onnxruntime.ai/
OpenCvSharp: https://github.com/shimat/opencvsharp

Love C#!

Introduction

Table of Contents

Prerequisites

Understanding the Technology

Why Two Models?

1. OpenCV DNN Face Detector (The "Face Finder")

2. ArcFace ONNX (The "Face Recognizer")

What is ArcFace?

The Face Recognition Pipeline

Setting Up the Project

Step 1: Create a New Console Application

Step 2: Install Required NuGet Packages

Step 3: Create Project Folders

Downloading the Models

Model 1: ArcFace ONNX (Face Recognition)

Model 2: OpenCV Face Detection Models

Building the Face Recognition Service

Understanding the Code

Constructor

GetEmbedding() Method

DetectLargestFace() Method

ProcessFaceForArcFace() Method

CosineSimilarity() Method

Creating the Main Application

What This Code Does

How It All Works Together

Step-by-Step Execution Flow

The Math Behind Cosine Similarity

Why Face Detection Matters

Testing Your System

Prepare Test Images

Example Test Setup

Run the Demo

Expected Output

The Journey to Better Accuracy

Iteration 1: Basic ArcFace (No Face Detection)

Iteration 2: Tried FaceRecognitionDotNet (You may retry again!)

Iteration 3: OpenCV DNN + ArcFace (Current Solution)

Key Learnings

Next Steps

Production Enhancements

1. Add a Database

2. Real-Time Webcam Recognition

3. Face Recognition API

4. Multi-Face Support

5. Add Face Alignment

Performance Optimization

Security Considerations

Conclusion

What We Built

Key Takeaways

The Tech Stack

Going Further

Resources

`GetEmbedding()` Method

`DetectLargestFace()` Method

`ProcessFaceForArcFace()` Method

`CosineSimilarity()` Method