How to Get Camera Frames for Image Processing in .NET MAUI Windows App

#dotnet #maui #windows #camera

Last week, we demonstrated how to create a Windows .NET MAUI app that can access the USB camera using Blazor web view. Although this solution provides a way to access the camera, for a Windows desktop application that requires optimal performance, it is recommended to prioritize native camera APIs over web APIs. In this article, we will guide you through the process of creating a .NET MAUI Windows application in C# that can capture camera frames for barcode and QR code scanning.

Required NuGet Packages

OpenCvSharp4: A .NET wrapper for OpenCV 4.x. It is used to capture camera frames.
OpenCvSharp4.runtime.win: Windows native binaries for OpenCvSharp4.
BarcodeQRCodeSDK: A .NET wrapper for Dynamsoft Barcode Reader SDK. It is used to detect 1D and 2D barcodes in camera frames.
SkiaSharp.Views.Maui.Controls: A set of views that can be used to draw on the screen.

Configure the above packages in your csproj file:

<ItemGroup>
  <PackageReference Include="BarcodeQRCodeSDK" Version="2.3.4" />
  <PackageReference Include="SkiaSharp.Views.Maui.Controls" Version="2.88.3" />
</ItemGroup>

<ItemGroup Condition="$([MSBuild]::IsOSPlatform('windows'))">
  <PackageReference Include="OpenCvSharp4" Version="4.6.0.20220608" />
  <PackageReference Include="OpenCvSharp4.runtime.win" Version="4.6.0.20220608" />
</ItemGroup>

The packages for Windows only are manually added to the ItemGroup with the condition $([MSBuild]::IsOSPlatform('windows')).

Steps to Capture Camera Frames for Barcode Detection in .NET MAUI Windows

We create a new MAUI content page and add the SKCanvasView to the page. The SKCanvasView is used to draw the camera frames on the screen.

<?xml version="1.0" encoding="utf-8" ?>
<ContentPage xmlns="http://schemas.microsoft.com/dotnet/2021/maui"
            xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
            xmlns:skia="clr-namespace:SkiaSharp.Views.Maui.Controls;assembly=SkiaSharp.Views.Maui.Controls"
            x:Class="BarcodeQrScanner.DesktopCameraPage"
            Title="DesktopCameraPage">
    <Grid x:Name="scannerView" Margin="0">
        <skia:SKCanvasView x:Name="canvasView"
                          Margin="0"
                          HorizontalOptions="FillAndExpand" VerticalOptions="FillAndExpand"
                          PaintSurface="OnCanvasViewPaintSurface" />

    </Grid>
</ContentPage>

Afterwards, instantiate the barcode SDK and register the OnDisappearing and OnAppearing callback functions in the constructor:

public partial class DesktopCameraPage : ContentPage
{
    private BarcodeQRCodeReader reader;
    private Thread thread;
    private volatile bool isCapturing;
    private VideoCapture capture;

    private ConcurrentQueue<SKBitmap> _bitmapQueue = new ConcurrentQueue<SKBitmap>();
    private SKBitmap _bitmap;

    private static object lockObject = new object();

    public DesktopCameraPage()
    {
      InitializeComponent();
      this.Disappearing += OnDisappearing;
      this.Appearing += OnAppearing;

      reader = BarcodeQRCodeReader.Create();
    }
}

The two callback functions are used to start and stop the camera capture thread when the page is shown or hidden. For example, when you press the back button in the navigation bar, the OnDisappearing function will be called to release the camera resources.

private void OnAppearing(object sender, EventArgs e)
{
    Create();
}

private void OnDisappearing(object sender, EventArgs e)
{
    Destroy();
}

private void Create()
{
    lock (lockObject)
    {
        capture = new VideoCapture(0);

        if (capture.IsOpened())
        {
            isCapturing = true;
            thread = new Thread(new ThreadStart(FrameCallback));
            thread.Start();
        }
    }
}

private void Destroy()
{
    lock (lockObject)
    {
        if (thread != null)
        {
            isCapturing = false;

            thread.Join();
            thread = null;
        }

        if (capture != null && capture.IsOpened())
        {
            capture.Release();
            capture = null;
        }

        ClearQueue();

        if (_bitmap != null)
        {
            _bitmap.Dispose();
            _bitmap = null;
        }
    }
}

As the camera capture thread is started, it will call the FrameCallback function to capture camera frames in a loop.

private void FrameCallback()
{
    while (isCapturing)
    {
        Decode();
    }
}

What does the Decode() function do:

Capture a camera frame to Mat using OpenCV VideoCapture.Read():
```
Mat mat = new Mat();
capture.Read(mat);
```

Convert the Mat to byte[]:

int length = mat.Cols * mat.Rows * mat.ElemSize();
if (length == 0) return;
byte[] bytes = new byte[length];
Marshal.Copy(mat.Data, bytes, 0, length);

Decode barcodes from the byte[] using the barcode SDK:

BarcodeQRCodeReader.Result[] results = reader.DecodeBuffer(bytes, mat.Cols, mat.Rows, (int)mat.Step(), BarcodeQRCodeReader.ImagePixelFormat.IPF_RGB_888);

Draw the barcode contours and text results on the camera frame using OpenCV Cv2.DrawContours and Cv2.PutText:

if (results != null)
{
    foreach (BarcodeQRCodeReader.Result result in results)
    {
        int[] points = result.Points;
        if (points != null)
        {
            OpenCvSharp.Point[] all = new OpenCvSharp.Point[4];
            int xMin = points[0], yMax = points[1];
            all[0] = new OpenCvSharp.Point(xMin, yMax);
            for (int i = 2; i < 7; i += 2)
            {
                int x = points[i];
                int y = points[i + 1];
                OpenCvSharp.Point p = new OpenCvSharp.Point(x, y);
                xMin = x < xMin ? x : xMin;
                yMax = y > yMax ? y : yMax;
                all[i / 2] = p;
            }
            OpenCvSharp.Point[][] contours = new OpenCvSharp.Point[][] { all };
            Cv2.DrawContours(mat, contours, 0, new Scalar(0, 0, 255), 2);
            if (result.Text != null) Cv2.PutText(mat, result.Text, new OpenCvSharp.Point(xMin, yMax), HersheyFonts.HersheySimplex, 1, new Scalar(0, 0, 255), 2);
        }
    }
}

Convert the color space of the frame from BGR to RGBA:

Mat newFrame = new Mat();
Cv2.CvtColor(mat, newFrame, ColorConversionCodes.BGR2RGBA);

Convert Mat to SKBitmap:

SKBitmap bitmap = new SKBitmap(mat.Cols, mat.Rows, SKColorType.Rgba8888, SKAlphaType.Premul);
bitmap.SetPixels(newFrame.Data);

Queue the SKBitmap to the _bitmapQueue and invalidate the SKCanvasView:

if (_bitmapQueue.Count == 2) ClearQueue();
_bitmapQueue.Enqueue(bitmap);
canvasView.InvalidateSurface();

After the InvalidateSurface() is called, the OnCanvasViewPaintSurface function will be triggered to draw the SKBitmap on the screen:

void OnCanvasViewPaintSurface(object sender, SKPaintSurfaceEventArgs args)
{
    lock (lockObject)
    {
        if (!isCapturing) return;

        SKSurface surface = args.Surface;
        SKCanvas canvas = surface.Canvas;

        canvas.Clear();

        _bitmapQueue.TryDequeue(out _bitmap);

        if (_bitmap != null)
        {
            try
            {
                canvas.DrawBitmap(_bitmap, new SKPoint(0, 0));
            }
            catch(Exception e)
            {
                Console.WriteLine(e.Message);
            }
            finally
            {
                _bitmap.Dispose();
                _bitmap = null;
            }
        }
    }

}