DEV Community

Xiao Ling
Xiao Ling

Posted on • Originally published at dynamsoft.com

How to Get Camera Frames for Image Processing in .NET MAUI Windows App

Last week, we demonstrated how to create a Windows .NET MAUI app that can access the USB camera using Blazor web view. Although this solution provides a way to access the camera, for a Windows desktop application that requires optimal performance, it is recommended to prioritize native camera APIs over web APIs. In this article, we will guide you through the process of creating a .NET MAUI Windows application in C# that can capture camera frames for barcode and QR code scanning.

.NET MAUI Windows Camera QR Code Scanner

Required NuGet Packages

Configure the above packages in your csproj file:

<ItemGroup>
  <PackageReference Include="BarcodeQRCodeSDK" Version="2.3.4" />
  <PackageReference Include="SkiaSharp.Views.Maui.Controls" Version="2.88.3" />
</ItemGroup>

<ItemGroup Condition="$([MSBuild]::IsOSPlatform('windows'))">
  <PackageReference Include="OpenCvSharp4" Version="4.6.0.20220608" />
  <PackageReference Include="OpenCvSharp4.runtime.win" Version="4.6.0.20220608" />
</ItemGroup>
Enter fullscreen mode Exit fullscreen mode

The packages for Windows only are manually added to the ItemGroup with the condition $([MSBuild]::IsOSPlatform('windows')).

Steps to Capture Camera Frames for Barcode Detection in .NET MAUI Windows

We create a new MAUI content page and add the SKCanvasView to the page. The SKCanvasView is used to draw the camera frames on the screen.

<?xml version="1.0" encoding="utf-8" ?>
<ContentPage xmlns="http://schemas.microsoft.com/dotnet/2021/maui"
            xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
            xmlns:skia="clr-namespace:SkiaSharp.Views.Maui.Controls;assembly=SkiaSharp.Views.Maui.Controls"
            x:Class="BarcodeQrScanner.DesktopCameraPage"
            Title="DesktopCameraPage">
    <Grid x:Name="scannerView" Margin="0">
        <skia:SKCanvasView x:Name="canvasView"
                          Margin="0"
                          HorizontalOptions="FillAndExpand" VerticalOptions="FillAndExpand"
                          PaintSurface="OnCanvasViewPaintSurface" />

    </Grid>
</ContentPage>
Enter fullscreen mode Exit fullscreen mode

Afterwards, instantiate the barcode SDK and register the OnDisappearing and OnAppearing callback functions in the constructor:

public partial class DesktopCameraPage : ContentPage
{
    private BarcodeQRCodeReader reader;
    private Thread thread;
    private volatile bool isCapturing;
    private VideoCapture capture;

    private ConcurrentQueue<SKBitmap> _bitmapQueue = new ConcurrentQueue<SKBitmap>();
    private SKBitmap _bitmap;

    private static object lockObject = new object();

    public DesktopCameraPage()
    {
      InitializeComponent();
      this.Disappearing += OnDisappearing;
      this.Appearing += OnAppearing;

      reader = BarcodeQRCodeReader.Create();
    }
}
Enter fullscreen mode Exit fullscreen mode

The two callback functions are used to start and stop the camera capture thread when the page is shown or hidden. For example, when you press the back button in the navigation bar, the OnDisappearing function will be called to release the camera resources.

private void OnAppearing(object sender, EventArgs e)
{
    Create();
}

private void OnDisappearing(object sender, EventArgs e)
{
    Destroy();
}

private void Create()
{
    lock (lockObject)
    {
        capture = new VideoCapture(0);

        if (capture.IsOpened())
        {
            isCapturing = true;
            thread = new Thread(new ThreadStart(FrameCallback));
            thread.Start();
        }
    }
}

private void Destroy()
{
    lock (lockObject)
    {
        if (thread != null)
        {
            isCapturing = false;

            thread.Join();
            thread = null;
        }

        if (capture != null && capture.IsOpened())
        {
            capture.Release();
            capture = null;
        }

        ClearQueue();

        if (_bitmap != null)
        {
            _bitmap.Dispose();
            _bitmap = null;
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

As the camera capture thread is started, it will call the FrameCallback function to capture camera frames in a loop.

private void FrameCallback()
{
    while (isCapturing)
    {
        Decode();
    }
}
Enter fullscreen mode Exit fullscreen mode

What does the Decode() function do:

  1. Capture a camera frame to Mat using OpenCV VideoCapture.Read():

    Mat mat = new Mat();
    capture.Read(mat);
    
  2. Convert the Mat to byte[]:

    int length = mat.Cols * mat.Rows * mat.ElemSize();
    if (length == 0) return;
    byte[] bytes = new byte[length];
    Marshal.Copy(mat.Data, bytes, 0, length);
    
  3. Decode barcodes from the byte[] using the barcode SDK:

    BarcodeQRCodeReader.Result[] results = reader.DecodeBuffer(bytes, mat.Cols, mat.Rows, (int)mat.Step(), BarcodeQRCodeReader.ImagePixelFormat.IPF_RGB_888);
    
  4. Draw the barcode contours and text results on the camera frame using OpenCV Cv2.DrawContours and Cv2.PutText:

    if (results != null)
    {
        foreach (BarcodeQRCodeReader.Result result in results)
        {
            int[] points = result.Points;
            if (points != null)
            {
                OpenCvSharp.Point[] all = new OpenCvSharp.Point[4];
                int xMin = points[0], yMax = points[1];
                all[0] = new OpenCvSharp.Point(xMin, yMax);
                for (int i = 2; i < 7; i += 2)
                {
                    int x = points[i];
                    int y = points[i + 1];
                    OpenCvSharp.Point p = new OpenCvSharp.Point(x, y);
                    xMin = x < xMin ? x : xMin;
                    yMax = y > yMax ? y : yMax;
                    all[i / 2] = p;
                }
                OpenCvSharp.Point[][] contours = new OpenCvSharp.Point[][] { all };
                Cv2.DrawContours(mat, contours, 0, new Scalar(0, 0, 255), 2);
                if (result.Text != null) Cv2.PutText(mat, result.Text, new OpenCvSharp.Point(xMin, yMax), HersheyFonts.HersheySimplex, 1, new Scalar(0, 0, 255), 2);
            }
        }
    }
    
  5. Convert the color space of the frame from BGR to RGBA:

    Mat newFrame = new Mat();
    Cv2.CvtColor(mat, newFrame, ColorConversionCodes.BGR2RGBA);
    
  6. Convert Mat to SKBitmap:

    SKBitmap bitmap = new SKBitmap(mat.Cols, mat.Rows, SKColorType.Rgba8888, SKAlphaType.Premul);
    bitmap.SetPixels(newFrame.Data);
    
  7. Queue the SKBitmap to the _bitmapQueue and invalidate the SKCanvasView:

    if (_bitmapQueue.Count == 2) ClearQueue();
    _bitmapQueue.Enqueue(bitmap);
    canvasView.InvalidateSurface();
    

After the InvalidateSurface() is called, the OnCanvasViewPaintSurface function will be triggered to draw the SKBitmap on the screen:

void OnCanvasViewPaintSurface(object sender, SKPaintSurfaceEventArgs args)
{
    lock (lockObject)
    {
        if (!isCapturing) return;

        SKSurface surface = args.Surface;
        SKCanvas canvas = surface.Canvas;

        canvas.Clear();

        _bitmapQueue.TryDequeue(out _bitmap);

        if (_bitmap != null)
        {
            try
            {
                canvas.DrawBitmap(_bitmap, new SKPoint(0, 0));
            }
            catch(Exception e)
            {
                Console.WriteLine(e.Message);
            }
            finally
            {
                _bitmap.Dispose();
                _bitmap = null;
            }
        }
    }

}
Enter fullscreen mode Exit fullscreen mode

.NET MAUI Windows Camera QR Code Scanner

Source Code

https://github.com/yushulx/dotnet-barcode-qr-code-sdk/tree/main/example/maui

Image of AssemblyAI tool

Challenge Submission: SpeechCraft - AI-Powered Speech Analysis for Better Communication

SpeechCraft is an advanced real-time speech analytics platform that transforms spoken words into actionable insights. Using cutting-edge AI technology from AssemblyAI, it provides instant transcription while analyzing multiple dimensions of speech performance.

Read full post

Top comments (0)

Image of AssemblyAI

Automatic Speech Recognition with AssemblyAI

Experience near-human accuracy, low-latency performance, and advanced Speech AI capabilities with AssemblyAI's Speech-to-Text API. Sign up today and get $50 in API credit. No credit card required.

Try the API

👋 Kindness is contagious

Explore a sea of insights with this enlightening post, highly esteemed within the nurturing DEV Community. Coders of all stripes are invited to participate and contribute to our shared knowledge.

Expressing gratitude with a simple "thank you" can make a big impact. Leave your thanks in the comments!

On DEV, exchanging ideas smooths our way and strengthens our community bonds. Found this useful? A quick note of thanks to the author can mean a lot.

Okay