Implementing Flutter QR Code Scanner with Swift and AVFoundation for iOS

#flutter #ios #swift #dart

In the previous article, we implemented a Flutter barcode and QR code scanner for Android using Kotlin and CameraX. Since the Dart code is platform-independent, no changes are necessary. In this article, we will take steps to implement the native camera and barcode scanning logic for iOS using Swift, AVFoundation, and the Dynamsoft Barcode Reader SDK.

Prerequisites

Dynamsoft Barcode Reader Trial license

Step 1: Installing Dynamsoft Barcode Reader for iOS

We use CocoaPods to install the Dynamsoft Barcode Reader SDK for iOS. If you haven't installed CocoaPods yet, please follow the official instructions here to do so.

Once CocoaPods is ready, create a Podfile in the iOS folder of your Flutter project:

cd ios
pod init

Next, edit the Podfile to include the Dynamsoft Barcode Reader SDK:

target 'Runner' do
  use_frameworks!

  pod 'DynamsoftBarcodeReader','9.6.40'

end

Save the Podfile and run pod install. This command will install or update the CocoaPods dependencies, including the Flutter framework required for your iOS project.

Step 2: Adding Camera Permission to Info.plist

To enable camera access on iOS, open the Info.plist file located in the ios/Runner folder. Add the following keys:

<key>NSCameraUsageDescription</key>
<string>your usage description here</string>
<key>NSMicrophoneUsageDescription</key>
<string>your usage description here</string>

Step 3: Implementing Camera Preview with Flutter Texture in Swift

The Runner/AppDelegate.swift file serves as the entry point of the Flutter application. By default, it contains the following boilerplate code:

override func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
  ) -> Bool {
    GeneratedPluginRegistrant.register(with: self)
    return super.application(application, didFinishLaunchingWithOptions: launchOptions)
  }

To integrate the camera functionality effectively, you'll need to enhance the application method with the following steps:

Establish a Flutter method channel. This is crucial for seamless communication between the Dart environment and Swift, allowing commands and data to be exchanged between the Flutter UI and native code.
Implement a startCamera() method. This method should initiate the camera preview and continuously render this preview into a Flutter texture. This involves setting up the camera capture session, configuring input and output, and linking the camera output to a Flutter texture that can be displayed in the UI.

Flutter Method Channel in Swift

The Flutter method channel is a named channel that facilitates the sending of data between Dart and platform-specific code.

private var channel: FlutterMethodChannel?
private var width = 1920
private var height = 1080

override func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
  ) -> Bool {

    channel = FlutterMethodChannel(name: CHANNEL, binaryMessenger: flutterViewController.binaryMessenger)
    channel?.setMethodCallHandler({
      (call: FlutterMethodCall, result: @escaping FlutterResult) -> Void in
      if call.method == "startCamera" {
        self.startCamera(result: result)
      } else if call.method == "getPreviewWidth" {
        result(self.width)
      } else if call.method == "getPreviewHeight" {
        result(self.height)
      }

      else {
        result(FlutterMethodNotImplemented)
      }
    })

    GeneratedPluginRegistrant.register(with: self)
    return super.application(application, didFinishLaunchingWithOptions: launchOptions)
  }

The channel variable is an instance of FlutterMethodChannel. It receives method calls from Dart using the setMethodCallHandler method. Additionally, the invokeMethod method is used to send data from Swift to Dart.
The width and height variables, which store the camera preview size, are hardcoded to 1920x1080 here. The methods getPreviewWidth and getPreviewHeight retrieve and return these values to Dart, respectively.

Creating Flutter Texture and Camera Preview

Define the CustomCameraTexture class that extends NSObject and implements the FlutterTexture protocol:

class CustomCameraTexture: NSObject, FlutterTexture {
  private weak var textureRegistry: FlutterTextureRegistry?
  var textureId: Int64?
  private var cameraPreviewLayer: AVCaptureVideoPreviewLayer?
  private let bufferQueue = DispatchQueue(label: "com.example.flutter/barcode_scan")
  private var _lastSampleBuffer: CMSampleBuffer?
  private var customCameraTexture: CustomCameraTexture?

  private var lastSampleBuffer: CMSampleBuffer? {
    get {
      var result: CMSampleBuffer?
      bufferQueue.sync {
        result = _lastSampleBuffer
      }
      return result
    }
    set {
      bufferQueue.sync {
        _lastSampleBuffer = newValue
      }
    }
  }

  init(cameraPreviewLayer: AVCaptureVideoPreviewLayer, registry: FlutterTextureRegistry) {
    self.cameraPreviewLayer = cameraPreviewLayer
    self.textureRegistry = registry
    super.init()
    self.textureId = registry.register(self)
  }

  func copyPixelBuffer() -> Unmanaged<CVPixelBuffer>? {
    guard let sampleBuffer = lastSampleBuffer, let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
      return nil
    }

    return Unmanaged.passRetained(pixelBuffer)
  }

  func update(sampleBuffer: CMSampleBuffer) {
    lastSampleBuffer = sampleBuffer
    textureRegistry?.textureFrameAvailable(textureId!)
  }

  deinit {
    if let textureId = textureId {
      textureRegistry?.unregisterTexture(textureId)
    }
  }
}

The textureRegistry variable is an instance of FlutterTextureRegistry. It is used to register and unregister the Flutter texture.
The textureId variable stores the Flutter texture ID, which will be used to render the camera preview in Flutter.
The copyPixelBuffer() method returns the latest pixel buffer for texture rendering.
The update() method appends a new camera frame and then notifies the Flutter texture that it needs to be updated. When the textureFrameAvailable() method is invoked, the copyPixelBuffer() method is triggered to fetch the latest pixel buffer.

Create a flutterTextureEntry variable in the AppDelegate class and obtain the Flutter texture registry within the application method:

private var flutterTextureEntry: FlutterTextureRegistry?

override func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
  ) -> Bool {
    ...

    guard let flutterViewController = window?.rootViewController as? FlutterViewController else {
      return super.application(application, didFinishLaunchingWithOptions: launchOptions)
    }
    flutterTextureEntry = flutterViewController.engine!.textureRegistry

    GeneratedPluginRegistrant.register(with: self)
    return super.application(application, didFinishLaunchingWithOptions: launchOptions)
  }

In the startCamera() initiate a camera session and add a video data output to capture the camera frames. When a new frame is captured in the captureOutput() method, update the CustomCameraTexture instance with the latest sample buffer:

private func startCamera(result: @escaping FlutterResult) {
    if cameraSession != nil {
      result(self.customCameraTexture?.textureId)
      return
    }

    cameraSession = AVCaptureSession()
    cameraSession?.sessionPreset = .hd1920x1080

    guard let backCamera = AVCaptureDevice.default(for: .video), let input = try? AVCaptureDeviceInput(device: backCamera) else {
      result(FlutterError(code: "no_camera", message: "No camera available", details: nil))
      return
    }

    cameraSession?.addInput(input)
    cameraPreviewLayer = AVCaptureVideoPreviewLayer(session: cameraSession!)
    cameraPreviewLayer?.videoGravity = .resizeAspectFill

    let cameraOutput = AVCaptureVideoDataOutput()
    cameraOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA]
    cameraOutput.setSampleBufferDelegate(self, queue: DispatchQueue(label: "camera_frame_queue"))
    cameraSession?.addOutput(cameraOutput)

    self.customCameraTexture = CustomCameraTexture(cameraPreviewLayer: cameraPreviewLayer!, registry: flutterTextureEntry!)
    cameraSession?.startRunning()

    result(self.customCameraTexture?.textureId)
  }

  func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    if connection.isVideoOrientationSupported {
      connection.videoOrientation = currentVideoOrientation()
    }
    self.customCameraTexture?.update(sampleBuffer: sampleBuffer)
  }

At this point, the camera preview should function correctly on iOS/iPadOS. Next, we will integrate the Dynamsoft Barcode Reader SDK to decode barcodes and QR codes.

Step 4: Integrating Dynamsoft Barcode Reader SDK for iOS in Swfit

Import the Dynamsoft Barcode Reader SDK in the AppDelegate.swift file:
```
import DynamsoftBarcodeReader
```

Create an instance of Dynamsoft Barcode Reader and activate it with a valid license key:

@UIApplicationMain
@objc class AppDelegate: FlutterAppDelegate, AVCaptureVideoDataOutputSampleBufferDelegate, DBRLicenseVerificationListener {

private let reader = DynamsoftBarcodeReader()

override func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
  ) -> Bool {
    DynamsoftBarcodeReader.initLicense("LICENSE-KEY", verificationDelegate: self)

    do {
      let settings = try? reader.getRuntimeSettings()
      settings!.expectedBarcodesCount = 999
      try reader.updateRuntimeSettings(settings!)
    } catch {
      print("Error getting runtime settings")
    }

    ...

    GeneratedPluginRegistrant.register(with: self)
    return super.application(application, didFinishLaunchingWithOptions: launchOptions)
  }

    func dbrLicenseVerificationCallback(_ isSuccess: Bool, error: Error?) {
        if isSuccess {
        print("License verification passed")
        } else {
        print("License verification failed: \(error?.localizedDescription ?? "Unknown error")")
        }
    }
}

Decode barcode and QR code from the camera frame in the captureOutput() method. Since the decoding API is CPU-intensive, to avoid blocking the camera preview rendering, we move the decoding logic to a separate thread:

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    if connection.isVideoOrientationSupported {
      connection.videoOrientation = currentVideoOrientation()
    }
    self.customCameraTexture?.update(sampleBuffer: sampleBuffer)

    if !isProcessing {
      isProcessing = true
      DispatchQueue.global(qos: .background).async {
        self.processImage(sampleBuffer)
        self.isProcessing = false
      }
    }
  }

The isProcessing boolean variable ensures that only one frame is processed at a time, and new frames are ignored until the processing is complete. This approach helps mitigate the accumulation of asynchronous tasks and prevents the app from crashing due to memory exhaustion.

Implement the processImage() method to decode barcodes from CMSampleBuffer and send the results to the Flutter UI via the method channel:

func processImage(_ sampleBuffer: CMSampleBuffer) {
    let imageBuffer:CVImageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
    CVPixelBufferLockBaseAddress(imageBuffer, .readOnly)
    let baseAddress = CVPixelBufferGetBaseAddress(imageBuffer)
    let bufferSize = CVPixelBufferGetDataSize(imageBuffer)
    let width = CVPixelBufferGetWidth(imageBuffer)
    let height = CVPixelBufferGetHeight(imageBuffer)
    let bpr = CVPixelBufferGetBytesPerRow(imageBuffer)
    CVPixelBufferUnlockBaseAddress(imageBuffer, .readOnly)
    let buffer = Data(bytes: baseAddress!, count: bufferSize)

    let imageData = iImageData.init()
    imageData.bytes = buffer
    imageData.width = width
    imageData.height = height
    imageData.stride = bpr
    imageData.format = .ARGB_8888
    imageData.orientation = 0

    let results = try? reader.decodeBuffer(imageData)
    DispatchQueue.main.async {
      self.channel?.invokeMethod("onBarcodeDetected", arguments: self.wrapResults(results: results))
    }
  }

  func wrapResults(results:[iTextResult]?) -> NSArray {
        let outResults = NSMutableArray(capacity: 8)
        if results == nil {
            return outResults
        }
        for item in results! {
            let subDic = NSMutableDictionary(capacity: 11)
            if item.barcodeFormat_2 != EnumBarcodeFormat2.Null {
                subDic.setObject(item.barcodeFormatString_2 ?? "", forKey: "format" as NSCopying)
            }else{
                subDic.setObject(item.barcodeFormatString ?? "", forKey: "format" as NSCopying)
            }
            let points = item.localizationResult?.resultPoints as! [CGPoint]
            subDic.setObject(Int(points[0].x), forKey: "x1" as NSCopying)
            subDic.setObject(Int(points[0].y), forKey: "y1" as NSCopying)
            subDic.setObject(Int(points[1].x), forKey: "x2" as NSCopying)
            subDic.setObject(Int(points[1].y), forKey: "y2" as NSCopying)
            subDic.setObject(Int(points[2].x), forKey: "x3" as NSCopying)
            subDic.setObject(Int(points[2].y), forKey: "y3" as NSCopying)
            subDic.setObject(Int(points[3].x), forKey: "x4" as NSCopying)
            subDic.setObject(Int(points[3].y), forKey: "y4" as NSCopying)
            subDic.setObject(item.localizationResult?.angle ?? 0, forKey: "angle" as NSCopying)
            subDic.setObject(item.barcodeBytes ?? "", forKey: "barcodeBytes" as NSCopying)
            outResults.add(subDic)
        }

        return outResults
    }