ML Kit in React Native (Part 1)

Machine learning isn't a science problem anymore. It's a common part of mobile apps nowadays. I found Google ML Kit is a quite interesting solution and decided to cover how we can use it in React Native. Since there is no dedicated library we will use native modules to wire up the JavaScript side with iOS and Android SDKs.

This tutorial will cover two topics - native modules basics for iOS and Android and Text Recognition API from the Google ML Kit. I also have it in video format on my YouTube channel

What are we building?

Boilerplate project

We are going to recognise text from the images. So I prepared project boilerplate which allow get image from Camera or Gallery. You can download it from here. We're using react-native-image-picker to select image and get its uri.

Android Native Module

Let's start with Android. And first of all, we need to create a Native Module. To do so we need simply follow the steps in the official docs.

Open the ./android/ project in the Android Studio.

Create a custom native module file ./android/app/src/main/java/com/rnmlkittutorial/mlkit/TextRecognitionModule.java

Then add the following content:

package com.rnmlkittutorial.mlkit;

import android.util.Log;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.bridge.ReactContextBaseJavaModule;
import com.facebook.react.bridge.ReactMethod;

public class TextRecognitionModule extends ReactContextBaseJavaModule {
    TextRecognitionModule(ReactApplicationContext context) {
        super(context);
    }

    @Override
    public String getName() {
        return "TextRecognitionModule";
    }

    @ReactMethod
    public void recognizeImage(String url) {
        Log.d("TextRecognitionModule", "Url: " + url);
    }
}

Our class TextRecognitionModule extends ReactContextBaseJavaModule which has inner implementation required by React Native.

Method getName returns our module which we will use in React JavaScript.

public String getName() {
    return "TextRecognitionModule";
}

@ReactMethod annotation means this method will be accessible on the JavaScript side. And we will use recognizeImage method to pass image url to the native Java side.

@ReactMethod
public void recognizeImage(String url) {
    Log.d("TextRecognitionModule", "Url: " + url);
}

To add your Native Module to ReactPackage, first create a new Java Class named TextRecognitionPackage.java that implements ReactPackage inside the ./android/app/src/main/java/com/rnmlkittutorial/mlkit/ folder. Then add the following content:

package com.rnmlkittutorial.mlkit;

import com.facebook.react.ReactPackage;
import com.facebook.react.bridge.NativeModule;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.uimanager.ViewManager;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

public class TextRecognitionPackage implements ReactPackage {
    @Override
    public List<ViewManager> createViewManagers(ReactApplicationContext reactContext) {
        return Collections.emptyList();
    }

    @Override
    public List<NativeModule> createNativeModules(
        ReactApplicationContext reactContext) {
        List<NativeModule> modules = new ArrayList<>();

        modules.add(new TextRecognitionModule(reactContext));

        return modules;
    }
}

Inside createNativeModules method we create an instance of our module and add it to the NativeModules array which this package class contains.
createViewManagers is responsible for UI components that we don't use. So simply return the empty list here.

To register the TextRecognitionModule package, we must add TextRecognitionPackage to the list of packages returned in ReactNativeHost's getPackages() method. Open up your MainApplication.java file, which can be found in the following path: ./android/app/src/main/java/com/rnmlkittutorial/MainApplication.java

@Override
protected List<ReactPackage> getPackages() {
  @SuppressWarnings("UnnecessaryLocalVariable")
  List<ReactPackage> packages = new PackageList(this).getPackages();
  // Packages that cannot be autolinked yet can be added manually here, for example:
   packages.add(new TextRecognitionPackage());
  return packages;
}

Here we can test what we have built. Let's back to our TypeScript side and create file src/mlkit/index.ts. And put the following code into it

import {NativeModules} from 'react-native';

const {TextRecognitionModule} = NativeModules;

export const recognizeImage = (url: string) => {
  return TextRecognitionModule.recognizeImage(url);
}

We take NativeModules from React Native which now contain our TextRecognitionModule. And we can invoke recognizeImage function we defined on the Java side. Let's go to the src/screens/ProcessImageScreen.tsx and use our module function like this

...
import {recognizeImage} from '../mlkit';
...
const proccessImage = async (url: string) => {
  if (url) {
    recognizeImage(url);
  }
};
...

Now run the android app npx react-native run-android or whatever you like. Select the image and tap Process Image.

Then you can back to the Android Studio and press Logcat tab. To find our log put TextRecognitionModule in the search field. This is first argument of Android Log.d function Log.d("TextRecognitionModule", "Url: " + url);