DEV Community: Vivi-clevercoder

How a Programmer Developed a Text Reader App for His 80-Year-Old Grandpa

Vivi-clevercoder — Fri, 06 Aug 2021 10:29:24 +0000

"John, have you seen my glasses?"

Our old friend John, a programmer at Huawei, has a grandpa who despite his old age, is an avid reader. Leaning back, struggling to make out what was written on the newspaper through his glasses, but unable to take his eyes off the text — this was how my grandpa used to read, John explained.

Reading this way was harmful on his grandpa's vision, and it occurred to John that the ears could take over the role of "reading" from the eyes. He soon developed a text-reading app that followed this logic, recognizing and then reading out text from a picture. Thanks to this app, John's grandpa now can ”read” from the comfort of his rocking chair, without having to strain his eyes.

How to Implement

The user takes a picture of a text passage. The app then automatically identifies the location of the text within the picture, and adjusts the shooting angle to an angle directly facing the text.
The app recognizes and extracts the text from the picture.
The app converts the recognized text into audio output by leveraging text-to-speech technology. These functions are easy to implement, when relying on three services in HUAWEI ML Kit: document skew correction, text recognition, and text to speech (TTS).

Preparations

buildscript {
    repositories {
        google()
        jcenter()
        maven {url 'https://developer.huawei.com/repo/'}
    }
    dependencies {
        classpath "com.android.tools.build:gradle:4.1.1"
        classpath 'com.huawei.agconnect:agcp:1.4.2.300'
        // NOTE: Do not place your app dependencies here; they belong
        // in the individual module build.gradle files.
    }
}
allprojects {
    repositories {
        google()
        jcenter()
        maven {url 'https://developer.huawei.com/repo/'}
    }
}

Add the build dependencies for the HMS Core SDK.

dependencies {

    // Import the base SDK.
    implementation 'com.huawei.hms:ml-computer-voice-tts:2.1.0.300'
    // Import the bee voice package.
    implementation 'com.huawei.hms:ml-computer-voice-tts-model-bee:2.1.0.300'
    // Import the eagle voice package.
    implementation 'com.huawei.hms:ml-computer-voice-tts-model-eagle:2.1.0.300'
    // Import a PDF file analyzer.
    implementation 'com.itextpdf:itextg:5.5.10'
}

Tap PREVIOUS or NEXT to turn to the previous or next page. Tap speak to start reading; tap it again to pause reading.

Development process

Create a TTS engine by using the custom configuration class MLTtsConfig. Here, on-device TTS is used as an example.

private void initTts() {
    // Set authentication information for your app to download the model package from the server of Huawei.
    MLApplication.getInstance().setApiKey(AGConnectServicesConfig.
            fromContext(getApplicationContext()).getString("client/api_key"));
    // Create a TTS engine by using MLTtsConfig.
    mlTtsConfigs = new MLTtsConfig()
            // Set the text converted from speech to English.
            .setLanguage(MLTtsConstants.TTS_EN_US)
            // Set the speaker with the English male voice (eagle).
            .setPerson(MLTtsConstants.TTS_SPEAKER_OFFLINE_EN_US_MALE_EAGLE)
            // Set the speech speed whose range is (0, 5.0]. 1.0 indicates a normal speed.
            .setSpeed(.8f)
            // Set the volume whose range is (0, 2). 1.0 indicates a normal volume.
            .setVolume(1.0f)
            // Set the TTS mode to on-device.
            .setSynthesizeMode(MLTtsConstants.TTS_OFFLINE_MODE);
    mlTtsEngine = new MLTtsEngine(mlTtsConfigs);
    // Update the configuration when the engine is running.
    mlTtsEngine.updateConfig(mlTtsConfigs);
    // Pass the TTS callback function to the TTS engine to perform TTS.
    mlTtsEngine.setTtsCallback(callback);
    // Create an on-device TTS model manager.
    manager = MLLocalModelManager.getInstance();
    isPlay = false;
}

Create a TTS callback function for processing the TTS result.

MLTtsCallback callback = new MLTtsCallback() {
    @Override   
    public void onError(String taskId, MLTtsError err) {
        // Processing logic for TTS failure.
    }
    @Override
    public void onWarn(String taskId, MLTtsWarn warn) {
        // Alarm handling without affecting service logic.
    }
    @Override
    // Return the mapping between the currently played segment and text. start: start position of the audio segment in the input text; end (excluded): end position of the audio segment in the input text.
    public void onRangeStart(String taskId, int start, int end) {
        // Process the mapping between the currently played segment and text.
    }
    @Override
    // taskId: ID of a TTS task corresponding to the audio.
    // audioFragment: audio data.
    // offset: offset of the audio segment to be transmitted in the queue. One TTS task corresponds to a TTS queue.
    // range: text area where the audio segment to be transmitted is located; range.first (included): start position; range.second (excluded): end position.
    public void onAudioAvailable(String taskId, MLTtsAudioFragment audioFragment, int offset,
                                 Pair<Integer, Integer> range, Bundle bundle) {
        // Audio stream callback API, which is used to return the synthesized audio data to the app.
    }
    @Override
    public void onEvent(String taskId, int eventId, Bundle bundle) {
        // Callback method of a TTS event. eventId indicates the event name.
        boolean isInterrupted;
        switch (eventId) {
            case MLTtsConstants.EVENT_PLAY_START:
                // Called when playback starts.
                break;
            case MLTtsConstants.EVENT_PLAY_STOP:
                // Called when playback stops.
                isInterrupted = bundle.getBoolean(MLTtsConstants.EVENT_PLAY_STOP_INTERRUPTED);
                break;
            case MLTtsConstants.EVENT_PLAY_RESUME:
                // Called when playback resumes.
                break;
            case MLTtsConstants.EVENT_PLAY_PAUSE:
                // Called when playback pauses.
                break;
            // Pay attention to the following callback events when you focus on only the synthesized audio data but do not use the internal player for playback.
            case MLTtsConstants.EVENT_SYNTHESIS_START:
                // Called when TTS starts.
                break;
            case MLTtsConstants.EVENT_SYNTHESIS_END:
                // Called when TTS ends.
                break;
            case MLTtsConstants.EVENT_SYNTHESIS_COMPLETE:
                // TTS is complete. All synthesized audio streams are passed to the app.
                isInterrupted = bundle.getBoolean(MLTtsConstants.EVENT_SYNTHESIS_INTERRUPTED);
                break;
            default:
                break;
        }
    }
};

Extract text from a PDF file.

private String loadText(String path) {
    String result = "";
    try {
        PdfReader reader = new PdfReader(path);
        result = result.concat(PdfTextExtractor.getTextFromPage(reader,
                mCurrentPage.getIndex() + 1).trim() + System.lineSeparator());
        reader.close();
    } catch (IOException e) {
        showToast(e.getMessage());
    }
    // Obtain the position of the header.
    int header = result.indexOf(System.lineSeparator());
    // Obtain the position of the footer.
    int footer = result.lastIndexOf(System.lineSeparator());
    if (footer != 0){
        // Do not display the text in the header and footer.
        return result.substring(header, footer - 5);
    }else {
        return result;
    }
}

Perform TTS in on-device mode.

// Create an MLTtsLocalModel instance to set the speaker so that the language model corresponding to the speaker can be downloaded through the model manager.
MLTtsLocalModel model = new MLTtsLocalModel.Factory(MLTtsConstants.TTS_SPEAKER_OFFLINE_EN_US_MALE_EAGLE).create();
manager.isModelExist(model).addOnSuccessListener(new OnSuccessListener<Boolean>() {
    @Override
    public void onSuccess(Boolean aBoolean) {
        // If the model is not downloaded, call the download API. Otherwise, call the TTS API of the on-device engine.
        if (aBoolean) {
            String source = loadText(mPdfPath);
            // Call the speak API to perform TTS. source indicates the text to be synthesized.
            mlTtsEngine.speak(source, MLTtsEngine.QUEUE_APPEND);
            if (isPlay){
                // Pause playback.
                mlTtsEngine.pause();
                tv_speak.setText("speak");
            }else {
                // Resume playback.
                mlTtsEngine.resume();
                tv_speak.setText("pause");
            }
            isPlay = !isPlay;
        } else {
            // Call the API for downloading the on-device TTS model.
            downloadModel(MLTtsConstants.TTS_SPEAKER_OFFLINE_EN_US_MALE_EAGLE);
            showToast("The offline model has not been downloaded!");
        }
    }
}).addOnFailureListener(new OnFailureListener() {
    @Override
    public void onFailure(Exception e) {
        showToast(e.getMessage());
    }
});

Release resources when the current UI is destroyed.

@Override
protected void onDestroy() {
    super.onDestroy();
    try {
        if (mParcelFileDescriptor != null) {
            mParcelFileDescriptor.close();
        }
        if (mCurrentPage != null) {
            mCurrentPage.close();
        }
        if (mPdfRenderer != null) {
            mPdfRenderer.close();
        }
        if (mlTtsEngine != null){
            mlTtsEngine.shutdown();
        }
    } catch (IOException e) {
        e.printStackTrace();
    }
}

Other Applicable Scenarios

TTS can be used across a broad range of scenarios. For example, you could integrate it into an education app to read bedtime stories to children, or integrate it into a navigation app, which could read out instructions aloud.

To learn more, visit the following links:
Documentation on the HUAWEI Developers website
https://developer.huawei.com/consumer/en/hms/huawei-MapKit

HUAWEI Developers official website

Development Guide

Redditto join developer discussions

GitHub or Gitee to download the demo and sample code

Stack Overflow to solve integration problems

How a Programmer Used 300 Lines of Code to Help His Grandma Shop Online with Voice Input

Vivi-clevercoder — Fri, 30 Jul 2021 08:38:24 +0000

"John, why the writing pad is missing again?"

John, programmer at Huawei, has a grandma who loves novelty, and lately she's been obsessed with online shopping. Familiarizing herself with major shopping apps and their functions proved to be a piece of cake, and she had thought that her online shopping experience would be effortless — unfortunately, however, she was hindered by product searching.

John's grandma tended to use handwriting input. When using it, she would often make mistakes, like switching to another input method she found unfamiliar, or tapping on undesired characters or signs.
Except for shopping apps, most mobile apps feature interface designs that are oriented to younger users — it's no wonder that elderly users often struggle to figure out how to use them.
John patiently helped his grandma search for products with handwriting input several times. But then, he decided to use his skills as a veteran coder to give his grandma the best possible online shopping experience. More specifically, instead of helping her adjust to the available input method, he was determined to create an input method that would conform to her usage habits.
Since his grandma tended to err during manual input, John developed an input method that converts speech into text. Grandma was enthusiastic about the new method, because it is remarkably easy to use. All she has to do is to tap on the recording button and say the product's name. The input method then recognizes what she has said, and converts her speech into text.

Real-time speech recognition and speech to text are ideal for a broad range of apps, including:

Game apps (online): Real-time speech recognition comes to users' aid when they team up with others. It frees up users' hands for controlling the action, sparing them from having to type to communicate with their partners. It can also free users from any potential embarrassment related to voice chatting during gaming.
Work apps: Speech to text can play a vital role during long conferences, where typing to keep meeting minutes can be tedious and inefficient, with key details being missed. Using speech to text is much more efficient: during a conference, users can use this service to convert audio content into text; after the conference, they can simply retouch the text to make it more logical.
Learning apps: Speech to text can offer users an enhanced learning experience. Without the service, users often have to pause audio materials to take notes, resulting in a fragmented learning process. With speech to text, users can concentrate on listening intently to the material while it is being played, and rely on the service to convert the audio content into text. They can then review the text after finishing the entire course, to ensure that they've mastered the content.

How to Implement

Two services in HUAWEI ML Kit: automatic speech recognition (ASR) and audio file transcription, make it easy to implement the above functions.

ASR can recognize speech of up to 60s, and convert the input speech into text in real time, with recognition accuracy of over 95%. It currently supports Mandarin Chinese (including Chinese-English bilingual speech), English, French, German, Spanish, Italian, and Arabic.
 Real-time result output
 Available options: with and without speech pickup UI
 Endpoint detection: Start and end points can be accurately located.
 Silence detection: No voice packet is sent for silent portions.
 Intelligent conversion to digital formats: For example, the year 2021 is recognized from voice input.
Audio file transcription can convert an audio file of up to five hours into text with punctuation, and automatically segment the text for greater clarity. In addition, this service can generate text with timestamps, facilitating further function development.

In this version, both Chinese and English are supported.

Development Procedures

Preparations

（1） Configure the Huawei Maven repository address, and put the agconnect-services.json file under the app directory.
Open the build.gradle file in the root directory of your Android Studio project.
Add the AppGallery Connect plugin and the Maven repository.
 Go to allprojects > repositories and configure the Maven repository address for the HMS Core SDK.
 Go to buildscript > repositories and configure the Maven repository address for the HMS Core SDK.
 If the agconnect-services.json file has been added to the app, go to buildscript > dependencies and add the AppGallery Connect plugin configuration.

buildscript {
    repositories {
        google()
        jcenter()
        maven { url 'https://developer.huawei.com/repo/' }
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:3.5.4'
        classpath 'com.huawei.agconnect:agcp:1.4.1.300'
        // NOTE: Do not place your app dependencies here; they belong
        // in the individual module build.gradle files.
    }
}

allprojects {
    repositories {
        google()
        jcenter()
        maven { url 'https://developer.huawei.com/repo/' }
    }
}
Set the app authentication information. For details, see Notes on Using Cloud Authentication Information.
（2） Add the build dependencies for the HMS Core SDK.
dependencies {
    //The audio file transcription SDK.
    implementation 'com.huawei.hms:ml-computer-voice-aft:2.2.0.300'
    // The ASR SDK.
    implementation 'com.huawei.hms:ml-computer-voice-asr:2.2.0.300'
    // Plugin of ASR.
    implementation 'com.huawei.hms:ml-computer-voice-asr-plugin:2.2.0.300'
    ...
}
apply plugin: 'com.huawei.agconnect'  // AppGallery Connect plugin.

（3） Configure the signing certificate in the build.gradle file under the app directory.

signingConfigs {
    release {
        storeFile file("xxx.jks")
        keyAlias xxx
        keyPassword xxxxxx
        storePassword xxxxxx
        v1SigningEnabled true
        v2SigningEnabled true
    }

}

buildTypes {
    release {
        minifyEnabled false
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
    }

    debug {
        signingConfig signingConfigs.release
        debuggable true
    }
}

（4） Add permissions in the AndroidManifest.xml file.

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />

<application
    android:requestLegacyExternalStorage="true"
  ...
</application>

Integrating the ASR Service

（1） Dynamically apply for the permissions.

if (ActivityCompat.checkSelfPermission(this, Manifest.permission.RECORD_AUDIO) != PackageManager.PERMISSION_GRANTED) {
    requestCameraPermission();
}

private void requestCameraPermission() {
    final String[] permissions = new String[]{Manifest.permission.RECORD_AUDIO};
    if (!ActivityCompat.shouldShowRequestPermissionRationale(this, Manifest.permission.RECORD_AUDIO)) {
        ActivityCompat.requestPermissions(this, permissions, Constants.AUDIO_PERMISSION_CODE);
        return;
    }
}

（2） Create an Intent to set parameters.

// Set authentication information for your app.
MLApplication.getInstance().setApiKey(AGConnectServicesConfig.fromContext(this).getString("client/api_key"));
//// Use Intent for recognition parameter settings.
Intent intentPlugin = new Intent(this, MLAsrCaptureActivity.class)
        // Set the language that can be recognized to English. If this parameter is not set, English is recognized by default. Example: "zh-CN": Chinese; "en-US": English.
        .putExtra(MLAsrCaptureConstants.LANGUAGE, MLAsrConstants.LAN_EN_US)
        // Set whether to display the recognition result on the speech pickup UI.
        .putExtra(MLAsrCaptureConstants.FEATURE, MLAsrCaptureConstants.FEATURE_WORDFLUX);
startActivityForResult(intentPlugin, "1");

（3） Override the onActivityResult method to process the result returned by ASR.

@Override
protected void onActivityResult(int requestCode, int resultCode, @Nullable Intent data) {
    super.onActivityResult(requestCode, resultCode, data);
    String text = "";
    if (null == data) {
        addTagItem("Intent data is null.", true);
    }
    if (requestCode == "1") {
        if (data == null) {
            return;
        }
        Bundle bundle = data.getExtras();
        if (bundle == null) {
            return;
        }
        switch (resultCode) {
            case MLAsrCaptureConstants.ASR_SUCCESS:
                // Obtain the text information recognized from speech.
                if (bundle.containsKey(MLAsrCaptureConstants.ASR_RESULT)) {
                    text = bundle.getString(MLAsrCaptureConstants.ASR_RESULT);
                }
                if (text == null || "".equals(text)) {
                    text = "Result is null.";
                    Log.e(TAG, text);
                } else {
                    // Display the recognition result in the search box.
                    searchEdit.setText(text);
                    goSearch(text, true);
                }
                break;
            // MLAsrCaptureConstants.ASR_FAILURE: Recognition fails.
            case MLAsrCaptureConstants.ASR_FAILURE:
                // Check whether an error code is contained.
                if (bundle.containsKey(MLAsrCaptureConstants.ASR_ERROR_CODE)) {
                    text = text + bundle.getInt(MLAsrCaptureConstants.ASR_ERROR_CODE);
                    // Troubleshoot based on the error code.
                }
                // Check whether error information is contained.
                if (bundle.containsKey(MLAsrCaptureConstants.ASR_ERROR_MESSAGE)) {
                    String errorMsg = bundle.getString(MLAsrCaptureConstants.ASR_ERROR_MESSAGE);
                    // Troubleshoot based on the error information.
                    if (errorMsg != null && !"".equals(errorMsg)) {
                        text = "[" + text + "]" + errorMsg;
                    }
                }
                // Check whether a sub-error code is contained.
                if (bundle.containsKey(MLAsrCaptureConstants.ASR_SUB_ERROR_CODE)) {
                    int subErrorCode = bundle.getInt(MLAsrCaptureConstants.ASR_SUB_ERROR_CODE);
                    // Troubleshoot based on the sub-error code.
                    text = "[" + text + "]" + subErrorCode;
                }
                Log.e(TAG, text);
                break;
            default:
                break;
        }
    }
}

3. Integrating the Audio File Transcription Service

（1） Dynamically apply for the permissions.

private static final int REQUEST_EXTERNAL_STORAGE = 1;
private static final String[] PERMISSIONS_STORAGE = {
        Manifest.permission.READ_EXTERNAL_STORAGE,
        Manifest.permission.WRITE_EXTERNAL_STORAGE };
public static void verifyStoragePermissions(Activity activity) {
    // Check if the write permission has been granted.
    int permission = ActivityCompat.checkSelfPermission(activity,
            Manifest.permission.WRITE_EXTERNAL_STORAGE);
    if (permission != PackageManager.PERMISSION_GRANTED) {
        // The permission has not been granted. Prompt the user to grant it.
        ActivityCompat.requestPermissions(activity, PERMISSIONS_STORAGE,
                REQUEST_EXTERNAL_STORAGE);
    }
}

（2） Create and initialize an audio transcription engine, and create an audio file transcription configurator.

// Set the API key.
MLApplication.getInstance().setApiKey(AGConnectServicesConfig.fromContext(getApplication()).getString("client/api_key"));
MLRemoteAftSetting setting = new MLRemoteAftSetting.Factory()
        // Set the transcription language code, complying with the BCP 47 standard. Currently, Mandarin Chinese and English are supported.
        .setLanguageCode("zh")
        // Set whether to automatically add punctuations to the converted text. The default value is false.
        .enablePunctuation(true)
        // Set whether to generate the text transcription result of each audio segment and the corresponding audio time shift. The default value is false. (This parameter needs to be set only when the audio duration is less than 1 minute.)
        .enableWordTimeOffset(true)
        // Set whether to output the time shift of a sentence in the audio file. The default value is false.
        .enableSentenceTimeOffset(true)
        .create();

// Create an audio transcription engine.
MLRemoteAftEngine engine = MLRemoteAftEngine.getInstance();
engine.init(this);
// Pass the listener callback to the audio transcription engine created beforehand.
engine.setAftListener(aftListener);

（3） Create a listener callback to process the audio file transcription result.

 Transcription of short audio files with a duration of 1 minute or shorter:

private MLRemoteAftListener aftListener = new MLRemoteAftListener() {
    public void onResult(String taskId, MLRemoteAftResult result, Object ext) {
        // Obtain the transcription result notification.
        if (result.isComplete()) {
            // Process the transcription result.
        }
    }
    @Override
    public void onError(String taskId, int errorCode, String message) {
        // Callback upon a transcription error.
    }
    @Override
    public void onInitComplete(String taskId, Object ext) {
        // Reserved.
    }
    @Override
    public void onUploadProgress(String taskId, double progress, Object ext) {
        // Reserved.
    }
    @Override
    public void onEvent(String taskId, int eventId, Object ext) {
        // Reserved.
    }
};

 Transcription of audio files with a duration longer than 1 minute:

private MLRemoteAftListener asrListener = new MLRemoteAftListener() {
    @Override
    public void onInitComplete(String taskId, Object ext) {
        Log.e(TAG, "MLAsrCallBack onInitComplete");
        // The long audio file is initialized and the transcription starts.
        start(taskId);
    }
    @Override
    public void onUploadProgress(String taskId, double progress, Object ext) {
        Log.e(TAG, " MLAsrCallBack onUploadProgress");
    }
    @Override
    public void onEvent(String taskId, int eventId, Object ext) {
        // Used for the long audio file.
        Log.e(TAG, "MLAsrCallBack onEvent" + eventId);
        if (MLAftEvents.UPLOADED_EVENT == eventId) { // The file is uploaded successfully.
            // Obtain the transcription result.
            startQueryResult(taskId);
        }
    }
    @Override
    public void onResult(String taskId, MLRemoteAftResult result, Object ext) {
        Log.e(TAG, "MLAsrCallBack onResult taskId is :" + taskId + " ");
        if (result != null) {
            Log.e(TAG, "MLAsrCallBack onResult isComplete: " + result.isComplete());
            if (result.isComplete()) {
                TimerTask timerTask = timerTaskMap.get(taskId);
                if (null != timerTask) {
                    timerTask.cancel();
                    timerTaskMap.remove(taskId);
                }
                if (result.getText() != null) {
                    Log.e(TAG, taskId + " MLAsrCallBack onResult result is : " + result.getText());
                    tvText.setText(result.getText());
                }
                List<MLRemoteAftResult.Segment> words = result.getWords();
                if (words != null && words.size() != 0) {
                    for (MLRemoteAftResult.Segment word : words) {
                        Log.e(TAG, "MLAsrCallBack word  text is : " + word.getText() + ", startTime is : " + word.getStartTime() + ". endTime is : " + word.getEndTime());
                    }
                }
                List<MLRemoteAftResult.Segment> sentences = result.getSentences();
                if (sentences != null && sentences.size() != 0) {
                    for (MLRemoteAftResult.Segment sentence : sentences) {
                        Log.e(TAG, "MLAsrCallBack sentence  text is : " + sentence.getText() + ", startTime is : " + sentence.getStartTime() + ". endTime is : " + sentence.getEndTime());
                    }
                }
            }
        }
    }
    @Override
    public void onError(String taskId, int errorCode, String message) {
        Log.i(TAG, "MLAsrCallBack onError : " + message + "errorCode, " + errorCode);
        switch (errorCode) {
            case MLAftErrors.ERR_AUDIO_FILE_NOTSUPPORTED:
                break;
        }
    }
};
// Upload a transcription task.
private void start(String taskId) {
    Log.e(TAG, "start");
    engine.setAftListener(asrListener);
    engine.startTask(taskId);
}
// Obtain the transcription result.
private Map<String, TimerTask> timerTaskMap = new HashMap<>();
private void startQueryResult(final String taskId) {
    Timer mTimer = new Timer();
    TimerTask mTimerTask = new TimerTask() {
        @Override
        public void run() {
            getResult(taskId);
        }
    };
    // Periodically obtain the long audio file transcription result every 10s.
    mTimer.schedule(mTimerTask, 5000, 10000);
    // Clear timerTaskMap before destroying the UI.
    timerTaskMap.put(taskId, mTimerTask);
}

（4） Obtain an audio file and upload it to the audio transcription engine.

// Obtain the URI of an audio file.
Uri uri = getFileUri();
// Obtain the audio duration.
Long audioTime = getAudioFileTimeFromUri(uri);
// Check whether the duration is longer than 60s.
if (audioTime < 60000) {
    // uri indicates audio resources read from the local storage or recorder. Only local audio files with a duration not longer than 1 minute are supported.
    this.taskId = this.engine.shortRecognize(uri, this.setting);
    Log.i(TAG, "Short audio transcription.");
} else {
    // longRecognize is an API used to convert audio files with a duration from 1 minute to 5 hours.
    this.taskId = this.engine.longRecognize(uri, this.setting);
    Log.i(TAG, "Long audio transcription.");
}

private Long getAudioFileTimeFromUri(Uri uri) {
    Long time = null;
    Cursor cursor = this.getContentResolver()
            .query(uri, null, null, null, null);
    if (cursor != null) {

        cursor.moveToFirst();
        time = cursor.getLong(cursor.getColumnIndexOrThrow(MediaStore.Video.Media.DURATION));
    } else {
        MediaPlayer mediaPlayer = new MediaPlayer();
        try {
            mediaPlayer.setDataSource(String.valueOf(uri));
            mediaPlayer.prepare();
        } catch (IOException e) {
            Log.e(TAG, "Failed to read the file time.");
        }
        time = Long.valueOf(mediaPlayer.getDuration());
    }
    return time;
}

To learn more, visit the following links:
Documentation on the HUAWEI Developers website
https://developer.huawei.com/consumer/en/hms/huawei-MapKit

HUAWEI Developers official website

Development Guide

Redditto join developer discussions

GitHub or Gitee to download the demo and sample code

Stack Overflow to solve integration problems

Building High-Precision Location Services with Location Kit

Vivi-clevercoder — Fri, 30 Jul 2021 06:43:44 +0000

HUAWEI Location Kit provides you with the tools to build ultra-precise location services into your apps, by utilizing GNSS, Wi-Fi, base stations, and a range of cutting-edge hybrid positioning technologies. Location Kit-supported solutions give your apps a leg up in a ruthlessly competitive marketplace, making it easier than ever for you to serve a vast, global user base.

Location Kit currently offers three main functions: fused location, geofence, and activity identification. When used in conjunction with the Map SDK, which is supported in 200+ countries and regions and 100+ languages, you'll be able to bolster your apps with premium mapping services that enjoy a truly global reach.

Fused location provides easy-to-use APIs that are capable of obtaining the user's location with meticulous accuracy, and doing so while consuming a minimal amount of power. HW NLP, Huawei's exclusive network location service, makes use of crowdsourced data to achieve heightened accuracy. Such high-precision, cost-effective positioning has enormous implications for a broad array of mobile services, including ride hailing navigation, food delivery, travel, and lifestyle services, providing customers and service providers alike with the high-value, real time information that they need.

To avoid boring you with the technical details, we've provided some specific examples of how positioning systems, geofence, activity identification, map display and route planning services can be applied in the real world.

For instance, you can use Location kit to obtain the user's current location and create a 500-meter geofence radius around it, which can be used to determine the user's activity status when the geofence is triggered, then automatically plan a route based on this activity status (for example, plan a walking route when the activity is identified as walking), and have it shown on the map.

*This article addresses the following functions: *

Fused location: Incorporates GNSS, Wi-Fi, and base station data via easy-to-use APIs, making it easy for your app to obtain device location information.
Activity identification: Identifies the user's motion status, using the acceleration sensor, network information, and magnetometer, so that you can tailor your app to account for the user's behavior.
Geofence: Allows you to set virtual geographic boundaries via APIs, to send out timely notifications when users enter, exit, or remain with the boundaries.
Map display: Includes the map display, interactive features, map drawing, custom map styles, and a range of other features.
Route planning: Provides HTTP/HTTPS APIs for you to initiate requests using HTTP/HTTPS, and obtain the returned data in JSON format.

Usage scenarios:

Using high-precision positioning technology to obtain real time location and tracking data for delivery or logistics personnel, for optimally efficient services. In the event of accidents or emergencies, the location of personnel could also be obtained with ease, to ensure their quick rescue.
Creating a geofence in the system, which can be used to monitor an important or dangerous area at all times. If someone enters such an area without authorization, the system could send out a proactive alert. This solution can also be linked with onsite video surveillance equipment. When an alert is triggered, the video surveillance camera could pop up to provide continual monitoring, free of any blind spots.
Tracking patients with special needs in hospitals and elderly residents in nursing homes, in order to provide them with the best possible care. Positioning services could be linked with wearable devices, for attentive 24/7 care in real time.
Using the map to directly find destinations, and perform automatic route planning.

I. Advantages of Location Kit and Map Kit

Low-power consumption (Location Kit): Implements geofence using the chipset, for optimized power efficiency
High precision (Location Kit): Optimizes positioning accuracy in urban canyons, correctly identifying the roadside of the user. Sub-meter positioning accuracy in open areas, with RTK (Real-time kinematic) technology support. Personal information, activity identification, and other data are not uploaded to the server while location services are performed. As the data processor, Location Kit only uses data, and does not store it.
Personalized map displays (Map Kit): Offers enriching map elements and a wide range of interactive methods for building your map.
Broad-ranging place searches (Map Kit): Covers 130+ million POIs and 150+ million addresses, and supports place input prompts.
Global coverage: Supports 200+ countries/regions, and 40+ languages.

For more information and development guides, please visit:

II. Demo App Introduction

In order to illustrate how to integrate Location Kit and Map Kit both easily and efficiently, we've provided a case study here, which shows the simplest coding method for running the demo.
This app is used to create a geofence on the map based on the location when the user opens the app. The user can drag on the red mark to set a destination. After being confirmed, when the user triggers the geofence condition, the app will automatically detect their activity status and plan a route for the user, such as planning a walking route if the activity status is walking, or cycling route if the activity status is cycling. You can also implement real-time voice navigation for the planned route.

III. Development Practice

You need to set the priority (which is 100 by default) before requesting locations. To request the precise GPS location, set the priority to 100. To request the network location, set the priority to 102 or 104. If you only need to passively receive locations, set the priority to 105.

Parameters related to activity identification include VEHICLE (100), BIKE (101), FOOT (102), and STILL (103).
Geofence-related parameters include ENTER_GEOFENCE_CONVERSION (1), EXIT_GEOFENCE_CONVERSION (2), and DWELL_GEOFENCE_CONVERSION (4).
The following describes how to run the demo using source code, helping you understand the implementation details.

Preparations

Preparing Tools

1) Huawei phones (It is recommended that multiple devices be tested)
2) Android Studio

Registering as a Developer

1) Register as a Huawei developer.
2) Create an app in AppGallery Connect.
Create an app in AppGallery Connect by referring to Location Kit development preparations or Map Kit development preparations.

 Enable Location Kit and Map Kit for the app on the Manage APIs page.
 Add the SHA-256 certificate fingerprint.
 Download the agconnect-services.json file and add it to the app directory of the project.
3) Create an Android demo project.
4) Learn about the function restrictions.
To use the route planning function of Map Kit, refer to Supported Countries/Regions (Route Planning).
To use other services of Map Kit, refer to Supported Countries/Regions

Running the Demo App

1) Install the app on the test device after debugging the project in Android Studio successfully
2) Replace the project package name and JSON file with those of your own.
3) Tap related button in the demo app to create a geofence which has a radius of 200 and is centered on the current location automatically pinpointed by the demo app.
4) Drag the mark point on the map to select a destination.
5) View the route that is automatically planned based on the current activity status when the geofence is triggered.
The following figure shows the demo effect:

Key Steps

Add the Huawei Maven repository to the project-level build.gradle file. Add the following Maven repository address to the project-level build.gradle file of your Android Studio project:

buildscript {
    repositories {
        maven { url 'http://developer.huawei.com/repo/'}
    }
dependencies {
        ...
        // Add the AppGallery Connect plugin configuration.
        classpath 'com.huawei.agconnect:agcp:1.4.2.300'
    }
}allprojects {
    repositories {
        maven { url 'http://developer.huawei.com/repo/'}
    }
}

Add dependencies on the SDKs in the app-level build.gradle file.

dependencies {
   implementation 'com.huawei.hms:location:5.1.0.300'
   implementation 'com.huawei.hms:maps:5.2.0.302' }

Add the following configuration to the next line under apply plugin: 'com.android.application' in the file header: apply plugin: 'com.huawei.agconnect'

Note:
• You must configure apply plugin: 'com.huawei.agconnect' under apply plugin: 'com.android.application'.
• The minimum Android API level (minSdkVersion) required for the HMS Core Map SDK is 19.

Declare system permissions in the AndroidManifest.xml file. Location Kit uses GNSS, Wi-Fi, and base station data for fused location, enabling your app to quickly and accurately obtain users' location information. Therefore, Location Kit requires permissions to access Internet, obtain the fine location, and obtain the coarse location. If your app needs to continuously obtain the location information when it runs in the background, you also need to declare the ACCESS_BACKGROUND_LOCATION permission in the AndroidManifest.xml file:

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
<uses-permission android:name="android.permission.WAKE_LOCK" />
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />
<uses-permission android:name="com.huawei.hms.permission.ACTIVITY_RECOGNITION" />
<uses-permission android:name="android.permission.ACTIVITY_RECOGNITION" />

Note: Because the ACCESS_FINE_LOCATION, WRITE_EXTERNAL_STORAGE, READ_EXTERNAL_STORAGE, and ACTIVITY_RECOGNITION permissions are dangerous system permissions, you need to dynamically apply for these permissions. If you do not have the permissions, Location Kit will reject to provide services for your app.

Key Code

I. Map Display
Currently, the Map SDK supports two map containers: SupportMapFragment and MapView. This document uses the SupportMapFragment container.

Add a Fragment object in the layout file (for example: activity_main.xml), and set map attributes in the file.

<fragment
    android:id="@+id/mapfragment_routeplanningdemo"
    android:name="com.huawei.hms.maps.SupportMapFragment"
    android:layout_width="match_parent"
    android:layout_height="match_parent" />

To use a map in your app, implement the OnMapReadyCallback API.

RoutePlanningActivity extends AppCompatActivity implements OnMapReadyCallback

Load SupportMapView in the onCreate method, call getMapAsync to register the callback.

Fragment fragment = getSupportFragmentManager().findFragmentById(R.id.mapfragment_routeplanningdemo);
if (fragment instanceof SupportMapFragment) {
    SupportMapFragment mSupportMapFragment = (SupportMapFragment) fragment;
    mSupportMapFragment.getMapAsync(this);
}

Call the onMapReady callback to obtain the HuaweiMap object.

@Override
public void onMapReady(HuaweiMap huaweiMap) {

    hMap = huaweiMap;
    hMap.setMyLocationEnabled(true);
    hMap.getUiSettings().setMyLocationButtonEnabled(true);
}

II. Function Implementation

Check the permissions.

if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.P) {
    if (ActivityCompat.checkSelfPermission(context,
            "com.huawei.hms.permission.ACTIVITY_RECOGNITION") != PackageManager.PERMISSION_GRANTED) {
        String[] permissions = {"com.huawei.hms.permission.ACTIVITY_RECOGNITION"};
        ActivityCompat.requestPermissions((Activity) context, permissions, 1);
        Log.i(TAG, "requestActivityTransitionButtonHandler: apply permission");
    }
} else {
    if (ActivityCompat.checkSelfPermission(context,
            "android.permission.ACTIVITY_RECOGNITION") != PackageManager.PERMISSION_GRANTED) {
        String[] permissions = {"android.permission.ACTIVITY_RECOGNITION"};
        ActivityCompat.requestPermissions((Activity) context, permissions, 2);
        Log.i(TAG, "requestActivityTransitionButtonHandler: apply permission");
    }
}

Check whether the location permissions have been granted. If no, the location cannot be obtained.

settingsClient.checkLocationSettings(locationSettingsRequest)
        .addOnSuccessListener(locationSettingsResponse -> {
                       fusedLocationProviderClient
                    .requestLocationUpdates(mLocationRequest, mLocationCallback, Looper.getMainLooper())
                    .addOnSuccessListener(aVoid -> {
                        //Processing when the API call is successful.
                    });
        })
        .addOnFailureListener(e -> {});
if (null == mLocationCallbacks) {
    mLocationCallbacks = new LocationCallback() {
        @Override
        public void onLocationResult(LocationResult locationResult) {
            if (locationResult != null) {
                List<HWLocation> locations = locationResult.getHWLocationList();
                if (!locations.isEmpty()) {
                    for (HWLocation location : locations) {
                        hMap.moveCamera(CameraUpdateFactory.newLatLngZoom(new LatLng(location.getLatitude(), location.getLongitude()), 14));
                        latLngOrigin = new LatLng(location.getLatitude(), location.getLongitude());
                        if (null != mMarkerOrigin) {
                            mMarkerOrigin.remove();
                        }
                        MarkerOptions options = new MarkerOptions()
                                .position(latLngOrigin)
                                .title("Hello Huawei Map")
                                .snippet("This is a snippet!");
                        mMarkerOrigin = hMap.addMarker(options);
                        removeLocationUpdatesWith();
                    }
                }
            }
        }

        @Override
        public void onLocationAvailability(LocationAvailability locationAvailability) {
            if (locationAvailability != null) {
                boolean flag = locationAvailability.isLocationAvailable();
                Log.i(TAG, "onLocationAvailability isLocationAvailable:" + flag);
            }
        }
    };
}

III. Geofence and Ground Overlay Creation

Create a geofence based on the current location and add a round ground overlay on the map.

GeofenceRequest.Builder geofenceRequest = new 
GeofenceRequest.Builder geofenceRequest = new GeofenceRequest.Builder();
geofenceRequest.createGeofenceList(GeoFenceData.returnList());
geofenceRequest.setInitConversions(7); 
try {
    geofenceService.createGeofenceList(geofenceRequest.build(), pendingIntent)
            .addOnCompleteListener(new OnCompleteListener<Void>() {
                @Override
                public void onComplete(Task<Void> task) {
                    if (task.isSuccessful()) {
                        Log.i(TAG, "add geofence success!");
                        if (null == hMap) {
                            return; }
                        if (null != mCircle) {
                            mCircle.remove();
                            mCircle = null;
                        }
                        mCircle = hMap.addCircle(new CircleOptions()
                                .center(latLngOrigin)
                                .radius(500)
                                .strokeWidth(1)
                                .fillColor(Color.TRANSPARENT));
                    } else {Log.w(TAG, "add geofence failed : " + task.getException().getMessage());}
                }
            });
} catch (Exception e) {
    Log.i(TAG, "add geofence error:" + e.getMessage());
}

// Geofence service
<receiver
    android:name=".GeoFenceBroadcastReceiver"
    android:exported="true">
    <intent-filter>
        <action android:name=".GeoFenceBroadcastReceiver.ACTION_PROCESS_LOCATION" />
    </intent-filter>
</receiver>

if (intent != null) {
    final String action = intent.getAction();
    if (ACTION_PROCESS_LOCATION.equals(action)) {
        GeofenceData geofenceData = GeofenceData.getDataFromIntent(intent);
        if (geofenceData != null && isListenGeofence) {
            int conversion = geofenceData.getConversion();
            MainActivity.setGeofenceData(conversion);
        }
    }
}

Mark the selected point on the map to obtain the destination information, check the current activity status, and plan routes based on the detected activity status.

hMap.setOnMapClickListener(latLng -> {
    latLngDestination = new LatLng(latLng.latitude, latLng.longitude);
    if (null != mMarkerDestination) {
        mMarkerDestination.remove();
    }
    MarkerOptions options = new MarkerOptions()
            .position(latLngDestination)
            .title("Hello Huawei Map");
    mMarkerDestination = hMap.addMarker(options);
    if (identification.getText().equals("To exit the fence,Your activity is about to be detected.")) {
        requestActivityUpdates(5000);
    }

});
// Activity identification API
activityIdentificationService.createActivityIdentificationUpdates(detectionIntervalMillis, pendingIntent)
        .addOnSuccessListener(new OnSuccessListener<Void>() {
            @Override
            public void onSuccess(Void aVoid) {
                Log.i(TAG, "createActivityIdentificationUpdates onSuccess");
            }
        })
        .addOnFailureListener(new OnFailureListener() {
            @Override
            public void onFailure(Exception e) {
                Log.e(TAG, "createActivityIdentificationUpdates onFailure:" + e.getMessage());
            }
        });
// URL of the route planning API (cycling route is used as an example): https://mapapi.cloud.huawei.com/mapApi/v1/routeService/bicycling?key=API KEY
 NetworkRequestManager.getBicyclingRoutePlanningResult(latLngOrigin, latLngDestination,
        new NetworkRequestManager.OnNetworkListener() {
            @Override
            public void requestSuccess(String result) {
                generateRoute(result);
            }

            @Override
            public void requestFail(String errorMsg) {
                Message msg = Message.obtain();
                Bundle bundle = new Bundle();
                bundle.putString("errorMsg", errorMsg);
                msg.what = 1;
                msg.setData(bundle);
                mHandler.sendMessage(msg);
            }
        });

Note:

The route planning function provides a set of HTTPS-based APIs used to plan routes for walking, cycling, and driving and calculate route distances. The APIs return route data in JSON format and provide the route planning capabilities.

The route planning function can plan walking, cycling, and driving routes.
You can try to plan a route from one point to another point and then draw the route on the map, achieving the navigation effects.

Related Parameters

In indoor environments, the navigation satellite signals are usually weak. Therefore, HMS Core (APK) will use the network location mode, which is relatively slow compared with the GNSS location. It is recommended that the test be performed outdoors.
In Android 9.0 or later, you are advised to test the geofence outdoors. In versions earlier than Android 9.0, you can test the geofence indoors.
Map Kit is unavailable in the Chinese mainland. Therefore, the Android SDK, JavaScript API, Static Map API, and Directions API are unavailable in the Chinese mainland. For details, please refer to Supported Countries/Regions.
In the Map SDK for Android 5.0.0.300 and later versions, you must set the API key before initializing a map. Otherwise, no map data will be displayed.
Currently, the driving route planning is unavailable in some countries and regions outside China. For details about the supported countries and regions, please refer to the Huawei official website.
Before building the APK, configure the obfuscation configuration file to prevent the HMS Core SDK from being obfuscated.  Open the obfuscation configuration file proguard-rules.pro in the app's root directory of your project and add configurations to exclude the HMS Core SDK from obfuscation.  If you are using AndResGuard, add its trustlist to the obfuscation configuration file.

For details, please visit the [following link]:

(https://developer.huawei.com/consumer/en/doc/development/HMSCore-Guides/android-sdk-config-obfuscation-scripts-0000001061882229 )

To learn more, visit the following links:
Documentation on the HUAWEI Developers website
https://developer.huawei.com/consumer/en/hms/huawei-MapKit

HUAWEI Developers official website

Development Guide

Redditto join developer discussions

GitHub or Gitee to download the demo and sample code

Stack Overflow to solve integration problems

How a Programmer Developed a Live-Streaming App with Gesture-Controlled Virtual Backgrounds

Vivi-clevercoder — Tue, 27 Jul 2021 11:42:48 +0000

"What's it like to date a programmer?"

John is a Huawei programmer. His girlfriend Jenny, a teacher, has an interesting answer to that question: "Thanks to my programmer boyfriend, my course ranked among the most popular online courses at my school".

Let's go over how this came to be. Due to COVID-19, the school where Jenny taught went entirely online. Jenny, who was new to live streaming, wanted her students to experience the full immersion of traveling to Tokyo, New York, Paris, the Forbidden City, Catherine Palace, and the Louvre Museum, so that they could absorb all of the relevant geographic and historical knowledge related to those places. But how to do so?

Jenny was stuck on this issue, but John quickly came to her rescue.

After analyzing her requirements in detail, John developed a tailored online course app that brings its users an uncannily immersive experience. It enables users to change the background while live streaming. The video imagery within the app looks true-to-life, as each pixel is labeled, and the entire body image — down to a single strand of hair — is completely cut out.

How to Implement

Changing live-streaming backgrounds by gesture can be realized by using image segmentation and hand gesture recognition in HUAWEI ML Kit

The image segmentation service segments specific elements from static images or dynamic video streams, with 11 types of image elements supported: human bodies, sky scenes, plants, foods, cats and dogs, flowers, water, sand, buildings, mountains, and others.
The hand gesture recognition service offers two capabilities: hand keypoint detection and hand gesture recognition. Hand keypoint detection is capable of detecting 21 hand keypoints (including fingertips, knuckles, and wrists) and returning positions of the keypoints. The hand gesture recognition capability detects and returns the positions of all rectangular areas of the hand from images and videos, as well as the type and confidence of a gesture. This capability can recognize 14 different gestures, including the thumbs-up/down, OK sign, fist, finger heart, and number gestures from 1 to 9. Both capabilities support detection from static images and real-time video streams.
Development Process

1. Add the AppGallery Connect plugin and the Maven repository.

    buildscript {
    repositories {
        google()
        jcenter()
        maven {url 'https://developer.huawei.com/repo/'}
    }
    dependencies {
        ...
        classpath 'com.huawei.agconnect:agcp:1.4.1.300'
    }
}

allprojects {
    repositories {
        google()
        jcenter()
        maven {url 'https://developer.huawei.com/repo/'}
    }
}

2. Integrate required services in the full SDK mode.

    dependencies{
     // Import the basic SDK of image segmentation.
    implementation 'com.huawei.hms:ml-computer-vision-segmentation:2.0.4.300'
    // Import the multiclass segmentation model package.
    implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-multiclass-model:2.0.4.300'
    // Import the human body segmentation model package.
    implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-body-model:2.0.4.300'
    // Import the basic SDK of hand gesture recognition.
    implementation 'com.huawei.hms:ml-computer-vision-handkeypoint:2.0.4.300'
    // Import the model package of hand keypoint detection.
    implementation 'com.huawei.hms:ml-computer-vision-handkeypoint-model:2.0.4.300'
}

3. Add configurations in the file header.

Add apply plugin: 'com.huawei.agconnect' after apply plugin: 'com.android.application'.

4. Automatically update the machine learning model.
Add the following statements to the AndroidManifest.xml file:

<manifest
    ...
    <meta-data
        android:name="com.huawei.hms.ml.DEPENDENCY"
        android:value="imgseg,handkeypoint" />
    ...
</manifest>

5. Create an image segmentation analyzer.

MLImageSegmentationAnalyzer imageSegmentationAnalyzer = MLAnalyzerFactory.getInstance().getImageSegmentationAnalyzer();// Image segmentation analyzer.
MLHandKeypointAnalyzer handKeypointAnalyzer = MLHandKeypointAnalyzerFactory.getInstance().getHandKeypointAnalyzer();// Hand gesture recognition analyzer.

MLCompositeAnalyzer analyzer = new MLCompositeAnalyzer.Creator()
                                    .add(imageSegmentationAnalyzer)
                                   .add(handKeypointAnalyzer)
                                   .create();

6. Create a class for processing the recognition result.

    public class ImageSegmentAnalyzerTransactor implements MLAnalyzer.MLTransactor<MLImageSegmentation> {
    @Override
    public void transactResult(MLAnalyzer.Result<MLImageSegmentation> results) {
        SparseArray<MLImageSegmentation> items = results.getAnalyseList();
        // Process the recognition result as required. Note that only the detection results are processed.
        // Other detection-related APIs provided by ML Kit cannot be called.
    }
    @Override
    public void destroy() {
        // Callback method used to release resources when the detection ends.
    }
}

public class HandKeypointTransactor implements MLAnalyzer.MLTransactor<List<MLHandKeypoints>> {
    @Override
    public void transactResult(MLAnalyzer.Result<List<MLHandKeypoints>> results) {
        SparseArray<List<MLHandKeypoints>> analyseList = results.getAnalyseList();
        // Process the recognition result as required. Note that only the detection results are processed.
        // Other detection-related APIs provided by ML Kit cannot be called.
    }
    @Override
    public void destroy() {
        // Callback method used to release resources when the detection ends.
    }
}

7. Set the detection result processor to bind the analyzer to the result processor.

    imageSegmentationAnalyzer.setTransactor(new ImageSegmentAnalyzerTransactor());
handKeypointAnalyzer.setTransactor(new HandKeypointTransactor());

8. Create a LensEngine object.

    Context context = this.getApplicationContext();
LensEngine lensEngine = new LensEngine.Creator(context,analyzer)
    // Set the front or rear camera mode. LensEngine.BACK_LENS indicates the rear camera, and LensEngine.FRONT_LENS indicates the front camera.
    .setLensType(LensEngine.FRONT_LENS)
    .applyDisplayDimension(1280, 720)
    .applyFps(20.0f)
    .enableAutomaticFocus(true)
    .create();

9. Start the camera, read video streams, and start recognition.

    // Implement other logics of the SurfaceView control by yourself.
SurfaceView mSurfaceView = new SurfaceView(this);
try {
    lensEngine.run(mSurfaceView.getHolder());
} catch (IOException e) {
    // Exception handling logic.
}

10. Stop the analyzer and release the recognition resources when recognition ends.

    if (analyzer != null) {
    try {
        analyzer.stop();
    } catch (IOException e) {
        // Exception handling.
    }
}
if (lensEngine != null) {
    lensEngine.release();
}

For more information, please visit:

HUAWEI Developers official website

Development Guide

Redditto join developer discussions

GitHub or Gitee to download the demo and sample code

Stack Overflow to solve integration problems

Implementing Real-Time Transcription in an Easy Way

Vivi-clevercoder — Fri, 23 Jul 2021 12:36:54 +0000

Background

The real-time onscreen subtitle is a must-have function in an ordinary video app. However, developing such a function can prove costly for small- and medium-sized developers. And even when implemented, speech recognition is often prone to inaccuracy. Fortunately, there's a better way — HUAWEI ML Kit, which is remarkably easy to integrate, and makes real-time transcription an absolute breeze!

Introduction to ML Kit

ML Kit allows your app to leverage Huawei's longstanding machine learning prowess to apply cutting-edge artificial intelligence (AI) across a wide range of contexts. With Huawei's expertise built in, ML Kit is able to provide a broad array of easy-to-use machine learning capabilities, which serve as the building blocks for tomorrow's cutting-edge AI apps. ML Kit capabilities include those related to:

Ø Text (including text recognition, document recognition, and ID card recognition)

Ø Language/Voice (such as real-time/on-device translation, automatic speech recognition, and real-time transcription)

Ø Image (such as image classification, object detection and tracking, and landmark recognition)

Ø Face/Body (such as face detection, skeleton detection, liveness detection, and face verification)

Ø Natural language processing (text embedding)

Ø Custom model (including the on-device inference framework and model development tool)

Real-time transcription is required to implement the function mentioned above. Let's take a look at how this works in practice:

Now let's move on to how to integrate this service.

Integrating Real-Time Transcription

Steps

1.Registering as a Huawei developer on HUAWEI Developers

2.Creating an app

Create an app in AppGallery Connect. For details, see Getting Started with Android.

We've provided some screenshots for your reference:

Enabling ML Kit

Integrating the HMS Core SDK

Add the AppGallery Connect configuration file by completing the steps below:

n Download and copy the agconnect-service.json file to the app directory of your Android Studio project.

n Call setApiKey during app initialization.

To learn more, go to Adding the AppGallery Connect Configuration File.

Configuring the maven repository address

n Add build dependencies.

implementation 'com.huawei.hms:ml-computer-voice-realtimetranscription:2.2.0.300'

n Add the AppGallery Connect plugin configuration.

Method 1: Add the following information under the declaration in the file header:

apply plugin: 'com.huawei.agconnect'

Method 2: Add the plugin configuration in the plugins block.

plugins {
id 'com.android.application'
// Add the following configuration:
id 'com.huawei.agconnect'
}

Please refer to Integrating the Real-Time Transcription SDK to learn more.

Setting the cloud authentication information

When using on-cloud services of ML Kit, you can set the API key or access token (recommended) in either of the following ways:

Access token

You can use the following API to initialize the access token when the app is started. The access token does not need to be set again once initialized.

MLApplication.getInstance().setAccessToken("your access token");

API key

You can use the following API to initialize the API key when the app is started. The API key does not need to be set again once initialized.

MLApplication.getInstance().setApiKey("your ApiKey");

For details, see Notes on Using Cloud Authentication Information.

Code Development

l Create and configure a speech recognizer.

MLSpeechRealTimeTranscriptionConfig config = new MLSpeechRealTimeTranscriptionConfig.Factory()

// Set the language. Currently, this service supports Mandarin Chinese, English, and French.

.setLanguage(MLSpeechRealTimeTranscriptionConstants.LAN_ZH_CN)

// Punctuate the text recognized from the speech.

.enablePunctuation(true)

// Set the sentence offset.

.enableSentenceTimeOffset(true)

// Set the word offset.

.enableWordTimeOffset(true)

// Set the application scenario. MLSpeechRealTimeTranscriptionConstants.SCENES_SHOPPING indicates shopping, which is supported only for Chinese. Under this scenario, recognition for the name of Huawei products has been optimized.

.setScenes(MLSpeechRealTimeTranscriptionConstants.SCENES_SHOPPING)

.create();

MLSpeechRealTimeTranscription mSpeechRecognizer = MLSpeechRealTimeTranscription.getInstance();

l Create a speech recognition result listener callback.

// Use the callback to implement the MLSpeechRealTimeTranscriptionListener API and methods in the API.

protected class SpeechRecognitionListener implements

MLSpeechRealTimeTranscriptionListener{
u/Override
public void onStartListening() {
// The recorder starts to receive speech.
}
u/Override
public void onStartingOfSpeech() {
// The user starts to speak, that is, the speech recognizer detects that the user starts to speak.
}
u/Override
public void onVoiceDataReceived(byte[] data, float energy, Bundle bundle) {
// Return the original PCM stream and audio power to the user. This API is not running in the main thread, and the return result is processed in a sub-thread.
}
u/Override
public void onRecognizingResults(Bundle partialResults) {
// Receive the recognized text from MLSpeechRealTimeTranscription.
}
u/Override
public void onError(int error, String errorMessage) {
// Called when an error occurs in recognition.
}
u/Override
public void onState(int state,Bundle params) {
// Notify the app of the status change.
}
}

The recognition result can be obtained from the listener callbacks, including onRecognizingResults. Design the UI content according to the obtained results. For example, display the text transcribed from the input speech.

mSpeechRecognizer.setRealTimeTranscriptionListener(new SpeechRecognitionListener());

l Call startRecognizing to start speech recognition.

mSpeechRecognizer.startRecognizing(config);

l Release resources after recognition is complete.

if (mSpeechRecognizer!= null) {
mSpeechRecognizer.destroy();
}
l (Optional) Obtain the list of supported languages.
MLSpeechRealTimeTranscription.getInstance()
.getLanguages(new MLSpeechRealTimeTranscription.LanguageCallback() {
u/Override
public void onResult(List<String> result) {
 Log.i(TAG, "support languages==" + result.toString());
}
u/Override
public void onError(int errorCode, String errorMsg) {
Log.e(TAG, "errorCode:" + errorCode + "errorMsg:" + errorMsg);
}
});

We've finished integration here, so let's test it out on a simple screen.

Tap START RECORDING. The text recognized from the input speech will display in the lower portion of the screen.

We've now built a simple audio transcription function.

Eager to build a fancier UI, with stunning animations, and other effects? By all means, take your shot!

For more information, please visit:
Real-Time Transcription

Sample Code for ML Kit

Documentation on the HUAWEI Developers website

HUAWEI Developers official website

Development Guide

Redditto join developer discussions

GitHub or Gitee to download the demo and sample code

Stack Overflow to solve integration problems

Monitor Real-time Health during Workouts with Body and Face Tracking

Vivi-clevercoder — Wed, 21 Jul 2021 12:06:54 +0000

Still wearing a smart watch to monitor health indicators during workouts? Curious at what makes AR apps so advanced? Still think that AR is only used in movies? With HUAWEI AR Engine, you can integrate AR capabilities into your own apps in just a few easy steps. If this has piqued your interest, read on to learn more!

What is AR Engine?

HUAWEI AR Engine is an engine designed for building augmented reality (AR) apps to be run on Android smartphones. It is based on the HiSilicon chipset, and integrates AR core algorithms to provide a range of basic AR capabilities, such as motion tracking, environment tracking, body tracking, and face tracking, enabling your app to bridge real and virtual worlds, by offering a brand new visually interactive user experience.

AR Engine provides for high-level health status detection, via facial information, and encompasses a range of different data indicators including heart rate, respiratory rate, facial health status, and heart rate waveform signals.

With the human body and face tracking capability, one of the engine's three major capabilities (the other two being motion tracking and environment tracking), HUAWEI AR Engine is able to monitor and display the user's real time health status during workouts.

Application scenarios:
Gym: Checking real-time body indicators during workouts.
Medical treatment: Monitoring patients' physical status in real time.
Caregiving: Monitoring health indicators of the elderly in real time.

Next, let's take a look at how to implement these powerful functions.

Advantages of AR monitoring and requirements for hardware:

Detects facial health information and calculates key health information, such as real time heart rate.
The human body and face tracking capabilities also equip your device to better understanding users. By locating hand locations and recognizing specific gestures, AR Engine can assist in placing a virtual object in the real world, or overlaying special effects on a hand. With the depth sensing components, the hand skeleton tracking capability is capable of tracking 21 hand skeleton points, to implement precise interactive controls and special effect overlays. With regard to body tracking, the capability can track 23 body skeleton points to detect human posture in real time, providing a strong foundation for motion sensing and fitness & health apps..
For details about supported models, please refer to the software and hardware dependencies on the HUAWEI Developers website.

Demo Introduction

A demo is offered here for you to learn how to integrate AR Engine with simplest code in the fastest way.
 Enable health check by using ENABLE_HEALTH_DEVICE.
 FaceHealthCheckStateEvent functions as a parameter of ServiceListener.handleEvent(EventObject eventObject) that passes health check status information to the app.
 The health check HealthParameter includes the heart rate, respiratory rate, facial attributes (like age and gender), and hear rate waveform signal.

Development Practice The following describes how to run the demo using source code, enabling you to understand the implementation details.

Preparations

Get the tools prepared.
a) A Huawei P30 running Android 11.
b) Development tool: Android Studio; development language: Java.
Register as a Huawei developer.
a) Register as a Huawei developer.
b) Create an app.
Follow instructions in the AR Engine Development Guide to add an app in AppGallery Connect.
c) Build the demo app.
 Import the source code to Android Studio.
 Download the agconnect-services.json file of the created app from AppGallery Connect, and add it to the app directory in the sample project.
Run the demo app.
a) Install the demo app on the test device.
b) After the app is started, access facial recognition. During recognition, the progress will be displayed on the screen in real time.
c) Your heart rate, respiratory rate, and real-time heart rate waveform will be displayed after successful recognition.
The results are as shown in the following figure.

Key Steps

Add the Huawei Maven repository to the project-level build.gradle file. Add the following Maven repository address to the project-level build.gradle file of your Android Studio project:

buildscript {
    repositories {
        maven { url 'http://developer.huawei.com/repo/'}
    }
dependencies {
        ...
        // Add the AppGallery Connect plugin configuration.
        classpath 'com.huawei.agconnect:agcp:1.4.2.300'
    }
}allprojects {
    repositories {
        maven { url 'http://developer.huawei.com/repo/'}
    }
}

2. Add dependencies on the SDKs in the app-level build.gradle file.

dependencies {
   implementation 'com.huawei.hms:arenginesdk: 2.15.0.1'
}

3. Declare system permissions in the AndroidManifest.xml file.

The required permissions include the camera permission and network permission.

Camera permission: android.permission.CAMERA, which is indispensable for using the AR Engine Server.
Network permission: android.permission.INTERNET, which is used to analyze API calling status and guide continuous capability optimization.

Note: The AR Engine SDK processes data only on the device side, and does not report data to the server.

Key Code Description

Check the AR Engine availability.

Check whether AR Engine has been installed on the current device. If yes, the app can run properly. If not, the app automatically redirects the user to AppGallery to install AR Engine. Sample code:

boolean isInstallArEngineApk = AREnginesApk.isAREngineApkReady(this);
        if (!isInstallArEngineApk) {
            // ConnectAppMarketActivity.class is the activity for redirecting to AppGallery.
            startActivity(new Intent(this, com.huawei.arengine.demos.common.ConnectAppMarketActivity.class));
            isRemindInstall = true;
        }

Create a ARFaceTrackingConfig scene.

// Create an ARSession.
mArSession = new ARSession(this);
// Select a specific Config to initialize the ARSession based on the application scenario.
ARWorldTrackingConfig config = new ARWorldTrackingConfig(mArSession);

Add the listener for passing information such as the health check status and progress.

mArSession.addServiceListener(new FaceHealthServiceListener() {
    @Override
    public void handleEvent(EventObject eventObject) {
        // FaceHealthCheckStateEvent passes the health check status information to the app.
        if (!(eventObject instanceof FaceHealthCheckStateEvent)) {
            return;
        }
        // Obtain the health check status.
        final FaceHealthCheckState faceHealthCheckState =
                ((FaceHealthCheckStateEvent) eventObject).getFaceHealthCheckState();
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                mHealthCheckStatusTextView.setText(faceHealthCheckState.toString());
            }
        });
    }
    //handleProcessProgressEvent Health check progress
    @Override
    public void handleProcessProgressEvent(final int progress) {
        mHealthRenderManager.setHealthCheckProgress(progress);
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                setProgressTips(progress);
            }
        });
    }
});

For more information, please visit:
Documentation on the HUAWEI Developers website

HUAWEI Developers official website

Development Guide

Redditto join developer discussions

GitHub or Gitee to download the demo and sample code

Stack Overflow to solve integration problems

How to Build a 3D Product Model Within Just 5 Minutes

Vivi-clevercoder — Mon, 19 Jul 2021 09:47:20 +0000

Displaying products with 3D models is something too great to ignore for an e-commerce app. Using those fancy gadgets, such an app can leave users with the first impression upon products in a fresh way!

The 3D model plays an important role in boosting user conversion. It allows users to carefully view a product from every angle, before they make a purchase. Together with the AR technology, which gives users an insight into how the product will look in reality, the 3D model brings a fresher online shopping experience that can rival offline shopping.

Despite its advantages, the 3D model has yet to be widely adopted. The underlying reason for this is that applying current 3D modeling technology is expensive:
 Technical requirements: Learning how to build a 3D model is time-consuming.
 Time: It takes at least several hours to build a low polygon model for a simple object, and even longer for a high polygon one.
 Spending: The average cost of building a simple model can be more than one hundred dollars, and even higher for building a complex one.

Luckily, 3D object reconstruction, a capability in 3D Modeling Kit newly launched in HMS Core, makes 3D model building straightforward. This capability automatically generates a 3D model with a texture for an object, via images shot from different angles with a common RGB-Cam. It gives an app the ability to build and preview 3D models. For instance, when an e-commerce app has integrated 3D object reconstruction, it can generate and display 3D models of shoes. Users can then freely zoom in and out on the models for a more immersive shopping experience.

Actual Effect

Technical Solutions

3D object reconstruction is implemented on both the device and cloud. RGB images of an object are collected on the device and then uploaded to the cloud. Key technologies involved in the on-cloud modeling process include object detection and segmentation, feature detection and matching, sparse/dense point cloud computing, and texture reconstruction. Finally, the cloud outputs an OBJ file (a commonly used 3D model file format) of the generated 3D model with 40,000 to 200,000 patches.

Preparations

1. Configuring a Dependency on the 3D Modeling SDK

Open the app-level build.gradle file and add a dependency on the 3D Modeling SDK in the dependencies block.

// Build a dependency on the 3D Modeling SDK.
implementation 'com.huawei.hms:modeling3d-object-reconstruct:1.0.0.300'

2. Configuring AndroidManifest.xml

Open the AndroidManifest.xml file in the main folder. Add the following information before to apply for the storage read and write permissions and camera permission.

<!-- Permission to read data from and write data into storage. -->
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<!-- Permission to use the camera. -->
<uses-permission android:name="android.permission.CAMERA" />

Development Procedure
1. Configuring the Storage Permission Application
In the onCreate() method of MainActivity, check whether the storage read and write permissions have been granted; if not, apply for them by using requestPermissions.

if (EasyPermissions.hasPermissions(MainActivity.this, PERMISSIONS)) {
    Log.i(TAG, "Permissions OK");
} else {
    EasyPermissions.requestPermissions(MainActivity.this, "To use this app, you need to enable the permission.",
            RC_CAMERA_AND_EXTERNAL_STORAGE, PERMISSIONS);
}

Check the application result. If the permissions are not granted, prompt the user to grant them.

@Override
public void onPermissionsGranted(int requestCode, @NonNull List<String> perms) {
    Log.i(TAG, "permissions = " + perms);
    if (requestCode == RC_CAMERA_AND_EXTERNAL_STORAGE &&              PERMISSIONS.length == perms.size()) {
        initView();
        initListener();
    }
}

@Override
public void onPermissionsDenied(int requestCode, @NonNull List<String> perms) {
    if (EasyPermissions.somePermissionPermanentlyDenied(this, perms)) {
        new AppSettingsDialog.Builder(this)
                .setRequestCode(RC_CAMERA_AND_EXTERNAL_STORAGE)
                .setRationale("To use this app, you need to enable the permission.")
                .setTitle("Insufficient permissions")
                .build()
                .show();
    }
}

2. Creating a 3D Object Reconstruction Configurator

// Set the PICTURE mode.
Modeling3dReconstructSetting setting = new Modeling3dReconstructSetting.Factory()
        .setReconstructMode(Modeling3dReconstructConstants.ReconstructMode.PICTURE)
        .create();

3. Creating a 3D Object Reconstruction Engine and Initializing the Task
Call getInstance() of Modeling3dReconstructEngine and pass the current context to create an instance of the 3D object reconstruction engine.

// Create an engine.
modeling3dReconstructEngine = Modeling3dReconstructEngine.getInstance(mContext);

Use the engine to initialize the task.

// Initialize the 3D object reconstruction task.
modeling3dReconstructInitResult = modeling3dReconstructEngine.initTask(setting);
// Obtain the task ID.
String taskId = modeling3dReconstructInitResult.getTaskId();

4. Creating a Listener Callback to Process the Image Upload Result

Create a listener callback that allows you to configure the operations triggered upon upload success and failure.

// Create an upload listener callback.
private final Modeling3dReconstructUploadListener uploadListener = new Modeling3dReconstructUploadListener() {
    @Override
    public void onUploadProgress(String taskId, double progress, Object ext) {
        // Upload progress.
    }

    @Override
    public void onResult(String taskId, Modeling3dReconstructUploadResult result, Object ext) {
        if (result.isComplete()) {
            isUpload = true;
            ScanActivity.this.runOnUiThread(new Runnable() {
                @Override
                public void run() {
                    progressCustomDialog.dismiss();
                    Toast.makeText(ScanActivity.this, getString(R.string.upload_text_success), Toast.LENGTH_SHORT).show();
                }
            });
            TaskInfoAppDbUtils.updateTaskIdAndStatusByPath(new Constants(ScanActivity.this).getCaptureImageFile() + manager.getSurfaceViewCallback().getCreateTime(), taskId, 1);
        }
    }

    @Override
    public void onError(String taskId, int errorCode, String message) {
        isUpload = false;
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                progressCustomDialog.dismiss();
                Toast.makeText(ScanActivity.this, "Upload failed." + message, Toast.LENGTH_SHORT).show();
                LogUtil.e("taskid" + taskId + "errorCode: " + errorCode + " errorMessage: " + message);
            }
        });

    }
};

5. Passing the Upload Listener Callback to the Engine to Upload Images

Pass the upload listener callback to the engine. Call uploadFile(),
pass the task ID obtained in step 3 and the path of the images to be uploaded. Then, upload the images to the cloud server.

// Pass the listener callback to the engine.
modeling3dReconstructEngine.setReconstructUploadListener(uploadListener);
// Start uploading.
modeling3dReconstructEngine.uploadFile(taskId, filePath);

6. Querying the Task Status

Call getInstance of Modeling3dReconstructTaskUtils to create a task processing instance. Pass the current context.

// Create a task processing instance.
modeling3dReconstructTaskUtils = Modeling3dReconstructTaskUtils.getInstance(Modeling3dDemo.getApp());

Call queryTask of the task processing instance to query the status of the 3D object reconstruction task.

// Query the task status, which can be: 0 (images to be uploaded); 1: (image upload completed);
2: (model being generated);
3( model generation completed);
4: (model generation failed).
Modeling3dReconstructQueryResult queryResult = modeling3dReconstructTaskUtils.queryTask(task.getTaskId());

7. Creating a Listener Callback to Process the Model File Download Result

Create a listener callback that allows you to configure the operations triggered upon download success and failure.

// Create a download listener callback.
private Modeling3dReconstructDownloadListener modeling3dReconstructDownloadListener = new Modeling3dReconstructDownloadListener() {
    @Override
    public void onDownloadProgress(String taskId, double progress, Object ext) {
        ((Activity) mContext).runOnUiThread(new Runnable() {
            @Override
            public void run() {
                dialog.show();
            }
        });
    }

    @Override
    public void onResult(String taskId, Modeling3dReconstructDownloadResult result, Object ext) {
        ((Activity) mContext).runOnUiThread(new Runnable() {
            @Override
            public void run() {
                Toast.makeText(getContext(), "Download complete", Toast.LENGTH_SHORT).show();
                TaskInfoAppDbUtils.updateDownloadByTaskId(taskId, 1);
                dialog.dismiss();
            }
        });
    }

    @Override
    public void onError(String taskId, int errorCode, String message) {
        LogUtil.e(taskId + " <---> " + errorCode + message);
        ((Activity) mContext).runOnUiThread(new Runnable() {
            @Override
            public void run() {
                Toast.makeText(getContext(), "Download failed." + message, Toast.LENGTH_SHORT).show();
                dialog.dismiss();
            }
        });
    }
};

8. Passing the Download Listener Callback to the Engine to Download the File of the Generated Model

Pass the download listener callback to the engine. Call downloadModel, pass the task ID obtained in step 3 and the path for saving the model file to download it.

// Pass the download listener callback to the engine.
modeling3dReconstructEngine.setReconstructDownloadListener(modeling3dReconstructDownloadListener);
// Download the model file.
modeling3dReconstructEngine.downloadModel(appDb.getTaskId(), appDb.getFileSavePath());

More Information

The object should have rich texture, be medium-sized, and a rigid body. The object should not be reflective, transparent, or semi-transparent. The object types include goods (like plush toys, bags, and shoes), furniture (like sofas), and cultural relics (such as bronzes, stone artifacts, and wooden artifacts).
The object dimension should be within the range from 15 x 15 x 15 cm to 150 x 150 x 150 cm. (A larger dimension requires a longer time for modeling.)
3D object reconstruction does not support modeling for the human body and face.
Ensure the following requirements are met during image collection: Put a single object on a stable plane in pure color. The environment shall not be dark or dazzling. Keep all images in focus, free from blur caused by motion or shaking. Ensure images are taken from various angles including the bottom, flat, and top (it is advised that you upload more than 50 images for an object). Move the camera as slowly as possible. Do not change the angle during shooting. Lastly, ensure the object-to-image ratio is as big as possible, and all parts of the object are present.
These are all about the sample code of 3D object reconstruction. Try to integrate it into your app and build your own 3D models!

To learn more, please visit:

HUAWEI Developers official website

Development Guide

Redditto join developer discussions

GitHub or Gitee to download the demo and sample code

Stack Overflow to solve integration problems

Communicating Between JavaScript and Java Through the Cordova Plugins in HMS Core Kits

Vivi-clevercoder — Thu, 15 Jul 2021 08:13:56 +0000

1. Background

Cordova is an open-source cross-platform development framework that allows you to use HTML and JavaScript to develop apps across multiple platforms, such as Android and iOS. So how exactly does Cordova enable apps to run on different platforms and implement the functions? The abundant plugins in Cordova are the main reason, and free you to focus solely on app functions, without having to interact with the APIs at the OS level.

HMS Core provides a set of Cordova-related plugins, which enable you to integrate kits with greater ease and efficiency.

2. Introduction

Here, I'll use the Cordova plugin in HUAWEI Push Kit as an example to demonstrate how to call Java APIs in JavaScript through JavaScript-Java messaging.
The following implementation principles can be applied to all other kits, except for Map Kit and Ads Kit (which will be detailed later), and help you master troubleshooting solutions.

3. Basic Structure of Cordova

When you call loadUrl in MainActivity, CordovaWebView will be initialized and Cordova starts up. In this case, CordovaWebView will create PluginManager, NativeToJsMessageQueue, as well as ExposedJsApi of JavascriptInterface. ExposedJsApi and NativeToJsMessageQueue will play a role in the subsequent communication.

During the plugin loading, all plugins in the configuration file will be read when the PluginManager object is created, and plugin mappings will be created. When the plugin is called for the first time, instantiation is conducted and related functions are executed.

A message can be returned from Java to JavaScript in synchronous or asynchronous mode. In Cordova, set async in the method to distinguish the two modes.

In synchronous mode, Cordova obtains data from the header of the NativeToJsMessageQueue queue, finds the message request based on callbackID, and returns the data to the success method of the request.

In asynchronous mode, Cordova calls the loop method to continuously obtain data from the NativeToJsMessageQueue queue, finds the message request, and returns the data to the success method of the request.
In the Cordova plugin of Push Kit, the synchronization mode is used.

4. Plugin Call

You may still be unclear on how the process works, based on the description above, so I've provided the following procedure:

1. Install the plugin.

Run the cordova plugin add @hmscore/cordova-plugin-hms-push command to install the latest plugin. After the command is executed, the plugin information is added to the plugins directory.

The plugin.xml file records all information to be used, such as JavaScript and Android classes. During the plugin initialization, the classes will be loaded to Cordova. If a method or API is not configured in the file, it is unable to be used.

2. Create a message mapping.

The plugin provides the methods for creating mappings for the following messages:

1) HmsMessaging

In the HmsPush.js file, call the runHmsMessaging API in asynchronous mode to transfer the message to the Android platform. The Android platform returns the result through Promise.
The message will be transferred to the HmsPushMessaging class. The execute method in HmsPushMessaging can transfer the message to a method for processing based on the action type in the message.

public void execute(String action, final JSONArray args, final CallbackContext callbackContext)
        throws JSONException {
    hmsLogger.startMethodExecutionTimer(action);
    switch (action) {
        case "isAutoInitEnabled":
            isAutoInitEnabled(callbackContext);
            break;
        case "setAutoInitEnabled":
            setAutoInitEnabled(args.getBoolean(1), callbackContext);
            break;
        case "turnOffPush":
            turnOffPush(callbackContext);
            break;
        case "turnOnPush":
            turnOnPush(callbackContext);
            break;
        case "subscribe":
            subscribe(args.getString(1), callbackContext);
            break;

The processing method returns the result to JavaScript. The result will be written to the nativeToJsMessageQueue queue.
callBack.sendPluginResult(new PluginResult(PluginResult.Status.OK,autoInit));

2) HmsInstanceId

In the HmsPush.js file, call the runHmsInstance API in asynchronous mode to transfer the message to the Android platform. The Android platform returns the result through Promise.
The message will be transferred to the HmsPushInstanceId class. The execute method in HmsPushInstanceId can transfer the message to a method for processing based on the action type in the message.

public void execute(String action, final JSONArray args, final CallbackContext callbackContext) throws JSONException {
    if (!action.equals("init"))
        hmsLogger.startMethodExecutionTimer(action);

    switch (action) {
        case "init":
            Log.i("HMSPush", "HMSPush initialized ");
            break;
        case "enableLogger":
            enableLogger(callbackContext);
            break;
        case "disableLogger":
            disableLogger(callbackContext);
            break;
        case "getToken":
            getToken(args.length() > 1 ? args.getString(1) : Core.HCM, callbackContext);
            break;
        case "getAAID":
            getAAID(callbackContext);
            break;
        case "getCreationTime":
            getCreationTime(callbackContext);
            break;

Similarly, the processing method returns the result to JavaScript. The result will be written to the nativeToJsMessageQueue queue.
callBack.sendPluginResult(new PluginResult(PluginResult.Status.OK,autoInit));

This process is similar to that for HmsPushMessaging. The main difference is that HmsInstanceId is used for HmsPushInstanceId-related APIs, and HmsMessaging is used for HmsPushMessaging-related APIs.

3) localNotification

In the HmsLocalNotification.js file, call the run API in asynchronous mode to transfer the message to the Android platform. The Android platform returns the result through Promise.
The message will be transferred to the HmsLocalNotification class. The execute method in HmsLocalNotification can transfer the message to a method for processing based on the action type in the message.

public void execute(String action, final JSONArray args, final CallbackContext callbackContext) throws JSONException {
    switch (action) {
        case "localNotification":
            localNotification(args, callbackContext);
            break;
        case "localNotificationSchedule":
            localNotificationSchedule(args.getJSONObject(1), callbackContext);
            break;
        case "cancelAllNotifications":
            cancelAllNotifications(callbackContext);
            break;
        case "cancelNotifications":
            cancelNotifications(callbackContext);
            break;
        case "cancelScheduledNotifications":
            cancelScheduledNotifications(callbackContext);
            break;
        case "cancelNotificationsWithId":
            cancelNotificationsWithId(args.getJSONArray(1), callbackContext);
            break;

Call sendPluginResult to return the result. However, for localNotification, the result will be returned after the notification is sent.

Perform message push event callback.

In addition to the method calling, message push involves listening for many events, for example, receiving common messages, data messages, and tokens.

The callback process starts from Android.
In Android, the callback method is defined in HmsPushMessageService.java.

Based on the SDK requirements, you can opt to redefine certain callback methods, such as onMessageReceived, onDeletedMessages, and onNewToken.

When an event is triggered, an event notification is sent to JavaScript.

public static void runJS(final CordovaPlugin plugin, final String jsCode) {
    if (plugin == null)
        return;
    Log.d(TAG, "runJS()");

    plugin.cordova.getActivity().runOnUiThread(() -> {
        CordovaWebViewEngine engine = plugin.webView.getEngine();
        if (engine == null) {
            plugin.webView.loadUrl("javascript:" + jsCode);

        } else {
            engine.evaluateJavascript(jsCode, (result) -> {

            });
        }
    });
}

Each event is defined and registered in HmsPushEvent.js.
exports.REMOTE_DATA_MESSAGE_RECEIVED = "REMOTE_DATA_MESSAGE_RECEIVED";
exports.TOKEN_RECEIVED_EVENT = "TOKEN_RECEIVED_EVENT";
exports.ON_TOKEN_ERROR_EVENT = "ON_TOKEN_ERROR_EVENT";
exports.NOTIFICATION_OPENED_EVENT = "NOTIFICATION_OPENED_EVENT";
exports.LOCAL_NOTIFICATION_ACTION_EVENT = "LOCAL_NOTIFICATION_ACTION_EVENT";
exports.ON_PUSH_MESSAGE_SENT = "ON_PUSH_MESSAGE_SENT";
exports.ON_PUSH_MESSAGE_SENT_ERROR = "ON_PUSH_MESSAGE_SENT_ERROR";
exports.ON_PUSH_MESSAGE_SENT_DELIVERED = "ON_PUSH_MESSAGE_SENT_DELIVERED";

function onPushMessageSentDelivered(result) {
  window.registerHMSEvent(exports.ON_PUSH_MESSAGE_SENT_DELIVERED, result);
}

exports.onPushMessageSentDelivered = onPushMessageSentDelivered;

Please note that the event initialization needs to be performed during app development. Otherwise, the event listening will fail. For more details, please refer to eventListeners.js in the demo.

If the callback has been triggered in Java, but is not received in JavaScript, check whether the event initialization is performed.

In doing so, when an event is triggered in Android, JavaScript will be able to receive and process the message. You can also refer to this process to add an event.

5. Summary
The description above illustrates how the plugin implements the JavaScript-Java communications. The methods of most kits can be called in a similar manner. However, Map Kit, Ads Kit, and other kits that need to display images or videos (such as maps and native ads) require a different method, which will be introduced in a later article.

To learn more, please visit:

HUAWEI Developers official website

Development Guide

Redditto join developer discussions

GitHub or Gitee to download the demo and sample code

Stack Overflow to solve integration problems

How Fingerprint and Facial Authentication in Mission: Impossible Can be Brought to Life

Vivi-clevercoder — Thu, 15 Jul 2021 02:51:23 +0000

Have you ever marveled at the impressive technology in sci-fi movies, such as the floating touchscreen in Iron Man and the fingerprint and iris authentication in Mission: Impossible?

Such cutting-edge technology has already entered our day-to-day lives, with fingerprint and facial authentication being widely used.

Users are paying more and more attention to individual privacy protection and thus have higher requirements about app security, which can be guaranteed with the help of authentication based on the unique nature of fingerprints and facial data.
Fingerprint and facial authentication effectively reduces the risk of account theft and information leakage when used for unlocking devices, making payments, and accessing files.

Such an authentication mode can be realized with HUAWEI FIDO: it arms your app with FIDO2 client capabilities based on the WebAuthn standard, as well as the fingerprint and facial authentication capabilities of BioAuthn.

FIDO ensures that the authentication result is secure and reliable by checking the system integrity and using cryptographic key verification. It allows password-free authentication during sign-in, a general solution that can be easily integrated with the existing account infrastructure.

Let's see how to integrate the fingerprint and facial authentication capabilities in FIDO.
Perform the steps below:

Click the hyperlinks of step 1 and 2 to learn more about them.
Note that in step 2 there are two SDKs:

Bioauthn-AndroidX: implementation 'com.huawei.hms:fido-bioauthn-androidx:5.2.0.301'

BioAuthn: implementation 'com.huawei.hms:fido-bioauthn:5.2.0.301'
They're slightly different from each other:
The BioAuthn-AndroidX SDK provides a unified UI for fingerprint authentication. You do not need to design a fingerprint authentication UI for your app, whereas the BioAuthn SDK requires you to design a fingerprint authentication UI for your app.

Below is the detailed description of the difference in the FAQs section of this kit:

This article gives an introduction about how to integrate the BioAuthn-AndroidX SDK. You can download its demo here.

Integrating the BioAuthn-AndroidX SDK
Notes:

The fingerprint and facial authentication capabilities cannot be used on a rooted device.
Before testing, make sure that you've enrolled facial data and a fingerprint in the testing device. Otherwise, an error code will be reported.
Go to Settings > Biometrics & password on the device to enroll facial data and a fingerprint.

Fingerprint Authentication
To use the fingerprint authentication capability, perform the following steps:

Initialize the BioAuthnPrompt object:

BioAuthnPrompt bioAuthnPrompt = new BioAuthnPrompt(this, ContextCompat.getMainExecutor(this), new BioAuthnCallback() {
    @Override
    public void onAuthError(int errMsgId, CharSequence errString) {
        showResult("Authentication error. errorCode=" + errMsgId + ",errorMessage=" + errString);
    }
    @Override
    public void onAuthSucceeded(BioAuthnResult result) {
        showResult("Authentication succeeded. CryptoObject=" + result.getCryptoObject());
    }
    @Override
    public void onAuthFailed() {
        showResult("Authentication failed.");
    }
});

2.Configure prompt information and perform authentication.

// Customize the prompt information.
BioAuthnPrompt.PromptInfo.Builder builder =
        new BioAuthnPrompt.PromptInfo.Builder().setTitle("This is the title.")
                .setSubtitle("This is the subtitle.")
                .setDescription("This is the description.");

// The user is allowed to authenticate with methods other than biometrics.
builder.setDeviceCredentialAllowed(true);

BioAuthnPrompt.PromptInfo info = builder.build();

// Perform authentication.
bioAuthnPrompt.auth(info);

After the configuration is complete, fingerprint authentication can be performed on a screen similar to the following image:

Facial Authentication
There are many restrictions on using the facial authentication capability. For details, please refer to the corresponding FAQs.

Check whether the camera permission has been granted to your app. (Note that this permission is not needed on devices running EMUI 10.1 or later.)

    int permissionCheck = 
    ContextCompat.checkSelfPermission(MainActivity.this, 
    Manifest.permission.CAMERA);
    if (permissionCheck != PackageManager.PERMISSION_GRANTED) {
        showResult("Grant the camera permission first.");

        ActivityCompat.requestPermissions(MainActivity.this, new 
    String[] {Manifest.permission.CAMERA}, 1);
        return;
    }

Check whether the device supports facial authentication.

FaceManager faceManager = new FaceManager(this);

int errorCode = faceManager.canAuth();
if (errorCode != 0) {
    resultTextView.setText("");
    showResult("The device does not support facial authentication. errorCode=" + errorCode);
    return;
}

Perform facial authentication.

int flags = 0;
Handler handler = null;
CryptoObject crypto = null;

faceManager.auth(crypto, cancellationSignal, flags, new BioAuthnCallback() {
    @Override
    public void onAuthError(int errMsgId, CharSequence errString) {
        showResult("Authentication error. errorCode=" + errMsgId + ",errorMessage=" + errString
                + (errMsgId == 1012 ? " The camera permission has not been granted." : ""));
    }

    @Override
    public void onAuthHelp(int helpMsgId, CharSequence helpString) {
        showResult("This is the prompt information during authentication. helpMsgId=" + helpMsgId + ",helpString=" + helpString + "\n");
    }

    @Override
    public void onAuthSucceeded(BioAuthnResult result) {
        showResult("Authentication succeeded. CryptoObject=" + result.getCryptoObject());
    }

    @Override
    public void onAuthFailed() {
        showResult("Authentication failed.");
    }
}, handler);

This is all the code for facial authentication. You can call it to perform this capability.
Note that there is no default UI for this capability. You need to design a UI as needed.

Application Scenarios
Fingerprint Authentication

Fingerprint authentication is commonly used before payments by users for security authentication.
It can also be integrated into file protection apps to allow only users passing fingerprint authentication to access relevant files.

Facial Authentication

This capability works well in scenarios where fingerprint authentication can be used. For file protection apps, facial authentication has a better performance than fingerprint authentication.

This is because such apps share a common flaw: they make it clear that a file is very important or sensitive.
Therefore, a hacker can access this file once they figure out a way to obtain the fingerprint authentication of the app, which can be done despite the difficulty in doing so.

To avoid this, in addition to fingerprint authentication, a file protection app can adopt facial authentication "secretly" — this capability does not require a UI. The app displays the real file after a user obtains both fingerprint and facial authentication, otherwise it will display a fake file.

In this way, it can improve the protection of user privacy.
The following is the sample code for developing such a function:

faceManager.auth(crypto, cancellationSignal, flags, new BioAuthnCallback() {
    @Override
    public void onAuthError(int errMsgId, CharSequence errString) {
        if(isFingerprintSuccess){// Fingerprint authentication succeeded but facial authentication failed.
            // Display a fake file.
            showFakeFile();
        }
    }

    @Override
    public void onAuthHelp(int helpMsgId, CharSequence helpString) {
    }

    @Override
    public void onAuthSucceeded(BioAuthnResult result) {
        if(isFingerprintSuccess){// Fingerprint authentication succeeded.
            // Display the real file.
            showRealFile();
        }else {// Fingerprint authentication failed.
            // Display a fake file.
            showFakeFile();
        }

    }

    @Override
    public void onAuthFailed() {
        if(isFingerprintSuccess){// Fingerprint authentication succeeded but facial authentication failed.
            // Display a fake file.
            showFakeFile();
        }

    }
}, handler);

To learn more, please visit:

HUAWEI Developers official website

Development Guide

Redditto join developer discussions

GitHub or Gitee to download the demo and sample code

Stack Overflow to solve integration problems

Audio File Transcription, for Super-Efficient Recording

Vivi-clevercoder — Mon, 12 Jul 2021 08:20:10 +0000

Introduction

Converting audio into text has a wide range of applications: generating video subtitles, taking meeting minutes, and writing interview transcripts. HUAWEI ML Kit's service makes doing so easier than ever before, converting audio files into meticulously accurate text, with correct punctuation as well!

Actual Effects

Build and run an app with audio file transcription integrated. Then, select a local audio file and convert it into text.

![Alt text of image]

Development Preparations
For details about configuring the Huawei Maven repository and integrating the audio file transcription SDK, please refer to the Development Guide of ML Kit on HUAWEI Developers.

Declaring Permissions in the AndroidManifest.xml File

Open the AndroidManifest.xml in the main folder. Add the network connection, network status access, and storage read permissions before Please note that these permissions need to be dynamically applied for. Otherwise, Permission Denied will be reported.

Development Procedure
Creating and Initializing an Audio File Transcription Engine**
Override onCreate in MainActivity to create an audio transcription engine.
private MLRemoteAftEngine mAnalyzer;

mAnalyzer = MLRemoteAftEngine.getInstance();
mAnalyzer.init(getApplicationContext());
mAnalyzer.setAftListener(mAsrListener);

Use MLRemoteAftSetting to configure the engine. The service currently supports Mandarin Chinese and English, that is, the options of mLanguage are zh and en.
MLRemoteAftSetting setting = new MLRemoteAftSetting.Factory()
.setLanguageCode(mLanguage)
.enablePunctuation(true)
.enableWordTimeOffset(true)
.enableSentenceTimeOffset(true)
.create();

enablePunctuation indicates whether to automatically punctuate the converted text, with a default value of false.
If this parameter is set to true, the converted text is automatically punctuated; false otherwise.

enableWordTimeOffset indicates whether to generate the text transcription result of each audio segment with the corresponding offset. The default value is false. You need to set this parameter only when the audio duration is less than 1 minute.
If this parameter is set to true, the offset information is returned along with the text transcription result. This applies to the transcription of short audio files with a duration of 1 minute or shorter.
If this parameter is set to false, only the text transcription result of the audio file will be returned.

enableSentenceTimeOffset indicates whether to output the offset of each sentence in the audio file. The default value is false.
If this parameter is set to true, the offset information is returned along with the text transcription result.
If this parameter is set to false, only the text transcription result of the audio file will be returned.

Creating a Listener Callback to Process the Transcription Result
private MLRemoteAftListener mAsrListener = new MLRemoteAftListener()

After the listener is initialized, call startTask in AftListener to start the transcription.

(

@Override
public void onInitComplete(String taskId, Object ext) {
    Log.i(TAG, "MLRemoteAftListener onInitComplete" + taskId);
    mAnalyzer.startTask(taskId);
}

Override onUploadProgress, onEvent, and onResult in MLRemoteAftListener.

@Override
public void onUploadProgress(String taskId, double progress, Object ext) {
    Log.i(TAG, " MLRemoteAftListener onUploadProgress is " + taskId + " " + progress);
}

@Override
public void onEvent(String taskId, int eventId, Object ext) {
    Log.e(TAG, "MLAsrCallBack onEvent" + eventId);
    if (MLAftEvents.UPLOADED_EVENT == eventId) { // The file is uploaded successfully.
        showConvertingDialog();
        startQueryResult(); // Obtain the transcription result.
    }
}

@Override
public void onResult(String taskId, MLRemoteAftResult result, Object ext) {
    Log.i(TAG, "onResult get " + taskId);
    if (result != null) {
        Log.i(TAG, "onResult isComplete " + result.isComplete());
        if (!result.isComplete()) {
            return;
        }
        if (null != mTimerTask) {
            mTimerTask.cancel();
        }
        if (result.getText() != null) {
            Log.e(TAG, result.getText());
            dismissTransferringDialog();
            showCovertResult(result.getText());
        }

        List<MLRemoteAftResult.Segment> segmentList = result.getSegments();
        if (segmentList != null && segmentList.size() != 0) {
            for (MLRemoteAftResult.Segment segment : segmentList) {
                Log.e(TAG, "MLAsrCallBack segment  text is : " + segment.getText() + ", startTime is : " + segment.getStartTime() + ". endTime is : " + segment.getEndTime());
            }
        }

        List<MLRemoteAftResult.Segment> words = result.getWords();
        if (words != null && words.size() != 0) {
            for (MLRemoteAftResult.Segment word : words) {
                Log.e(TAG, "MLAsrCallBack word  text is : " + word.getText() + ", startTime is : " + word.getStartTime() + ". endTime is : " + word.getEndTime());
            }
        }

        List<MLRemoteAftResult.Segment> sentences = result.getSentences();
        if (sentences != null && sentences.size() != 0) {
            for (MLRemoteAftResult.Segment sentence : sentences) {
                Log.e(TAG, "MLAsrCallBack sentence  text is : " + sentence.getText() + ", startTime is : " + sentence.getStartTime() + ". endTime is : " + sentence.getEndTime());
            }
        }
    }

}
(```

)


Processing the Transcription Result in Polling Mode
After the transcription is completed, call getLongAftResult to obtain the transcription result. Process the obtained result every 10 seconds.


(

```)
private void startQueryResult() {
    Timer mTimer = new Timer();
    mTimerTask = new TimerTask() {
        @Override
        public void run() {
            getResult();
        }
    };
    mTimer.schedule(mTimerTask, 5000, 10000); // Process the obtained long speech transcription result every 10s.
}

private void getResult() {
    Log.e(TAG, "getResult");
    mAnalyzer.setAftListener(mAsrListener);
    mAnalyzer.getLongAftResult(mLongTaskId);
}

(```

)


(https://stackoverflow.com/questions/tagged/huawei-mobile-services)

Follow our official account for the latest HMS Core-related news and updates.

AppGallery and HMS Core Enable Game Developers in Vietnam

Vivi-clevercoder — Fri, 25 Dec 2020 03:42:35 +0000

On November 19 2020, Vietnam's Huawei Developers Day ended in Hanoi. The theme of the seminar was Unlock the global market, focusing on the game field, attracting nearly 60 leading game developers from Vietnam.

Huawei Advertising Network: Leading Game Revenue Growth Strategy in Vietnam

Maintaining the ability to develop revenue is critical for long-term games. But when foreign giants make up a large share of their investment in the domestic market, this becomes a difficulty for Vietnamese developers. In this case, for game developers who want to grow in the long term, expansion into the international market is an inevitable trend.

To address this issue, Huawei AppGallery, a mobile app store, has invested in the development of Huawei's advertising network. For developers, multiple sources of revenue can be realized, even for free applications. In a short period of time, developers can establish advertising partnerships with Huawei. They can integrate Ads Kit of HMS Core to gain profits from advertisements in video game applications. Ads kit will enable developers to have high-quality promotions from advertisers in more than 220 countries.

One of the guests at this event, Mr. Dang Thanh Long, Chief Technology Officer of Segu Company, a well-known developer, said, "Huawei AppGallery is a huge partner for us and will help us quickly realize our desire to bring value to global users through mobile apps and make life better, Happier. "With the advantages of Huawei AppGallery, Vietnamese developers will no longer rely on a single market for revenue. Instead, they are entirely proactive in looking for opportunities to increase sales and penetrate global markets.

Local policy support and technical support: Huawei AppGallery Plus HMS Core

The local developer support policy is an innovation of Huawei AppGallery's competitive edge. “The Huawei Developer Fund” established to help developers obtain resources and reinvest game quality and content for long-term development. From the technical perspective, the HMS Core can be connected to applications to meet four key requirements, including graphics development, network acceleration, technical innovation, and monetization. In this process, the Vietnamese technical team and local experts will also be provided, to help them overcome language barriers and cultural differences when releasing games in the marketplace.

In addition to improving the content discovery and recommendation system, Huawei AppGallery also highlights the difference in revenue sharing rates between the app store and developers. Specifically, by working with Huawei AppGallery, developers can earn up to 90% of the revenue from ads on their apps. This share will continue until the end of 2020 and will reach 80-20% by 2021.

Huawei AppGallery will become a very good platform for developers in Vietnam who want to achieve long-term growth and gain access to the international market, and HMS Core will help its applications achieve more functions after integration.