Develop your HelloAR app in Android studio using ARCore and Sceneform

AR turning accessible with the increasing interest within the metaverse trend daily. It is changing the traditional way of interactions into more digital.

What is Augmented Reality

Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory — Wikipedia

What is ARCore

ARCore is a software development kit developed by Google that allows for augmented reality applications to be built.
ARCore uses three key technologies to integrate virtual content with the real world as seen through the camera of a smartphone.

• Six degrees of freedom allow the phone to understand and track its position relative to the world.
• Environmental understanding allows the phone to detect the size and location of flat horizontal surfaces like the ground or a coffee table.
• Light estimation allows the phone to estimate the environment’s current lighting conditions.

Sceneform

Sceneform is the Google official AR SDK to enable developers to make AR apps while not having to be told OpenGL.
Sceneform comes with options such as:

• An automatic compatibility check for ARCore enabled phones.
• Checking for camera permissions.
• A plugin for manipulating 3D assets.
• A scene graph API to abstract all the complexities.

Getting Started

You first need to enable ARCore in your project. This is simple as we will be using Android Studio and Sceneform SDK. There are two major operations Sceneform performs automatically:
Checking for availability of ARCore
Asking for camera permission
Create a new Android Studio project and select an empty activity.

Adding Dependencies

Add the following dependency to your project level build.gradle file:

dependencies {
    ...
    classpath 'com.google.ar.sceneform:plugin:1.17.1'
}

Add the following dependency to your app level build.gradle file:

dependencies {
    ...
    implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.17.1'
    implementation 'com.google.ar:core:1.27.0'
}

Add the following lines in your AndroidManifest.xml file:

<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.INTERNET"/>
<uses-feature android:name="android.hardware.camera.ar" android:required="true"/>

<application
    ...>
    ...
    <meta-data
        android:name="com.google.ar.core"
        android:value="required" />
</application>

Then sync your project with Gradle files and wait for the build to finish (click on the sync icon). This will install the Sceneform SDK to the project and Sceneform plugin to AndroidStudio.

Now, we can begin with creating our first ARCore application.
In the first place, we have to add the Sceneform fragment to our design layout. This will be the container for rendering the 3d models and deals with the camera instantiation and other ar related functionalities.
Add the sceneform ar fragment into the activity_main.xml like this,

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical"
    tools:context=".MainActivity">

    <fragment
        android:id="@+id/ux_fragment"
        android:name="com.google.ar.sceneform.ux.ArFragment"
        android:layout_width="match_parent"
        android:layout_height="match_parent" />

</LinearLayout>

Checking compatibility at runtime

Add the method below in your class for the compatibility check:

private boolean checkIsSupportedDeviceOrFinish(final Activity activity) {
        if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) {
            Log.e(TAG, "Sceneform requires Android N or later");
            Toast.makeText(activity, "Sceneform requires Android N or later", Toast.LENGTH_LONG).show();
            activity.finish();
            return false;
        }
        String openGlVersionString =
                ((ActivityManager) activity.getSystemService(Context.ACTIVITY_SERVICE))
                        .getDeviceConfigurationInfo()
                        .getGlEsVersion();
        if (Double.parseDouble(openGlVersionString) < MIN_OPENGL_VERSION) {
            Log.e(TAG, "Sceneform requires OpenGL ES 3.0 later");
            Toast.makeText(activity, "Sceneform requires OpenGL ES 3.0 or later", Toast.LENGTH_LONG)
                    .show();
            activity.finish();
            return false;
        }
        return true;
    }

Adding 3D models to our application

Now, we can download and import the 3D models into our application. You will get 3D models from the repositories like cgtrader, free3d and turboquid. Google also provided an excellent repository POLY to download 3D models for your application. You can download the models in .fbx, .obj or .gltf format. We will be doing it in the .fbx format.
In this demo, we will be augmenting a 3D cube on the user tap.
Open the project and expand the app folder in the project view. Create a folder named sampledata inside the app folder. Then you have to add the downloaded model file inside this sampledata folder. Here I have added a cube.fbx model. You can add your own model.

Importing the model using the Sceneform plugin

Apply sceneform ar plugin in your app’s build.gradle file.
Add the following dependencies:

apply plugin: 'com.google.ar.sceneform.plugin'

Add the following lines at the end of the app’s build.gradle file:

sceneform.asset('sampledata/cube.obj',
        'default',
        'sampledata/cube.sfa',
        'src/main/res/raw/cube')

Finally, the build.gradle of the app looks like this:

apply plugin: 'com.android.application'
apply plugin: 'com.google.ar.sceneform.plugin'

android {
    compileSdkVersion 31
    defaultConfig {
        applicationId "com.example.arcorefirst"
        minSdkVersion 24
        targetSdkVersion 31
        versionCode 1
        versionName "1.0"
        testInstrumentationRunner 'androidx.test.runner.AndroidJUnitRunner'
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
    }
}

dependencies {
    implementation 'androidx.appcompat:appcompat:1.3.1'
    implementation 'androidx.constraintlayout:constraintlayout:2.1.1'
    testImplementation 'junit:junit:4.13.2'
    androidTestImplementation 'androidx.test.ext:junit:1.1.3'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.4.0'
    implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.17.1'
    implementation 'com.google.ar:core:1.27.0'
}

sceneform.asset('sampledata/cube.obj',
        'default',
        'sampledata/cube.sfa',
        'src/main/res/raw/cube')

Integrating the Model

Add the following code to your java file:

@Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        if (!checkIsSupportedDeviceOrFinish(this))
            return;

        setContentView(R.layout.activity_main);

        arFragment = (ArFragment) getSupportFragmentManager().findFragmentById(R.id.ux_fragment);
        arFragment.setOnTapArPlaneListener(
                (HitResult hitresult, Plane plane, MotionEvent motionevent) -> {
                    if (plane.getType() != Plane.Type.HORIZONTAL_UPWARD_FACING)
                        return;

                    Anchor anchor = hitresult.createAnchor();
                    placeObject(arFragment, anchor, R.raw.cube);
                }
        );
    }

Adding the Model to the AR Scene

The AR fragment is a container to hold all 3d objects. So, we will add a model to the fragment whenever it is clicked by implementing onTapListener in our fragment.

private void placeObject(ArFragment arFragment, Anchor anchor, int uri) {
        ModelRenderable.builder()
            .setSource(arFragment.getContext(), uri)
            .build()
            .thenAccept(modelRenderable -> addNodeToScene(arFragment, anchor, modelRenderable))
            .exceptionally(throwable -> {
                    Toast.makeText(arFragment.getContext(), "Error:" + throwable.getMessage(), Toast.LENGTH_LONG).show();
                    return null;
                }
            );
        }

        private void addNodeToScene(ArFragment arFragment, Anchor anchor, Renderable renderable) {
        AnchorNode anchorNode = new AnchorNode(anchor);
        TransformableNode node = new TransformableNode(arFragment.getTransformationSystem());
        node.setRenderable(renderable);
        node.setParent(anchorNode);
        arFragment.getArSceneView().getScene().addChild(anchorNode);
        node.select();
    }

Let’s understand the code:
Using the hitResult, get the tapped location value and create an anchor node on that position. The anchor node is used to place the object in the real world position but it can’t perform any operation over the model. So, we create a TransformableNode which can enable the user to apply the various transformation on the object like translate, rotate and scale.
Let’s have a look at some terminologies here:
Scene: It’s the place where our 3D world will be rendered.
HitResult: It is an imaginary ray of light coming from infinity and its first point of intersection with the real world is the point of the tap.
Anchor: A fixed location in the real world. Used to transform local coordinates into real-world coordinates.
TransformableNode: A node that can react to user interactions such as translation, rotation, scale and drag.
Finally, the MainActivity.java file looks like this:

package com.example.arcorefirst;

import android.app.Activity;
import android.app.ActivityManager;
import android.content.Context;
import android.net.Uri;
import android.os.Build;
import android.os.Bundle;
import androidx.appcompat.app.AppCompatActivity;
import android.util.Log;
import android.view.MotionEvent;
import android.widget.Toast;

import com.google.ar.core.Anchor;
import com.google.ar.core.HitResult;
import com.google.ar.core.Plane;
import com.google.ar.sceneform.AnchorNode;
import com.google.ar.sceneform.rendering.ModelRenderable;
import com.google.ar.sceneform.rendering.Renderable;
import com.google.ar.sceneform.ux.ArFragment;
import com.google.ar.sceneform.ux.TransformableNode;

public class MainActivity extends AppCompatActivity {
    private static final String TAG = MainActivity.class.getSimpleName();
    private static final double MIN_OPENGL_VERSION = 3.0;

    ArFragment arFragment;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        if (!checkIsSupportedDeviceOrFinish(this))
            return;

        setContentView(R.layout.activity_main);

        arFragment = (ArFragment) getSupportFragmentManager().findFragmentById(R.id.ux_fragment);
        arFragment.setOnTapArPlaneListener(
                (HitResult hitresult, Plane plane, MotionEvent motionevent) -> {
                    if (plane.getType() != Plane.Type.HORIZONTAL_UPWARD_FACING)
                        return;

                    Anchor anchor = hitresult.createAnchor();
                    placeObject(arFragment, anchor, R.raw.cube);
                }
        );
    }

    private void placeObject(ArFragment arFragment, Anchor anchor, int uri) {
        ModelRenderable.builder()
            .setSource(arFragment.getContext(), uri)
            .build()
            .thenAccept(modelRenderable -> addNodeToScene(arFragment, anchor, modelRenderable))
            .exceptionally(throwable -> {
                    Toast.makeText(arFragment.getContext(), "Error:" + throwable.getMessage(), Toast.LENGTH_LONG).show();
                    return null;
                }
            );
    }

    private void addNodeToScene(ArFragment arFragment, Anchor anchor, Renderable renderable) {
        AnchorNode anchorNode = new AnchorNode(anchor);
        TransformableNode node = new TransformableNode(arFragment.getTransformationSystem());
        node.setRenderable(renderable);
        node.setParent(anchorNode);
        arFragment.getArSceneView().getScene().addChild(anchorNode);
        node.select();
    }

    private boolean checkIsSupportedDeviceOrFinish(final Activity activity) {
        if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) {
            Log.e(TAG, "Sceneform requires Android N or later");
            Toast.makeText(activity, "Sceneform requires Android N or later", Toast.LENGTH_LONG).show();
            activity.finish();
            return false;
        }
        String openGlVersionString =
                ((ActivityManager) activity.getSystemService(Context.ACTIVITY_SERVICE))
                        .getDeviceConfigurationInfo()
                        .getGlEsVersion();
        if (Double.parseDouble(openGlVersionString) < MIN_OPENGL_VERSION) {
            Log.e(TAG, "Sceneform requires OpenGL ES 3.0 later");
            Toast.makeText(activity, "Sceneform requires OpenGL ES 3.0 or later", Toast.LENGTH_LONG)
                    .show();
            activity.finish();
            return false;
        }
        return true;
    }
}

Let’s see what’s happening here:

1. First, we need to get the ar fragment with its id from the layout file using supportFragmentManager .

2. Then we need to load the model into the scene. For this, we use the ModelRenderable class provided by the Sceneform SDK. We load the model by passing the name of the generated .sfb file as the argument of the setSource() method.

3. The model is being built on a background thread. After that, it renders in the scene using the main thread.

4. We receive the model inside the thenAccept method. An exception is thrown if there’s an error in building the model.

5. Now, we create an anchorNode and a transformableNode. Also, refer to the renderable object inside the setRenderable(). Finally, we set anchorNode as the parent of the transformableNode and add the anchorNode to the scene.

There you have it! Your own AR App in Android studio :)

Thanks for reading this article.
If you enjoyed this article, please click on the heart button ♥ and share to help others find it!
The full code of the project in this tutorial is available on

codemaker2015/ar-object-augmentation-android-studio

Originally posted on Medium -
Develop your HelloAR app in Android studio using ARCore and Sceneform