How I Built an Offline AR Video-Book App on Android (Without Cloud APIs or Premium SDKs)

Abstract

• A detailed breakdown of developing an image recognition AR application using Google's ARCore.

Overview

My previously released AR application VideoPhotoBook was built using the Vuforia SDK. While it worked well, Vuforia requires pre-registering reference images on their web developer portal.

When I realized that Google's official ARCore supports dynamic image database generation directly on the device, I decided to rebuild the project. This article documents the development of VideoPhotoBookv3, a completely local-first AR app that allows users to create dynamic AR experiences on the fly.

• App: VideoPhotoBookv3 (TBD)
• GitHub: VideoPhotoBookv3
• Zenn Article (Japanese): ARCoreを使った画像認識ARアプリを公開してみた
• DEV.to Article: (This Post)

1. Requirements

• Core Feature: Match any image (marker) with a video from the user's gallery. When the camera points at the image, overlay the corresponding video on top of the physical image target in real-time.
• Multiple Pairs: Support listing and registering multiple "Image ⇄ Video" pairs, allowing the AR session to simultaneously track and play multiple videos.
• Design: Minimalist UI.

2. UI Design

Designed under minimalist principles: generous spacing, clean typography, and a cohesive, simple color scheme.

Settings List Screen (App Entry Point):

• Pair List (Card List):
  • Displays a list of configured pairs.
  • Each card displays the marker image thumbnail, video file name, target physical width, and video scale factor.
  • Interactive Edit: Tapping any card opens the "Edit Pair Dialog", allowing the user to update the image, video, and physical width individually.
  • All active pairs in the list will be target markers for AR tracking.
• Add/Edit Pair Dialog:
  • Lets users configure new pairs with preview capabilities.
  • Shows a thumbnail preview of the selected marker image to ensure visual clarity.
• Launch AR Button:
  • A slim, elegant black button positioned at the bottom of the screen, displayed only when one or more pairs are configured.

AR Camera Screen:

• Camera Preview: Full-screen view.
• Back Button: A clean arrow icon in the top-left corner.
• Status Indicator: Subtle overlay text indicating whether the app is searching for markers or actively tracking them.

3. Technical & Functional Design

(1) Data Persistence (Data Model)

To save multiple image-video pairs locally, we define the following data structure:

data class ArKeyPair(
    val id: String,          // Unique Identifier (UUID)
    val markerUri: String,   // Gallery URI of the marker image
    val videoUri: String,    // Gallery URI of the video file
    val physicalWidth: Float, // Actual physical width of the marker (meters, default = 0.1m = 10cm)
    val scaleFactor: Float   // Scale multiplier (default = 1.0 = 100%)
)

• The pairs are serialized to a JSON string and persisted using SharedPreferences.

(2) AR Rendering via Sceneview

We use Sceneview (built on Google's Filament 3D renderer) to handle the 3D scene graph and ARCore integration.

Video Overlay Architecture Overview:

1. ARScene Composable: Sceneview provides a Jetpack Compose wrapper called ARScene to host the camera preview and handle the ARCore lifecycle out of the box.
2. Asynchronous Database Generation: During initialization, the app decodes the gallery image URIs into Bitmaps on a background thread and compiles them into ARCore's AugmentedImageDatabase via database.addImage(id, bitmap, physicalWidth).
3. Anchor and Node Binding: When ARCore detects a marker, it creates an Anchor. We then instantiate an AnchorNode at this anchor's position.
4. VideoNode for Media Playback:
   • Sceneview's VideoNode allows rendering video streams onto a 3D plane.
   • We instantiate a standard MediaPlayer (or ExoPlayer), assign the videoUri as the data source, and bind it to the VideoNode.
   • The video will overlay precisely on top of the physical target image based on its coordinates and width.

4. Sequences

1. App Launch & Initialization

Sequence Diagram

Explanation

When the app launches, it asynchronously loads the image-video pair configurations (JSON string) from SharedPreferences. The deserialized list updates the uiState (StateFlow) in MainScreenViewModel, triggering Jetpack Compose to render the list of cards.

Code Implementation

• MainScreenViewModel.kt:

  class MainScreenViewModel(private val repository: KeyPairRepository) : ViewModel() {
      private val _uiState = MutableStateFlow<List<ArKeyPair>>(emptyList())
      val uiState: StateFlow<List<ArKeyPair>> = _uiState.asStateFlow()

      init { loadPairs() }

      fun loadPairs() {
          viewModelScope.launch {
              _uiState.value = repository.getPairs()
          }
      }
  }

• KeyPairRepository.kt:

  fun getPairs(): List<ArKeyPair> {
      val json = prefs.getString(key, null) ?: return emptyList()
      return try {
          val type = object : TypeToken<List<ArKeyPair>>() {}.type
          gson.fromJson(json, type) ?: emptyList()
      } catch (e: Exception) {
          emptyList()
      }
  }

---

2. Selecting Media & Persisting URI Access

Sequence Diagram

Explanation

When selecting images and videos via the Android Photo Picker, the returned Uri loses its read permissions once the app process restarts. To bypass this, we immediately request a persistable read permission using takePersistableUriPermission. This ensures the app can access the selected local assets across device restarts.

Code Implementation

• MainScreen.kt (Photo Picker & URI Persistence):

  // Helper to persist URI read permission
  private fun persistUriAccess(context: Context, uri: Uri) {
      try {
          val takeFlags: Int = Intent.FLAG_GRANT_READ_URI_PERMISSION
          context.contentResolver.takePersistableUriPermission(uri, takeFlags)
      } catch (e: SecurityException) {
          // Handle exception
      }
  }

  // Photo Picker Launcher
  val pickImageLauncher = rememberLauncherForActivityResult(
      contract = ActivityResultContracts.PickVisualMedia()
  ) { uri ->
      if (uri != null) {
          persistUriAccess(context, uri)
          markerUri = uri
      }
  }

---

3. AR Startup & ARCore Session Initialization

Sequence Diagram

Explanation

When transitioning to the AR screen, the app requests camera permission if it has not been granted. Next, it uses a coroutine running on Dispatchers.IO to load and decode the target images into Bitmaps asynchronously. Once the AR session starts, these Bitmaps are dynamically added to the AugmentedImageDatabase and configured on the session along with auto-focus configurations.

Code Implementation

• ArViewScreen.kt (Async Bitmaps & Database Init):

  // Decode marker images asynchronously
  LaunchedEffect(pairs) {
      withContext(Dispatchers.IO) {
          for (pair in pairs) {
              val bitmap = loadBitmapFromUri(context, pair.markerUri)
              if (bitmap != null) {
                  bitmaps[pair.id] = bitmap
              }
          }
          isBitmapsLoaded = true
      }
  }

  // Build the image database when session initializes
  LaunchedEffect(arSession, isBitmapsLoaded) {
      val session = arSession
      if (session != null && isBitmapsLoaded && isLoading) {
          withContext(Dispatchers.IO) {
              val database = AugmentedImageDatabase(session)
              for (pair in pairs) {
                  val bitmap = bitmaps[pair.id]
                  if (bitmap != null) {
                      database.addImage(pair.id, bitmap, pair.physicalWidth)
                  }
              }
              val config = session.config
              config.augmentedImageDatabase = database
              config.focusMode = Config.FocusMode.AUTO
              session.configure(config)
              withContext(Dispatchers.Main) {
                  isLoading = false
              }
          }
      }
  }

---

4. Tracking and Dynamic Video Lifecycle Control

Sequence Diagram

Explanation

We capture frames from the AR session callback onSessionUpdated. When target images are detected, their tracking states (TRACKING, PAUSED, STOPPED) drive the video lifecycle:

• Newly Detected (TRACKING): We create an Anchor at the center of the image. We extract the video's aspect ratio via the MediaPlayer and compute the correct fitting bounds within the marker's physical size. The video plays, and the AnchorNode is reactively rendered onto the screen.
• Lost Sight (PAUSED): The video pauses to save resources, resuming once tracking is restored.
• Destroyed (STOPPED): The player resources are released, and the node is removed from the layout hierarchy.

Code Implementation

• ArViewScreen.kt (Tracking Lifecycle Handler):

  onSessionUpdated = { session, frame ->
      val updatedImages = frame.getUpdatedTrackables(AugmentedImage::class.java)
      for (image in updatedImages) {
          val id = image.name
          val pair = pairs.firstOrNull { it.id == id } ?: continue

          when (image.trackingState) {
              TrackingState.TRACKING -> {
                  val activeVideo = activeVideos[id]
                  if (activeVideo == null) {
                      val anchor = image.createAnchor(image.centerPose)
                      val mediaPlayer = MediaPlayer().apply {
                          setDataSource(context, Uri.parse(pair.videoUri))
                          isLooping = true
                          prepare()
                      }

                      // Preserve video aspect ratio within the marker bounds
                      val videoWidth = mediaPlayer.videoWidth.toFloat()
                      val videoHeight = mediaPlayer.videoHeight.toFloat()
                      val videoRatio = if (videoWidth > 0f && videoHeight > 0f) videoWidth / videoHeight else (image.extentX / image.extentZ)
                      val imageRatio = image.extentX / image.extentZ
                      val baseSize = if (videoRatio > imageRatio) {
                          Size(image.extentX, image.extentX / videoRatio)
                      } else {
                          Size(image.extentZ * videoRatio, image.extentZ)
                      }
                      val scale = if (pair.scaleFactor <= 0f) 1f else pair.scaleFactor
                      val size = Size(baseSize.x * scale, baseSize.y * scale)

                      mediaPlayer.start()
                      activeVideos[id] = ActiveVideo(id, anchor, mediaPlayer, size)
                  } else {
                      if (!activeVideo.mediaPlayer.isPlaying) {
                          activeVideo.mediaPlayer.start()
                      }
                  }
              }
              TrackingState.PAUSED -> {
                  activeVideos[id]?.let {
                      if (it.mediaPlayer.isPlaying) it.mediaPlayer.pause()
                  }
              }
              TrackingState.STOPPED -> {
                  activeVideos[id]?.let {
                      try {
                          if (it.mediaPlayer.isPlaying) it.mediaPlayer.stop()
                          it.mediaPlayer.release()
                      } catch (e: Exception) {}
                      activeVideos.remove(id)
                  }
              }
          }
      }
  }

• ArViewScreen.kt (Declarative Scene Graph rendering):

  ARSceneView(
      // ...,
      onSessionUpdated = { /* ... */ }
  ) {
      activeVideos.values.forEach { activeVideo ->
          AnchorNode(anchor = activeVideo.anchor) {
              VideoNode(
                  player = activeVideo.mediaPlayer,
                  size = activeVideo.size,
                  // Rotates -90 degrees on the X axis to lie flat on the horizontal X-Z plane
                  rotation = Rotation(x = -90f, y = 0f, z = 0f)
              )
          }
      }
  }

---

Key Implementation Summary

1. Persisting URI read permission: Crucial when working with Android's system pickers. Bypasses permission expiration on app process restarts via takePersistableUriPermission.
2. Asynchronous Bitmap Loading: De-clutters the main thread by preparing Bitmaps off the main thread before starting the ARCore engine.
3. Runtime Augmented Image Database creation: Builds the database dynamically, removing the need for pre-calculated static .imgdb files.
4. Declarative 3D Scene Graph via Compose: Sceneview's Compose layout eliminates imperative parent.addChild() commands. 3D nodes reactively follow the Compose lifecycle.

Conclusion & Takeaways

• Development is much simpler without proprietary platforms: Moving away from Vuforia meant getting rid of web portal pre-registration. Everything is now processed on-device, offering a much cleaner development and user flow.
• ARCore Tracking is highly reliable: Even when tracking targets generated dynamically at runtime, ARCore's image tracking accuracy and latency are highly performant.

I hope this walkthrough assists you in building your next local-first AR app. Happy coding!