DEV Community

Cover image for Running Real-ESRGAN and NAFNet on Android: Engineering a 100% On-Device AI Video Upscaler
Nabil Ansari
Nabil Ansari

Posted on

Running Real-ESRGAN and NAFNet on Android: Engineering a 100% On-Device AI Video Upscaler

#android #machinelearning #onnx #kotlin

A technical breakdown of how OnDevice - Video Enhancer achieves deep-learning video super-resolution and motion deblurring locally on mobile GPUs and NPUs with zero cloud infrastructure.

Introduction: The Cloud Paradigm vs. Edge Intelligence

Over the past decade, state-of-the-art super-resolution networks like Real-ESRGAN and image restoration networks like NAFNet (Nonlinear Activation Free Network) have set the gold standard in computer vision. However, deploying these architectures for consumer mobile applications has historically relied almost exclusively on cloud servers.

The rationale seemed obvious: video enhancement requires decoding hundreds or thousands of high-resolution video frames, running billions of floating-point operations per second (FLOPs), and re-encoding with hardware codecs like H.264 or HEVC without exceeding mobile thermal or memory envelopes.

Yet the cloud paradigm carries massive trade-offs:

  1. Severe Privacy Hazards: Users must upload sensitive personal, family, and confidential recordings to remote cloud buckets.
  2. Bandwidth & Latency Bottlenecks: Uploading and downloading multi-gigabyte video files over cellular data is slow and expensive.
  3. Escalating Cloud Costs: GPU compute hours on cloud providers (A100/H100 instances) force app developers into exorbitant recurring subscription models.

This article examines how OnDevice - Video Enhancer (com.nabil.ondevice.videoenhancer) bypasses the cloud entirely, deploying optimized ONNX neural models directly onto Android devices running consumer mobile chipsets (Qualcomm Snapdragon, MediaTek Dimensity, Samsung Exynos, and Google Tensor).

Architecture Overview

The local pipeline is structured into four decoupled, memory-safe stages:

[ Input MP4/MOV Video ]
           │
           ▼
┌────────────────────────────────────────┐
│ 1. Native MediaCodec Decoder Pipeline  │
│    Extracts raw RGB/YUV bitstream      │
└────────────────────────────────────────┘
           │
           ▼
┌────────────────────────────────────────┐
│ 2. Edge Neural Network Inference       │
│    • Real-ESRGAN Compact 4x            │
│    • OpenCV NAFNet Deblur 1x           │
│    • MobileSR 2x (Fast Convolutional)  │
│    • AnimeSharp 2x (Line-Art Restore)  │
│    • MobileClear (Laplacian DSP)       │
└────────────────────────────────────────┘
           │
           ▼
┌────────────────────────────────────────┐
│ 3. Foreground Service & Thermal Policy │
│    • Android Game Mode GPU Lock        │
│    • Chunked In-Memory Frame Buffer    │
└────────────────────────────────────────┘
           │
           ▼
┌────────────────────────────────────────┐
│ 4. Hardware Encoder & Muxer            │
│    • MediaMuxer + MediaCodec (H.264)   │
│    • Audio Stream Preservation         │
└────────────────────────────────────────┘
           │
           ▼
[ Output 2x / 4K Enhanced Video ]
Enter fullscreen mode Exit fullscreen mode

1. The Model Suite: Balancing Quality, Compute, and Model Footprint

Deploying deep learning models to edge devices requires matching the model complexity to the user's specific degradation profile and hardware tier.

A. Real-ESRGAN Cinematic 4x (Photorealistic Texture Synthesis)

  • Model Size: 4.8 MB (Optimized compact architecture)
  • Scale Factor: 4×
  • Architecture: Compact Residual-in-Residual Dense Block (RRDB) network optimized with FP16 precision.
  • Role: Reconstructs fine high-frequency details (foliage, facial pores, fabric weave) from compressed or downscaled source clips.

B. OpenCV NAFNet Unblur & Restore 1x (Motion & Defocus Inversion)

  • Model Size: 87.5 MB
  • Scale Factor: 1× (Preserves native canvas dimensions)
  • Architecture: Nonlinear Activation Free Network. By replacing conventional non-linear activations (like ReLU or GELU) with simple element-wise multiplications (Gated Mechanism) and simplified channel attention, NAFNet achieves state-of-the-art deblurring performance with a fraction of the computational complexity of transformer-based restoration models.
  • Role: Eliminates camera motion shake, vehicle motion blur, and out-of-focus camera distortion.

C. MobileSR 2x (Ultra-Fast Lightweight Super-Resolution)

  • Model Size: 7.6 MB (Pre-bundled in app assets)
  • Scale Factor: 2×
  • Architecture: Shallow convolutional neural network tuned for sub-second frame inference.
  • Role: High-speed upscaling designed for mid-range and budget devices where thermal throttling and battery consumption are top priorities.

D. MobileClear Fast 1x (Native DSP Engine)

  • Model Size: 0 MB (Zero external download required)
  • Scale Factor: 1×
  • Architecture: Native digital signal processing leveraging Laplacian high-pass spatial filtering and contrast-adaptive unsharp masking.
  • Role: Instant real-time clarity boosting without requiring neural tensor allocation.

2. Solving Mobile Constraints: Memory, Thermal, and Throttling

Running deep convolutional networks on video frames introduces severe challenges that don't exist in desktop computing:

Memory Management & Out-Of-Memory (OOM) Prevention

A naive implementation might load multiple frames into a queue as uncompressed 32-bit bitmaps. A single 1080p frame consumes ~8.3 MB of RAM; buffering 60 frames would easily breach Android's per-app memory limit (256MB–512MB on standard devices).

Solution: OnDevice implements a strict backpressure pipeline. The decoder extracts a single frame, copies the normalized tensor directly into the ONNX Runtime input tensor, executes inference, and pipes the output directly to the hardware encoder's input surface before releasing the buffer. Peak RAM usage remains constant regardless of whether the video is 10 seconds or 10 minutes long.

Thermal Throttling & Android Game Mode

Long-running AI workloads cause mobile SoCs to quickly heat up, triggering Android's thermal daemon to throttle CPU/GPU frequencies down to 30% of their base clocks.

Solution: The app implements the Android Game Mode API (android.game_mode_config) and utilizes foreground wake locks. By signaling the OS that the application is performing an intensive, continuous compute task, the scheduler prevents erratic frequency downscaling, ensuring consistent FPS throughput across long export sessions.

3. Performance Benchmarks

Typical frame inference speeds across various mobile silicon tiers:

Device Chipset Architecture Tier Model Average Per-Frame Latency 10-Second 30 FPS Clip Time
Snapdragon 8 Gen 3 Flagship Real-ESRGAN Compact 4x ~85 ms ~25.5 seconds
Snapdragon 8 Gen 2 Flagship MobileSR 2x ~32 ms ~9.6 seconds
Dimensity 9200+ Flagship NAFNet Unblur 1x ~140 ms ~42.0 seconds
Snapdragon 778G Mid-Range MobileSR 2x ~65 ms ~19.5 seconds
Helio G99 / Budget Budget MobileClear Fast 1x ~8 ms ~2.4 seconds

4. Why On-Device AI is the Future of Mobile Multimedia

The success of OnDevice - Video Enhancer demonstrates that consumer mobile devices have reached the performance inflection point where desktop-class neural restoration models can run entirely locally.

By eliminating remote server infrastructure:

  • Users gain ironclad privacy: Family memories and sensitive footage never traverse the internet.
  • Zero latency over mobile networks: The app works on a flight, in a rural cabin, or with no SIM card installed.
  • Fair economics: Without multi-thousand-dollar monthly server bills, developers can offer perpetual licenses rather than predatory weekly subscriptions.

How to Try It

OnDevice - Video Enhancer is built using modern Kotlin, Jetpack Compose Material 3, and ONNX Runtime. The app is publicly available on the Google Play Store:

  • Package Name: com.nabil.ondevice.videoenhancer
  • Minimum Requirements: Android 8.0 (Oreo) or higher; 4GB RAM recommended for 4x neural upscaling.
  • Play Store Link: OnDevice - Video Enhancer on Google Play

Top comments (0)