Beck_Moulton

Posted on Jun 3

Beyond the Cloud: Building an Ultra-Fast Skin Lesion Screener with MobileNetV3 and TensorFlow Lite

#android #machinelearning #ai #tensorflow

In the world of digital health, privacy and latency aren't just features—they are requirements. When dealing with sensitive medical data like skin images, sending every pixel to a cloud server is often a bottleneck for user experience and a nightmare for data sovereignty.

Today, we are diving deep into on-device machine learning to build a skin lesion screening tool. By leveraging MobileNetV3, TensorFlow Lite (TFLite), and MediaPipe, we can achieve near-zero latency and high-accuracy classification directly on a smartphone. This tutorial covers the end-to-end pipeline: from training a lightweight CNN to deploying it as a high-performance mobile application.

Why On-Device ML for Dermatology?

Cloud-based AI is powerful, but edge AI offers three critical advantages:

Privacy: Sensitive medical images never leave the user's device.
Offline Capability: Works in remote areas without an internet connection.
Real-time Feedback: Instant results (sub-100ms) allow for better UX and follow-up tracking.

To build a robust production-ready system, we focus on MobileNetV3, which uses hardware-aware platform-aware NAS (Neural Architecture Search) to optimize performance for mobile CPUs.

The Architecture Flow

Before we write code, let’s visualize how the data moves from the camera sensor to the classification result.

graph TD
    A[Camera Input / Gallery Image] --> B[MediaPipe Image Processing]
    B --> C{Region of Interest?}
    C -- Yes --> D[Resize to 224x224]
    C -- No --> A
    D --> E[MobileNetV3 TFLite Model]
    E --> F[Softmax Classification Output]
    F --> G[Threshold Validation]
    G --> H[UI: Result & Follow-up Recommendation]

    subgraph On-Device Execution
    B
    D
    E
    F
    end

Prerequisites

To follow along, you’ll need:

Tech Stack: Python 3.9+, TensorFlow 2.x, Keras.
Mobile: Android Studio (Kotlin) or Xcode (Swift).
Dataset: A dataset like HAM10000 (ensure you have the rights to use it).

Step 1: Training the Lightweight Champion (Keras)

We use MobileNetV3Small because it provides an incredible balance between accuracy and parameter count.

import tensorflow as tf
from tensorflow.keras import layers, models

def build_model(num_classes):
    # Use MobileNetV3Small for maximum efficiency
    base_model = tf.keras.applications.MobileNetV3Small(
        input_shape=(224, 224, 3),
        include_top=False,
        weights='imagenet'
    )

    # Freeze the base model for transfer learning
    base_model.trainable = False

    model = models.Sequential([
        base_model,
        layers.GlobalAveragePooling2D(),
        layers.Dense(128, activation='relu'),
        layers.Dropout(0.2),
        layers.Dense(num_classes, activation='softmax')
    ])

    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
    return model

# Assume we have 7 classes of skin lesions
model = build_model(num_classes=7)
print(model.summary())

Step 2: Optimizing for the Edge (TFLite Conversion)

Standard models are too heavy for mobile memory. We use Float16 Quantization to reduce the model size by ~50% with negligible accuracy loss.

# Convert the Keras model to TFLite
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16]

tflite_model = converter.convert()

# Save the model
with open('skin_screener.tflite', 'wb') as f:
    f.write(tflite_model)

Pro-Tip: For even more advanced patterns and production-ready architectures—including int8 quantization for DSP acceleration—be sure to check out the technical deep dives at the WellAlly Blog. It’s a goldmine for scaling AI health tech.

Step 3: Mobile Implementation (Kotlin Example)

On Android, we use the TensorFlow Lite Task Library to simplify the inference logic.

// Initialize the TFLite Interpreter
val options = Interpreter.Options().apply {
    setNumThreads(4)
    useNNAPI = true // Use hardware acceleration
}

val tflite = Interpreter(loadModelFile("skin_screener.tflite"), options)

fun runInference(bitmap: Bitmap): String {
    // 1. Preprocess: Resize bitmap to 224x224
    val resizedBitmap = Bitmap.createScaledBitmap(bitmap, 224, 224, true)

    // 2. Convert Bitmap to ByteBuffer
    val inputBuffer = convertBitmapToByteBuffer(resizedBitmap)

    // 3. Prepare Output Buffer (7 classes)
    val outputArray = Array(1) { FloatArray(7) }

    // 4. Run Inference
    tflite.run(inputBuffer, outputArray)

    // 5. Post-process: Get highest probability class
    return labels[outputArray[0].indices.maxByOrNull { outputArray[0][it] } ?: 0]
}

Step 4: Enhancing Precision with MediaPipe

A common issue in dermatology apps is "junk in, junk out." Users might take photos of their dog or a blurry background. We can use MediaPipe Object Detection to ensure a "Skin Lesion" is actually in the frame before running our classification.

import mediapipe as mp

# Initialize MediaPipe for custom ROI detection
mp_hands = mp.solutions.hands # Or use a custom TFLite model for skin patches
# This ensures the user is pointing at a specific area

Conclusion: The Future of Pocket Diagnostics

Building a skin lesion screener isn't just about the model; it's about the pipeline. By combining MobileNetV3's efficiency with TFLite's optimization, we bring powerful diagnostic tools directly to the user's pocket.

Summary Checklist for your project:

[x] Use MobileNetV3 for the base.
[x] Apply Float16 or Int8 quantization.
[x] Implement hardware acceleration (NNAPI/GPU Delegate).
[x] Add a pre-processing validation step with MediaPipe.

If you are looking for more comprehensive guides on HIPAA-compliant cloud synchronization or integrating these models with wearable data, head over to wellally.tech/blog.

What are you building next with On-Device ML? Let me know in the comments! 👇

DEV Community