Real-Time Object Detection and Tracking with Kotlin and YOLO on Android

Real-time computer vision is one of the most useful applications of machine learning on mobile devices. Android phones provide cameras, GPU acceleration, and enough processing power to run optimized detection models locally.

In this tutorial, we will build the architecture for a Kotlin application that captures camera frames, runs a YOLO-style object detector, and tracks detected objects across frames.

Architecture

CameraX
   |
ImageAnalysis
   |
Frame Conversion
   |
YOLO Detector
   |
Non-Maximum Suppression
   |
Object Tracker
   |
UI Overlay

The important part is to keep camera capture, inference, tracking, and rendering separate.

CameraX Setup

Add the CameraX dependencies compatible with your project:

dependencies {
    implementation("androidx.camera:camera-camera2:<version>")
    implementation("androidx.camera:camera-lifecycle:<version>")
    implementation("androidx.camera:camera-view:<version>")
}

Create an ImageAnalysis use case:

val analysis = ImageAnalysis.Builder()
    .setBackpressureStrategy(
        ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST
    )
    .build()

KEEP_ONLY_LATEST is important for real-time applications. If inference is slower than the camera, processing every old frame creates a growing queue and increases latency.

Analyzer

Implement an analyzer that receives camera frames:

class ObjectAnalyzer(
    private val detector: YoloDetector
) : ImageAnalysis.Analyzer {

    override fun analyze(image: ImageProxy) {
        try {
            detector.detect(image)
        } finally {
            image.close()
        }
    }
}

Always close ImageProxy when processing is complete.

YOLO Detector

Your detector should convert camera input into the tensor format expected by your model.

Conceptually:

class YoloDetector {

    fun detect(image: ImageProxy): List<Detection> {
        val input = preprocess(image)
        val output = runModel(input)

        return postprocess(output)
    }
}

A detection normally contains a bounding box, class ID, confidence score, and optionally additional metadata.

data class Detection(
    val classId: Int,
    val confidence: Float,
    val box: RectF
)

Preprocessing

Most object detection models expect a fixed input size.

For example:

Camera frame
    ↓
Rotate
    ↓
Crop / resize
    ↓
Normalize
    ↓
Tensor

Be careful with aspect ratio. Incorrect scaling can make objects appear distorted and reduce detection accuracy.

Postprocessing

YOLO-style models can return many candidate boxes. Filter low-confidence detections:

val filtered = detections.filter {
    it.confidence >= 0.5f
}

Then apply non-maximum suppression (NMS) to remove overlapping boxes representing the same object.

The exact threshold should be tuned using your target model and dataset.

Tracking Objects Across Frames

Detection tells you what is visible in a frame. Tracking attempts to maintain an object's identity across multiple frames.

A simplified tracked object could be:

data class TrackedObject(
    val id: Int,
    var box: RectF,
    var classId: Int,
    var confidence: Float
)

A basic tracker can associate the current detection with an existing object using Intersection over Union (IoU).

fun iou(a: RectF, b: RectF): Float {
    val left = maxOf(a.left, b.left)
    val top = maxOf(a.top, b.top)
    val right = minOf(a.right, b.right)
    val bottom = minOf(a.bottom, b.bottom)

    if (right <= left || bottom <= top) return 0f

    val intersection =
        (right - left) * (bottom - top)

    val union =
        a.width() * a.height() +
        b.width() * b.height() -
        intersection

    return intersection / union
}

For more robust tracking, consider established algorithms such as SORT or ByteTrack.

Drawing Bounding Boxes

Use a custom Android view to draw detections over the camera preview.

class DetectionOverlay : View(context) {

    var detections: List<Detection> = emptyList()

    override fun onDraw(canvas: Canvas) {
        super.onDraw(canvas)

        for (detection in detections) {
            canvas.drawRect(
                detection.box,
                paint
            )
        }
    }
}

You will need coordinate transformation because model coordinates and preview coordinates may differ.

Improving FPS

Real-time inference requires careful optimization.

Useful techniques include:

• Resize frames before inference
• Process only the latest frame
• Avoid unnecessary bitmap allocations
• Reuse buffers
• Use hardware acceleration where supported
• Quantize the model
• Reduce model input resolution
• Run inference off the UI thread

Do not optimize only for FPS. A smaller model with slightly lower accuracy may provide a much better mobile user experience.

Measuring Performance

Track at least:

Camera FPS
Inference latency
End-to-end latency
Memory usage
CPU/GPU utilization

For example:

val start = System.nanoTime()

val detections = detector.detect(image)

val elapsedMs =
    (System.nanoTime() - start) / 1_000_000

Use real devices for benchmarking because emulator performance does not represent typical mobile hardware.

Handling Rotation

Camera frames can have different orientations depending on the device.

Pass rotation information into preprocessing:

val rotation = image.imageInfo.rotationDegrees

Failing to handle rotation correctly can produce incorrect bounding boxes or significantly reduce accuracy.

Production Considerations

A production application should also handle:

• Camera permission failures
• Model initialization errors
• Unsupported hardware
• Low-memory devices
• App backgrounding
• Camera lifecycle changes
• Device rotation

Conclusion

Combining Kotlin, CameraX, YOLO, and object tracking creates a powerful on-device computer vision pipeline.

The key to a production-ready implementation is not simply running a model. You must also control frame backpressure, coordinate transformations, memory allocations, inference latency, and camera lifecycle.

This foundation can be extended into applications such as traffic monitoring, retail analytics, industrial inspection, sports analysis, and smart navigation.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx