DEV Community

Jose
Jose

Posted on

Benchmarking Batch YOLO Object Detection on AWS

Overview

This is my first step into the Computer Vision MLOps Engineering field. I wanted to explore an MLOps subject that I might see as a senior Computer Vision or MLOps engineer:

Process a large set of images from COCO 2017 validation set(5000 images) as cheaply and quickly as possible on AWS. Images land in S3, queue up in SQS, and workers on ECS run them through YOLOv8m in batches, writing results to S3 and DynamoDB. The project is about throughput, cost and reliability for bulk work, not per-request latency.

Source Code

Github Repo

Goals

  • Find the best batch size: the batch size on a T4 GPU (g4dn.xlarge) that gives the most images per second and the lowest cost per 1,000 images, and the point where bigger batches stop helping.
  • Quantify GPU vs. CPU for bulk work: how much cheaper and faster a 5,000-image job is on a GPU worker than on a CPU worker.
  • Measure an optimized runtime: what ONNX Runtime adds on top of batching, without losing accuracy.
  • Build a pipeline you'd trust in production: it scales with queue depth, isolates bad images in a dead-letter queue, and can be re-run without processing any image twice.

Architecture Diagram

High-level Architecture

Headline results

  • Best setup: pipelined PyTorch FP16 at batch 32 processes 85.7 images/s for $0.0017 per 1,000 images on one g4dn.xlarge. Its mAP50-95 is 0.5021, against 0.5026 for FP32 (the published YOLOv8m value is 0.502). It's the GPU service's default.
  • GPU vs. CPU: about 30× faster and 20× cheaper per image than an unoptimized CPU baseline at a similar hourly price (c6i.2xlarge: 2.8 images/s, $0.0339 per 1,000).
  • Pipelining beat batch size. Larger batches took FP32 from 6 to 27 images/s, but the GPU was still only 49% busy. Overlapping download/decode, inference and writes in separate threads doubled that to 52 images/s with the GPU 93% busy.
  • TensorRT is the open issue. It has the fastest model execution (5.4 ms per image vs. 10 ms for PyTorch FP16) but the slowest end-to-end FP16 result (53.7 images/s), because image loading slows down while it runs.

Inference Pipeline Setups

One step at a time: the GPU (green) is busy only during each batch's infer.

Processing Images All steps in sequence.

Pipelined: each step runs in its own thread, so the GPU goes straight from one batch to the
next.

Batch Processing images on the GPU.

Results

Setup Precision Batch size Images/sec Time for N images Cost per 1,000 images Avg GPU utilization mAP50-95 Failed images
ECS workers, CPU (c6i.2xlarge, PyTorch, unoptimized) FP32 1 2.8 29 min 57 s (5,000) $0.0339 — 0.5026 0
ECS workers, GPU (g4dn.xlarge), PyTorch FP32 1 6.0 14 min 00 s (5,000) $0.0245 11% 0.5026 0
ECS workers, GPU (g4dn.xlarge), PyTorch FP32 8 18.3 4 min 33 s (5,000) $0.0080 39% 0.5026 0
ECS workers, GPU (g4dn.xlarge), PyTorch FP32 16 22.4 3 min 43 s (5,000) $0.0065 44% 0.5026 0
ECS workers, GPU (g4dn.xlarge), PyTorch FP32 32 25.5 3 min 16 s (5,000) $0.0057 46% 0.5026 0
ECS workers, GPU (g4dn.xlarge), PyTorch FP32 64 26.9 3 min 06 s (5,000) $0.0054 49% 0.5026 0
ECS workers, GPU (g4dn.xlarge), PyTorch, pipelined FP32 16 46.8 1 min 47 s (5,000) $0.0031 94% 0.5026 0
ECS workers, GPU (g4dn.xlarge), PyTorch, pipelined FP32 32 51.8 1 min 37 s (5,000) $0.0028 93% 0.5026 0
ECS workers, GPU (g4dn.xlarge), PyTorch, pipelined FP32 64 48.5 1 min 43 s (5,000) $0.0030 92% 0.5026 0
ECS workers, GPU (g4dn.xlarge), PyTorch, pipelined FP16 32 85.7 58 s (5,000) $0.0017 83% 0.5021 0
ECS workers, GPU (g4dn.xlarge), ONNX Runtime (CUDA), pipelined FP16 32 81.8 1 min 01 s (5,000) $0.0018 81% 0.5021 0
ECS workers, GPU (g4dn.xlarge), ONNX Runtime (TensorRT), pipelined FP16 32 53.7 1 min 33 s (5,000) $0.0027 28% 0.5022 0

Images per second and Cost of 1000 images by batch size.

Inference Rate and cost per 1000 images by Pipeline Setup

StreamLit App ScreenShot

Batch Results

Top comments (0)