<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Jose</title>
    <description>The latest articles on DEV Community by Jose (@jagx4669).</description>
    <link>https://dev.to/jagx4669</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4248449%2Fdbbf7161-af95-40cb-a12d-f9dc65daf10f.png</url>
      <title>DEV Community: Jose</title>
      <link>https://dev.to/jagx4669</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/jagx4669"/>
    <language>en</language>
    <item>
      <title>Benchmarking Batch YOLO Object Detection on AWS</title>
      <dc:creator>Jose</dc:creator>
      <pubDate>Fri, 20 Nov 2026 05:19:49 +0000</pubDate>
      <link>https://dev.to/jagx4669/benchmarking-batch-yolo-object-detection-on-aws-4bj6</link>
      <guid>https://dev.to/jagx4669/benchmarking-batch-yolo-object-detection-on-aws-4bj6</guid>
      <description>&lt;h1&gt;
  
  
  Overview
&lt;/h1&gt;

&lt;p&gt;This is my first step into the Computer Vision MLOps Engineering field.  I wanted to explore an MLOps subject that I might see as a senior Computer Vision or MLOps engineer:  &lt;/p&gt;

&lt;p&gt;Process a large set of images from COCO 2017 validation set(5000 images) as cheaply and quickly as possible on AWS. Images land in S3, queue up in SQS, and workers on ECS run them through YOLOv8m in batches, writing results to S3 and DynamoDB. The project is about throughput, cost and reliability for bulk work, not per-request latency.&lt;/p&gt;

&lt;h1&gt;
  
  
  Source Code
&lt;/h1&gt;

&lt;p&gt;&lt;a href="https://github.com/jose-guevarra/batch-obj-det-aws-public" rel="noopener noreferrer"&gt;Github Repo&lt;/a&gt;&lt;/p&gt;

&lt;h1&gt;
  
  
  Goals
&lt;/h1&gt;

&lt;ul&gt;
&lt;li&gt;Find the best batch size: the batch size on a T4 GPU (g4dn.xlarge) that gives the most images per second and the lowest cost per 1,000 images, and the point where bigger batches stop helping.&lt;/li&gt;
&lt;li&gt;Quantify GPU vs. CPU for bulk work: how much cheaper and faster a 5,000-image job is on a GPU worker than on a CPU worker.&lt;/li&gt;
&lt;li&gt;Measure an optimized runtime: what ONNX Runtime adds on top of batching, without losing accuracy.&lt;/li&gt;
&lt;li&gt;Build a pipeline you'd trust in production: it scales with queue depth, isolates bad images in a dead-letter queue, and can be re-run without processing any image twice.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1&gt;
  
  
  Architecture Diagram
&lt;/h1&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4l0if8q8whsuubq1f6cx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4l0if8q8whsuubq1f6cx.png" alt="High-level Architecture" width="798" height="147"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h1&gt;
  
  
  Headline results
&lt;/h1&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Best setup:&lt;/strong&gt; pipelined PyTorch FP16 at batch 32 processes &lt;strong&gt;85.7 images/s for $0.0017 per 1,000 images&lt;/strong&gt; on one &lt;code&gt;g4dn.xlarge&lt;/code&gt;. Its mAP50-95 is 0.5021, against 0.5026 for FP32 (the published YOLOv8m value is 0.502). It's the GPU service's default.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GPU vs. CPU:&lt;/strong&gt; about &lt;strong&gt;30× faster and 20× cheaper&lt;/strong&gt; per image than an unoptimized CPU baseline at a similar hourly price (&lt;code&gt;c6i.2xlarge&lt;/code&gt;: 2.8 images/s, $0.0339 per 1,000).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pipelining beat batch size.&lt;/strong&gt; Larger batches took FP32 from 6 to 27 images/s, but the GPU was still only 49% busy. Overlapping download/decode, inference and writes in separate threads doubled that to 52 images/s with the GPU 93% busy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;TensorRT is the open issue.&lt;/strong&gt; It has the fastest model execution (5.4 ms per image vs. 10 ms for PyTorch FP16) but the slowest end-to-end FP16 result (53.7 images/s), because image loading slows down while it runs.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1&gt;
  
  
  Inference Pipeline Setups
&lt;/h1&gt;

&lt;p&gt;&lt;strong&gt;One step at a time:&lt;/strong&gt; the GPU (green) is busy only during each batch's &lt;em&gt;infer&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F585rnfqahf2vi7b0yxgl.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F585rnfqahf2vi7b0yxgl.png" alt="Processing Images All steps in sequence." width="800" height="157"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pipelined:&lt;/strong&gt; each step runs in its own thread, so the GPU goes straight from one batch to the&lt;br&gt;
next.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm5z9s5hqv1ijrvuqe84d.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm5z9s5hqv1ijrvuqe84d.png" alt="Batch Processing images on the GPU." width="800" height="234"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h1&gt;
  
  
  Results
&lt;/h1&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Setup&lt;/th&gt;
&lt;th&gt;Precision&lt;/th&gt;
&lt;th&gt;Batch size&lt;/th&gt;
&lt;th&gt;Images/sec&lt;/th&gt;
&lt;th&gt;Time for N images&lt;/th&gt;
&lt;th&gt;Cost per 1,000 images&lt;/th&gt;
&lt;th&gt;Avg GPU utilization&lt;/th&gt;
&lt;th&gt;mAP50-95&lt;/th&gt;
&lt;th&gt;Failed images&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;ECS workers, CPU (&lt;code&gt;c6i.2xlarge&lt;/code&gt;, PyTorch, unoptimized)&lt;/td&gt;
&lt;td&gt;FP32&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;2.8&lt;/td&gt;
&lt;td&gt;29 min 57 s (5,000)&lt;/td&gt;
&lt;td&gt;$0.0339&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;0.5026&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ECS workers, GPU (&lt;code&gt;g4dn.xlarge&lt;/code&gt;), PyTorch&lt;/td&gt;
&lt;td&gt;FP32&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;6.0&lt;/td&gt;
&lt;td&gt;14 min 00 s (5,000)&lt;/td&gt;
&lt;td&gt;$0.0245&lt;/td&gt;
&lt;td&gt;11%&lt;/td&gt;
&lt;td&gt;0.5026&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ECS workers, GPU (&lt;code&gt;g4dn.xlarge&lt;/code&gt;), PyTorch&lt;/td&gt;
&lt;td&gt;FP32&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;18.3&lt;/td&gt;
&lt;td&gt;4 min 33 s (5,000)&lt;/td&gt;
&lt;td&gt;$0.0080&lt;/td&gt;
&lt;td&gt;39%&lt;/td&gt;
&lt;td&gt;0.5026&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ECS workers, GPU (&lt;code&gt;g4dn.xlarge&lt;/code&gt;), PyTorch&lt;/td&gt;
&lt;td&gt;FP32&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;22.4&lt;/td&gt;
&lt;td&gt;3 min 43 s (5,000)&lt;/td&gt;
&lt;td&gt;$0.0065&lt;/td&gt;
&lt;td&gt;44%&lt;/td&gt;
&lt;td&gt;0.5026&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ECS workers, GPU (&lt;code&gt;g4dn.xlarge&lt;/code&gt;), PyTorch&lt;/td&gt;
&lt;td&gt;FP32&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;25.5&lt;/td&gt;
&lt;td&gt;3 min 16 s (5,000)&lt;/td&gt;
&lt;td&gt;$0.0057&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;td&gt;0.5026&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ECS workers, GPU (&lt;code&gt;g4dn.xlarge&lt;/code&gt;), PyTorch&lt;/td&gt;
&lt;td&gt;FP32&lt;/td&gt;
&lt;td&gt;64&lt;/td&gt;
&lt;td&gt;26.9&lt;/td&gt;
&lt;td&gt;3 min 06 s (5,000)&lt;/td&gt;
&lt;td&gt;$0.0054&lt;/td&gt;
&lt;td&gt;49%&lt;/td&gt;
&lt;td&gt;0.5026&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ECS workers, GPU (&lt;code&gt;g4dn.xlarge&lt;/code&gt;), PyTorch, pipelined&lt;/td&gt;
&lt;td&gt;FP32&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;46.8&lt;/td&gt;
&lt;td&gt;1 min 47 s (5,000)&lt;/td&gt;
&lt;td&gt;$0.0031&lt;/td&gt;
&lt;td&gt;94%&lt;/td&gt;
&lt;td&gt;0.5026&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ECS workers, GPU (&lt;code&gt;g4dn.xlarge&lt;/code&gt;), PyTorch, pipelined&lt;/td&gt;
&lt;td&gt;FP32&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;51.8&lt;/td&gt;
&lt;td&gt;1 min 37 s (5,000)&lt;/td&gt;
&lt;td&gt;$0.0028&lt;/td&gt;
&lt;td&gt;93%&lt;/td&gt;
&lt;td&gt;0.5026&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ECS workers, GPU (&lt;code&gt;g4dn.xlarge&lt;/code&gt;), PyTorch, pipelined&lt;/td&gt;
&lt;td&gt;FP32&lt;/td&gt;
&lt;td&gt;64&lt;/td&gt;
&lt;td&gt;48.5&lt;/td&gt;
&lt;td&gt;1 min 43 s (5,000)&lt;/td&gt;
&lt;td&gt;$0.0030&lt;/td&gt;
&lt;td&gt;92%&lt;/td&gt;
&lt;td&gt;0.5026&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ECS workers, GPU (&lt;code&gt;g4dn.xlarge&lt;/code&gt;), PyTorch, pipelined&lt;/td&gt;
&lt;td&gt;FP16&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;85.7&lt;/td&gt;
&lt;td&gt;58 s (5,000)&lt;/td&gt;
&lt;td&gt;$0.0017&lt;/td&gt;
&lt;td&gt;83%&lt;/td&gt;
&lt;td&gt;0.5021&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ECS workers, GPU (&lt;code&gt;g4dn.xlarge&lt;/code&gt;), ONNX Runtime (CUDA), pipelined&lt;/td&gt;
&lt;td&gt;FP16&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;81.8&lt;/td&gt;
&lt;td&gt;1 min 01 s (5,000)&lt;/td&gt;
&lt;td&gt;$0.0018&lt;/td&gt;
&lt;td&gt;81%&lt;/td&gt;
&lt;td&gt;0.5021&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ECS workers, GPU (&lt;code&gt;g4dn.xlarge&lt;/code&gt;), ONNX Runtime (TensorRT), pipelined&lt;/td&gt;
&lt;td&gt;FP16&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;53.7&lt;/td&gt;
&lt;td&gt;1 min 33 s (5,000)&lt;/td&gt;
&lt;td&gt;$0.0027&lt;/td&gt;
&lt;td&gt;28%&lt;/td&gt;
&lt;td&gt;0.5022&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsyx0yuuikheapfa63w1o.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsyx0yuuikheapfa63w1o.png" alt="Images per second and Cost of 1000 images by batch size." width="800" height="336"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvvystnqxebyee0u04ym7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvvystnqxebyee0u04ym7.png" alt="Inference Rate and cost per 1000 images by Pipeline Setup" width="800" height="348"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  StreamLit App ScreenShot
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6gaakjm14ykigt9gq46s.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6gaakjm14ykigt9gq46s.png" alt="Batch Results" width="800" height="878"&gt;&lt;/a&gt;&lt;/p&gt;

</description>
      <category>aws</category>
      <category>infrastructure</category>
      <category>machinelearning</category>
      <category>performance</category>
    </item>
  </channel>
</rss>
