<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Hemkesh</title>
    <description>The latest articles on DEV Community by Hemkesh (@hemkesh2021dotcom).</description>
    <link>https://dev.to/hemkesh2021dotcom</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3989202%2Fcf128d60-77b0-44be-857b-a30fc5f74c11.png</url>
      <title>DEV Community: Hemkesh</title>
      <link>https://dev.to/hemkesh2021dotcom</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/hemkesh2021dotcom"/>
    <language>en</language>
    <item>
      <title>Your slowest model shouldn't make your fastest decisions: lessons from an edge AI camera on a Jetson</title>
      <dc:creator>Hemkesh</dc:creator>
      <pubDate>Tue, 17 Nov 2026 12:33:41 +0000</pubDate>
      <link>https://dev.to/hemkesh2021dotcom/your-slowest-model-shouldnt-make-your-fastest-decisions-lessons-from-an-edge-ai-camera-on-a-jetson-5ep6</link>
      <guid>https://dev.to/hemkesh2021dotcom/your-slowest-model-shouldnt-make-your-fastest-decisions-lessons-from-an-edge-ai-camera-on-a-jetson-5ep6</guid>
      <description>&lt;p&gt;One camera. One NVIDIA Jetson Orin Nano Super. Every model runs on the board.&lt;/p&gt;

&lt;p&gt;That was the brief for &lt;strong&gt;Sentinel&lt;/strong&gt;, our final-year project at Dayananda Sagar University: spot people, follow them across the frame, and open an incident when someone steps into a zone you care about. No cloud AI anywhere in the loop.&lt;/p&gt;

&lt;p&gt;We expected the hard part to be the models. It wasn't.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;See it first:&lt;/strong&gt; the &lt;a href="https://hemkesh2021-dotcom.github.io/sentinel-site/" rel="noopener noreferrer"&gt;project website&lt;/a&gt; has a hardware tour and a demo dashboard you can click around in. It is the real dashboard code replaying a scripted scene, so no Jetson is needed.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft4wuhjbcxlf27mwf5cng.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft4wuhjbcxlf27mwf5cng.gif" alt="Sentinel dashboard tour, recorded with synthetic demonstration data" width="800" height="556"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The dashboard with synthetic demo data. No camera or model was running.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The pipeline
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Step&lt;/th&gt;
&lt;th&gt;What does it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;See&lt;/td&gt;
&lt;td&gt;YOLOv8n on TensorRT detects people&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Follow&lt;/td&gt;
&lt;td&gt;ByteTrack keeps each person's track&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Decide&lt;/td&gt;
&lt;td&gt;Zone rules open incidents in a local SQLite database&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tell&lt;/td&gt;
&lt;td&gt;A restart-safe outbox delivers alerts (Telegram in our tests)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show&lt;/td&gt;
&lt;td&gt;A dashboard that listens on 127.0.0.1 only&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two optional lanes sit beside it: face recognition (YuNet + Facenet512) and a local vision-language model (LFM3-VL-2B through llama.cpp). Both are still under validation, so they are off by default. More on that below.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lesson 1: your slowest model shouldn't make your fastest decisions
&lt;/h2&gt;

&lt;p&gt;On a small board, models finish at very different speeds. The detector keeps up with the stream. Face matching is slower. A vision-language model is slower still.&lt;/p&gt;

&lt;p&gt;Now imagine the camera drops and reconnects. A slow model is still chewing on a frame from &lt;em&gt;before&lt;/em&gt; the reconnect. When its answer finally lands, the naive pipeline treats it as news about &lt;em&gt;now&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;That is how an old frame fires a new alert.&lt;/p&gt;

&lt;p&gt;Our rule: &lt;strong&gt;every observation is evidence about one specific frame, never about "now".&lt;/strong&gt; So every observation carries three stamps:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the frame's &lt;strong&gt;source time&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;the &lt;strong&gt;boot ID&lt;/strong&gt; of the run that produced it&lt;/li&gt;
&lt;li&gt;a &lt;strong&gt;stream epoch&lt;/strong&gt; that changes on every reconnect&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Before any result can drive a decision, it has to pass a check like this (a simplified sketch, not the project's exact code):&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;def can_drive_live_decision(obs, now, current):
    if obs.boot_id != current.boot_id:
        return False          # from a previous run
    if obs.stream_epoch != current.stream_epoch:
        return False          # from before a reconnect
    if now - obs.frame_time &amp;gt; MAX_AGE:
        return False          # simply too old
    return True
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;A result that fails is not thrown away. It can still annotate history. It just never gets to act on the present.&lt;/p&gt;

&lt;p&gt;One more detail that mattered: age is measured with an injected monotonic clock, not wall time. That makes the rule testable. A fake clock lets a test replay "the model answered late" without waiting for a real model to be slow.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lesson 2: "I don't know" is a valid answer
&lt;/h2&gt;

&lt;p&gt;A monitoring system that guesses is worse than one that stays quiet. Three rules came out of that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A face is tied to &lt;strong&gt;at most one&lt;/strong&gt; tracked person. If ownership is ambiguous, it stays unresolved.&lt;/li&gt;
&lt;li&gt;An unknown face is &lt;strong&gt;never&lt;/strong&gt; labelled a stranger. Not recognising someone is not evidence about who they are.&lt;/li&gt;
&lt;li&gt;Fire or smoke reported by the scene model alone is &lt;strong&gt;never&lt;/strong&gt; a critical alarm.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The dashboard follows the same idea. Empty, fresh, stale and offline are four different states, and it shows each one explicitly instead of leaving an old picture on screen.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lesson 3: an alert that dies with the process was never an alert
&lt;/h2&gt;

&lt;p&gt;An in-memory queue loses everything on a restart. So incidents and their pending alerts are written to SQLite, and a worker delivers from that outbox with retries. If Sentinel restarts between "incident opened" and "alert sent", the alert still goes out afterwards.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lesson 4: test without the hardware
&lt;/h2&gt;

&lt;p&gt;Most of Sentinel can be tested on a laptop. The portable suite needs no camera, GPU or model, and CI runs it on every push under Python 3.10 and 3.12. On the commit the beta was built from, 4,224 tests passed.&lt;/p&gt;

&lt;p&gt;The trick is keeping hardware adapters at the edges, so importing the core never requires CUDA or TensorRT.&lt;/p&gt;

&lt;h2&gt;
  
  
  What isn't done
&lt;/h2&gt;

&lt;p&gt;Sentinel is an experimental beta, and we would rather say so than oversell it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Face recognition has not passed its formal validation.&lt;/li&gt;
&lt;li&gt;Chat with the local vision-language model is not admitted for normal use.&lt;/li&gt;
&lt;li&gt;The install package has run only on our own Jetson.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The README keeps a feature table that says what is admitted and what is not, with the evidence behind each claim.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;In a browser:&lt;/strong&gt; the &lt;a href="https://hemkesh2021-dotcom.github.io/sentinel-site/" rel="noopener noreferrer"&gt;project website&lt;/a&gt; and its demo dashboard&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On any computer with Python 3.10+:&lt;/strong&gt; clone the repo and run the test suite&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On a Jetson Orin Nano:&lt;/strong&gt; the experimental install package in the repo&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Code (MIT): &lt;a href="https://github.com/hemkesh2021-dotcom/Sentinel_Surveillance" rel="noopener noreferrer"&gt;https://github.com/hemkesh2021-dotcom/Sentinel_Surveillance&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If you run a slow model next to a fast one, how do you keep the slow one's answers honest? I'd like to hear how others handle it.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>computervision</category>
      <category>opensource</category>
      <category>nvidia</category>
    </item>
    <item>
      <title>Why cudaMalloc fails on NVIDIA Jetson Orin Nano Super — and the one flag that fixes it</title>
      <dc:creator>Hemkesh</dc:creator>
      <pubDate>Wed, 17 Jun 2026 13:57:24 +0000</pubDate>
      <link>https://dev.to/hemkesh2021dotcom/why-cudamalloc-fails-on-nvidia-jetson-orin-nano-super-and-the-one-flag-that-fixes-it-1b0n</link>
      <guid>https://dev.to/hemkesh2021dotcom/why-cudamalloc-fails-on-nvidia-jetson-orin-nano-super-and-the-one-flag-that-fixes-it-1b0n</guid>
      <description>&lt;h1&gt;
  
  
  Why cudaMalloc fails on NVIDIA Jetson Orin Nano Super — and the one flag that fixes it
&lt;/h1&gt;

&lt;p&gt;If you've tried running a GGUF model via llama.cpp on a Jetson Orin Nano Super and hit this error:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NvMapMemAllocInternalTagged: error 12
cudaMalloc failed: out of memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;...while tegrastats shows the GPU idle and several GB of RAM free — this post is for you.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hardware context
&lt;/h2&gt;

&lt;p&gt;The Jetson Orin Nano Super (8 GB) uses unified memory. There is no separate VRAM pool — the CPU and GPU share one physical 8 GB block. This is how all Jetson Orin-series SoCs work, and it's what makes them cost-effective for edge AI.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why stock llama.cpp fails
&lt;/h2&gt;

&lt;p&gt;When llama.cpp allocates GPU tensor buffers, it calls &lt;code&gt;cudaMalloc&lt;/code&gt;. On a discrete GPU, &lt;code&gt;cudaMalloc&lt;/code&gt; carves out memory from a dedicated VRAM pool. On the Orin Nano Super, that dedicated pool doesn't exist — the allocator hits the NvMap interface, which rejects the request with error 12 (ENOMEM) even though the unified pool has plenty of space.&lt;/p&gt;

&lt;p&gt;The result: the model fails to load with a misleading out-of-memory error despite free RAM.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;Set &lt;code&gt;GGML_CUDA_ENABLE_UNIFIED_MEMORY=1&lt;/code&gt; as an environment variable when you launch the server. This is a &lt;strong&gt;runtime flag, not a build option&lt;/strong&gt; — in llama.cpp's CUDA backend it switches GPU allocations from &lt;code&gt;cudaMalloc&lt;/code&gt; to &lt;code&gt;cudaMallocManaged&lt;/code&gt;, which works with the unified pool.&lt;/p&gt;

&lt;p&gt;First, build llama.cpp with CUDA for the Orin's compute capability (sm_87):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/ggml-org/llama.cpp
&lt;span class="nb"&gt;cd &lt;/span&gt;llama.cpp
cmake &lt;span class="nt"&gt;-B&lt;/span&gt; build &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;-DGGML_CUDA&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;ON &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;-DCMAKE_CUDA_ARCHITECTURES&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;87
cmake &lt;span class="nt"&gt;--build&lt;/span&gt; build &lt;span class="nt"&gt;--config&lt;/span&gt; Release &lt;span class="nt"&gt;-j&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;nproc&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then enable unified memory at launch by setting the environment variable:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;GGML_CUDA_ENABLE_UNIFIED_MEMORY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1 ./build/bin/llama-server &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;-m&lt;/span&gt; LFM2-VL-1.6B-Q4_0.gguf &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;--mmproj&lt;/span&gt; mmproj-LFM2-VL-1.6B-Q8_0.gguf &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;--n-gpu-layers&lt;/span&gt; 999
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's it. The model loads and runs fully GPU-accelerated.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I was building
&lt;/h2&gt;

&lt;p&gt;I ran into this while building SENTINEL — a self-hosted AI surveillance system for my B.Tech minor project. It runs YOLOv8n + TensorRT for person detection, DeepFace for face recognition, and LFM2-VL 1.6B for scene understanding — all on the Jetson, zero cloud dependency.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Full project and complete BUILD.md:&lt;/strong&gt;&lt;br&gt;
&lt;a href="https://github.com/hemkesh2021-dotcom/Sentinel_Surveillance" rel="noopener noreferrer"&gt;https://github.com/hemkesh2021-dotcom/Sentinel_Surveillance&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Hope this saves someone a few hours of debugging.&lt;/p&gt;

</description>
      <category>jetson</category>
      <category>nvidia</category>
      <category>cpp</category>
      <category>edgeai</category>
    </item>
  </channel>
</rss>
