DEV Community

Cover image for Your slowest model shouldn't make your fastest decisions: lessons from an edge AI camera on a Jetson
Hemkesh
Hemkesh

Posted on

Your slowest model shouldn't make your fastest decisions: lessons from an edge AI camera on a Jetson

One camera. One NVIDIA Jetson Orin Nano Super. Every model runs on the board.

That was the brief for Sentinel, our final-year project at Dayananda Sagar University: spot people, follow them across the frame, and open an incident when someone steps into a zone you care about. No cloud AI anywhere in the loop.

We expected the hard part to be the models. It wasn't.

See it first: the project website has a hardware tour and a demo dashboard you can click around in. It is the real dashboard code replaying a scripted scene, so no Jetson is needed.

Sentinel dashboard tour, recorded with synthetic demonstration data

The dashboard with synthetic demo data. No camera or model was running.

The pipeline

Step What does it
See YOLOv8n on TensorRT detects people
Follow ByteTrack keeps each person's track
Decide Zone rules open incidents in a local SQLite database
Tell A restart-safe outbox delivers alerts (Telegram in our tests)
Show A dashboard that listens on 127.0.0.1 only

Two optional lanes sit beside it: face recognition (YuNet + Facenet512) and a local vision-language model (LFM3-VL-2B through llama.cpp). Both are still under validation, so they are off by default. More on that below.

Lesson 1: your slowest model shouldn't make your fastest decisions

On a small board, models finish at very different speeds. The detector keeps up with the stream. Face matching is slower. A vision-language model is slower still.

Now imagine the camera drops and reconnects. A slow model is still chewing on a frame from before the reconnect. When its answer finally lands, the naive pipeline treats it as news about now.

That is how an old frame fires a new alert.

Our rule: every observation is evidence about one specific frame, never about "now". So every observation carries three stamps:

  • the frame's source time
  • the boot ID of the run that produced it
  • a stream epoch that changes on every reconnect

Before any result can drive a decision, it has to pass a check like this (a simplified sketch, not the project's exact code):

def can_drive_live_decision(obs, now, current):
    if obs.boot_id != current.boot_id:
        return False          # from a previous run
    if obs.stream_epoch != current.stream_epoch:
        return False          # from before a reconnect
    if now - obs.frame_time > MAX_AGE:
        return False          # simply too old
    return True
Enter fullscreen mode Exit fullscreen mode

A result that fails is not thrown away. It can still annotate history. It just never gets to act on the present.

One more detail that mattered: age is measured with an injected monotonic clock, not wall time. That makes the rule testable. A fake clock lets a test replay "the model answered late" without waiting for a real model to be slow.

Lesson 2: "I don't know" is a valid answer

A monitoring system that guesses is worse than one that stays quiet. Three rules came out of that:

  • A face is tied to at most one tracked person. If ownership is ambiguous, it stays unresolved.
  • An unknown face is never labelled a stranger. Not recognising someone is not evidence about who they are.
  • Fire or smoke reported by the scene model alone is never a critical alarm.

The dashboard follows the same idea. Empty, fresh, stale and offline are four different states, and it shows each one explicitly instead of leaving an old picture on screen.

Lesson 3: an alert that dies with the process was never an alert

An in-memory queue loses everything on a restart. So incidents and their pending alerts are written to SQLite, and a worker delivers from that outbox with retries. If Sentinel restarts between "incident opened" and "alert sent", the alert still goes out afterwards.

Lesson 4: test without the hardware

Most of Sentinel can be tested on a laptop. The portable suite needs no camera, GPU or model, and CI runs it on every push under Python 3.10 and 3.12. On the commit the beta was built from, 4,224 tests passed.

The trick is keeping hardware adapters at the edges, so importing the core never requires CUDA or TensorRT.

What isn't done

Sentinel is an experimental beta, and we would rather say so than oversell it:

  • Face recognition has not passed its formal validation.
  • Chat with the local vision-language model is not admitted for normal use.
  • The install package has run only on our own Jetson.

The README keeps a feature table that says what is admitted and what is not, with the evidence behind each claim.

Try it

  • In a browser: the project website and its demo dashboard
  • On any computer with Python 3.10+: clone the repo and run the test suite
  • On a Jetson Orin Nano: the experimental install package in the repo

Code (MIT): https://github.com/hemkesh2021-dotcom/Sentinel_Surveillance

If you run a slow model next to a fast one, how do you keep the slow one's answers honest? I'd like to hear how others handle it.

Top comments (0)