DEV Community

Dimitar Apostolov
Dimitar Apostolov

Posted on Edited on

Which day should we drive? Watching border queues with Home Assistant and a vision model

#homeassistant #homelab #ai #selfhosted

Twice a year we drive home. Same route, same border crossings, same question in the queue: should we have left a day earlier?

I wanted to know which day and hour were best at each crossing. A navigation app only shows the queue right now, so I decided to record it myself.

Public traffic cams exist for the crossings on our route, and I already had Home Assistant running. So I made it photograph every camera on a schedule, ask a vision model how bad the queue looks, and keep the answer as a sensor.

This post covers the idea, the build, and what I'd do differently.

Contents

Overview

The idea

Peak travel follows the school calendar. In countries where school holidays are staggered by region (Germany is the textbook case), traffic comes in waves.

Outbound, traffic usually starts on the Friday before the first region's holidays begin. On the way back, it usually starts building about a week before the first region's holidays end.

So each direction gets its own collection window. Outside the windows I don't need data. Inside them I want a reading every 30 minutes from every camera on the route. Then I look at the graphs and pick a day.

route:   origin ──► crossing A ──► crossing B ──► home
                       │              │
                  cameras per     cameras per
                    direction       direction
Enter fullscreen mode Exit fullscreen mode

Each camera watches one direction of one crossing, which matters when the cameras are grouped later.

What you end up with 

every 30 min, daylight only, only inside the collection windows
        │
        ▼
 for each camera:
   snapshot ──► JPEG on disk (/media/traffic/<camera>/<timestamp>.jpg)
        │
   is it a real image?  no ──► retry once ──► still no ──► "no data"
        │ yes
        ▼
   vision model ──► { congestion 0-4, image_kind, note, overlay_time }
        │
   image_kind != live?  ──► "no data"
        │
        ▼
   helpers + one sensor per camera ──► history graph, per direction
Enter fullscreen mode Exit fullscreen mode

Each camera gets a sensor with a 0 to 4 score, the crossing and direction as attributes, and a note like "long queue of heavy trucks along the left side". Two history graphs show every crossing for one direction on one time axis.

I have 13 cameras and one image each per run. No GPU involved.

Tech stack

Layer What I used Used for
Platform Home Assistant OS in a VM Runs everything below
Cameras Generic Camera integration, public still-image URLs One entity per camera, all in one area
Capture camera.snapshot Saves a JPEG per camera to /media
Image check shell_command (wc, head, od) Rejects error pages and truncated files
Vision model ai_task.generate_data via the core OpenRouter integration, a small cheap model Score, note, image_kind, overlay_time as typed fields
Scheduling Automation with time_pattern and a sun condition, plus a date-window list in the script Every 30 min, daylight only, inside the windows
Storage input_number and input_text helpers, JPEGs on disk Latest reading per camera, plus the raw images
Charting Template sensors with state_class: measurement, history-graph cards One graph per travel direction, long-term statistics for export
Tests Python and Jinja2 against the real YAML, plus a live scenario script Gate logic offline, the full flow on the real instance
Generator A small Python script Writes the per-camera helpers and sensors from one camera table

The setup

Step 1: The cameras

I used a public site that lists traffic cams for border crossings. Each camera there has a still-image URL.

Add each one in Home Assistant as a Generic Camera:

Settings → Devices & services → Add integration → Generic Camera

Use the still image URL from the site. You don't need a stream URL, we only want stills. Pick a boring, consistent naming pattern, because the name becomes the entity id and everything else is derived from it.

Put all the cameras in one area (mine is called Traffic Cams). The automation loops over the area, so adding a camera later is a UI-only change.

Gotcha: the entity id comes from the name you type when you add the camera, and the friendly name you see afterwards can look different. Check the id in Settings → Entities after adding each one. If you rename a camera later, everything keyed on the old id stops working, without an error.

Verify it worked: in Developer tools → Template:

{{ area_entities('traffic_cams') | select('match', 'camera\.') | list | length }}
Enter fullscreen mode Exit fullscreen mode

It should print your camera count.

Step 2: Snapshot and check the image

camera.snapshot fetches the still and writes it to disk. Home Assistant only writes to allow-listed folders, so /media has to be on the list. On my HAOS install it already was.

- action: camera.snapshot
  target:
    entity_id: "{{ cam }}"
  data:
    filename: "/media/traffic/{{ slug }}/{{ now().strftime('%Y-%m-%d_%H%M') }}.jpg"
Enter fullscreen mode Exit fullscreen mode

The file name carries the camera and the timestamp. That's the whole "label the images" requirement, and it keeps the archive browsable without a database.

Next, check that you actually got an image.

A snapshot can "succeed" and still save an error page as .jpg. In my first audit, 1 fetch out of 65 came back as an HTTP 500 with a 26-byte text body. Left alone, it would have gone straight to the model.

Home Assistant templates can't look inside files, so I used a tiny shell_command:

shell_command:
  # prints "ok" only if the file exists, is >= 5000 bytes, and starts with the JPEG magic bytes
  traffic_cams_probe: >-
    sh -c 'f="{{ path }}"; [ -f "$f" ] && [ "$(wc -c < "$f")" -ge 5000 ] &&
    [ "$(head -c 3 "$f" | od -An -tx1 | tr -d " \n")" = "ffd8ff" ] && echo ok'
Enter fullscreen mode Exit fullscreen mode

If the probe fails, take the snapshot once more. If it fails again, skip the model call and record "no data".

Gotcha: a templated shell_command runs without a shell, so pipes and && do nothing unless you wrap everything in sh -c. Check that wc, head, od and tr exist in the Home Assistant container before you trust the probe.

Verify it worked: run the script by hand and open Media → My media. You should see one folder per camera.

Step 3: Ask the model

Home Assistant has an ai_task.generate_data action. It takes an image attachment and a structure, so you get typed fields back instead of a paragraph.

- action: ai_task.generate_data
  data:
    entity_id: ai_task.my_vision_model
    task_name: traffic
    instructions: >-
      This is a still from a border-crossing traffic camera. Rate the queue
      from 0 (empty, free flow) to 4 (standstill, long queue).
      If the image is not a live road view (poster, offline, black, test
      pattern) say so in image_kind.
    structure:
      congestion:
        selector: { number: { min: 0, max: 4 } }
        required: true
      image_kind:
        selector: { select: { options: [live, poster, offline, night, unclear] } }
        required: true
      note:
        selector: { text: }
      overlay_time:
        selector: { text: }
    attachments:
      media_content_id: "media-source://media_source/local/traffic/{{ slug }}/{{ stamp }}.jpg"
      media_content_type: image/jpeg
  response_variable: result
Enter fullscreen mode Exit fullscreen mode

I added two fields late, and I'd now keep them in any version of this.

image_kind makes the model say whether it's looking at a real road. A camera could show a poster, an "offline" tile or a black frame, and a model could rate that "0, empty road". That would put a false zero in the series. My cameras were all live when I checked, but I didn't want the pipeline to depend on that.

overlay_time is the clock most cameras burn into the frame. The model reads it, and later you can compare it with the real time to catch frozen or delayed feeds.

For the model I used a small, cheap one through OpenRouter, which Home Assistant has a core integration for. You add the integration, then an AI task sub-entry with the model you want. Swapping models later is one dropdown.

I stuck to one rule: the model reports and the code decides.

image_kind != live  →  congestion = -1  (no data)
live                →  congestion clamped to 0..4
Enter fullscreen mode Exit fullscreen mode

That gate is plain template logic and I test it offline. The model doesn't decide whether a reading is stored.

Verify it worked: run the script once. Each camera's note should match what you see when you open its JPEG.

Step 4: Store it so you can chart it

Per camera I write four helpers: congestion (input_number, range -1 to 4), plus note, image_kind and overlay_time (all input_text).

The helpers are only the write target. What you chart is a template sensor on top:

template:
  - sensor:
      - name: "Crossing A south→north"
        unique_id: traffic_crossing_a_sn
        state_class: measurement          # long-term statistics, so you can export a CSV later
        state: >-
          {% set v = states('input_number.traffic_crossing_a_sn_congestion') | float(-1) %}
          {{ v if v >= 0 else none }}
        attributes:
          crossing: Crossing A
          direction_group: southbound
          image_kind: "{{ states('input_text.traffic_crossing_a_sn_image_kind') }}"
Enter fullscreen mode Exit fullscreen mode

-1 becomes none, so the history graph shows a gap instead of a fake zero. state_class: measurement makes Home Assistant keep long-term statistics, which outlive the 10-day recorder default and export to CSV.

Then two history-graph cards give me what I wanted in the first place, one per travel direction with a line per crossing:

type: history-graph
title: Traffic southbound
hours_to_show: 72
entities:
  - sensor.traffic_crossing_a_sn
  - sensor.traffic_crossing_b_sn
Enter fullscreen mode Exit fullscreen mode

Gotcha: helpers defined in YAML can't be grouped under a device, because only integrations own devices. I wanted a tidy "Traffic" device and ended up with a naming prefix and a dashboard. If you need a real device, MQTT discovery gives you one, but you have to run a broker.

Gotcha: dashboards have URL paths, and Home Assistant insists on a hyphen in them. traffic-cams works, traffic doesn't.

Verify it worked: open the dashboard after a few runs. Gaps in a line are fine. A flat zero for a crossing you know is busy is not.

Step 5: Only run when it matters

13 cameras every 30 minutes, all year, adds cost and fills the disk for no benefit. The traffic I care about lives in two or three windows a year.

So the windows sit in one list at the top of the script:

windows:
  - name: christmas
    start: "2030-12-21"
    end:   "2031-01-08"
    directions: [southbound, northbound]
  - name: summer_southbound
    start: "2031-06-24"          # the Friday before the first region's holidays start
    end:   "2031-08-02"          # the day we drive
    directions: [southbound]
  - name: summer_northbound
    start: "2031-07-29"          # a week before the first region's holidays end
    end:   "2031-09-13"          # school starts again in the last region
    directions: [northbound]
Enter fullscreen mode Exit fullscreen mode

Every camera carries a direction. A camera is processed only if today falls inside a window that lists its direction. The date is now().date(), so it follows the Home Assistant timezone, not UTC.

The two summer windows overlap for a few days. During the overlap both directions run.

One window per direction roughly halved the number of model calls compared with "run everything all summer". The cameras facing the other way would have recorded weeks of data I don't need.

The automation itself stays dumb:

automation:
  - id: traffic_cams_daylight
    alias: Traffic cams (daylight)
    triggers:
      - trigger: time_pattern
        minutes: "/30"
    conditions:
      - condition: state
        entity_id: sun.sun
        state: above_horizon
    actions:
      - action: script.traffic_cams_run
    mode: single
    initial_state: true
Enter fullscreen mode Exit fullscreen mode

The sun condition is there because I decided night frames aren't worth rating.

The script also has an ignore_windows field. It exists for testing. Outside every window the script correctly does nothing, and you can't prove that without a way to make it do something.

Gotcha: initial_state is re-applied on every Home Assistant restart. A toggle in the UI won't survive it. Whatever the YAML says wins.

Gotcha: an automation's entity id comes from its id. I renamed mine once, because the first name had the interval in it, and the old entity hung around as an unavailable orphan until I deleted it from the entity registry. Don't put anything in an id that might change.

Verify it worked: run the script with ignore_windows: false on a day outside every window. No new files should appear. Run it again with true and every camera should produce one.

Running it

What it costs

I measured, then extrapolated. All my testing cost $0.04 in total. I didn't count the calls exactly, roughly 85, so call it $0.0005 per call. One run is one call per camera.

Window Cameras Daylight runs/day Days Calls Cost
Christmas, both directions 13 ~17 19 ~4,200 ~$2
Summer, southbound 7 ~31 40 ~8,700 ~$4.40
Summer, northbound 6 ~31 47 ~8,500 ~$4.20

That's about $11 for a whole season. Put a credit limit on the API key at roughly double that, so a bug can't turn into a bill.

Going from every 20 to every 30 minutes cut the cost by a third, so I took the saving.

Storage is small. The JPEGs are 15 to 55 KB each, around 640 MB for the season. I don't delete anything. The images are the raw data if I ever want to re-score them with a better model.

Testing

Everything deterministic gets tested without Home Assistant. The test script renders the real templates from the YAML with Jinja2, so it exercises the same code and not a copy of it.

It checks that:

  • the YAML parses with no duplicate keys,
  • the expected helpers exist for every camera,
  • scores are clamped (-3 → 0, 7 → 4, abc → 0),
  • the image gate works (non-live becomes -1, an unknown kind becomes unclear and then -1),
  • the window gate behaves on the edge days: day before, first day, last day, day after, the overlap, and a random day in November,
  • the camera table in the README matches what the generator prints.

That's about 300 checks.

Then there's a live scenario against the real instance. It runs the script gated and expects nothing, runs it forced and expects one image and one fresh sensor per camera, and checks the automation exists.

Gotcha: my first live test failed for a camera whose note was word for word the same as the previous run. Home Assistant doesn't bump last_updated when a value doesn't change, and an empty road looks the same every time. Test "did it run" with something that can't repeat, like the image file.

Troubleshooting

Rows marked * I haven't hit myself. They follow from how the pieces work.

Symptom Cause Fix
* Every camera shows "no data" The file probe fails inside the HA container (a missing wc or od, or no sh -c) Run the probe command by hand with docker exec and fix what's missing
* camera.snapshot fails with "not allowed" The target folder isn't in allowlist_external_dirs Add homeassistant: allowlist_external_dirs: [/media]
One camera never updates Its entity id doesn't match the helper names Rename the entity id in Settings → Entities
* The model rates an empty frame "0" It's a poster, an offline tile or a night frame Look at image_kind. If it says live, tighten the prompt
Config check warns about another package An unrelated package in the same folder (often a missing custom integration) Ignore it, or fix that package
A file you wrote isn't picked up Wrong folder, because the same directory has three names See below

On that last row: Home Assistant OS has three names for the same config folder:

Where you stand Path
Inside the Home Assistant core container /config
Inside add-ons (File editor and friends) /homeassistant
On the HAOS host (what the Proxmox guest agent sees) /mnt/data/supervisor/homeassistant

Retrospective

What I learned

  1. A successful fetch isn't proof of a good image. One of my 65 fetches returned an error page. I also ran a 12-minute audit of the cameras, five rounds three minutes apart. Every image was different, which shows the snapshots come from live broadcasts, and every one was a real scene. But one camera's burned-in clock was 26 minutes behind, another's was two hours ahead, and one camera was mounted upside down. Stale or poster frames didn't show up, but nothing in the pipeline would have caught them if they had.

  2. I let the model describe and the code decide. The model says what it sees, and plain code applies the thresholds. That kept the rules testable, and swapping the model touched no logic.

  3. Direction turned out to be the right unit of analysis. My first grouping was "into the country / out of the country", and it was wrong for what I needed. What matters is which way you're travelling along the route, because a crossing has cameras for both directions, and their traffic differs. Splitting by southbound and northbound fixed that, and it gave me the per-direction windows for free.

  4. I got most of the cost saving by limiting when I collect, not how much. The windows were a small change.

  5. I used -1 for "no data" instead of a fake zero. With none in the sensor, the chart shows a gap. A made-up 0 would put false data in it.

  6. I should have checked the system before documenting it. At one point I had documentation naming the wrong AI provider. The model's display name looked like it came from one vendor but was served through another. The answer was one API call away (the list of loaded components).

  7. I needed to test the bypass as well as the gate. An automation that is correctly silent most of the year needs a switch that makes it run, or there is no way to show that it works.

What I would change

Starting over, knowing what I know now:

  1. I'd run the freshness audit on day one. Five fetches per camera, three minutes apart, compare hashes, look at the images. A short audit that would have shaped the design earlier: the gate, the image_kind field, the retry.
  2. I'd store the model and prompt version with every reading. Right now I keep the answer but not which model and prompt produced it. If I change either mid-season, the series has a silent seam in it, and a version field costs nothing.
  3. I'd add a staleness check outside Home Assistant. It can't hash a file. A small script that compares each image with the previous one (a perceptual hash) and flags "unchanged for N runs" would catch frozen feeds that the model still calls live. I keep every image, so I can do this after the fact, but I'd rather have it in the loop.
  4. I'd calibrate the score. "0 to 4" comes from a prompt, not from data, and a single frame shows density, not speed. Before trusting the numbers I'd label a few dozen frames by hand and compare them with the model's scores.
  5. I'd set the windows from data, not from a calendar. Mine come from holiday dates plus a feel for when people start moving. A cheap all-year run, say every 3 hours, would show when the waves really start.
  6. I'd put the three folder names on the first page of the README. It got in the way once.
  7. I'd keep anything that might change out of names. Intervals, dates and providers don't belong in an entity id.
  8. I'd look at pictures earlier. I built most of the pipeline before I'd looked at enough frames to see what the data would really be like: low resolution, hazy, one camera rotated.

The YAML above is trimmed and renamed from the real package.

Top comments (0)