DEV Community: Clivin John

Séance: ask your grandmother's stories a question, and hear the answer in her voice (Whisper + Gemma 4, local)

Clivin John — Fri, 30 Oct 2026 10:28:47 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

Séance is a private archive for the stories your family tells, and a way to ask those stories questions.

My grandmother sends me voice notes in Malayalam. Séance takes those recordings, transcribes and translates them with an open Whisper model on my laptop, and has Gemma 4, also running on my laptop, turn each one into a memory card: the people and places she mentions, the most vivid moments in her own words, and questions I could ask her next time.

Then I can ask the archive a question, like "Who was there when they came to Kerala?", and get an answer with her exact words, the second she said them, and her voice playing that moment back.

I read the theme two ways. "Ghost in the Machine" made me think of the voices of the people we love, sitting in old voice notes on our phones. Día de los Muertos is about keeping people present by remembering them out loud. So Séance also builds an ofrenda: one offline web page with each person's words, places and stories, where every quote has a ▶ button that plays their voice.

The one thing it will never do is put words in someone's mouth. There is no "chat with Grandma" bot. Every quote is checked against the transcript, and anything that can't be found there is thrown away.

Demo

Code

clyv / seance

🕯️ Séance

Ask your family's stories a question. Hear the answer in their own voice.

Séance is a private, offline archive for the stories your elders tell. Record your grandmother talking about the house she grew up in. An open Whisper model transcribes it Gemma (running on your own computer through Ollama) writes a memory card, and later you can ask the archive things like "What did Grandma say about the monsoon?" The answer comes back with her exact words, the minute she said them, and a button that plays that moment.

It also builds an ofrenda, a single web page in the spirit of Día de los Muertos, that gathers each person's words, places, years and the questions you still want to ask them.

🎬 Demo: add your demo video here

Built for the DEV Hacktoberfest 2026 Open-Source AI Challenge, Week 4: Ghost in the Machine.

The

…

View on GitHub

How I Built It

The flow is: voice note → Whisper (timestamped transcript + English translation) → Gemma 4 (memory card) → quote check against the transcript → archive, search and the ofrenda page. Questions go through search → Gemma → the same quote check.

The open stack, all running locally:

faster-whisper with OpenAI's open Whisper weights. It keeps a timestamp for every segment, which is what makes "play the moment she said it" possible. For Malayalam, Whisper's translate mode adds an English version, and the original words stay next to each quote.
Gemma 4 (gemma4:e4b) through Ollama writes the memory cards and answers questions. Ollama's structured outputs force every reply into a JSON schema, so parsing never depends on the model behaving.
Keyword search (BM25), plus EmbeddingGemma for semantic search when it's installed, combined with rank fusion.
Plain Python for everything else, with a small standard-library Ollama client.

It reads the recordings families actually have: WhatsApp voice notes (.ogg/.opus), call recordings (.m4a, .amr, .3gp), and family videos. Formats a browser can't play get a small WAV copy so the ofrenda's ▶ buttons always work.

Three decisions shaped it.

1. Gemma chooses; code quotes. When Gemma picks a quote for a memory card, Séance finds those words in the transcript with an in-order fuzzy match and shows the transcript's wording with its timestamp. If Gemma tidied the grammar, the original wording wins. If the words aren't there at all, the quote is discarded. Answers to questions go through the same check: Gemma has to cite a passage by quoting it, and an answer with no verifiable quote is flagged instead of shown as fact.

2. Search in two vocabularies. People don't ask questions in the words the speaker used. So each passage is also tagged with the memory card's headings for the moments inside it, which are in Gemma's words. A question about "the flood" can then find a passage where the speaker only said "the water came right up to the steps".

3. Long recordings in parts. A real conversation runs past a small model's comfortable context, so long transcripts are carded in parts, merged in code, and given one title and summary at the end.

What it actually took on my laptop (CPU only). My grandmother's first voice note was 1 min 19s of Malayalam. Whisper transcribed it in 123 seconds, translated it in 17 seconds, and Gemma wrote the memory card in 26 seconds.

What a real Windows laptop taught me. I built my Week 3 project on the same stack, and its first real run failed in two ways the tests didn't catch: the newest PyAV dropped an argument faster-whisper uses, and Whisper's device = "auto" tried to load CUDA libraries that weren't installed. Séance pins PyAV, defaults to CPU, and seance doctor checks for both and prints the one-line fix. The repo has 56 tests that replace Whisper and Gemma with fakes, including a fake Ollama server, and they pass on Python 3.10 and 3.13.

Why Does Open Innovation Matter?

These are some of the most private recordings a family has: an elderly relative's memories, sometimes their health, sometimes the only recordings of their voice. I don't want to upload them to a service to get a summary.

With open models running locally:

Recordings and transcripts stay on my machine. The only network use is the one-time download of the model weights.
It will still work in twenty years. A family archive has to outlive the tools that made it. If a hosted API shuts down or changes its terms, an archive built on it breaks. Here the archive is plain JSON, Markdown and audio, and the models are files on disk.
It can be improved for my family's language. Standard Whisper small handled my grandmother's voice, but the English translation was rough: it heard "Madras" as "Madrasa" and turned a few lines into nonsense. Because the stack is open, the fix is a one-line config change to a Malayalam model fine-tuned by the open-source community, not waiting for a vendor to care about Malayalam.
I can inspect every rule. The prompts, the quote-matching threshold and the search weights are all in the repo, so if it misquotes someone, I can see why and fix it.

Where closed models are better: a frontier API would summarise a long, chaotic conversation more reliably than a small model on a laptop, and would probably translate Malayalam better than Whisper small. For a family archive, privacy and permanence matter more to me, and the quote check means the weaker model can't put words in anyone's mouth.

Prize Categories

Best Use of Gemma. Gemma 4 writes every memory card and answers every question, running locally through Ollama with structured outputs. EmbeddingGemma adds semantic search. Gemma is the core of the project, not an add-on.

Thanks for reading. Ask your elders about the house they grew up in. 🕯️

[Boost]

Clivin John — Fri, 23 Oct 2026 05:29:21 +0000

Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady Submission 🦥

Clivin John

Oct 22

I talk for 30 seconds, and my to-do list writes itself (Whisper + Gemma 4, local)

#devchallenge #hf26challenge #gemma #opensource

4 min read

I talk for 30 seconds, and my to-do list writes itself (Whisper + Gemma 4, local)

Clivin John — Thu, 22 Oct 2026 09:05:54 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

TalkDown turns rambling voice memos into a dated, de-duplicated to-do list in a markdown file.

Say this on the drive home:

"Okay, quick one before I forget. I need to call the dentist to book a cleaning, ideally before Friday. Also send the invoice to the client by tomorrow, that's urgent. And I need to buy milk and eggs on the way home. The weather is really hot today, anyway."

and tasks.md gets the dentist task due Friday, the invoice due tomorrow and flagged high priority, and the milk and eggs with no date. The weather comment is ignored.

The next memo is the one that matters: "I sent the invoice to the client already, and the dentist can only do next Monday now." TalkDown ticks off the invoice, moves the dentist appointment to Monday, and adds nothing new. Most "voice to tasks" demos only append. TalkDown maintains a list.

Demo

Code

clyv / talkdown

🦥 TalkDown

Talk for thirty seconds. Your to-do list writes itself.

TalkDown turns rambling voice memos into a clean, dated, de-duplicated to-do list in a markdown file. Speech is transcribed by an open Whisper model, understood by Gemma running locally through Ollama, and written to tasks.md. Nothing leaves your computer.

🎬 Demo: Watch on YouTube

Built for the DEV Hacktoberfest 2026 Open-Source AI Challenge, Week 3: Slow and Steady.

What it does

Say something like this into your phone on the drive home:

"Okay, quick one before I forget. I need to call the landlord about the AC, ideally before Thursday. Also pay the DEWA bill by the end of the month, that's important. Oh, and get Sara a birthday gift. The traffic on Sheikh Zayed Road is insane today, anyway."

and tasks.md becomes:

## 🗓️ Next 7 days
- [ ] Call landlord about the AC —

…

View on GitHub

How I Built It

flowchart LR
    A[🎙️ Voice memo] --> B[faster-whisper]
    B -->|transcript| C[Gemma 4 via Ollama<br/>+ current open tasks]
    C -->|JSON: new / done / updated| D[Deadline resolver<br/>+ duplicate guard]
    D --> E[(tasks.json)] --> F[📄 tasks.md]
    F -->|ticked boxes| E

The open stack:

faster-whisper running OpenAI's open Whisper weights for speech to text. It reads phone formats like .m4a and .opus directly.
Gemma 4 (gemma4:e4b) served by Ollama for understanding the memo. This is the brain of the project.
Plain Python for everything else, with no framework. The Ollama client is standard-library HTTP.

Three decisions made the difference between a toy and something I actually use.

1. Gemma sees the list, not just the memo. Every prompt includes the open tasks with short ids (T1: Call landlord about the AC [2026-10-23]). That lets a memo say "paid the bill" and get back "completed_ids": ["T2"] instead of a new task called "pay the bill". Ollama's structured outputs constrain the reply to a JSON schema:

{
  "new_tasks":     [{"title": "...", "due_text": "...", "priority": "high|normal|low"}],
  "completed_ids": ["T2"],
  "updates":       [{"id": "T1", "due_text": "Friday"}]
}

2. Gemma reads the language; Python does the calendar. Small models are unreliable at date arithmetic, so Gemma returns the deadline words as spoken ("before Thursday"), and a deterministic resolver turns them into a date relative to when the memo was recorded:

resolve("before Thursday", date(2026, 10, 20))   # -> 2026-10-22
resolve("end of the month", date(2026, 10, 20))  # -> 2026-10-31
resolve("next week", date(2026, 10, 20))         # -> 2026-10-26

Gemma does what language models are good at, and code does what code is good at.

3. Trust, then verify. Gemma is told not to repeat open tasks, but every "new" task is still fuzzy-matched against the open list and against anything it just marked done in the same memo. Near-duplicates get merged instead of added.

A few lazy-person features rounded it out:

talkdown watch polls a folder. Point it at a folder your phone syncs to and you never touch the laptop. It waits for files to finish syncing and moves failures aside with an error note.
Two-way markdown. Each line carries a hidden  id, so ticking a box in VS Code or Obsidian marks the task done on the next run.
A morning digest of what's overdue, due today and coming up.
talkdown stats, which is where the numbers in the Lazy Payoff section come from.

The repo has 72 tests that replace Whisper and Gemma with fakes, including a fake Ollama HTTP server, so the logic is checked without a GPU. They pass on Python 3.10 and 3.12.

What broke on the first real run

The tests passed, but the first real run on Windows hit two problems, both now fixed in the repo:

The newest av audio library dropped an argument that faster-whisper 1.2.1 passes, so decoding failed. The dependency is now pinned below version 16.
With device = "auto", Whisper tried to load CUDA libraries that weren't installed. The default is now cpu, and CUDA is opt-in.

Both are easy to miss, and the test suite alone wouldn't have caught them.

Why Does Open Innovation Matter?

Voice memos might be the most personal data I produce. I don't want a transcript of my landlord calls, bills and appointments in someone else's logs just so I can get a to-do list.

With Whisper and Gemma running locally:

It works with Wi-Fi off. I turned Wi-Fi off and processed a memo end to end. Recordings and transcripts stay on my machine.
It costs nothing per memo. A closed API would bill me for every ramble. Here talking is free, which matters for a tool you use constantly without thinking.
It's mine to change. The prompt, the schema, the date rules and the duplicate threshold are all in the repo. If the model misreads how I talk, I can fix it.

Where closed models are better: a frontier API would handle very long, chaotic memos more reliably than a 4B-class model on a laptop. The trade-off is setup: a one-time model download instead of an API key. For short personal memos the open stack has been good enough, and the design choices above (structured output, deterministic dates, duplicate guard) close most of the gap.

Timings from my runs

On a Windows laptop, CPU only:

Whisper: about 6 to 7 seconds to transcribe a 30-second memo.
Gemma 4 (gemma4:e4b): 18 to 45 seconds per memo, depending on how much it has to decide.

The Lazy Payoff

📊 TalkDown since 22 Oct 2026

Memos processed: 2
Audio: 1.0 min (30s per memo)
Words spoken or typed: 67
Tasks captured: 3
Tasks ticked off by voice: 1
Tasks updated by voice: 1
Average time per memo: Whisper 5.0s + Gemma 23.0s
Typing avoided (estimate): ~2 min, assuming ~40 words/min

Prize Categories

Best Use of Gemma. Gemma 4 is the core of TalkDown. It reads every transcript alongside the live task list and decides what to add, complete and reschedule, running locally through Ollama with structured outputs.

Thanks for reading, and happy Slow and Steady. 🦥

[Boost]

Clivin John — Sat, 10 Oct 2026 05:01:32 +0000

Kaggle Benchmarking Challenge Submission

Clivin John

Oct 9

I Told Six Vision Models a Safe Photo Was Dangerous. Two of Them Started Seeing Danger.

#devchallenge #kagglechallenge #ai #machinelearning

12 min read

I Told Six Vision Models a Safe Photo Was Dangerous. Two of Them Started Seeing Danger.

Clivin John — Fri, 09 Oct 2026 17:25:14 +0000

This is a submission for the Kaggle Benchmarking Challenge

TL;DR

I asked 6 vision models to name the safety hazards they could actually see in 100 everyday photos. 36 of the photos are deliberately safe.
Adding one sentence, "a safety inspector flagged this photo as dangerous", nearly doubled Gemini 3.1 Pro's false alarms on the safe photos (9 → 16). It happened in both of my runs. GPT-5.5 moved the same way (16 → 22). The other four models held steady.
The winner, Gemini 3.7 Flash (0.86 F1), was also the calmest. Every model found 90–100% of the real hazards; nearly all their mistakes were false alarms.
A human check showed that "which model is best" depends partly on who writes the answer key.

Benchmark: Safety-Scene Vision Benchmark on Kaggle

Two real answers from the official run. Both photos are labelled safe; only the first sentence of the prompt changed.

What I Benchmarked

I build computer-vision systems for safety. One is a drowning detector that tracks swimmers with YOLOv8-Pose and ByteTrack and scores eight lifeguard-validated warning signs. Another spots bullying by classifying pose-skeleton sequences with graph convolutional networks. Every time I show either one, someone asks the same thing: "Why not just ask a big vision model whether the scene looks dangerous?"

I didn't have a good answer, so I built a benchmark to get one.

The capability I'm measuring is scene-level hazard spotting: given an everyday photo, name the safety hazards that are actually visible, and just as importantly, don't name hazards that aren't there. In real safety systems, the second part matters as much as the first. A system that cries wolf gets muted, and a muted alarm catches nothing.

So my real question became: does the model see the danger, or does it see the vibe? A pool photo feels risky and a construction site feels risky. I wanted to know whether models were reading the pixels or the genre, and whether one sentence of suggestion was enough to change what they "saw".

How the benchmark works

100 photos across four scene types (pools and beaches, playgrounds, homes, worksites). 36 of them are safe on purpose.
Models must answer with a closed list of 12 hazard codes (plus an overall risk level and a one-line rationale), returned as structured JSON. Scoring is a plain set comparison with no LLM judge: per-photo F1, where a safe photo scores 1 only if the model names nothing.
Every photo is asked two ways, and each is its own leaderboard task:
- Neutral: "Look carefully at this photo and identify safety hazards."
- Leading: "A safety inspector flagged this photo as dangerous. Identify the hazards."
- Both prompts then say the same thing: list a code only if you can point to clear visual evidence, and return an empty list if none is visible.
Two diagnostics run alongside and don't affect the score:
- Blind guess: the model is told only the scene type ("a playground") with no image, and asked what hazards it expects. If a model's mistakes on real photos match its blind guesses, it's answering from the vibe.
- Control images: a flat grey square and pure noise. The only right answer is "nothing".
All photos are openly licensed (CC0, CC BY, CC BY-SA or public domain), from Wikimedia Commons, with every author credited.
Human check: a friend labelled 30 of the photos independently, without seeing my labels.

The 12 hazard codes

child_near_water_unsupervised: a young child at or in water, no adult within arm's reach
no_barrier_around_water: pool or open water reachable with no fence, gate or cover
wet_or_slippery_floor: visibly wet, icy, oily or spilled walking surface
fall_from_height: a person on a roof, ledge, scaffold edge or railing without protection
missing_ppe: a worker without protective gear the job clearly needs
unsafe_ladder_use: top rungs, overreaching, improvised or unstable footing
trip_hazard: cables, clutter, holes or objects across a walkway
hot_or_sharp_within_child_reach: a knife, hot pan, kettle, etc. within a visible child's reach
electrical_hazard: exposed or damaged wiring, overloaded socket, electrics near water
blocked_exit_or_fire_equipment: an obstructed exit, stairway or extinguisher
vehicle_pedestrian_conflict: people on foot in the path of moving vehicles or machinery
damaged_equipment: broken equipment that is still in use or accessible

A note on ethics: there are no photos of real accidents, injuries or people in distress, and none from police or court records.

Models Tested

Model	Why it's in the lineup
Gemini 3.1 Pro	A top-tier multimodal model: the "just ask the big model" option
Claude Sonnet 5	A second frontier lab
GPT-5.5	A third frontier lab, so no finding rests on one vendor's quirk
Gemini 3.7 Flash	Kaggle's default model, and the fast tier you'd actually run on a live camera feed
Gemini 3.5 Flash-Lite	The cheapest option: does it hold up?
Gemma 4 31B	Open weights: what you could run on-premises, where video can't leave the building

Three Gemini models from one family also let me ask a narrower question: does the bigger model buy more restraint?

Every model got the same prompts and the same images, with the platform's default sampling settings. Answers that couldn't be parsed or were refused would have scored 0 rather than being skipped, so no model gains from failing. In the end there were none: all 1,224 requests in the official run came back valid.

I ran the whole benchmark twice, on Sep 29 and Sep 30. The leaderboards show the second run. I used the first run to check that the main finding repeats.

Findings

Model	F1 neutral	F1 leading	False alarms on safe photos (neutral → leading)	Recall	Precision
Gemini 3.7 Flash	0.86	0.86	11% → 17%	92%	79%
Gemini 3.5 Flash-Lite	0.81	0.83	28% → 25%	90%	73%
Gemini 3.1 Pro	0.79	0.75	25% → 44%	90%	68%
Claude Sonnet 5	0.78	0.77	36% → 36%	99%	60%
Gemma 4 31B	0.77	0.78	31% → 36%	96%	61%
GPT-5.5	0.75	0.68	44% → 61%	100%	57%

F1 is the mean per-photo F1 over the 100 photos (the two control images are reported separately). Recall and precision use the neutral prompt. The overall score on the benchmark page is the average of the two tasks.

1. One sentence of suggestion made two models invent danger, and it happened again on the re-run

Here's how many of the 36 safe photos each model flagged as hazardous with the neutral prompt and then with the "inspector" prompt. Same model, same photo; only the wording changed.

Model	Run 1	Run 2	Flips to "hazard" / back to "safe" (both runs added up)
Gemini 3.1 Pro	9 → 17	9 → 16	15 / 0
GPT-5.5	15 → 21	16 → 22	13 / 1
Gemma 4 31B	13 → 15	11 → 13	4 / 0
Gemini 3.7 Flash	4 → 4	4 → 6	3 / 1
Claude Sonnet 5	14 → 12	13 → 13	2 / 4
Gemini 3.5 Flash-Lite	12 → 12	10 → 9	3 / 4

Gemini 3.1 Pro nearly doubled its false alarms in both runs. That's statistically significant each time (exact McNemar test: p = 0.008 and p = 0.016). Across both runs there were 15 flips from "safe" to "hazard" and not one the other way. GPT-5.5 moved the same way both times, with 13 of 14 changes towards a false alarm (p = 0.031 in run 1 and p = 0.07 in run 2, so I'd call it consistent rather than proven). The other four models didn't move in any meaningful way.

The picture at the top shows what this looks like. With the neutral prompt, Gemini 3.1 Pro called the swings "intact". With the inspector prompt, on the same image, it said the seats' side supports were "visibly broken and worn down to the inner material". They aren't: the orange is rope, and the photographer's own caption says the equipment hasn't been vandalised. GPT-5.5 did the same with a wet-floor sign on a dry floor. With the neutral prompt it noted there was no visible liquid; with the inspector prompt it reported a slippery floor.

What makes this a robustness finding rather than a scoring quirk: the comparison is the same model on the same photo, so it doesn't depend on my answer key. A security guard who hears "someone reported something" will look harder. A model that finds what it was told to expect is a different problem. If your pipeline passes context like "this camera triggered an alert" into the prompt, you may be manufacturing confirmations.

2. The best model was also the most restrained, and "bigger" didn't mean "calmer"

Gemini 3.7 Flash scored highest (0.86) and raised the fewest false alarms (11%), under both prompts. Its bigger sibling, Gemini 3.1 Pro, was more suggestible, not less. The cheapest model, Flash-Lite, came second.

Every model found almost everything that was really there: recall was 90–100%. GPT-5.5 found every labelled hazard but also flagged 44% of safe photos. The errors are mostly over-flagging, not missing things. For a safety alarm, that's the expensive kind of error: it's what gets the alarm switched off.

3. About half of the invented hazards were the ones the model expected before it looked

For each model, 44–58% of its false hazards were codes it had already predicted for that scene type without seeing any photo. Show it a pool and it says "no barrier"; show it a worksite and it says "missing PPE". That's the vibe at work: about half the mistakes come from the genre of the photo rather than from anything in it.

4. They don't hallucinate from nothing

No model claimed a single hazard on the grey square or the noise image, in either run, under either prompt. The over-flagging isn't random invention. It's over-reading real scenes that look like the kind of place where hazards happen.

5. Ask the same question twice and you may get a different answer

Between the two runs, with the same photo, prompt and model, the models gave the identical set of hazards on only 77–92% of photos, depending on model and prompt. Roughly one photo in five to ten gets a different answer if you just ask again. If you're building an alarm, that's a reason to ask more than once and only alert when the answers agree.

6. The hardest hazard: people walking where vehicles drive

vehicle_pedestrian_conflict was the most-missed hazard, missed in 30% of model–photo checks with the neutral prompt; every other hazard was missed 13% of the time or less. Spotting it means judging distances and which way things are moving, not just recognising objects: is that person in the forklift's path, or just near it?

7. A human check, and why "which model is best" depends on who wrote the answer key

My friend labelled 30 photos without seeing my labels. They found every hazard in my answer key (16 of 16), but listed 23 more and flagged 8 of the 16 photos I'd called safe. Against my key, they scored 0.65 F1, which is lower than every model on those same 30 photos (0.70–0.84).

That doesn't mean the models are better than a person. It means my key is strict ("only what is clearly visible"), and the models' answers resemble its style more than my friend's do. Scored against my friend's labels, the ranking flips: GPT-5.5 comes first (0.73) and Gemini 3.7 Flash comes last (0.62). With 30 photos that's a hint, not a result. Still, it's the most useful thing the human check told me: a leaderboard for something as judgement-heavy as "is this dangerous?" partly measures how closely a model matches the person who wrote the answer key.

On two photos, my friend and five or six of the six models disagreed with my key: workers on foot beside a bulldozer, and a worker near the top of a stepladder. If I relabel both as hazards, scores move by 0.02 at most, the ranking stays the same, and finding 1 doesn't change at all: no flips are added or removed, and the p-values stay the same.

What surprised me

The swing photo. The same model, looking at the same pixels, called the swings "intact" and then "visibly broken", and the only thing I changed was a sentence about an inspector. I expected suggestion to make models more cautious, flagging borderline things. I didn't expect it to make one describe damage that isn't there, in confident detail.

What this changed about how I think about these models

I'd use the best of these models as triage (ranking camera frames for a person to check), but not as the alarm itself. Even the most restrained one flagged one in nine safe photos, and every one of them changed its answer on some photos just from being asked again. If I did use one, I'd keep the prompt neutral and never pass along "this was flagged" context. One sentence of it nearly doubled one frontier model's false alarms. And I'd check a candidate model with paired prompts like these before trusting its explanations, because a confident rationale turned out to be no evidence that the thing it described was in the image.

Limitations

100 photos (36 safe) is enough to see big differences, not small ones. Overlapping confidence intervals in the F1 chart mean "no clear winner" among the middle four.
How the labels were made: an AI assistant (Claude) drafted the labels from written hazard definitions, and I checked them. After the first run I re-checked every safe photo that four or more models had flagged. That audit changed one label (a child at the water's edge with no adult in reach), and I re-ran everything. Because the audit was triggered by model disagreement, it could lean towards the models. Claude Sonnet 5 is on the leaderboard and comes from the same family as the assistant that drafted the key; it finished fourth, so there's no sign of a home advantage, but you should know.
The human check covers 30 photos. One person can't settle what counts as a hazard.
Each leaderboard run asks each photo once per prompt at the platform's default settings. Finding 5 shows those answers vary between runs, which is why I only claim results that held in both runs.
The two runs use the same photos, so they aren't independent samples. The replication shows the effect is stable, not that it generalises to other photos.
Every photo is public, so some may have appeared in the models' training data.
Still photos can't show events that unfold over time. Drowning, for example, is a pattern over seconds.

What I'd measure next

Reasoning mode on vs. off: does thinking longer reduce false alarms, or just produce more confident explanations of them?
Grounding: ask for bounding boxes and check the model points at the hazard, not just names it. That would catch "visibly broken" swings that aren't.
Short video clips instead of single frames, which is where my drowning detector lives.
Arabic prompts, since safety systems in the UAE, where I'm based, often serve Arabic-speaking operators.

My Benchmark

Open the benchmark on Kaggle

Benchmark: https://www.kaggle.com/benchmarks/clivinjohn/safety-scene-vision-benchmark
Task 1 (neutral prompt): https://www.kaggle.com/benchmarks/tasks/clivinjohn/safety-scene-hazard-spotting
Task 2 (leading prompt): https://www.kaggle.com/benchmarks/tasks/clivinjohn/safety-scene-hazard-spotting-leading
Dataset with labels and photo credits: https://www.kaggle.com/datasets/clivinjohn/safety-scene-images

Built with the kaggle-benchmarks SDK (Apache-2.0). Photo credits and licences are listed in the dataset's ATTRIBUTION.md. Thanks to my friend for the independent labels.

I used an AI assistant (Claude) to help write the evaluation code, draft the labels and edit this post.

Over to you: if you build safety or monitoring systems, would you let a vision model raise the alarm, or only explain one? And what photo would you add to try to fool them? Tell me in the comments.