What I Built

Somebody leaves a backpack on the grass outside the canteen. Hours later, security has to find out when, and who was near it. The usual answer is a person staring at a CCTV timeline, scrubbing back and forth until their eyes give up.

WatchTower is a local AI agent that does the screen part so people can do the outside part. You type the incident in plain English ("a blue backpack was left near the canteen around lunchtime"). WatchTower plans an investigation, searches the footage, follows the object over time, finds who was closest to it, pulls evidence frames and writes a report with a timeline. Then it drafts an alert for the right person and waits for a human to approve it. Staff spend their time walking to the actual spot, not sitting in front of a monitor.

It is for campus, facility and event security teams, and for anyone with long recordings who needs to find one moment.

Code

VINAYMADIVAL / WatchTower

WatchTower: an open-weight agent for video incident investigation

Staff describe an incident in plain English ("a blue backpack was left near the canteen around lunchtime") WatchTower plans the investigation, calls tools on the recorded footage, recovers from failures writes an evidence report, and asks a human to approve before any alert is sent. Everything runs locally; no footage leaves the machine.

Built at Hacktoberfest Hack Day Bengaluru '26 (IEEE CIS, RIT).

What the AI does

Role	Model
Planner: chooses tools, rewrites weak queries, writes the summary	Gemma 4 (open-weight, via Ollama)
Visual verifier: confirms what a saved frame actually shows	Gemma 4 (multimodal)
Detection + tracking	YOLOv8n + ByteTrack (Ultralytics)
Text-to-image retrieval	CLIP ViT-B/32 + FAISS

Model terms: Gemma 4 Terms of Use · CLIP: MIT · YOLOv8: AGPL-3.0 · FAISS: MIT

See docs/ARCHITECTURE.md for the full system diagram.

Agent tools

list_objects · search_video · track_history · nearby_tracks…

View on GitHub

How I Built It

The whole thing runs locally on a laptop. No cloud API sees the footage.

Gemma 4 (gemma4:e4b, via Ollama) is the agent's brain. It reads the request, chooses tools, rewrites weak searches and writes the summary. It also acts as a visual verifier, looking at a candidate frame to confirm what is actually in it.

YOLOv8n + ByteTrack detect objects and give each one a persistent track ID across frames.

CLIP ViT-B/32 + FAISS turn detected objects into embeddings so plain-English queries can search the video.

FastAPI + Streamlit + SQLite hold the tools, the trace and the operations screen.

The agent has tools for searching the video, following an object's track, finding nearby people, grabbing evidence frames, verifying a frame, writing a report and drafting an alert. Tool arguments are validated, and errors go back to the agent as data. Failures don't crash the run: it retries, changes its query, and if it cannot finish it still produces a partial report. Ollama's native tool calling failed once on my machine, so I added a structured JSON fallback that works with the same tools.

One honest moment from testing: on a clip with no backpack, the agent searched three ways and then reported that it found no evidence of one, instead of making something up.

Why Does Open Innovation Matter?

Security footage is exactly the data you should not upload to a closed API. With open-weight Gemma 4, YOLO and CLIP, WatchTower runs fully on the user's own machine, so nobody has to trade privacy for capability. I could also inspect and fix the pieces: when native tool calling failed, I could build a fallback, which is not possible with a black-box endpoint. And because the stack is open, a campus or small business can run it without paying per query.

WatchTower never identifies people: it describes objects and clothing only, and results are treated as leads for a human to verify, not proof.

DEV Community

WatchTower : an open-weight agent for video incident investigation