Freelance editors get a card full of interviews, B-roll, and one moment they cannot find. Smart Proxy Tagger is a localhost tool for that Side Hustle: paste a path, index the footage on the same machine, and search for a timestamp. Client media stays on the laptop. The only outbound media call is one spoken sentence when the index is finished, and that sentence never includes a frame or a file path.
The product is for a local library that can grow large. The judging demo is a short clip, sampled about every 5 seconds, with at most 24 moments per file. That interval lives next to FRAME_SAMPLE_INTERVAL_SECONDS in backend/frames.py. This post does not treat the demo clip as a multi-terabyte run.
Repo: github.com/devinuwa/smart-proxy-tagger
Demo:
Why open source matters
Uploading hours of 4K to a closed vision API is slow, and the bill grows with the length of the shoot. A freelance editor also should not send a client's unreleased footage off the laptop.
The indexer is built so a local open-weight model reads each sampled frame. In this submission, caption_frame is an explicit stand-in: it writes the file name and the timestamp, and the string says it is a stand-in. PaliGemma is the Gemma-family model that replaces that function. The workflow, the embedding step, and pgvector are already in place, so the swap does not add a cloud vision call. There is no per-minute hosted caption API in this build. Search embeddings are a local 384-dimension vector, stored in TimescaleDB with pgvector on 127.0.0.1.
Other people on DEV have been making the same trade. Kabilarasan B kept video annotation local with SAM 2.1. swmcc used a local vision model to index a photo archive. FacePika treated "no cloud uploads" as the constraint, not a feature to add later. Mukundakatta showed why a long job should checkpoint an index and resume there. That last idea is why the caption step is one Temporal activity with a heartbeat, not one activity per frame.
The tech stack and the sponsor tracks
-
Gemma. The caption activity calls
caption_frame. The running build labels its textphase2-stand-in. PaliGemma is the local model that function is written to call next, on this machine, with no frame leaving the laptop. - Temporal. Frame discovery, extraction, captioning, and the voice notification are one workflow. The caption step is a single long activity. It heartbeats the next frame index and resumes there if the worker retries, including after the laptop sleeps. The API returns a workflow id and does not wait for the footage to finish.
- Tiger Data. Search rows live in TimescaleDB with pgvector, published on localhost. Keyword full-text and vector similarity are fused, and each hit says which signal matched. A vector neighbor at cosine distance 0.85 or more is dropped, so a query with no overlap does not list every stored moment. This is not a hosted database.
- ElevenLabs. When the index finishes, one activity speaks: "Your footage is indexed." A missing key or a voice error fails that activity only. The moments stay searchable. The key is read on the server from the environment.
- Entire / DevRelay. The build session for this submission is embedded at the bottom of this post.
Show the code
The workflow stays free of database and HTTP imports, because Temporal replays it in a sandbox. Voice is a separate activity so a speech failure does not re-caption the footage.
self._progress["status"] = "captioning"
stored = await workflow.execute_activity(
"caption_and_store",
{**job, "frames": frames},
start_to_close_timeout=_CAPTION,
heartbeat_timeout=_HEARTBEAT,
retry_policy=RetryPolicy(maximum_attempts=5),
)
self._progress["frames_captioned"] = stored["frames_captioned"]
self._progress["status"] = "indexed"
Inside that activity, the heartbeat detail is the next frame index. Completed rows are already in the database, so a retry does not redo them.
start_index = 0
details = activity.info().heartbeat_details
if details:
start_index = int(details[0])
for index in range(start_index, len(frames)):
frame = frames[index]
caption = caption_frame(
frame["video_path"],
float(frame["timestamp_seconds"]),
frame.get("image_path"),
)
embedding = embed_text(caption)
await db.upsert_segment(
frame["video_path"],
float(frame["timestamp_seconds"]),
caption,
embedding,
)
activity.heartbeat(index + 1)
Try it
The README at github.com/devinuwa/smart-proxy-tagger has the local run steps: Docker for TimescaleDB and Temporal, a Python worker, FastAPI, and the Vite app at http://127.0.0.1:5173. Paste a full path, click Index footage, wait until the status is complete, then search a word from the caption. Closing the browser does not stop the worker.
Agent session
Build Smart Proxy Tagger, a localhost tool for videographers to index and search raw footage, for the Hacktoberfest Week 2 Side Hustle challenge. Keep footage on the machine. Use Temporal, local TimescaleDB with pgvector, and ElevenLabs only for a completion sentence.
The design is a local loop: the React app asks FastAPI to start a Temporal workflow, one caption activity heartbeats a frame cursor, and search reads TimescaleDB plus pgvector. ElevenLabs receives one sentence and never a frame or a client path.
The caption step is one activity, not one activity per frame. It heartbeats the next frame index and resumes there after a retry. A missing ElevenLabs key fails only the voice activity, so the footage stays searchable.
Search should show keyword and vector hits. A nearest-neighbor query with no shared words was still returning every row, so vector hits at cosine distance 0.85 or more are dropped before fusion.
I added the ElevenLabs API key and upgraded to the Creator plan. Will the existing key still work?
The same key still works after the plan upgrade. A library voice that previously returned HTTP 402 now returns audio. The key stays in .env and is never sent to the browser.
Group the scattered Python files in a folder named backend, then push to GitHub under the devinuwa account.
The modules now live in backend/. The worker and API start from the repository root with python -m backend.worker and uvicorn backend.server:app, so .env and var stay beside the frontend. Pushed to github.com/devinuwa/smart-proxy-tagger. Captions in this build are still the local stand-in: file name and timestamp. PaliGemma is the model that replaces caption_frame later. The demo samples about every 5 seconds, up to 24 frames, and is not a terabyte run.
Top comments (0)