EARWITNESS

Ernesto Casablanca — Sun, 08 Nov 2026 11:58:53 +0000

Every photo and video is a witness.

A video is rarely just a sequence of pictures and sounds. Its frame timing can expose an edit. A faint electrical hum can narrow down when and where it was recorded. Street signs, accents, weather, compression artefacts, reflections and even stars can all reveal details that were never deliberately placed in the shot.

Professional forensic analysts know how to investigate those traces, but the relevant tools are fragmented, highly technical and often inaccessible to the people who first need an answer: journalists, fact-checkers and anyone handed a clip that supposedly “proves” something.

For HackTheBurgh 2026, we built EARWITNESS: a web-based forensics suite that questions a photo or video through 15 independent “witnesses” and turns their findings into a plain-language report.

Try it: earwitness.ussr.pl
Source code: github.com/HTB2026/HackTheBurgh2026

This project is a submission for HackTheBurgh 2026

The idea: leads, not proof

The central design decision behind EARWITNESS was not a model or an algorithm. It was a rule:

Every result must say what was measured, how confident the system is and where the method can fail.

Media forensics is full of signals that are useful but not conclusive. Metadata may contain a GPS position, but metadata can be removed or rewritten. Pixel statistics may look suspicious, but ordinary phone compression can create similar patterns. An AI model may recognise a city, but it can also be confidently wrong.

We therefore avoided building a single black-box “real or fake” detector. Instead, EARWITNESS treats each technique as an independent witness. Every witness submits a finding, supporting evidence, confidence and limitations. The final case summary weighs agreement and disagreement across the panel.

That structure makes the result more useful and more honest. One weak clue remains a lead. Several independent clues pointing to the same conclusion make a stronger case.

What the witnesses examine

The available checks depend on whether the uploaded file is a photograph or a video.

File and image evidence

EARWITNESS begins with evidence already present in the file:

File history reads metadata such as the recording device, date, location and software that has handled the file.
Compression anatomy examines JPEG quantisation, encoder characteristics, block structure and signs of repeated saves.
Hidden data accounts for bytes outside the visible image, including appended content, comments, private blocks and embedded thumbnails.
Pixel tampering searches for copied, pasted or differently processed regions.
AI-generation signals inspect declared provenance information and weaker statistical traces associated with synthetic imagery.
People analysis finds usable faces and examines details such as eye reflections without attempting to identify anyone.

These checks can expose useful inconsistencies, but none is treated as infallible. Re-uploading through a messaging platform may strip metadata and recompress an image, while normal camera processing can resemble editing.

Video and audio evidence

Video gives us temporal and acoustic signals as well:

Frame timing checks spacing, sudden visual changes, duplicated frames, dropped frames and possible joins.
Sound analysis converts audio into a visual representation and searches for tones, silences and abrupt changes.
Electrical hum looks for traces of the local power grid, which may indicate a grid frequency and, in favourable conditions, a recording time.
Light flicker examines artificial lighting for mains-related flicker and possible sensor timing clues.
Heartbeat looks for tiny periodic colour changes in visible skin.
Sound versus picture compares sharp sounds with visible events to detect synchronisation problems or replaced audio.
Speech transcribes spoken content with timestamps and identifies potentially useful names, places and times.
Sound sleuth uses AI to suggest background sounds, accents and languages that may point towards a location or time of day.

Some of these signals are extremely fragile. Compression, music, movement, filtered phone audio and poor lighting can erase them. EARWITNESS reports that absence rather than forcing a conclusion.

Location witnesses

For many investigations, the first question is simply: where was this recorded?

EARWITNESS combines visible text, scene understanding and external corroboration:

Text in the picture reads signs, screens, number plates and other labels.
AI eyewitness describes architecture, vehicles, uniforms, weather, terrain and other visible details.
Location from pixels estimates a likely country, region or place using only scene content.
Weather witness compares the claimed place and time with recorded local conditions.

The report leads with a proposed location, uncertainty radius and confidence level. It also shows the frame used, the matching reference imagery, and the clues that agreed or disagreed.

A second, sceptical AI is used as a verifier rather than another source of unchecked confidence. It challenges the initial hypothesis against satellite imagery, landmark photographs and street-level references before a claim is marked as verified.

Designing the investigation flow

A technically impressive detector is not very helpful if its output is a folder of plots and JSON files. We wanted the investigation itself to be understandable.

The user flow is deliberately small:

Upload a photo or video.
Choose Quick or Deep analysis.
Watch the witnesses run live.
Open each finding to inspect its evidence and limitations.
Read the combined verdict and location report.

Quick mode aims to produce useful leads in roughly a minute. Deep mode allows up to around ten minutes for more expensive analysis and corroboration.

Showing progress matters. A forensic pipeline can otherwise look frozen while it extracts frames, analyses audio or waits for an external model. Streaming each witness’s status makes the system feel like an active investigation and helps the user understand where the eventual answer came from.

From heterogeneous tools to one evidence model

The hardest engineering problem was not implementing one detector. It was making very different detectors behave like parts of the same product.

A metadata parser may return exact key-value pairs. A signal-processing check may produce frequencies, peaks and a diagnostic plot. OCR returns uncertain text with bounding boxes. A vision model returns prose. A weather lookup returns structured observations. Those outputs cannot be dropped directly into one interface.

We needed a shared conceptual contract for every witness:

status: waiting, running, complete, failed or inconclusive;
claim: the plain-language finding;
confidence: how strongly this witness supports its own claim;
evidence: frames, measurements, text or references behind the claim;
limitations: conditions that weaken or invalidate the method;
provenance: whether a statement came from deterministic analysis, an external source or AI inference.

This evidence model let us build the report around meaning rather than around whichever library produced the result. It also made failures local: if one witness cannot analyse a file, the remaining witnesses can still complete the case.

Building an honest verdict

We intentionally avoided reducing the entire report to a mysterious percentage.

For editing, the interface uses verdicts such as:

Edited — independent checks indicate that the file was cut or altered.
Possibly edited — something is inconsistent, but alteration is not established.
No signs of editing — the checks found no measured indication of a cut or alteration.
AI-generated — the file itself contains provenance declaring AI generation.

The wording is important. “No signs of editing” is not the same as “authentic.” A staged video can be recorded in one take. A generated image can be photographed from a screen. A synthetic file can also lose its provenance labels after re-encoding.

Likewise, pixel statistics may raise suspicion, but they do not by themselves prove AI generation. EARWITNESS reserves the strongest AI-generated verdict for explicit file-level provenance and labels weaker model-based indications as uncertain.

What we learned during the hackathon

1. Independent weak signals can be stronger than one impressive model

The most valuable result is often agreement between methods that fail differently. Metadata and OCR can be manipulated, geolocation models can hallucinate, and weather stations can be distant from the scene. When several of them converge, the user has a much better lead than any one method could provide.

2. Confidence needs context

A confidence number without an explanation creates false precision. We found it more useful to pair confidence with the evidence, the method’s assumptions and a visible uncertainty radius.

3. “Inconclusive” is a successful result

Many recordings simply do not contain measurable electrical hum, readable text or a usable face. Reporting that honestly is better than manufacturing an answer. A robust forensic product needs graceful abstention as much as it needs detection.

4. AI works better as a participant than as the judge

AI is useful for generating hypotheses from messy scenes, sounds and language. It is much less trustworthy when asked to produce an unquestioned final answer. In EARWITNESS, AI-generated observations are labelled, challenged and combined with deterministic measurements and external evidence.

5. Re-uploads destroy evidence

Messaging apps and social networks often strip metadata and recompress media. That can remove exactly the clues an investigation needs. Whenever possible, the original file should be analysed rather than a downloaded social-media copy.

What EARWITNESS cannot prove

We wanted the limitations to be part of the product rather than buried in a disclaimer.

EARWITNESS cannot guarantee that a file depicts a truthful event. It cannot prove authenticity merely because it found no edit. It cannot assume metadata is genuine. It cannot turn a model’s location guess into a fact without corroboration. Its case summary is only as strong as the witnesses underneath it.

The goal is not to replace a forensic analyst or the original source. It is to make sophisticated investigative leads available quickly, present the evidence behind them and help users decide what to verify next.

Where we would take it next

With more time, we would focus on evaluation and reproducibility rather than simply adding more detectors:

benchmark each witness against labelled, real-world media;
calibrate confidence separately for different devices and compression levels;
preserve complete evidence provenance and reproducible analysis parameters;
expand reference-data coverage for geolocation;
improve adversarial testing against deliberately manipulated files;
let investigators export a portable case bundle for independent review;
add stronger privacy controls and configurable data-retention policies.

A larger panel is only useful if each witness is measurable, auditable and clear about uncertainty.

Technical summary

Infrastructure as code on Vultr: We deployed EARWITNESS on Vultr and defined the cloud infrastructure using Terraform. This made the environment reproducible and allowed us to provision or rebuild the required compute and networking without relying on manual server configuration.
TigerCloud PostgreSQL persistence: Analysis jobs, extracted evidence, intermediate findings and final reports are stored in a managed PostgreSQL database provided by TigerCloud. Persisting each witness’s structured output allows the frontend to display live progress and lets users return to previously submitted cases.
Agentic development with Entire: We used Entire to organise and preserve our agentic development workflows. Its checkpoints helped connect AI-assisted coding sessions with the repository’s Git history, making generated changes easier to inspect, continue and audit.
A hybrid forensic pipeline: EARWITNESS combines deterministic extraction and signal processing with AI-enhanced analysis rather than relying on one black-box model. File metadata, compression characteristics, frame timing, audio signals and other measurable properties are evaluated alongside OCR, visual interpretation, sound classification and geolocation hypotheses. Every technique becomes an independent “witness,” so corroborating signals can strengthen a prediction while conflicting evidence remains visible.
Multiple interchangeable LLM backends: The AI layer supports OpenAI, Google Gemini and a self-hosted Gemma 4 deployment (on Vultr). Keeping model providers behind a common interface lets individual workflows use the most suitable backend, reduces dependence on a single external API and gives us a self-hosted option for greater control over cost and data processing.
Authenticated access through Auth0: Media analysis is computationally expensive, especially when several witnesses process video frames, audio tracks and external reference data concurrently. We therefore require users to authenticate through Auth0, helping us associate jobs with users, control access and protect the service against automated misuse and unbounded compute consumption.
Independent, failure-tolerant witnesses: Each forensic check produces a standard result containing its execution status, claim, confidence, evidence, limitations and provenance. This common contract allows very different tools—from metadata parsers to multimodal models—to feed the same report, while ensuring that one failed or inconclusive check does not prevent the rest of the investigation from completing.
Asynchronous live analysis: Uploaded media passes through multiple independent and potentially long-running checks. The interface exposes witness-level progress as the analysis runs, rather than making users wait for the entire pipeline to finish before seeing any result.
Two analysis depths: Quick mode prioritises fast, high-value checks, while Deep mode permits more computationally expensive processing and corroboration. This gives users a practical trade-off between turnaround time and investigative depth.
Explicit AI provenance: AI-derived observations are labelled separately from deterministic measurements and externally verified facts. A second, sceptical stage can challenge geolocation hypotheses against satellite, landmark and street-level reference imagery before they are presented as verified.

DEV Community: Ernesto Casablanca