We run Kahve Tabela, an open atlas of 32,000+ registered heritage sites in Türkiye — castles, ancient cities, mosques, museums. Like everyone building a dataset on a budget, we filled the photo gaps from the usual open sources: Wikipedia article images, Wikimedia Commons geosearch, Google Places, Mapillary, the national heritage inventory.
Then a reader reported that a photo on one of our pages showed the wrong building. We checked. They were right. So we asked the obvious question nobody budgets for: how many of our 14,512 photos actually show the place they claim to show?
We put every single one through a local vision model (Qwen3-VL 30B, MLX, one Mac Studio, no API bill). Two findings — one about the sources, one about the audit itself — and the second one is the reason to read this.
Finding 1: error rate is a property of the SOURCE
First pass: show the model the photo, the place name, the province, and ask "could this photo belong to this place?"
| source | photos | flagged "mismatch" (pass 1) |
|---|---|---|
| Wikipedia article images (scraped from the relevant article body) | 300 | 64.3% |
| Wikimedia (title/filename matched) | 1,140 | 55.6% |
| Wikimedia Commons geosearch | 1,549 | 17.2% |
| National heritage inventory (Kültür Envanteri) | 6,210 | 9.5% |
| Google Places | 3,070 | 7.2% |
| Mapillary | 126 | 6.3% |
Read that top row again. Images we pulled from the Wikipedia article about the place itself were flagged nine times more often than photos from Google Places. The intuition "it's on the article, so it depicts the subject" is simply false at scale: articles carry maps, related buildings, portraits of people, the view from the site, the neighboring mosque. Scrape the article body and you inherit all of it.
Finding 2: our auditor was mostly wrong about being wrong
Before deleting anything we re-tried every flagged photo with a harder, humbler question. Pass 2 hid the place name entirely — the model saw only the image and the category: "does this show a castle or castle ruins? yes/no". No name, no province, nothing to reason from.
Pass 2 confirmed only 37% of pass 1's accusations.
The transcripts show why: given a name, the model reasons about the NAME. "Karbala is in Iraq." "This record is in Denizli but the photo looks like Elazığ." A photograph cannot testify about where it was taken — but a model that knows the place name will happily hallucinate geography and call the photo wrong. Identity questions invite hallucination; category questions are name-proof.
After requiring both passes to independently agree, the confirmed-wrong rates:
| source | confirmed wrong (both passes) |
|---|---|
| Wikipedia article images | 29.3% |
| Wikimedia Commons geosearch | 1.5% |
| Wikimedia (title-matched) | 1.2% |
| National heritage inventory | 1.2% |
| Mapillary | 0.8% |
| Google Places | 0.1% |
Two things survive the strictest standard: Wikipedia-article scraping is still catastrophically dirty — nearly one in three images does not show the subject at all — and Google Places is ~200x cleaner than it. And notice what happened to Wikimedia title-matching: 55.6% flagged, 1.2% confirmed. Most of those photos were fine; the auditor was the problem. If we had trusted pass 1 we would have deleted six hundred correct photographs.
What we actually did
- Deleted the 264 two-pass-confirmed wrong photos.
- Kept a per-photo provenance field so every image can be re-audited when models improve.
- Downscaled every image to 512px before inference — 10x throughput on one Mac Studio, no measurable verdict change on a 512-photo validation set.
- Rewrote our ingestion so source trust is explicit: Google Places and Mapillary photos auto-attach; anything scraped from a Wikipedia article body goes to a review queue.
If you're building a dataset (or a RAG pipeline)
- Provenance is a feature. Store where every asset came from; error rate is a property of the source, and you can't fix what you can't slice.
- Never audit with the name in the prompt. Ask what the model SEES, in a closed vocabulary, and compare against the record yourself.
- One model opinion is an accusation, not a verdict. Require two independent passes with different framings to agree before you delete.
- "It's from the article about X" is not evidence it depicts X.
The atlas and its data are open: the corpus is on Zenodo (DOI, CC-layered licensing per source) and Kaggle, and the whole thing is browsable at kahvetabela.com.
Methods note: pass 1 ran name+province+photo over all 14,512 records; pass 2 re-ran flagged photos name-blind with category-only questions on a 1,560-photo slice of the 2,118 flags. Confirmed-wrong = flagged by pass 1 AND rejected name-blind by pass 2 AND surviving a human spot-check of edge buckets. Model: mlx-community Qwen3-VL-30B (8-bit), temperature 0.
Top comments (0)