DEV Community: Artur Azevedo

Building a State-Level Rollup From OpenStreetMap Surveillance Tags

Artur Azevedo — Fri, 06 Nov 2026 11:40:37 +0000

If you have ever built a "counts by state" page on top of OpenStreetMap data, you learn quickly that the hard part is not the query. It is deciding what a count means, and then making every derived number carry that meaning with it.

This is a write-up of the invariants I found useful. No numbers here, because any total I printed would be stale by the time you read it.

Start with the tag contract, not the map

The source is a shared, volunteer-maintained dataset. Contributors place a node at the physical location of a device and tag it as a surveillance feature, with a further tag identifying the device type as ALPR so it can be separated from generic CCTV. Where someone could tell, they also record the manufacturer, the compass bearing, the mounting, and a check date.

That is a loose schema. Fields are optional, values are free text, and contributors do not coordinate. Your pipeline has to assume that any field can be missing or malformed. A record with no bearing is still a valid record. A record with a free-text manufacturer you cannot normalize is still a valid record. Do not filter those out unless you are explicit that your page shows a subset.

The data methodology page for this map documents which records are included and why, which is the first thing a reader should be able to check.

The invariant that catches most bugs

Once you derive per-state rows from a single extract, every row must be traceable to the same source snapshot:

every derived row has the same as_of date as its source extract
sum of per_state counts == count of included nodes in that extract
count of state pages == count of distinct state keys in that extract

If those three do not hold, you either double-counted nodes that sit on a boundary, dropped rows with a missing state key, or mixed two extracts. Assert all three in the build step and fail loudly. It is a boring check and it catches nearly everything.

Boundaries are a data problem, not a geometry problem

Assigning a point to a state sounds like a point-in-polygon call until you hit the edges: water, roads that run along a border, and nodes tagged slightly off-position by a contributor standing on the opposite kerb. Pick one rule, write it down, and apply it identically to every node. Whether you use administrative boundaries or a nearest-centroid fallback matters less than being consistent and saying which one you used.

A point near a border that you assign differently from a competing map is not a bug in either project. It is a different documented rule. That is one reason published totals disagree.

Why totals disagree between projects

If two sites built on the same open dataset show different numbers, the usual causes are, in order:

Different extract dates. The dataset changes daily as volunteers add entries.
Different inclusion rules: all surveillance nodes, or only those with the ALPR type tag; all vendors, or one manufacturer.
Different geography: US only, or worldwide; whether the District of Columbia is its own row.
Different deduplication, especially where a contributor added both a node and a way.
Different boundary assignment, as above.

None of these implies someone is wrong. It means "count of documented cameras" is a derived metric with a definition attached. Print the definition next to the number, and print the extract date with it. A count without an as-of date is not a fact you can compare.

The state-level breakdown is an example of putting the rollup next to its own footnote instead of letting the number stand alone.

License and attribution are part of the schema

The dataset is distributed under the Open Database License. If you republish a derived extract, attribution and share-alike obligations travel with it. Put the credit and the license in the page template, not in a changelog nobody reads, and keep a plain link back to the source project.

What I would do differently

On a first pass I treated "unknown manufacturer" as a category to display. It is not a category, it is an absence, and readers consistently misread it as "a different vendor". Same for missing bearings: rendering a cone only for nodes that have one is fine, but the page has to say that cones represent recorded bearings, not an estimate.

The general lesson: every derived field should either carry its provenance or be visibly marked as unknown. In a volunteer dataset, absence of evidence is the most common value you will ever handle.

How a Daily Silver Benchmark Is Computed (And Why Unit Conversion Is the Hard Part)

Artur Azevedo — Fri, 06 Nov 2026 06:34:37 +0000

A daily metal "rate" looks like one number on a page. Under the hood it is a small data pipeline, and in my experience most of the bugs are not in the fetch step. They are in the units.

This is a purely methodological note. I am not quoting any live prices, tax rates or dealer margins here, because those change and are exactly the parts people copy-paste without checking.

Start from one fixing, not from a scraped list

The cleanest design is to pick a single authoritative wholesale fixing for a defined fineness, in a defined unit, for a defined trading day. Everything else on the page is derived from that one value.

If you instead scrape several "silver price today" widgets and average them, you have silently mixed:

different fineness assumptions,
different units (per gram vs per 10 grams vs per kilogram),
wholesale versus retail quotes,
and different as-of dates.

You cannot reconcile those after the fact. Pick one input, document it, and derive.

The conversion table is the real schema

Once the base is fixed, the derived numbers are arithmetic. A useful mental model is to treat every published figure as a row with four columns: unit, fineness, quote type, as-of date.

For a base expressed per kilogram of 999 fine metal:

Published unit	Divisor from the base
per 100 grams	10
per 10 grams	100
per gram	1000
per tola	1000 / grams-per-tola

Two traps worth writing tests for:

Fineness is a multiplier, not a new source. 925 and 999 describe the metal content of an object, not a different market price. Applying the fineness fraction to a number that is already retail-quoted double counts it.
Making charges are not part of the metal. A jewellery bill contains a craftsmanship component that resale usually does not return. If you fold that into a "price per gram" and then compare it with a wholesale fixing, the comparison is meaningless.

So the invariant to assert in code is boring but effective: per_kg == per_g * 1000 == per_10g * 100, and every derived row must carry the same as-of date as its source.

Why a retail quote never equals the benchmark

Readers often treat the difference as an error. It is not. A retail quote can legitimately include a dealer premium, a making charge, applicable tax, and a buyback policy that prices the same object differently on the way out. A benchmark is a reference for the metal content, not a price you can transact at.

That distinction belongs in the UI, not in a footnote. The most useful thing a price page can show is the basis of each number it prints: what unit, what fineness, what quote type, what date.

Data quality problems that actually bite

Trading day vs calendar day. A fixing published on Monday may be Friday's session after a holiday. Store both the publication date and the reference date.
AM vs PM sessions. Some sources publish more than one fixing per day. If you pick one, record which one; if you switch, you have silently changed the series.
Time zones. "Today" in the source's locale is not "today" for the reader. Normalise to a single zone before generating slugs or filenames.
Caching and staleness. A cached HTML widget can look perfectly valid and be a day old. Persist the fetched timestamp next to the value, and refuse to render if it is older than your threshold.
Anti-bot responses. A 403 is not a missing price. Treat "blocked" as a distinct state from "no data", because retrying a blocked fetch with a different parser will not help.
Silent unit changes upstream. If a source switches from per-kilogram to per-10-grams, an unconverted pipeline will happily publish a number 100× too small. Assert plausible ranges.

A small validation checklist

One documented base source, one fineness, one unit, one as-of date.
Derived values recomputed from the base, never re-scraped.
Unit-consistency assertions in the build step.
Explicit separation of benchmark, spot-derived conversion, and retail quote.
Reference date displayed next to every table, not just in the page footer.
A visible "no current data" state instead of a stale number.

None of this is glamorous, but it is the difference between a page that survives an audit and one that quietly publishes the wrong number for weeks.

Wrapping up

If you maintain any kind of daily price or rate page, the highest-value work is almost never adding another source. It is making the units, the fineness and the as-of date impossible to misunderstand — for the reader and for your own future self.

General methodology notes, not financial, tax or purchase advice. Rules and costs change; verify against primary source documents before relying on any figure.