Someone asks for yesterday's maximum temperature in Belgrade. We don't have it. We won't have it for another four days, and when we do have it, the number will be wrong — not badly wrong, but wrong enough that we'll replace it a month later, and replace it again a year after that.
That's not a bug report. That's the design.
I want to explain why, because it turns out the three-pass thing is the most consequential decision in the whole system, and every other piece — the API, the storage layout, the way a chart gets drawn — bends around it.
What the data actually is
DailyMeteo is a gridded daily climate archive. Four variables — maximum, minimum and mean temperature, and precipitation — at 1 km resolution, for all land on Earth, every day since 1 January 1961.
Some numbers, because the shape of the problem is mostly numbers:
| Resolution | 1 km, daily |
| Period | 1961-01-01 → present |
| Variables |
tmax, tmin, tmean, prcp
|
| Continental zones | 6, in the Equi7 projection |
| Rasters per day | 24 (6 zones × 4 variables) |
| One European tile | 8229 × 5588 px, 10.2 MB |
| Archive | ~11 TB |
Twenty-four rasters per day, roughly 24,000 days. That's around 580,000 GeoTIFFs, and it's why "just put it in a database" stopped being an option early.
The projection deserves a note. Equi7 splits the land surface into seven continental zones, each with its own equidistant projection, so that a kilometre is actually a kilometre wherever you are. A single global grid can't do that — it either stretches at the poles or bunches at the equator. The cost is that "the world" is six separate rasters that don't share a coordinate system, and every query has to work out which zone the point falls in before it can read a pixel.
How one day gets made
Weather stations don't cover the world. They cover the parts of the world that historically built weather stations, which is a very uneven map. Turning some thousands of point observations into a continuous 1 km surface is the actual work.
The pipeline pulls station observations from several archives — OGIMET, GSOD, GHCN-D, ECA&D, MeteoManz — because none of them alone is complete, and each has a different idea of what "yesterday" means. On a normal day that's something like 18,000 stations queried for a single variable.
Then, per day and per variable:
- Fit a trend. A linear model on three covariates: elevation from a DEM, topographic wetness index, and a geometric temperature trend — a deterministic function of latitude and day of year that carries most of the seasonal signal before any interpolation happens.
- Take the residuals. Observed minus trend, at each station.
- Krige the residuals in space and time. Not just "what do the neighbouring stations say today" but "what did this neighbourhood say yesterday and the day before", which matters enormously when a station is missing for a day.
- Add the two rasters back together. Trend surface plus interpolated residual surface.
Precipitation gets a second pass, because rain isn't like temperature. Temperature always has a value; rain is mostly zero. So it's modelled twice — first whether it rained at all, then how much, given that it did. Interpolating rainfall directly gives you a light drizzle over an entire continent, which is both wrong and hard to notice.
None of this is my invention. The method comes from Kilibarda et al. (2014), where Milan Kilibarda and co-authors established spatio-temporal regression kriging for global daily temperature at 1 km, and the interpolation runs through the meteo R package developed by Milan Kilibarda and Aleksandar Sekulić. What's ours is the part that turns it into something that runs every night and answers HTTP requests. It is not fast. A single day, all zones, all variables, takes hours on a machine with a terabyte of RAM.
Why the same day is computed three times
Here's the tension. Station data arrives late, and it arrives incrementally.
The archive that gives you a station's observation within a day or two is not the archive that eventually gives you the quality-controlled version. Some networks publish a preliminary value and revise it. Some publish nothing for weeks and then backfill a month at once. If you wait until the data is final, "today's weather" is a year old.
So we don't wait. Every day gets computed three times:
- early — about four days behind, from whatever stations have reported. Runs nightly.
- late — about four months behind, once the monthly archives have settled. Runs monthly.
- final — the following year, on the fully quality-controlled record. Runs yearly.
Each pass overwrites nothing. The three versions sit side by side on disk, distinguished by a suffix in the filename:
tmax_day_20200715_equi7_early.tif
tmax_day_20200715_equi7_late.tif
tmax_day_20200715_equi7.tif # final, no suffix
The API ranks them — final beats late beats early — and serves the best one that exists for the date you asked about. Right now that means anything before 2024 comes back final, and the last few weeks come back early. The transition is invisible in the response, which is either elegant or dishonest depending on your mood; I'll come back to that.
The ratio is heavily in favour of finished data. In the European maximum-temperature series: 23,010 final rasters, 851 late, 585 early. Ninety-five percent of the archive is settled. It's the leading edge that churns.
What it costs
Three variants of every file, ranked at read time, is a rule that lives in exactly one function. Rules like that get broken by accident.
We added the late stage after early and final were already running. Somewhere in the code that assembles a time series for a point query, the ranking knew about two variants and the directory now contained three. The unknown file didn't lose the ranking — it wasn't in the ranking at all, so it came through as a separate, additional row.
The result was a chart with two values for the same date. Not a crash, not an error, not a log line. Two dots where there should be one, on 3,624 days' worth of data, sitting there for however long it took someone to look closely at a chart.
The fix is three lines. The lesson isn't about the three lines. It's that "prefer the best available version" is a domain rule, and if it's implemented as an incidental sort somewhere in a view, it will drift the moment the domain gains a version. It belongs in one named function that every read path goes through, and adding a variant should be a change to that function and nothing else.
What you can actually do with it
The data is the product, but nobody wants an 11 TB tarball. What's on top of it:
Point queries. Click a location, get a series. The API takes a latitude, longitude, variable, and either a range or a list of dates, and returns timestamps and values. Daily, monthly or annual, aggregated or as long-term means over the two standard climate normals — 1961–1990 and 1991–2020.
GET /meteo/v2/pq/?var=tmax&agg_level=agg&time_scale=day
&from=2020-01-01&to=2020-12-31&lat=44.81&lon=20.46&api_key=…
Areas. Draw a polygon, get the aggregate over it, or export the clipped rasters. Large exports are priced by area and period and delivered by email when they're ready, because a polygon over Asia for a decade is not a request you answer inside an HTTP timeout.
Long-term means and anomalies. The monthly and annual aggregates and the two climate normals are precomputed — roughly 750 GB of them — so "how does this July compare to the 1991–2020 average" is a lookup, not a computation over 30 years of daily rasters.
R in the browser. There's an R environment on the site that runs entirely client-side, compiled to WebAssembly. You write R, it fetches from the API and plots, and your code never touches our servers. There's an assistant next to it that turns a question in English into R code against the same helpers. That one has enough engineering in it to deserve its own post, which is the next one.
Embeddable charts, because half the time what someone actually wants is a temperature curve on their own page.
The part I'd rather be honest about
Two things about this design are genuinely awkward and I don't have clean answers.
The seam is invisible. A response doesn't tell you which pass produced each value. Ask for a series spanning 2023 to now and you get final data and early data in one array, with no marker. For a chart, fine. For a paper, not fine. Exposing the processing level per value is the obvious fix and it's the kind of obvious fix that stays on the list because nothing visibly breaks without it.
"Available" and "complete" aren't the same thing, and we conflated them. A date is only fully covered when all 24 rasters exist. When one source fails — and sources fail; a station archive returned zero rows out of 18,000 for several days recently — the pipeline still produces the zones it can. So a date can have 22 of 24 rasters: present, queryable, and quietly missing two continents.
We were tracking "latest date with any raster". By that measure everything looked five days behind, which is normal. By the measure that matters — latest date with all 24 — we were six weeks behind and nobody had noticed, because no counter anywhere was counting that. Now there's an internal page that shows both numbers side by side, and the gap between them is the number I look at first.
That's the recurring shape of this whole system, honestly. The hard part was never the kriging. It's that a pipeline which degrades gracefully also fails quietly, and you have to go out of your way to build the thing that tells you it did.
The interpolation method and the meteo R package behind it are the work of Milan Kilibarda and Aleksandar Sekulić at the University of Belgrade, Faculty of Civil Engineering.
DailyMeteo is at dailymeteo.com. Next post: how 11 TB of rasters get served without a raster database, and what happened when we tried to run R inside a browser tab.
Top comments (0)