Eleven Terabytes and No Raster Database

DailyMeteo is about 11 TB of GeoTIFFs — 1 km daily temperature and precipitation for all land on Earth, back to 1961. There is no raster database anywhere in the system. No PostGIS raster tables, no tile server in the read path for point queries, no object store. A Django app reads files off a disk.

That sounds like something we haven't gotten around to fixing. It isn't. Here's the reasoning, and the places where it bit us.

Two machines, one job each

The archive is produced on one machine and served from another.

The production machine is a big box — a terabyte of RAM, most of it in use — that does nothing but run the interpolation. Space-time kriging over a continent is not a workload you want sharing a server with anything user-facing, because it will happily consume every core for six hours and then a request times out for reasons no one will connect to weather.

The serving machine runs the API, the frontend, the database, GeoServer, the workers, and the reverse proxy, all as a Docker Swarm stack.

Between them, a rsync that runs every 30 minutes. Two details in it are load-bearing:

It pulls, it doesn't push. The serving machine reaches into the production machine and takes what's missing. Nothing runs on the production box on our behalf except a remote rsync under nice -n 19 ionice -c3, with a bandwidth cap. If the transfer is slow, the interpolation doesn't care. A push would have put the scheduling decision on the machine whose whole job is to not be interrupted.

It transfers incomplete dates. It used to only take dates where all 24 rasters existed. The effect was that when one zone failed, the perfectly good rasters for the other five stayed on the production box and the API reported the date as empty. Missing rasters are missing either way; this way, the ones that exist are at least reachable.

Why the filename is the index

A point query needs to answer: for this coordinate, this variable, and these 400 dates, which files do I open?

With files on a disk and a naming convention, that's string formatting:

{var}_day_{YYYYMMDD}_equi7{_early|_late|}.tif

Work out which continental zone the point falls in, build 400 paths, open the ones that exist. No index to keep in sync, no ingest step, no migration when a year of backfill lands, no second system that can disagree with the disk about what exists. The pipeline finishes writing a file and it is served, with no further action.

What that buys is worth being explicit about, because it's easy to reach for a database out of habit. There is no state anywhere that can drift from reality. A file is either on the disk or it isn't. When we needed to know exactly what we could serve, the answer was os.scandir over 24 directories — about a second for 580,000 entries — and it was the truth, not a cached projection of it.

What it costs:

• No query planner. "Every date where the July mean exceeded X anywhere in Europe" is not a question this shape can answer. It answers "give me these pixels from these files" very well and nothing else at all.
• Directories with 24,000 files. Fine on ext4, and scandir doesn't care, but ls in a terminal will make you wait and any tool that stats every entry becomes the bottleneck.
• The convention is the schema, and it's enforced by nothing. One file written with a different suffix is a silent data bug, not a constraint violation. This is exactly how we ended up serving duplicate values for 3,624 days — a variant appeared that the read path's ranking didn't know about.

Given the access pattern — read a handful of pixels from an arbitrary set of files, never scan or join — I'd make the same call again. The mitigation for the last point isn't a database. It's that every read path goes through one function that knows the convention, and that function is the only place the convention exists.

The file format does the work a database would

The reason reading raw files is fast enough is Cloud Optimized GeoTIFF.

Each raster is Int16, LZW-compressed, internally tiled in 512×512 blocks, with five levels of overviews baked in. A European tile is 8229 × 5588 pixels and 10.2 MB on disk.

The tiling is the point. To read one pixel you read one 512×512 block, not 10 MB. To draw a zoomed-out map you read an overview level that's already there, instead of downsampling the full raster. The layout means a reader can seek to the bytes it needs, and everything downstream — the point query, GeoServer's mosaics, the polygon clip — gets that for free.

Int16 instead of a float is a factor-of-two on 11 TB, which is worth roughly five and a half terabytes of disk. Temperature stored in tenths of a degree loses nothing anyone can measure.

What a request actually does

A point query is Django REST Framework, and the interesting parts are all about what not to do.

Requests are authenticated with an API key, priced in credits, and cached in Redis — a keyed lookup of an immutable historical value is the ideal cache entry, so the TTL is a week. Bulk exports don't happen in the request at all; they're Celery tasks that write a file and email a link, because "clip 20 years of daily rasters to this polygon" and "answer within an HTTP timeout" are incompatible requirements.

The polygon pricing had a nice bug in it, and it's the kind that only shows up at the edges. Cost is computed from pixels times days. Pixels came from the mask of the clipped raster — except the mask counts pixels inside the polygon, and a polygon over the ocean is full of nodata pixels that are inside it and contain nothing. Draw a box over the Atlantic and you'd be charged for a full raster's worth of nothing. The fix is one line — exclude nodata as well as masked — but the shape of the mistake is general: "how much data is here" and "how much of this rectangle is here" are different questions, and the second one is easier to compute, so it's the one you accidentally write.

Running R in someone else's browser

The part of the system I find most interesting isn't on the server at all.

There's an R environment on the site — a console, an editor, plots, tables — and it doesn't run R on our machines. It runs webR, R compiled to WebAssembly, inside the user's browser tab. Their code fetches from our API, computes locally, and renders locally. We never see what they ran.

That was a deliberate call. The alternative is an RStudio-shaped service where users execute arbitrary R on our infrastructure, and that is a sandbox problem, a resource-limits problem, a queueing problem and a security problem, forever. Moving it into the browser makes all four somebody else's — specifically, the browser's, which is very good at exactly this.

The bill comes as about 45 MB of WebAssembly and R packages, downloaded before anything runs. We warm the runtime up in the background as soon as the page loads, so by the time someone has typed a question the worker is usually alive. "Usually" is doing real work in that sentence, and the first visit on a slow connection is not a great experience.

The second cost is subtler. webR's default communication channel uses SharedArrayBuffer, and browsers only expose that to pages that are cross-origin isolated — which means serving two specific headers:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

The catch is that require-corp also blocks every cross-origin subresource that doesn't explicitly opt in. Turn it on site-wide and the map's vector tiles stop loading and half the images on the marketing pages disappear. So it's scoped to the R pages only, which means the R runtime lives on an island with different security semantics from the rest of the app, and every asset it needs has to be reachable from that island.

It's worth the trouble for one reason: SharedArrayBuffer is the only channel that supports interrupting a running computation. Without it, a runaway while (TRUE) can only be stopped by destroying the worker and booting a fresh 45 MB one. With it, there's a stop button that actually stops.

Two failures worth stealing

Both of these cost more time than they should have, and both look like someone else's bug until you find them.

nginx doesn't know what .mjs is. The bundled mime.types maps .js and stops there. Every ES module served straight off disk — including both worker entry points, the map's and R's — went out as application/octet-stream, and the browser refused them: "Strict MIME type checking is enforced for module scripts." Every other asset was fine, so the app looked healthy while two of its major features were dead.

The fix is a line in the Dockerfile. The part worth copying is the second line:

RUN sed -i 's|\(application/javascript[[:space:]]\+\)js;|\1js mjs;|' /etc/nginx/mime.types \
 && grep -q 'js mjs;' /etc/nginx/mime.types

If a future base image formats that entry differently, the sed silently does nothing and we're back to a broken worker discovered by a user. The grep turns that into a failed build. A patch applied by pattern-matching someone else's file should always be followed by an assertion that it took.

A URL nothing would accept. The assistant generates R against a helper with this signature:

get_data(var, agg_level, time_scale, from = NULL, to = NULL, time = NULL, lat, lon)

time takes a comma-separated list of dates, as an alternative to from/to. Asked for six years of daily data, the model enumerated every date into time. That's 2,192 dates and a 24,222-character URL.

webR failed with problem writing module_download template in internet module, which reads like a webR bug and sent me looking in the wrong place. It isn't. Our own API returns 414 above roughly 8,190 characters — Apache's default LimitRequestLine — so the same generated code, copied into RStudio, would have failed too:

  700 dates ( 7,810 chars) → 400   (request accepted)
1,000 dates (11,110 chars) → 414 URI Too Long
2,192 dates (24,222 chars) → 414

The tempting fix is to teach the model not to do that. We fixed the helper instead. Before building any URL, get_data now normalises time: a contiguous run collapses into a single from/to request, and a list with gaps is split into batches that keep every URL under 2,000 characters. The failing case went from 24,222 characters to 165, in one request, returning exactly the same 2,192 rows as from/to would.

The signature didn't change — it's the contract the model was trained against, and it's written into its system prompt. Only the body did.

That distinction is the whole point. When a language model generates code against your helpers, the helpers are the place to be defensive, because they're the part you control and the part that doesn't need retraining. Prompt changes are a request. Helper changes are a guarantee.

The blind spot

The last thing this architecture taught me is that it degrades quietly, everywhere.

A missing raster isn't an error; the date is just thinner. A failed source isn't an error; the day is produced with five zones instead of six. A transfer that skips an incomplete date isn't an error; the API simply has nothing for it. Every one of those is a reasonable local decision, and stacked together they mean the system can be substantially broken while every component reports success.

The number that finally made it visible was the difference between two dates: the most recent day with any raster, and the most recent day with all 24. The first was five days back, which is normal and looked fine. The second was six weeks back.

Nothing was alerting, because nothing was subtracting.

---

DailyMeteo is at dailymeteo.com. The previous post covers what the data is and why every day is computed three times.