DEV Community: Srdjan Popovic

Ten Containers, One Developer: The Architecture of a Web GIS

Srdjan Popovic — Thu, 03 Sep 2026 10:43:50 +0000

I built a web GIS by myself. It holds 2.7 million road features, 51 GB of LiDAR and imagery, and runs as ten containers on one machine.

This is the map of it: what each piece does, why the boundaries are where they are, and — since I've spent several posts on this — where I got it wrong.

What it does

Road inventory. Surveyors drive a vehicle with a LiDAR scanner and a panoramic camera, and the result is a catalogue of everything alongside a road: signs, poles, kerbs, drains, cameras, bins, street lights. Users then work with that catalogue in a browser — draw, correct, measure, export.

Two things about that shape drive every decision below:

Users edit. This isn't a published dataset that refreshes nightly. Someone moves a sign and expects the map to show it moved.

The binary dwarfs the records. The database is 2.3 GB. One survey is a gigabyte on its own.

The containers

proxy        TLS, routing, caching
frontend     the browser app
backend      Django + DRF — auth, projects, layers, features
importer     FastAPI — uploads, imports, exports, mosaics
martin       vector tiles from PostGIS
titiler      raster tiles from Cloud-Optimised GeoTIFFs
db           PostgreSQL + PostGIS
pgbouncer    connection pooling
file-drop    bulk data ingest
ci           builds and deploys

Ten containers for one developer looks like over-engineering until you notice most of them are off-the-shelf processes doing one job. Only three contain code I wrote.

Why the boundaries are where they are

Django and FastAPI, both. Not fashion — a split by workload. Django holds the domain: users, roles, projects, layers, feature CRUD. It's a request/response application and the ORM and admin earn their keep.

Imports are not request/response. A 200 MB shapefile occupies a worker for minutes, and a synchronous endpoint gives the client no way to ask how it's going. So that work went to a separate async service with a job-and-poll shape, asyncio.to_thread for the blocking GDAL calls, and a semaphore to bound concurrency — because heavy geospatial work is memory-bound, and unbounded parallelism converts "slow" into "OOM-killed". → A Shapefile Is Four Files

Two tile servers. Martin renders vector tiles from PostGIS live. TiTiler serves raster tiles from COGs on disk. They share nothing but the reverse proxy, because vector and raster have nothing in common at this layer: one is a query, the other is a byte range in a file.

Features live in three tables, not one per layer. Points, lines and polygons, each with a layer_id. A table per layer would mean DDL every time a user clicks "new layer". The cost lands on tiles: Martin publishes views, so every layer gets a generated view filtered to its id, and auto_publish picks it up within five seconds. Create a layer in the browser, it's a live tile endpoint before you've finished naming it. → One View Per Layer

Big binary never enters the database. Point clouds, panoramas and orthophotos sit on disk and are served as static files. The database holds the trajectory — the line the survey vehicle drove — plus metadata and paths. That trajectory is 0.01% of the bytes and answers every question anyone asks. → One Gigabyte per Survey

pgbouncer in front of Postgres. Martin alone opens a hundred connections. Postgres does not enjoy that; a pooler does.

One pipeline per service repo. Push to the deploy branch, the image is built and the service restarted. Keeping each service in its own repository with its own pipeline means deploys are independent — a change to the tile server config never waits on a backend build, and a broken pipeline takes one service out of date rather than all of them.

The four things I got wrong

Writing these posts turned into an audit, and the audit found more than the posts did.

A cache with no TTL isn't a cache. Martin's built-in tile cache has no invalidation hook. A tile cached before an edit is served, byte-identical, for the life of the process — I measured the same 242 B tile long after the geometry under it had changed. It's off now, and nginx caches the same tiles with a five-second TTL instead, which absorbs the burst from panning a map without ever showing yesterday's geometry. → My Tile Cache Has No Invalidation

Users write SQL identifiers. Layer names become view names, and they were being interpolated into DDL with an f-string, unquoted and unvalidated. The tell wasn't a security scan — it was a layer somebody had named 1, whose view silently failed to exist because an unquoted identifier can't start with a digit. The same string that breaks the syntax could have completed it.

One dead container stopped nginx from starting. proxy_pass with a literal hostname resolves at config load, and nginx refuses to start if any name is missing. A stopped background service took down the entire entry point. Moving addresses into variables fixes it — and then quietly breaks trailing-slash path stripping, and then quietly breaks any if block sitting below the rewrite you added to compensate. → One Dead Container

A dropdown had the wrong EPSG code. Two coordinate systems in this region describe the same Gauss-Krüger zone with identical projection parameters and different datums — 427 metres apart, which lands in the right city and is therefore the dangerous kind of wrong. A third option was labelled as that zone but was actually a neighbouring country's grid, 5,000 km out. → Same Zone, Same Projection, 427 Metres Apart

What building it alone actually changes

Not the architecture. The boundaries above are the ones I'd argue for on a team.

What changes is who notices. Every one of those four defects had been in production for months, behind an error that was caught and logged and never read. No code review would have caught the EPSG label — you'd have to know the region. But a second person wondering aloud why one layer never renders would have found the injection in an afternoon.

So the thing I'd tell anyone in the same position: you are the code review, and you have to schedule it. Not "read your own code" — that finds nothing. Pick a subsystem, write down what it does as if explaining it to somebody, and check every claim against the running system as you go.

I set out to write a blog post about a tile server. What I got was a security fix, four bugs, and a config file that no longer has a password in it.

The stack, plainly


Backend	Django, DRF, PostGIS
Import/export	FastAPI, geopandas, GDAL, async jobs
Vector tiles	Martin, one view per layer, auto-publish
Raster tiles	TiTiler over COGs
Database	PostgreSQL + PostGIS, pgbouncer
Edge	nginx — TLS, routing, tile and raster caching
CI	one pipeline per service repo
Data	2.7M features · 102 layers · 51 GB imagery and point clouds

One Gigabyte per Survey, of Which 108 KB Goes in the Database

Srdjan Popovic — Thu, 27 Aug 2026 15:32:20 +0000

Here is the disk layout of one mobile mapping survey — a vehicle with a LiDAR scanner and a panoramic camera, driven along a road:

data/001_MMS/            507 MB    point cloud
orbit/oblak/             566 MB    spherical photos
trajectory/*.gpkg        108 KB    the path the vehicle drove

Just over a gigabyte. The database this feeds holds 2.3 GB in total — for 2.7 million road features across a hundred layers. Two more surveys and the binary data outweighs everything the database has ever stored.

So the question isn't how to put a point cloud in Postgres. It's what you put in Postgres instead.

The trajectory is the index

Of that gigabyte, one file goes into the database: the 108 KB trajectory, a GeoPackage holding the line the vehicle drove.

That line is what makes the survey findable. It draws on the map with everything else. You can ask which surveys cover a junction, which are newest, whether a stretch of road has been captured since the resurfacing. All the questions people actually ask are questions about where and when, and the trajectory answers every one of them at 0.01% of the storage.

The heavy files never enter the database. The row holds paths:

class Cloud(models.Model):
    name            = models.CharField(max_length=120, db_index=True)
    path_name       = models.CharField(max_length=120)   # -> octree metadata JSON
    orbit_url       = models.CharField(max_length=255)   # -> spherical photo index
    spherical_photo = models.BooleanField(default=False)
    recording_date  = models.DateField(null=True)
    source_srid     = models.IntegerField(null=True, choices=SOURCE_SRID_CHOICES)
    available       = models.BooleanField(default=True)

Metadata, geometry, and pointers. That's the whole trick, and it isn't clever — it's just the discipline to not reach for a bytea column.

Why not in the database

Postgres will happily store a gigabyte. It's the access pattern that kills you.

A browser point cloud viewer doesn't fetch a point cloud. It fetches an octree: a tree of small files, and as the user moves the camera it pulls the nodes covering what's in view at the detail level that's visible. Zoom in, it fetches deeper nodes. Pan away, it drops them. A single session issues hundreds of small ranged reads driven by mouse movement.

That is precisely the workload a static file server is built for, and precisely the one a database connection pool is not. Serving it through Django would mean an application worker occupied for every node fetch, connection pool pressure from mouse movement, and no benefit whatsoever — there is no query, no join, no permission decision per node beyond the one already made when the survey was opened.

nginx serves the directory directly:

location /media/ {
    alias /app/media/;

    # CORS - needed for local frontend dev servers (different origin/port)
    # fetching point cloud (Potree octree/hierarchy) and other media files
    # directly via fetch()/XHR. No credentials involved, so a wildcard is safe.
    add_header 'Access-Control-Allow-Origin' '*' always;
}

That wildcard needs the justification written next to it, which is why the comment is there. It's safe because no credentials ride along: the octree nodes are opaque binary that mean nothing without the metadata, and the metadata comes from the authenticated API. Change either of those facts and the wildcard becomes a mistake.

Where the size actually goes

27 GB   orthophotos
24 GB   prepared point clouds
74 MB   symbology (icons for signs, poles, cameras)
1.5 MB  project thumbnails

The orthophotos are the bigger half, and they follow a different path — served as Cloud-Optimised GeoTIFFs through a raster tile server, cached at nginx for a day. Different data, different access pattern, different tool. What they share is that neither one is in Postgres.

Note the shape of the tail: two entries measured in gigabytes, everything else in megabytes. That's typical, and it's the argument for treating "large binary" as its own tier rather than a column type. The 74 MB of symbology icons could live in the database without anyone noticing. The 24 GB could not.

The escape hatch, and what it costs

Not all data can be copied. Some surveys are enormous and already sitting on a storage array, and duplicating them to bring them into the system is not worth 500 GB.

So there's a symlink path: point the system at data that lives elsewhere, and it appears under media/external/<hash>/ as if it had been uploaded.

This works, and it has a cost that must be paid explicitly. A symlink is a reference the database doesn't own, so deleting the row has to clean up the link too — otherwise media/external slowly fills with pointers to nothing:

@receiver(post_delete, sender=Cloud)
def delete_cloud_symlinks(sender, instance, **kwargs):
    """Remove the symlink folders under media/external when a Cloud is deleted."""

Every "just point at the existing files" shortcut buys you disk and sells you a lifecycle problem. Worth it here — media/external is 116 KB of links standing in for far more — but the cleanup is not optional, and a signal is the cheapest place to guarantee it runs.

The field that has to be asked for

One more field in that model earns its place: source_srid.

The trajectory arrives in whatever coordinate system the surveyor worked in, and — as with shapefiles that ship without a .prj — the file often doesn't say. So the model offers the choices, and the pipeline prefers the file's declared CRS, falls back to the user's selection, and refuses to guess.

That dropdown is small and easy to get wrong. I found one of its options labelled with an EPSG code from the wrong country, which would have put a trajectory about 5,000 km from the road it was recorded on.

The rule

Put in the database what you'll query. Put on disk what you'll stream.

For survey data that means: the trajectory, the recording date, the coordinate system, the paths, and a flag for whether the thing is ready to view. Not the octree, not the panoramas, not the orthophoto.

The test I'd apply to any large asset: is there a question someone will ask that requires this to be in a table? For a point cloud the honest answer is no — every question is about the trajectory, which is 0.01% of the bytes and answers all of them.

A Shapefile Is Four Files, and the Important One Is Optional

Srdjan Popovic — Wed, 26 Aug 2026 11:10:44 +0000

A user clicks "upload" and picks a file. In most systems that sentence is the whole story.

In a geospatial system, the thing they picked is one of four files that only mean something together, the one carrying the most important piece of information is optional and frequently absent, and the geometry inside it almost certainly doesn't match your database schema.

This is what I learned building the ingest path for a web GIS that now holds about 2.7 million features.

First: it left Django

Import used to be a synchronous Django endpoint. Upload a file, the request thread parses it, writes rows, returns.

That works until the file is 200 MB of line geometry. Then the request occupies a worker for minutes, the client times out with no way to find out whether the import survived, and a second user doing the same thing takes out a second worker. Nothing is wrong; the shape is just incompatible with the work.

So imports moved to a separate FastAPI service. Not because FastAPI is faster — the parsing is done by GDAL and geopandas either way, and they don't care which framework called them — but because the async model makes the right structure cheap to express:

# the blocking part goes to a thread, not the event loop
footprints = await asyncio.to_thread(_compute_all_footprints, tiff_list, default_epsg=32634)

# and concurrency is bounded on purpose
semaphore = asyncio.Semaphore(4)

That Semaphore(4) is the important line. Heavy geospatial work is memory-bound before it's CPU-bound — a raster mosaic or a large shapefile can hold hundreds of megabytes while it's being read. Unbounded concurrency doesn't make imports finish sooner, it makes the container get OOM-killed while five of them are half-done.

The endpoint returns a job id immediately. The client polls. There's a progression table so "importing" can say what it's doing rather than just spinning.

The same job-plus-poll shape is now used for raster mosaics and for exports. That consistency is worth more than any of the individual implementations: three features, one mental model, one place to look when something is stuck.

A file is not a file

A shapefile is a set:

REQUIRED_SHAPEFILE_EXTENSIONS = ['.shp', '.shx', '.dbf']

.shp holds geometry. .shx is the index into it. .dbf holds the attributes. Hand a parser the .shp alone and you get an error that's about a missing index, not about a missing upload.

So validation runs over the set of filenames before anything is parsed:

def validate_shapefile_components(filenames):
    extensions = {Path(f).suffix.lower() for f in filenames}
    missing = [e for e in REQUIRED_SHAPEFILE_EXTENSIONS if e not in extensions]
    if missing:
        return False, f"Missing required shapefile components: {', '.join(missing)}"
    return True, ''

Users mostly upload a ZIP, which the service extracts and then validates. The error message names the missing extension, because "invalid shapefile" tells someone nothing they can act on and "missing .dbf" tells them exactly what to go find.

The important file is the optional one

Notice what isn't in that required list: .prj.

.prj is the file that says which coordinate system the numbers are in. Without it, a shapefile contains coordinates like 7457000, 4958000 and no statement of what they mean. Those could be metres in one of several national grids, and picking wrong puts the data hundreds of metres — or thousands of kilometres — from where it belongs.

It isn't required because it's genuinely, frequently absent. Decades of surveying data was delivered as three files, in a projection everyone in the room already knew. That knowledge lived in people, not in the archive.

Which is why the upload form has to ask, and why "what coordinate system is this in?" is a dropdown the user must answer rather than something the software detects. That dropdown is its own story — I wrote about the day I found one of its options was labelled with the wrong EPSG code, pointing at a different country's grid.

The pipeline handles it in the right order: use the file's own CRS if it declares one; fall back to what the user selected; refuse to guess.

Your schema will not match their data

The three feature tables are typed. Deliberately:

project_pointfeature     POINT            3 dims   SRID 4326
project_linefeature      MULTILINESTRING  3 dims   SRID 4326
project_polygonfeature   MULTIPOLYGON     3 dims   SRID 4326

Uploaded data essentially never arrives in that shape, so every import performs three coercions.

Reproject to 4326. One SRID in storage, always. Anything else means every query has to know what it's holding.

Single to multi. A shapefile can contain a Polygon where the next feature is a MultiPolygon — the format allows both, and real files mix them. A column typed MULTIPOLYGON rejects the plain one. Promoting every geometry to its multi form is a one-line transform that removes an entire class of "some features imported and some didn't".

2D to 3D. This one is the least obvious and the one I'd defend hardest:

def _ensure_3d(geom, z_value=0.0):
    if isinstance(geom, Point):
        return geom if geom.has_z else Point(geom.x, geom.y, z_value)
    ...

The columns are three-dimensional, and today every single row has a Z value — 1,820,288 points, 697,009 lines, 171,830 polygons, without exception. Not because all the source data had elevation, but because anything that didn't got lifted to z = 0 on the way in.

That looks like storing a fake number. It's a deliberate trade: this data sits alongside LiDAR surveys where Z is real and load-bearing, and a table where some geometries have Z is worse than either alternative. Every query that touches elevation would need to know which rows to trust, and PostGIS functions behave differently on mixed-dimension inputs. One dimensionality, one code path, and z = 0 that is explicitly a placeholder.

If you have no 3D data at all, store 2D. What you should not do is let the dimension depend on whichever file happened to be uploaded.

What I'd carry to the next one

Move long work out of the request before you're forced to. The rewrite is cheap while it's one endpoint and expensive once three features have grown their own half-solutions. Job id, poll, progress — same shape every time.

Bound your concurrency explicitly. For memory-heavy work, unbounded parallelism converts "slow" into "OOM-killed", which is much harder to debug because it takes unrelated jobs down with it.

Validate the set, not the file. Where a format is really several artefacts, check them together and name what's missing.

Normalise aggressively at the boundary. Reproject, promote to multi, fix the dimension — once, on the way in, where you can still reject the whole upload cleanly. Every check you skip there becomes a conditional in every query that reads the table afterwards.

Ask for what the file can't tell you. Some information genuinely isn't in the data. Building a place for the user to supply it beats inferring it, and inferring it beats a default that's silently wrong.

One Dead Container Stopped nginx From Starting At All

Srdjan Popovic — Tue, 25 Aug 2026 08:35:05 +0000

Eight containers behind one nginx. One of them was stopped. nginx would not start.

Not "returned 502 for that service" — would not start, at all, with every other service running fine:

host not found in upstream "importer"

That is one dead background service taking down the entire public entry point: the frontend, the API, the tile server, all of it. This is a short post about why, the three-line fix, and the two things the fix breaks that nobody mentions.

Why it happens

The usual framing is "nginx caches DNS". That's true but it undersells it. With a literal hostname:

location /importer/ {
    proxy_pass http://importer:8001/;
}

nginx resolves importer once, while parsing the config, and bakes the address into the running configuration. Two consequences follow, and the second is the dangerous one:

If the container restarts and Docker gives it a new IP, nginx keeps sending traffic to the old one until you reload.
If the name doesn't resolve at parse time, the config is invalid. nginx exits.

Point 2 makes the blast radius total. Every upstream becomes a startup dependency of every route. Restart your reverse proxy at the wrong moment — during a deploy, after a host reboot when containers come up in an unlucky order — and it will refuse to come back because of a service nothing on the critical path needs.

The fix

Put the address in a variable and give nginx a resolver:

resolver 127.0.0.11 valid=10s ipv6=off;
resolver_timeout 5s;

set $svc_importer http://importer:8001;
set $svc_api      http://api:8000;
set $svc_tiles    http://tiles:3000;

location /importer/ {
    proxy_pass $svc_importer;
}

When proxy_pass contains a variable, nginx defers resolution to request time. Now a missing container is a 502 on its own route, which is what you wanted all along. Everything else keeps serving.

Two details in that resolver line:

127.0.0.11 is Docker's embedded DNS server, present inside every container on a user-defined network. On another platform it's whatever your service discovery exposes.

ipv6=off is not optional decoration. Without it nginx asks for AAAA records too, and if it gets one for a service listening only on IPv4, the connection fails in a way that looks nothing like a DNS problem. Turn it off unless your services actually speak IPv6.

valid=10s caps how long a resolved address is reused. Short enough that a container restart heals on its own; long enough that you're not doing a DNS lookup per request.

What the fix breaks, part one: the trailing slash

This is where it stops being a three-line change.

These two are not the same directive:

proxy_pass http://tiles:3000/;   # note the trailing slash
proxy_pass $svc_tiles;           # variable

With a literal URI and a trailing slash, nginx strips the matched location prefix before forwarding. /tiles/catalog arrives upstream as /catalog.

With a variable, nginx cannot do that — it doesn't know at parse time what the URI part is — so it forwards the full original path. /tiles/catalog arrives upstream as /tiles/catalog, and your tile server returns 404 for everything.

You have to do the stripping yourself:

location /tiles/ {
    rewrite ^/tiles/(.*)$ /$1 break;
    proxy_pass $svc_tiles;
}

Nothing warns you. The config is valid, nginx starts, and the route 404s.

What the fix breaks, part two: `if` below `rewrite`

And now the second-order one, which cost me considerably more time.

CORS preflight is normally handled with a short-circuit:

location /tiles/ {
    rewrite ^/tiles/(.*)$ /$1 break;

    if ($request_method = 'OPTIONS') {
        add_header 'Access-Control-Allow-Origin' '*';
        add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
        return 204;
    }

    proxy_pass $svc_tiles;
}

That if never runs.

rewrite and if both belong to ngx_http_rewrite_module, which evaluates its directives in order — and the break flag stops that evaluation for the rest of the block. Every rewrite-module directive after it, if included, is skipped.

So the preflight falls through to proxy_pass, the browser gets whatever the upstream says about OPTIONS, and your CORS headers never appear. The failure shows up in a browser console as a CORS error, which sends you looking at add_header and origins — the two things that were correct all along.

The fix is ordering, not content:

location /tiles/ {
    # OPTIONS must come ABOVE the rewrite: `rewrite ... break` halts
    # ngx_http_rewrite_module, and `if` is a directive of that same module.
    if ($request_method = 'OPTIONS') {
        add_header 'Access-Control-Allow-Origin' '*';
        return 204;
    }

    rewrite ^/tiles/(.*)$ /$1 break;
    proxy_pass $svc_tiles;
}

Note that this ordering trap only appears because of the earlier fix. With a literal proxy_pass and a trailing slash, there is no rewrite, so there is nothing to halt the module and the if works wherever you put it. Fixing the startup dependency introduced the rewrite; the rewrite introduced the ordering constraint.

Worth knowing

Test your reverse proxy with a service stopped. Not a service returning errors — a service not running. That's the state a host reboot produces, and it's the one that turns a single failure into a total one. It takes thirty seconds to check and it's the only way to find this before it finds you.

A variable in proxy_pass changes URI handling, not just resolution timing. If you're converting an existing config, every proxy_pass that ended in a slash needs a rewrite to replace it.

In nginx, module ordering beats block ordering. Directives from the same module run as a sequence, and break ends that sequence. if, rewrite, return and set all belong to the rewrite module, so their relative order matters in a way that add_header and proxy_set_header don't.

Three lines to fix. Two more to keep it working.

One View Per Layer: Four Sharp Edges I Found in My Own Code

Srdjan Popovic — Mon, 24 Aug 2026 12:21:48 +0000

There is a layer in my database called 1.

Somebody created it, presumably by accident, and it sat there for months looking harmless. It was the only layer in the system that never served a single tile, and nobody noticed, because it was empty anyway.

That layer turned out to be a symptom of a SQL injection vulnerability. This post is about the design that produced it — which I still think is a good design — and the four things I got wrong inside it.

The setup

A web GIS with about 2.7 million features: 1.8 million points, 697,000 lines, 172,000 polygons. Users create layers through the UI, upload data into them, edit geometry, and expect to see it on a map.

The features do not live in a table per layer. They live in three tables — one for points, one for lines, one for polygons — with a layer_id foreign key and a JSON column for attributes:

project_pointfeature     1,820,288 rows
project_linefeature        697,009 rows
project_polygonfeature     171,830 rows

That's a deliberate trade. A table per layer means DDL every time a user clicks "new layer", a migration story that never ends, and a schema that drifts. Three generic tables mean one schema, one set of indexes, and layers that are just rows in a metadata table.

The cost lands on the tile server.

The pattern

Martin serves vector tiles from PostGIS. Point it at a database and it discovers spatial tables and views and publishes each as an MVT endpoint. It can be told to publish views but not tables:

postgres:
  auto_publish:
    from_schemas: [public]
    publish_tables: false
  reload_interval: 5s

So: give every layer its own view. A Django post_save signal on the Layer model creates it:

CREATE OR REPLACE VIEW t19_saobracajni_znakovi AS
    SELECT f.id, f.feature_attrs, f.geom, f.layer_id,
           l.name AS layer_name, lg.name AS layer_group_name,
           p.title AS project_title
    FROM project_pointfeature f
        JOIN project_layer l  ON f.layer_id = l.id
        JOIN project_layergroup lg ON l.layer_group_id = lg.id
        JOIN project_project p ON lg.project_id = p.id
    WHERE f.layer_id = 81

A user creates a layer in the browser. Five seconds later — one reload_interval — it is a live tile endpoint. No migration, no deploy, no restart. There are 106 of these now.

I still like this. Everything below is what it cost.

Edge 1: the layer named `1`

The signal built the view name like this:

view_name = f"{instance.name.replace(' ', '_')}"
cursor.execute(f"CREATE OR REPLACE VIEW {view_name} AS ...")

Layer.name is a CharField filled in by users, with no validation on it. It goes straight into DDL through an f-string.

A layer named x" AS SELECT 1; DROP TABLE project_layer; -- executes on save, with whatever privileges the application's database role happens to hold.

I did not find this by thinking about attackers. I found it because of the layer called 1, whose view was missing:

CREATE OR REPLACE VIEW 1 AS ...
-- ERROR: syntax error at or near "1"

An unquoted identifier can't start with a digit. The exception was caught, logged, and swallowed, and that layer quietly had no tiles. A user-supplied string that breaks SQL syntax is the same string that could complete it. The failure was the tell.

The fix is psycopg2.sql:

from psycopg2 import sql

stmt = sql.SQL("CREATE OR REPLACE VIEW {view} AS ... WHERE f.layer_id = {lid}").format(
    view=sql.Identifier(view_name),
    lid=sql.Literal(layer.id),
)

Rendered with a hostile name, the whole thing lands inside one quoted identifier with the quote doubled:

CREATE OR REPLACE VIEW "x""_as_select_1;_drop_table_project_layer;_--" AS SELECT ...

And "1" is now a perfectly legal view name, so that layer works too.

Edge 2: quoting changes your names

This is the part that would have caused a bad afternoon if I'd shipped the obvious fix.

An unquoted identifier in Postgres is folded to lower case. A quoted one is not. Every one of those 106 views was created unquoted, so they're all lower case in the catalog — while 23 layers have capital letters in their names.

Switch naively to sql.Identifier and T53_Traffic_Cameras stops resolving to the existing t53_traffic_cameras and creates a second view beside it. The old one keeps existing. Martin keeps publishing both. Half your layers quietly fork.

So the derivation has to reproduce what Postgres was doing implicitly:

def normalize_layer_name(layer_name):
    name = layer_name.strip().replace(' ', '_')
    # Postgres folds only ASCII A-Z. str.lower() would also fold Cyrillic
    # and diverge from the names already in the catalog.
    return ''.join(c.lower() if 'A' <= c <= 'Z' else c for c in name)

That ASCII-only detail matters here: four layers have Cyrillic names, and str.lower() would have renamed them. The existing views were created by Postgres's rule, not Python's, and the two disagree outside ASCII.

I verified it before merging by running the new function over every layer and diffing against the catalog: 102 layers, 0 names changed. That check took a minute and was the only thing standing between me and 23 orphaned views.

Edge 3: two code paths, two shapes

There were two places that created these views: the signal, and a management command for bulk regeneration. Over time they diverged. The command included symbology columns for point styling; the signal didn't.

The database ended up holding two shapes of the same thing — 15 views with symbology columns, 24 identical point layers without.

That's not just untidy, because of a rule worth memorising:

CREATE OR REPLACE VIEW can add columns at the end. It cannot remove them, and it cannot reorder them.

So saving a layer whose view had the "wrong" shape failed with cannot drop columns from view. Caught, logged, swallowed — and the view silently kept its old definition. This had already caused a 500 on an unrelated endpoint, because the failed statement poisoned the caller's transaction.

Two fixes. First, the column set is no longer a flag anyone can pass; it's derived from the geometry type, because only the point table has a symbology_id column — the command had been passing with_symbology=True for lines and polygons too, where it could only ever have failed.

Second, and this is the useful bit: the optional columns moved to the end of the SELECT list. Because CREATE OR REPLACE can append, 24 views could be brought into line with no interruption at all. Only the 15 that needed reordering required DROP + CREATE:

102 layers, 63 already aligned
 24 replaced in place   (no downtime)
 15 required DROP       (brief lock, inside a transaction)

Column order is invisible to clients — MVT attributes are named. Choosing it deliberately turned most of a migration into a no-op.

Edge 4: the view namespace is global, the layer namespace wasn't

Layer names were unique per project. View names are unique per database.

So two projects both had a layer called tacke. Both mapped to one view. Whichever was saved last owned it, and the other layer served the wrong project's features — with no error anywhere, because from Postgres's point of view nothing was wrong.

The tempting fix is to rename the views: layer_81 instead of tacke. It's correct, and it's a coordinated deploy — tiles are requested by name, so every client has to change on the same day.

The cheaper fix follows from noticing why the frontend works at all: it builds the tile URL from the layer name it reads from the API. If layer names are unique, tile addresses are unique for free. So the constraint belongs on the layer name, not on a new naming scheme:

def validate_name(self, value):
    holder = layer_holding_view_name(tile_view_name(value),
                                     exclude_pk=self.instance.pk if self.instance else None)
    if holder:
        raise ValidationError(...)

Comparing the derived name, so Tacke and tacke collide, and so do centralne linije and centralne_linije. One of the two existing duplicates was an empty layer in a test project; renaming it cost nothing, and the frontend followed automatically because it reads the name from the API rather than remembering it.

While I was in there: renaming a layer created a view under the new name and left the old one behind, publishing a source no layer pointed at. A pre_save now remembers the previous name so post_save can drop it — unless another layer is using it.

Would I build it this way again

Yes, with the edges filed off.

A generic feature table plus a view per layer gets you user-created layers that become tile endpoints in seconds, without DDL migrations or deploys, and Martin's reload_interval does the discovery for free. For an internal tool where people create layers as part of their work, that's the right shape.

But it means your users write DDL identifiers, indirectly, by typing a name into a form. Once you accept that, four things follow, and I got all four wrong first:

Compose DDL with sql.Identifier, never an f-string. The failure that reveals it may look like a syntax error, not an attack.
If you're adding quoting to something that ran unquoted, reproduce the old folding exactly and diff every existing name before you ship.
Put optional columns last, so CREATE OR REPLACE can add them without a drop.
Check whether your derived namespace is wider than the namespace you enforce uniqueness in. Ours was, by exactly one level.

The layer named 1 serves tiles now. It's still empty.

Same Zone, Same Projection, 427 Metres Apart

Srdjan Popovic — Sun, 23 Aug 2026 07:36:47 +0000

I was writing a blog post about the architecture of a system I built when I found a bug in it. This is that bug, and the much more interesting thing standing behind it.

The system ingests mobile mapping surveys — LiDAR trajectories with spherical photos along them, driven along roads. The files arrive in whatever coordinate system the surveyor was working in, which in this part of the world is not a settled question. So the upload form asks the user to pick one:

SOURCE_SRID_CHOICES = [
    (32634, 'UTM zone 34N (EPSG:32634)'),
    (31277, 'Gauss-Kruger zone 7 (EPSG:31277)'),
    (8686,  'Gauss-Kruger zone 7 (EPSG:8686)'),
]

Two codes, one label. That was deliberate — I knew there were two definitions floating around for the same zone and wanted users to be able to pick the one matching their source.

I was wrong about what the second one was.

What the database says

You don't have to guess about an EPSG code. If you're running PostGIS, the definition is sitting in spatial_ref_sys and you can just read it.

SELECT srid, proj4text FROM spatial_ref_sys WHERE srid IN (31277, 8686);

31277 | +proj=tmerc +lat_0=0 +lon_0=21 +k=0.9999 +x_0=7500000 +y_0=0
        +datum=hermannskogel +units=m +no_defs

 8686 | +proj=tmerc +lat_0=0 +lon_0=15 +k=0.9999 +x_0=500000 +y_0=0
        +ellps=bessel +towgs84=476.08,125.947,417.81,... +units=m +no_defs

Look at lon_0. One is on the 21st meridian, the other on the 15th. Look at x_0: 7,500,000 versus 500,000.

These are not two definitions of the same zone. 8686 is MGI 1901 / Slovenia Grid — a different country's national grid, six degrees west.

How far wrong? Take a coordinate pair from Belgrade and push it through both:

SELECT ST_AsText(ST_Transform(ST_SetSRID(ST_MakePoint(7457000, 4958000), 31277), 4326));
-- POINT(20.456 44.765)   Belgrade. Correct.

SELECT ST_AsText(ST_Transform(ST_SetSRID(ST_MakePoint(7457000, 4958000), 8686), 4326));
-- POINT(76.635 25.100)   Rajasthan, India. About 5,400 km off.

Nobody had used the option yet — every record in the table had the field null — so this cost nothing except my confidence. But it's in a dropdown, labelled invitingly, waiting.

Why that mistake was easy to make

Here's the part that turns a careless bug into something worth writing about.

The modern EPSG codes for this family are the MGI 1901 / Balkans zone N series. If you go looking for them in spatial_ref_sys, you get this:

SELECT srid, srtext FROM spatial_ref_sys WHERE srtext LIKE '%MGI 1901%Balkans%';

3907  MGI 1901 / Balkans zone 5   lon_0=15
3908  MGI 1901 / Balkans zone 6   lon_0=18
3909  MGI 1901 / Balkans zone 7   lon_0=21
3910  MGI 1901 / Balkans zone 8   lon_0=24

8677  MGI 1901 / Balkans zone 5   lon_0=15
8678  MGI 1901 / Balkans zone 6   lon_0=18
8679  MGI 1901 / Balkans zone 8   lon_0=24

Read the second block again. 8677 is zone 5, 8678 is zone 6, and 8679 is zone 8.

The 867x series skips zone 7. There is no 8679 = zone 7 to find, because zone 7 already had 3909 and the later batch didn't re-issue it. So anyone scanning that numeric range for "the modern zone 7 code" finds a gap where their pattern-matching brain expects a hit, and the nearby numbers are all real, plausible, MGI-1901-family codes.

8686 is one of those nearby numbers. It is a real code. It belongs to the same datum family. It is in the same region of Europe. It is simply the wrong country.

The correct code for Gauss-Krüger zone 7 in this region is EPSG:3909.

The trap that actually matters

Fixing the label is a one-line change. The thing worth your attention is the other pair in that dropdown, because it fails in a much more dangerous way.

31277 and 3909 both describe Gauss-Krüger zone 7. Same central meridian, same scale factor, same false easting, same ellipsoid:

3909   +proj=tmerc +lat_0=0 +lon_0=21 +k=0.9999 +x_0=7500000 +ellps=bessel
       +towgs84=682,-203,480,0,0,0,0

31277  +proj=tmerc +lat_0=0 +lon_0=21 +k=0.9999 +x_0=7500000
       +datum=hermannskogel

Every projection parameter is identical. The only difference is how the datum reaches WGS 84: an explicit three-parameter shift in one, PROJ's built-in Hermannskogel datum in the other.

Same input through both:

3909   POINT(20.451342 44.765252)
31277  POINT(20.456732 44.765230)

-- distance between them
426.73 metres

427 metres. From an identical-looking projection definition, differing only in a datum parameter you have to scroll sideways to notice.

And here is why that is worse than the 5,400 km error: it looks fine.

Five thousand kilometres announces itself. You load the layer, it isn't on the map, you find the problem in thirty seconds. Four hundred metres puts your data in the right country, the right city, the right neighbourhood. Zoom out to a national view and both versions are the same pixel. Everything renders. Nothing errors.

It only becomes visible when someone zooms in far enough to notice that the road inventory is on the wrong side of the road — in a dataset whose entire value proposition is sub-metre positional accuracy.

Why there are two in the first place

Briefly, because it explains why you can't just delete the old one.

31277 is marked deprecated in the EPSG registry. It's the older realization, and PROJ resolves its datum through Hermannskogel. 3909 is the current code with an explicit shift to WGS 84.

But deprecated doesn't mean unused. Decades of surveying in this region was done, stored, and delivered in the older definition, and files still arrive that way. If you only offer the modern code, you silently apply the wrong datum shift to legacy data — the same 427 m error, in the other direction.

So both belong in the dropdown. What doesn't belong is giving them the same label.

What I changed

The codes stay. The labels stop lying:

SOURCE_SRID_CHOICES = [
    (32634, 'UTM zone 34N — WGS 84 (EPSG:32634)'),
    (3909,  'Gauss-Krüger zone 7 — MGI 1901 (EPSG:3909)'),
    (31277, 'Gauss-Krüger zone 7 — MGI/Hermannskogel, legacy (EPSG:31277)'),
]

8686 is gone. 3909 replaces it. The two remaining zone 7 entries name their datum, because the datum is the entire difference between them.

What I'd take from this

Read spatial_ref_sys instead of trusting the label. The definition is in your database. SELECT proj4text takes five seconds and tells you the central meridian, the false easting, and the datum. Every mistake in this post was visible in that one column.

Compare lon_0 and x_0 first. They're the parameters that produce catastrophic, obvious errors, so they're the cheapest to check. If those match and you still have two codes, your difference is in the datum — which is the expensive, quiet kind.

Add a plausibility check on transformed coordinates. One bounding box test — does the result land inside the country this data is supposed to be in — would have caught the Slovenia Grid mistake automatically, at upload time, before it ever reached a map. It would not have caught the 427 m one, which is the point: cheap checks catch the loud failures, and you need to know they don't catch the quiet ones.

Never label two coordinate systems identically. If the user has to choose between them, the label has to contain the thing that differs. "Gauss-Krüger zone 7" twice is not a choice, it's a coin flip with a 427 m stake.

Gap-filled numbering, deprecated-but-still-in-use codes, and neighbouring countries sharing a datum family — none of these are anyone's fault. They're the residue of a century of national surveying, encoded into a flat integer namespace. The registry is doing its job. The label was doing mine, badly.

My Tile Cache Has No Invalidation, So I Set It to Zero

Srdjan Popovic — Thu, 20 Aug 2026 19:30:13 +0000

Here is a line from the config of a vector tile server that has been in production for months, serving 2.7 million features to a web GIS:

cache_size_mb: 0

That's a cache, deliberately disabled. Underneath it, a comment I wrote for whoever touches this next — probably me, having forgotten:

MUST stay 0. Martin's in-memory tile cache has no invalidation hook for underlying data changes.

This post is about how I ended up there, because "turn off the cache" is the kind of decision that looks lazy until you've watched the alternative fail.

What the system is

Martin is a vector tile server written in Rust. Point it at PostGIS and it will discover your spatial tables and views and serve each one as an MVT endpoint. It is genuinely excellent, and none of what follows is a complaint about it.

In my case it sits in front of a road-inventory database: 1.8 million point features, 697,000 lines, 172,000 polygons, spread across roughly a hundred layers. Users don't just look at that data. They edit it — move a sign, redraw a kerb line, correct an attribute — and expect the map to show the change.

That last sentence is the whole problem.

The default that looks free

Martin's tile cache defaults to 512 MB, on. For most tile workloads that's an obvious win: your data is a static extract refreshed nightly, tiles are expensive to build, memory is cheap. Cache everything.

I left it on initially for exactly that reasoning. Then a user reported that a geometry they had edited still looked wrong on the map.

The natural suspects, in order: the browser cached the tile, nginx cached the tile, the edit never committed, or the edit landed in the wrong layer. I checked all four. The database had the new geometry. The layer view returned the new geometry. The API returned the new geometry. The map did not.

The experiment

The useful thing about a tile is that it's a file. You can fetch it and look at its size.

So: pick a feature, note the tile that contains it, fetch the tile, record the byte count. Edit the feature's geometry so the shape visibly changes. Fetch the same tile again.

before edit:  242 B
after edit:   242 B
30s later:    242 B

Byte-identical. Not "similar size" — identical. Meanwhile the database and the layer view were both returning the new shape the entire time.

I waited. It stayed 242 B. I waited longer. Still 242 B. The tile was still 242 B when I restarted the container, at which point it immediately became the correct tile.

A cached tile is served unchanged for the lifetime of the process. There is no TTL to wait out and no hook that notices the underlying rows moved.

The setting that looks like it would help, and doesn't

There is a reload_interval in the config, and when you are staring at a stale tile it reads like the answer:

reload_interval: 5s

It isn't. reload_interval re-discovers the catalog — which tables and views exist, what their geometry columns are, what SRID they carry. It's what makes a newly created layer show up as a tile source without a restart, which in a system where users create layers is genuinely valuable.

It does not touch cached tiles. The catalog and the cache are different things, and only one of them refreshes.

This is worth stating plainly because the two settings sit near each other in the config file and it is very easy to assume one covers the other. I assumed it for a while.

What "no invalidation" actually means

It's tempting to file this as a missing feature. It isn't, really — it's a hard problem wearing a simple name.

To invalidate correctly, the tile server would have to know that a row changed, work out which tiles at which zoom levels contained the old geometry and which contain the new one, and evict all of them. That means the tile server needs a change feed from Postgres, plus geometry-to-tile-index math for both the before and after state, at every zoom level.

That is a substantial amount of machinery for a server whose main job is to be a fast, boring translator between PostGIS and MVT. Most deployments don't need it, because most tile data doesn't change under the reader.

Mine does. So the cache is the wrong tool for my workload, and the right move is to not use it.

What it costs to turn off

This is the part I want to be honest about, because "I disabled the cache and everything was fine" would be a suspiciously tidy ending.

Every tile request now goes to Postgres. With cache_size_mb: 0, Martin builds each tile with ST_AsMVT on demand, on every request, through pgbouncer, against tables holding millions of rows.

That is survivable for a specific reason: the data is not read by the public. This is an internal tool with a bounded number of authenticated users working on a bounded number of projects. The requests-per-second ceiling is set by how fast a few dozen people can pan a map, not by the internet.

If this were a public basemap, the answer would be completely different — you'd cache aggressively at the edge and accept that edits take minutes to appear, or you'd pre-render tiles and rebuild on write.

So the real lesson isn't "disable your cache". It's that cache correctness is a property of your workload, not of your tile server, and the default is tuned for the workload where readers vastly outnumber writers.

What replaced it

I left something out above, and it changes the conclusion.

Turning off Martin's cache did not leave the system uncached. One layer up, nginx caches the same tiles:

location ~ ^/(.+)/(\d+)/(\d+)/(\d+)$ {
    proxy_cache       martin_tiles_cache;
    proxy_cache_key   $scheme$host$request_uri;
    proxy_cache_lock  on;
    proxy_cache_valid 200 302 5s;
}

Five seconds.

That number looks almost pointless until you think about what a map client does. Panning a map fires dozens of tile requests, many of them repeats, within a second or two. proxy_cache_lock on collapses concurrent requests for the same tile into one upstream fetch. The five-second window absorbs exactly that burst and nothing more.

So the real difference between the two caches was never "cached vs uncached". It was:

	Martin's internal cache	nginx cache
TTL	none — process lifetime	5s
Eviction	memory pressure only	expiry, and the cache dir is purgeable
Effect of an edit	invisible until restart	visible within 5s

A cache without a TTL is not a cache with a long TTL. It is a different thing. The first is a promise that data never changes; the second is a bet that it changes slower than the window. Only one of those was true here.

The lesson isn't "don't cache tiles that users edit". It's that the acceptable staleness window is a product decision — five seconds is fine, five minutes probably is too, and "until someone restarts a container" isn't a window at all.

The other number in that file

While I was in there, one more setting was wrong in the opposite direction:

worker_processes: 4

It had been 18. Martin, like a lot of servers, picks a worker count from the host's visible CPUs — and inside a container, the host's CPU count is not your CPU allowance. Eighteen workers were contending for two cores' worth of scheduling.

More workers than cores does not add throughput once you're saturated; it adds context switching and queueing, and it shows up in tail latency rather than in the average. The p50 looks fine. The p99 is where users live.

Setting the worker count to the actual CPU allowance is one of those changes that produces no visible improvement in a benchmark and a real one in how the application feels.

What I'd tell someone setting this up

Ask what your staleness window is, and whether the cache can honour it. If users edit rows and expect to see the result, a cache with no TTL and no invalidation is not a performance optimisation — it's a correctness bug you configured on purpose. A cache with a short TTL, one layer out, gets you most of the benefit and bounds the damage.

Test invalidation by byte count. Fetch a tile, change the data, fetch again, compare sizes. It takes two minutes and gives you a fact instead of a belief. I spent longer than that theorising about nginx.

Read what your reload setting actually reloads. Catalog discovery and cache eviction are different, and adjacent config keys imply a relationship that isn't there.

Give containerised servers your CPU allowance, not the host's. Anything that auto-detects nproc inside a container is auto-detecting the wrong number.

And write the reason in the config file. That comment is the only thing standing between this setting and someone — me, in a year, looking at a cache set to zero and thinking it must be a mistake.

Eleven Terabytes and No Raster Database

Srdjan Popovic — Wed, 19 Aug 2026 08:53:56 +0000

DailyMeteo is about 11 TB of GeoTIFFs — 1 km daily temperature and precipitation for all land on Earth, back to 1961. There is no raster database anywhere in the system. No PostGIS raster tables, no tile server in the read path for point queries, no object store. A Django app reads files off a disk.

That sounds like something we haven't gotten around to fixing. It isn't. Here's the reasoning, and the places where it bit us.

Two machines, one job each

The archive is produced on one machine and served from another.

The production machine is a big box — a terabyte of RAM, most of it in use — that does nothing but run the interpolation. Space-time kriging over a continent is not a workload you want sharing a server with anything user-facing, because it will happily consume every core for six hours and then a request times out for reasons no one will connect to weather.

The serving machine runs the API, the frontend, the database, GeoServer, the workers, and the reverse proxy, all as a Docker Swarm stack.

Between them, a rsync that runs every 30 minutes. Two details in it are load-bearing:

It pulls, it doesn't push. The serving machine reaches into the production machine and takes what's missing. Nothing runs on the production box on our behalf except a remote rsync under nice -n 19 ionice -c3, with a bandwidth cap. If the transfer is slow, the interpolation doesn't care. A push would have put the scheduling decision on the machine whose whole job is to not be interrupted.

It transfers incomplete dates. It used to only take dates where all 24 rasters existed. The effect was that when one zone failed, the perfectly good rasters for the other five stayed on the production box and the API reported the date as empty. Missing rasters are missing either way; this way, the ones that exist are at least reachable.

Why the filename is the index

A point query needs to answer: for this coordinate, this variable, and these 400 dates, which files do I open?

With files on a disk and a naming convention, that's string formatting:

{var}_day_{YYYYMMDD}_equi7{_early|_late|}.tif

Work out which continental zone the point falls in, build 400 paths, open the ones that exist. No index to keep in sync, no ingest step, no migration when a year of backfill lands, no second system that can disagree with the disk about what exists. The pipeline finishes writing a file and it is served, with no further action.

What that buys is worth being explicit about, because it's easy to reach for a database out of habit. There is no state anywhere that can drift from reality. A file is either on the disk or it isn't. When we needed to know exactly what we could serve, the answer was os.scandir over 24 directories — about a second for 580,000 entries — and it was the truth, not a cached projection of it.

What it costs:

No query planner. "Every date where the July mean exceeded X anywhere in Europe" is not a question this shape can answer. It answers "give me these pixels from these files" very well and nothing else at all.
Directories with 24,000 files. Fine on ext4, and scandir doesn't care, but ls in a terminal will make you wait and any tool that stats every entry becomes the bottleneck.
The convention is the schema, and it's enforced by nothing. One file written with a different suffix is a silent data bug, not a constraint violation. This is exactly how we ended up serving duplicate values for 3,624 days — a variant appeared that the read path's ranking didn't know about.

Given the access pattern — read a handful of pixels from an arbitrary set of files, never scan or join — I'd make the same call again. The mitigation for the last point isn't a database. It's that every read path goes through one function that knows the convention, and that function is the only place the convention exists.

The file format does the work a database would

The reason reading raw files is fast enough is Cloud Optimized GeoTIFF.

Each raster is Int16, LZW-compressed, internally tiled in 512×512 blocks, with five levels of overviews baked in. A European tile is 8229 × 5588 pixels and 10.2 MB on disk.

The tiling is the point. To read one pixel you read one 512×512 block, not 10 MB. To draw a zoomed-out map you read an overview level that's already there, instead of downsampling the full raster. The layout means a reader can seek to the bytes it needs, and everything downstream — the point query, GeoServer's mosaics, the polygon clip — gets that for free.

Int16 instead of a float is a factor-of-two on 11 TB, which is worth roughly five and a half terabytes of disk. Temperature stored in tenths of a degree loses nothing anyone can measure.

What a request actually does

A point query is Django REST Framework, and the interesting parts are all about what not to do.

Requests are authenticated with an API key, priced in credits, and cached in Redis — a keyed lookup of an immutable historical value is the ideal cache entry, so the TTL is a week. Bulk exports don't happen in the request at all; they're Celery tasks that write a file and email a link, because "clip 20 years of daily rasters to this polygon" and "answer within an HTTP timeout" are incompatible requirements.

The polygon pricing had a nice bug in it, and it's the kind that only shows up at the edges. Cost is computed from pixels times days. Pixels came from the mask of the clipped raster — except the mask counts pixels inside the polygon, and a polygon over the ocean is full of nodata pixels that are inside it and contain nothing. Draw a box over the Atlantic and you'd be charged for a full raster's worth of nothing. The fix is one line — exclude nodata as well as masked — but the shape of the mistake is general: "how much data is here" and "how much of this rectangle is here" are different questions, and the second one is easier to compute, so it's the one you accidentally write.

Running R in someone else's browser

The part of the system I find most interesting isn't on the server at all.

There's an R environment on the site — a console, an editor, plots, tables — and it doesn't run R on our machines. It runs webR, R compiled to WebAssembly, inside the user's browser tab. Their code fetches from our API, computes locally, and renders locally. We never see what they ran.

That was a deliberate call. The alternative is an RStudio-shaped service where users execute arbitrary R on our infrastructure, and that is a sandbox problem, a resource-limits problem, a queueing problem and a security problem, forever. Moving it into the browser makes all four somebody else's — specifically, the browser's, which is very good at exactly this.

The bill comes as about 45 MB of WebAssembly and R packages, downloaded before anything runs. We warm the runtime up in the background as soon as the page loads, so by the time someone has typed a question the worker is usually alive. "Usually" is doing real work in that sentence, and the first visit on a slow connection is not a great experience.

The second cost is subtler. webR's default communication channel uses SharedArrayBuffer, and browsers only expose that to pages that are cross-origin isolated — which means serving two specific headers:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

The catch is that require-corp also blocks every cross-origin subresource that doesn't explicitly opt in. Turn it on site-wide and the map's vector tiles stop loading and half the images on the marketing pages disappear. So it's scoped to the R pages only, which means the R runtime lives on an island with different security semantics from the rest of the app, and every asset it needs has to be reachable from that island.

It's worth the trouble for one reason: SharedArrayBuffer is the only channel that supports interrupting a running computation. Without it, a runaway while (TRUE) can only be stopped by destroying the worker and booting a fresh 45 MB one. With it, there's a stop button that actually stops.

Two failures worth stealing

Both of these cost more time than they should have, and both look like someone else's bug until you find them.

nginx doesn't know what .mjs is. The bundled mime.types maps .js and stops there. Every ES module served straight off disk — including both worker entry points, the map's and R's — went out as application/octet-stream, and the browser refused them: "Strict MIME type checking is enforced for module scripts." Every other asset was fine, so the app looked healthy while two of its major features were dead.

The fix is a line in the Dockerfile. The part worth copying is the second line:

RUN sed -i 's|\(application/javascript[[:space:]]\+\)js;|\1js mjs;|' /etc/nginx/mime.types \
 && grep -q 'js mjs;' /etc/nginx/mime.types

If a future base image formats that entry differently, the sed silently does nothing and we're back to a broken worker discovered by a user. The grep turns that into a failed build. A patch applied by pattern-matching someone else's file should always be followed by an assertion that it took.

A URL nothing would accept. The assistant generates R against a helper with this signature:

get_data(var, agg_level, time_scale, from = NULL, to = NULL, time = NULL, lat, lon)

time takes a comma-separated list of dates, as an alternative to from/to. Asked for six years of daily data, the model enumerated every date into time. That's 2,192 dates and a 24,222-character URL.

webR failed with problem writing module_download template in internet module, which reads like a webR bug and sent me looking in the wrong place. It isn't. Our own API returns 414 above roughly 8,190 characters — Apache's default LimitRequestLine — so the same generated code, copied into RStudio, would have failed too:

  700 dates ( 7,810 chars) → 400   (request accepted)
1,000 dates (11,110 chars) → 414 URI Too Long
2,192 dates (24,222 chars) → 414

The tempting fix is to teach the model not to do that. We fixed the helper instead. Before building any URL, get_data now normalises time: a contiguous run collapses into a single from/to request, and a list with gaps is split into batches that keep every URL under 2,000 characters. The failing case went from 24,222 characters to 165, in one request, returning exactly the same 2,192 rows as from/to would.

The signature didn't change — it's the contract the model was trained against, and it's written into its system prompt. Only the body did.

That distinction is the whole point. When a language model generates code against your helpers, the helpers are the place to be defensive, because they're the part you control and the part that doesn't need retraining. Prompt changes are a request. Helper changes are a guarantee.

The blind spot

The last thing this architecture taught me is that it degrades quietly, everywhere.

A missing raster isn't an error; the date is just thinner. A failed source isn't an error; the day is produced with five zones instead of six. A transfer that skips an incomplete date isn't an error; the API simply has nothing for it. Every one of those is a reasonable local decision, and stacked together they mean the system can be substantially broken while every component reports success.

The number that finally made it visible was the difference between two dates: the most recent day with any raster, and the most recent day with all 24. The first was five days back, which is normal and looked fine. The second was six weeks back.

Nothing was alerting, because nothing was subtracting.

DailyMeteo is at dailymeteo.com. The previous post covers what the data is and why every day is computed three times.

We Publish the Data Before It's Correct. On Purpose.

Srdjan Popovic — Wed, 19 Aug 2026 08:52:13 +0000

Someone asks for yesterday's maximum temperature in Belgrade. We don't have it. We won't have it for another four days, and when we do have it, the number will be wrong — not badly wrong, but wrong enough that we'll replace it a month later, and replace it again a year after that.

That's not a bug report. That's the design.

I want to explain why, because it turns out the three-pass thing is the most consequential decision in the whole system, and every other piece — the API, the storage layout, the way a chart gets drawn — bends around it.

What the data actually is

DailyMeteo is a gridded daily climate archive. Four variables — maximum, minimum and mean temperature, and precipitation — at 1 km resolution, for all land on Earth, every day since 1 January 1961.

Some numbers, because the shape of the problem is mostly numbers:


Resolution	1 km, daily
Period	1961-01-01 → present
Variables	`tmax`, `tmin`, `tmean`, `prcp`
Continental zones	6, in the Equi7 projection
Rasters per day	24 (6 zones × 4 variables)
One European tile	8229 × 5588 px, 10.2 MB
Archive	~11 TB

Twenty-four rasters per day, roughly 24,000 days. That's around 580,000 GeoTIFFs, and it's why "just put it in a database" stopped being an option early.

The projection deserves a note. Equi7 splits the land surface into seven continental zones, each with its own equidistant projection, so that a kilometre is actually a kilometre wherever you are. A single global grid can't do that — it either stretches at the poles or bunches at the equator. The cost is that "the world" is six separate rasters that don't share a coordinate system, and every query has to work out which zone the point falls in before it can read a pixel.

How one day gets made

Weather stations don't cover the world. They cover the parts of the world that historically built weather stations, which is a very uneven map. Turning some thousands of point observations into a continuous 1 km surface is the actual work.

The pipeline pulls station observations from several archives — OGIMET, GSOD, GHCN-D, ECA&D, MeteoManz — because none of them alone is complete, and each has a different idea of what "yesterday" means. On a normal day that's something like 18,000 stations queried for a single variable.

Then, per day and per variable:

Fit a trend. A linear model on three covariates: elevation from a DEM, topographic wetness index, and a geometric temperature trend — a deterministic function of latitude and day of year that carries most of the seasonal signal before any interpolation happens.
Take the residuals. Observed minus trend, at each station.
Krige the residuals in space and time. Not just "what do the neighbouring stations say today" but "what did this neighbourhood say yesterday and the day before", which matters enormously when a station is missing for a day.
Add the two rasters back together. Trend surface plus interpolated residual surface.

Precipitation gets a second pass, because rain isn't like temperature. Temperature always has a value; rain is mostly zero. So it's modelled twice — first whether it rained at all, then how much, given that it did. Interpolating rainfall directly gives you a light drizzle over an entire continent, which is both wrong and hard to notice.

None of this is my invention. The method comes from Kilibarda et al. (2014), where Milan Kilibarda and co-authors established spatio-temporal regression kriging for global daily temperature at 1 km, and the interpolation runs through the meteo R package developed by Milan Kilibarda and Aleksandar Sekulić. What's ours is the part that turns it into something that runs every night and answers HTTP requests. It is not fast. A single day, all zones, all variables, takes hours on a machine with a terabyte of RAM.

Why the same day is computed three times

Here's the tension. Station data arrives late, and it arrives incrementally.

The archive that gives you a station's observation within a day or two is not the archive that eventually gives you the quality-controlled version. Some networks publish a preliminary value and revise it. Some publish nothing for weeks and then backfill a month at once. If you wait until the data is final, "today's weather" is a year old.

So we don't wait. Every day gets computed three times:

early — about four days behind, from whatever stations have reported. Runs nightly.
late — about four months behind, once the monthly archives have settled. Runs monthly.
final — the following year, on the fully quality-controlled record. Runs yearly.

Each pass overwrites nothing. The three versions sit side by side on disk, distinguished by a suffix in the filename:

tmax_day_20200715_equi7_early.tif
tmax_day_20200715_equi7_late.tif
tmax_day_20200715_equi7.tif        # final, no suffix

The API ranks them — final beats late beats early — and serves the best one that exists for the date you asked about. Right now that means anything before 2024 comes back final, and the last few weeks come back early. The transition is invisible in the response, which is either elegant or dishonest depending on your mood; I'll come back to that.

The ratio is heavily in favour of finished data. In the European maximum-temperature series: 23,010 final rasters, 851 late, 585 early. Ninety-five percent of the archive is settled. It's the leading edge that churns.

What it costs

Three variants of every file, ranked at read time, is a rule that lives in exactly one function. Rules like that get broken by accident.

We added the late stage after early and final were already running. Somewhere in the code that assembles a time series for a point query, the ranking knew about two variants and the directory now contained three. The unknown file didn't lose the ranking — it wasn't in the ranking at all, so it came through as a separate, additional row.

The result was a chart with two values for the same date. Not a crash, not an error, not a log line. Two dots where there should be one, on 3,624 days' worth of data, sitting there for however long it took someone to look closely at a chart.

The fix is three lines. The lesson isn't about the three lines. It's that "prefer the best available version" is a domain rule, and if it's implemented as an incidental sort somewhere in a view, it will drift the moment the domain gains a version. It belongs in one named function that every read path goes through, and adding a variant should be a change to that function and nothing else.

What you can actually do with it

The data is the product, but nobody wants an 11 TB tarball. What's on top of it:

Point queries. Click a location, get a series. The API takes a latitude, longitude, variable, and either a range or a list of dates, and returns timestamps and values. Daily, monthly or annual, aggregated or as long-term means over the two standard climate normals — 1961–1990 and 1991–2020.

GET /meteo/v2/pq/?var=tmax&agg_level=agg&time_scale=day
    &from=2020-01-01&to=2020-12-31&lat=44.81&lon=20.46&api_key=…

Areas. Draw a polygon, get the aggregate over it, or export the clipped rasters. Large exports are priced by area and period and delivered by email when they're ready, because a polygon over Asia for a decade is not a request you answer inside an HTTP timeout.

Long-term means and anomalies. The monthly and annual aggregates and the two climate normals are precomputed — roughly 750 GB of them — so "how does this July compare to the 1991–2020 average" is a lookup, not a computation over 30 years of daily rasters.

R in the browser. There's an R environment on the site that runs entirely client-side, compiled to WebAssembly. You write R, it fetches from the API and plots, and your code never touches our servers. There's an assistant next to it that turns a question in English into R code against the same helpers. That one has enough engineering in it to deserve its own post, which is the next one.

Embeddable charts, because half the time what someone actually wants is a temperature curve on their own page.

The part I'd rather be honest about

Two things about this design are genuinely awkward and I don't have clean answers.

The seam is invisible. A response doesn't tell you which pass produced each value. Ask for a series spanning 2023 to now and you get final data and early data in one array, with no marker. For a chart, fine. For a paper, not fine. Exposing the processing level per value is the obvious fix and it's the kind of obvious fix that stays on the list because nothing visibly breaks without it.

"Available" and "complete" aren't the same thing, and we conflated them. A date is only fully covered when all 24 rasters exist. When one source fails — and sources fail; a station archive returned zero rows out of 18,000 for several days recently — the pipeline still produces the zones it can. So a date can have 22 of 24 rasters: present, queryable, and quietly missing two continents.

We were tracking "latest date with any raster". By that measure everything looked five days behind, which is normal. By the measure that matters — latest date with all 24 — we were six weeks behind and nobody had noticed, because no counter anywhere was counting that. Now there's an internal page that shows both numbers side by side, and the gap between them is the number I look at first.

That's the recurring shape of this whole system, honestly. The hard part was never the kriging. It's that a pipeline which degrades gracefully also fails quietly, and you have to go out of your way to build the thing that tells you it did.

The interpolation method and the meteo R package behind it are the work of Milan Kilibarda and Aleksandar Sekulić at the University of Belgrade, Faculty of Civil Engineering.

DailyMeteo is at dailymeteo.com. Next post: how 11 TB of rasters get served without a raster database, and what happened when we tried to run R inside a browser tab.

Our Documentation Was Lying. The Model Believed It.

Srdjan Popovic — Wed, 19 Aug 2026 08:26:58 +0000

There is a variable called slp — sea-level pressure. Our API documentation lists it as available. Our error messages list it among the valid options. Our fine-tuned model, asked about air pressure over Belgrade, will happily write you fifteen lines of R to fetch it.

The request comes back 400.

It has been coming back 400 the whole time.

What this is

We run dailymeteo.com — a daily meteorological archive for Europe, gridded at 1 km, running from 1961 to roughly five days ago. There's a chat endpoint where you ask a question in plain language and get back R code that queries the archive and answers it. Behind that sits a fine-tuned GPT model, trained on a few hundred question-and-code pairs.

Last week I set out to retrain it on a better dataset. I expected to spend the day on hyperparameters. I spent it finding out that the model had been taught things that were not true.

The training data was built from the documentation

This is the part worth stopping on, because I suspect it is extremely common.

When you build a fine-tuning set for "write code against our API", the natural move is to sit down with the API documentation and write examples from it. That is what had happened. Each example carried a system prompt describing what the API does, a question, and the R code that answers it.

The trouble is that documentation is a claim about a system, not the system. And nobody had checked the claim in a while.

So before touching anything, I did the boring thing: I called the service and wrote down what actually came back.

Three of its claims were wrong.

Claim 1: `slp` is an available variable

The data exists. There are 21,916 daily rasters sitting on disk, covering 1961 to 2020, Europe only.

The API will not serve any of them. The allow-list in the view is:

VARS = ['tmax', 'tmin', 'tmean', 'prcp']

and the check var not in VARS guards every entry point. On top of that, the continent mapping points at a newer data folder that never received the pressure rasters at all. Two separate reasons for the same 400.

The best detail: the error message you get back still lists slp among the available variables. The message is older than the list.

The model had been taught to ask for it. Two examples in the training set did exactly that. So on any question about pressure, the model produced confident, well-formed, non-functional code.

Claim 2: "data from 1960 to 2024"

Both ends wrong.

1960 returns "There's no data for date range." The archive starts in 1961. And the far end isn't a year at all — the archive is kept near-real-time. Daily data run to about five days ago, monthly to the previous month, annual to the last complete year. Writing a fixed end year into a system prompt guarantees it will be wrong within twelve months, silently.

While measuring this I found something genuinely useful: asking for a range outside the archive is not an error. The API quietly clips.

ann  1901 → 1970   returns  1961 → 1970
ann  1961 → 2035   returns  1961 → 2025
mon  1961-01 → 2030-12   returns through 2026-07

That changes what "correct" code looks like. Generated code doesn't need to know where the archive ends — it can ask wide and read the actual extent back out of the response. That's a pattern that never goes stale. Several of our training examples had been hardcoding an end year instead, which ages badly and quietly.

Claim 3: the timestamp format

The docs said the returned timestamp is formatted like the input date. For aggregated data, true. For long-term means, not remotely:

ann   "1961-1990"          "1991-2020"
mon   "05.1961-1990"       "05.1991-2020"
day   "25.07.1961-1990"    "25.07.1991-2020"

It carries the climate period, not a date. Any code doing substr(timestamp, 1, 4) to pull a year out gets nonsense. Some of ours did.

And the thing that wasn't documented at all

Long-term means — the most semantically awkward corner of the API — had no description whatsoever. The rule, once measured, is simple: the year inside the date selects which climate period you get. Pass 1961, get 1961–1990. Pass 1991, get 1991–2020. Omit the year, get both.

Nobody had written that down. So across 23 calls in the training set, long-term means were invoked seven different ways, two of them mutually contradictory. The model wasn't learning a convention. It was learning that there isn't one.

The subtler poison: examples whose answer the question doesn't determine

This one I didn't expect, and it's the one I'd most like other people to check for.

Consider this pair from the training set:

Question: plotting temperatures during autumn (September 1 to November 30) in Prague

Code: from = "1999-09-01", to = "1999-11-30"

Why 1999? No reason. The question doesn't say. Whoever wrote the example picked a year.

There were 28 examples like this. And here is why they matter beyond tidiness: the model cannot possibly predict the answer from the question. No amount of training reduces the error on that example, because the target contains information the input doesn't.

You can see it in the metrics. In the previous training run, the loss spikes that survived all the way to the final epoch — steps 223, 238, 249, 253, 260, still spiking at 0.46–0.66 while everything around them sat at 0.13 — were these. They're not a hyperparameter problem. They're irreducible.

Worse, what the model does learn from them is the behaviour: invent a year, say nothing. In production that's a model quietly answering a different question than the one asked.

The fix cost nothing and didn't change a single choice the code makes. We just made it say so:

cat("No period was specified in the question - using 1961 to 2020.\n\n")

Same year. Now the answer is determined by the question, plus a disclosure the model can actually learn.

What we changed

Nine steps, but the shape is simple:

Measure the API. Every boundary verified by calling the service. The result is one Python file that every other script imports — a single source of truth that is not the documentation.
Audit against it. A script that walks each example and flags what doesn't work, what wastes model capacity, and what's cosmetic. First run: 13 examples the API rejects, 28 inventing periods, and a training set split 135-to-62 between two different code formatting styles for identical tasks.
Fix what's unambiguously wrong. The 13 rejects, and 23 silent period choices turned into disclosed ones.
Normalize style. Half the set broke calls across multiple lines, half kept them on one — same task, double the tokens. That split was costing model capacity on a question with no informational value.
Fill the gaps. There was not a single example covering questions the API can't answer — wind, humidity, forecasts. The model had been improvising. Now it answers "I don't have that, here's what I do have" — still as runnable R, because the contract is that every answer executes.
Derive the explanations from the code, not by hand, so they stay in sync when the code changes.
Split off a validation set. 29 examples the model never sees.
Preflight. Refuse to upload if anything above regressed.
Train, then choose.

The previous run had no validation set

This is the single change that mattered most, and it's the cheapest one.

The earlier job ran three epochs with no validation file. Its training loss fell nicely, from 0.38 to 0.16, and everyone was happy.

Training loss falls whether the model is learning or memorising. Without a held-out set, those two are indistinguishable. You are looking at a number that goes down in both the good case and the bad case, and concluding things about it.

With validation, this run:

Epoch	Training loss	Validation loss
1	0.368	0.285
2	0.133	0.131
3	0.071	0.140
4	0.038	0.152

Training loss keeps falling all the way to 0.038 — near-perfect reproduction of what it was shown. Validation bottoms out in epoch 2 and turns back up. Textbook. Invisible without the held-out set.

One incidental finding: the platform's auto hyperparameter selection picked a learning-rate multiplier of 0.51 on the previous run and 2.0 on this one. The only meaningful difference was that our examples got longer. If you rely on auto, know that it can quadruple your learning rate because you added a paragraph to your system prompt.

Then loss and behaviour disagreed

Validation loss says take epoch 2. So I built a second evaluation that asks a different question: does the generated code actually work? Each checkpoint answers the 29 held-out questions, and we measure whether R parses it, whether the API arguments are valid, whether it invents packages that aren't installed, and whether it discloses the period it chose.

Checkpoint	Validation loss	Failures out of 29
Epoch 2	0.131	4
Epoch 3	0.140	1
Epoch 4	0.152	0

Exactly inverted.

The explanation is that validation loss measures token-level similarity to a reference answer. It does not measure whether code runs. As training continues, the model drifts away from the reference answer's exact wording — which raises the loss — while the code stays correct and the behaviours we deliberately added keep consolidating.

Epoch 2 has the best number and cannot do the thing we trained it to do: on three of four questions with no stated period, it silently picks a year.

We shipped epoch 4. Had we trusted the loss, we'd have shipped the one that doesn't work.

And then I got the measurement wrong too

Worth including because it's the mistake I'd most likely repeat.

The behavioural evaluation gave epoch 4 a perfect score on disclosure: 4 out of 4. I reported 100%.

It asked each question once.

Re-running the same question against the deployed model several times:

temperature = 1.0    discloses 3/4
temperature = 0      discloses 1/3

The real figure is somewhere around 50–75%, not 100%. The behaviour is genuinely learned and genuinely better than the 25% the previous model managed — but it is not reliable, and a single sample per question cannot tell you that. One sample measures what a model did once. It says nothing about what it does.

(temperature = 0 also isn't deterministic without a seed. Three calls, three different answers.)

What I'd tell anyone fine-tuning against their own API

Your model inherits your documentation's lies. If the training data was written from the docs, every stale claim in them is now a learned behaviour. Call the service and write down what comes back. It took an afternoon and found three errors in a prompt that had been in production for months.

Check whether the question determines the answer. Any example where the target contains information absent from the input is teaching the model to make things up. It also shows up as loss spikes that never come down, which is a cheap way to find them.

Ship a validation split, even a tiny one. Twenty-nine examples were enough to reveal an overfitting turn that was completely invisible for three epochs previously.

Measure whether the output works, not whether it matches. Loss is a proxy. Parse rate, valid arguments, no hallucinated dependencies — those are the thing itself. When the two disagree, the proxy is not the one to trust.

Sample more than once. A percentage from one draw per question is not a measurement, it's an anecdote with a decimal point.

The model was maybe a fifth of the work. The rest was going back and asking the system what it actually does — which, in hindsight, is what anyone should have done before writing the documentation the model learned from.