A Shapefile Is Four Files, and the Important One Is Optional

A user clicks "upload" and picks a file. In most systems that sentence is the whole story.

In a geospatial system, the thing they picked is one of four files that only mean something together, the one carrying the most important piece of information is optional and frequently absent, and the geometry inside it almost certainly doesn't match your database schema.

This is what I learned building the ingest path for a web GIS that now holds about 2.7 million features.

First: it left Django

Import used to be a synchronous Django endpoint. Upload a file, the request thread parses it, writes rows, returns.

That works until the file is 200 MB of line geometry. Then the request occupies a worker for minutes, the client times out with no way to find out whether the import survived, and a second user doing the same thing takes out a second worker. Nothing is wrong; the shape is just incompatible with the work.

So imports moved to a separate FastAPI service. Not because FastAPI is faster — the parsing is done by GDAL and geopandas either way, and they don't care which framework called them — but because the async model makes the right structure cheap to express:

# the blocking part goes to a thread, not the event loop
footprints = await asyncio.to_thread(_compute_all_footprints, tiff_list, default_epsg=32634)

# and concurrency is bounded on purpose
semaphore = asyncio.Semaphore(4)

That Semaphore(4) is the important line. Heavy geospatial work is memory-bound before it's CPU-bound — a raster mosaic or a large shapefile can hold hundreds of megabytes while it's being read. Unbounded concurrency doesn't make imports finish sooner, it makes the container get OOM-killed while five of them are half-done.

The endpoint returns a job id immediately. The client polls. There's a progression table so "importing" can say what it's doing rather than just spinning.

The same job-plus-poll shape is now used for raster mosaics and for exports. That consistency is worth more than any of the individual implementations: three features, one mental model, one place to look when something is stuck.

A file is not a file

A shapefile is a set:

REQUIRED_SHAPEFILE_EXTENSIONS = ['.shp', '.shx', '.dbf']

.shp holds geometry. .shx is the index into it. .dbf holds the attributes. Hand a parser the .shp alone and you get an error that's about a missing index, not about a missing upload.

So validation runs over the set of filenames before anything is parsed:

def validate_shapefile_components(filenames):
    extensions = {Path(f).suffix.lower() for f in filenames}
    missing = [e for e in REQUIRED_SHAPEFILE_EXTENSIONS if e not in extensions]
    if missing:
        return False, f"Missing required shapefile components: {', '.join(missing)}"
    return True, ''

Users mostly upload a ZIP, which the service extracts and then validates. The error message names the missing extension, because "invalid shapefile" tells someone nothing they can act on and "missing .dbf" tells them exactly what to go find.

The important file is the optional one

Notice what isn't in that required list: .prj.

.prj is the file that says which coordinate system the numbers are in. Without it, a shapefile contains coordinates like 7457000, 4958000 and no statement of what they mean. Those could be metres in one of several national grids, and picking wrong puts the data hundreds of metres — or thousands of kilometres — from where it belongs.

It isn't required because it's genuinely, frequently absent. Decades of surveying data was delivered as three files, in a projection everyone in the room already knew. That knowledge lived in people, not in the archive.

Which is why the upload form has to ask, and why "what coordinate system is this in?" is a dropdown the user must answer rather than something the software detects. That dropdown is its own story — I wrote about the day I found one of its options was labelled with the wrong EPSG code, pointing at a different country's grid.

The pipeline handles it in the right order: use the file's own CRS if it declares one; fall back to what the user selected; refuse to guess.

Your schema will not match their data

The three feature tables are typed. Deliberately:

project_pointfeature     POINT            3 dims   SRID 4326
project_linefeature      MULTILINESTRING  3 dims   SRID 4326
project_polygonfeature   MULTIPOLYGON     3 dims   SRID 4326

Uploaded data essentially never arrives in that shape, so every import performs three coercions.

Reproject to 4326. One SRID in storage, always. Anything else means every query has to know what it's holding.

Single to multi. A shapefile can contain a Polygon where the next feature is a MultiPolygon — the format allows both, and real files mix them. A column typed MULTIPOLYGON rejects the plain one. Promoting every geometry to its multi form is a one-line transform that removes an entire class of "some features imported and some didn't".

2D to 3D. This one is the least obvious and the one I'd defend hardest:

def _ensure_3d(geom, z_value=0.0):
    if isinstance(geom, Point):
        return geom if geom.has_z else Point(geom.x, geom.y, z_value)
    ...

The columns are three-dimensional, and today every single row has a Z value — 1,820,288 points, 697,009 lines, 171,830 polygons, without exception. Not because all the source data had elevation, but because anything that didn't got lifted to z = 0 on the way in.

That looks like storing a fake number. It's a deliberate trade: this data sits alongside LiDAR surveys where Z is real and load-bearing, and a table where some geometries have Z is worse than either alternative. Every query that touches elevation would need to know which rows to trust, and PostGIS functions behave differently on mixed-dimension inputs. One dimensionality, one code path, and z = 0 that is explicitly a placeholder.

If you have no 3D data at all, store 2D. What you should not do is let the dimension depend on whichever file happened to be uploaded.

What I'd carry to the next one

Move long work out of the request before you're forced to. The rewrite is cheap while it's one endpoint and expensive once three features have grown their own half-solutions. Job id, poll, progress — same shape every time.

Bound your concurrency explicitly. For memory-heavy work, unbounded parallelism converts "slow" into "OOM-killed", which is much harder to debug because it takes unrelated jobs down with it.

Validate the set, not the file. Where a format is really several artefacts, check them together and name what's missing.

Normalise aggressively at the boundary. Reproject, promote to multi, fix the dimension — once, on the way in, where you can still reject the whole upload cleanly. Every check you skip there becomes a conditional in every query that reads the table afterwards.

Ask for what the file can't tell you. Some information genuinely isn't in the data. Building a place for the user to supply it beats inferring it, and inferring it beats a default that's silently wrong.