One Gigabyte per Survey, of Which 108 KB Goes in the Database

Here is the disk layout of one mobile mapping survey — a vehicle with a LiDAR scanner and a panoramic camera, driven along a road:

data/001_MMS/            507 MB    point cloud
orbit/oblak/             566 MB    spherical photos
trajectory/*.gpkg        108 KB    the path the vehicle drove

Just over a gigabyte. The database this feeds holds 2.3 GB in total — for 2.7 million road features across a hundred layers. Two more surveys and the binary data outweighs everything the database has ever stored.

So the question isn't how to put a point cloud in Postgres. It's what you put in Postgres instead.

The trajectory is the index

Of that gigabyte, one file goes into the database: the 108 KB trajectory, a GeoPackage holding the line the vehicle drove.

That line is what makes the survey findable. It draws on the map with everything else. You can ask which surveys cover a junction, which are newest, whether a stretch of road has been captured since the resurfacing. All the questions people actually ask are questions about where and when, and the trajectory answers every one of them at 0.01% of the storage.

The heavy files never enter the database. The row holds paths:

class Cloud(models.Model):
    name            = models.CharField(max_length=120, db_index=True)
    path_name       = models.CharField(max_length=120)   # -> octree metadata JSON
    orbit_url       = models.CharField(max_length=255)   # -> spherical photo index
    spherical_photo = models.BooleanField(default=False)
    recording_date  = models.DateField(null=True)
    source_srid     = models.IntegerField(null=True, choices=SOURCE_SRID_CHOICES)
    available       = models.BooleanField(default=True)

Metadata, geometry, and pointers. That's the whole trick, and it isn't clever — it's just the discipline to not reach for a bytea column.

Why not in the database

Postgres will happily store a gigabyte. It's the access pattern that kills you.

A browser point cloud viewer doesn't fetch a point cloud. It fetches an octree: a tree of small files, and as the user moves the camera it pulls the nodes covering what's in view at the detail level that's visible. Zoom in, it fetches deeper nodes. Pan away, it drops them. A single session issues hundreds of small ranged reads driven by mouse movement.

That is precisely the workload a static file server is built for, and precisely the one a database connection pool is not. Serving it through Django would mean an application worker occupied for every node fetch, connection pool pressure from mouse movement, and no benefit whatsoever — there is no query, no join, no permission decision per node beyond the one already made when the survey was opened.

nginx serves the directory directly:

location /media/ {
    alias /app/media/;

    # CORS - needed for local frontend dev servers (different origin/port)
    # fetching point cloud (Potree octree/hierarchy) and other media files
    # directly via fetch()/XHR. No credentials involved, so a wildcard is safe.
    add_header 'Access-Control-Allow-Origin' '*' always;
}

That wildcard needs the justification written next to it, which is why the comment is there. It's safe because no credentials ride along: the octree nodes are opaque binary that mean nothing without the metadata, and the metadata comes from the authenticated API. Change either of those facts and the wildcard becomes a mistake.

Where the size actually goes

27 GB   orthophotos
24 GB   prepared point clouds
74 MB   symbology (icons for signs, poles, cameras)
1.5 MB  project thumbnails

The orthophotos are the bigger half, and they follow a different path — served as Cloud-Optimised GeoTIFFs through a raster tile server, cached at nginx for a day. Different data, different access pattern, different tool. What they share is that neither one is in Postgres.

Note the shape of the tail: two entries measured in gigabytes, everything else in megabytes. That's typical, and it's the argument for treating "large binary" as its own tier rather than a column type. The 74 MB of symbology icons could live in the database without anyone noticing. The 24 GB could not.

The escape hatch, and what it costs

Not all data can be copied. Some surveys are enormous and already sitting on a storage array, and duplicating them to bring them into the system is not worth 500 GB.

So there's a symlink path: point the system at data that lives elsewhere, and it appears under media/external/<hash>/ as if it had been uploaded.

This works, and it has a cost that must be paid explicitly. A symlink is a reference the database doesn't own, so deleting the row has to clean up the link too — otherwise media/external slowly fills with pointers to nothing:

@receiver(post_delete, sender=Cloud)
def delete_cloud_symlinks(sender, instance, **kwargs):
    """Remove the symlink folders under media/external when a Cloud is deleted."""

Every "just point at the existing files" shortcut buys you disk and sells you a lifecycle problem. Worth it here — media/external is 116 KB of links standing in for far more — but the cleanup is not optional, and a signal is the cheapest place to guarantee it runs.

The field that has to be asked for

One more field in that model earns its place: source_srid.

The trajectory arrives in whatever coordinate system the surveyor worked in, and — as with shapefiles that ship without a .prj — the file often doesn't say. So the model offers the choices, and the pipeline prefers the file's declared CRS, falls back to the user's selection, and refuses to guess.

That dropdown is small and easy to get wrong. I found one of its options labelled with an EPSG code from the wrong country, which would have put a trajectory about 5,000 km from the road it was recorded on.

The rule

Put in the database what you'll query. Put on disk what you'll stream.

For survey data that means: the trajectory, the recording date, the coordinate system, the paths, and a flag for whether the thing is ready to view. Not the octree, not the panoramas, not the orthophoto.

The test I'd apply to any large asset: is there a question someone will ask that requires this to be in a table? For a point cloud the honest answer is no — every question is about the trajectory, which is 0.01% of the bytes and answers all of them.

Comments

[Kane Lim]

This is a very solid architecture decision, especially the principle:

“Put in the database what you’ll query. Put on disk what you’ll stream.”

For LiDAR, panoramic imagery, orthophotos, and other large geospatial assets, the important design question isn't whether PostgreSQL can store the binary data—it can. The real question is whether the database should participate in every read.

I particularly like treating the trajectory as the spatial index. It keeps the database responsible for metadata, spatial relationships, temporal filtering, permissions, and asset discovery, while object/file storage handles the high-volume streaming workload.

For a production system, I'd extend this architecture with a few additional considerations:

Object storage over local disk where possible (S3/Azure Blob/MinIO), with immutable asset paths and lifecycle policies.
Signed URLs or short-lived access tokens instead of exposing /media/ publicly when datasets become sensitive.
CDN/cache layers for octree nodes and frequently accessed imagery.
Asset manifests + checksums so ingestion can be idempotent and corrupted/missing files can be detected.
Explicit ingestion states such as pending → processing → ready → failed, rather than relying only on available.
Spatial indexes on trajectories (PostGIS) so coverage queries remain fast as the survey catalog grows.
Background processing for generating Potree/LOD structures, COGs, thumbnails, and metadata rather than tying those operations to web requests.

The CRS point is also critical. Never silently guess a coordinate reference system. A technically valid geometry in the wrong CRS is often more dangerous than an obvious failure because it can look correct at first glance while being geographically meaningless.

One additional pattern I'd consider is separating the logical asset identity from its physical storage location. That makes migrations from local filesystem → NAS → S3 → cloud archive much easier without changing the application domain model.

The same architectural principle applies well beyond mapping: video, medical imaging, satellite imagery, CAD files, ML datasets, and other large immutable assets.

The interesting engineering problem isn't “How do I store 500 GB?”

It's “What information belongs in the database so that I never need to move 500 GB to answer a 50-byte question?”

That's where good architecture starts. I'd be very interested in discussing how you're approaching the next stage of this system—especially storage abstraction, ingestion pipelines, and scaling the survey catalog.