Developed Python CLI to easily download Dutch point cloud data

HideBa — Sat, 04 May 2024 09:41:53 +0000

Building a CLI Tool for Downloading AHN Point Cloud Data

Hey there, fellow developers! I recently had the opportunity to work on a really cool project—developing a Command-Line Interface (CLI) tool that makes it a breeze to download Actueel Hoogtebestand Nederland (AHN) point cloud data for specific cities. Here is the GitHub link. https://github.com/HideBa/ahn_cli I thought I'd share my experience with you all, so buckle up and let's dive in!

The Objective

The main goal was to create a tool that would streamline the process of getting geospatial data, making it easier for various GIS applications like urban planning and environmental analysis. I mean, who doesn't love a good time-saver, right?

The Journey

Step 1: Simplifying City Polygons

First things first, I had to create simplified polygons of Dutch cities to make processing smoother. I found a comprehensive list of cities from the Nationaal Georegister GeoNetwork. But here's the catch—the original polygons were so detailed that the software couldn't handle them efficiently. So, I put on my problem-solving hat and used the Douglas-Peucker Algorithm in QGIS to simplify those polygons. Voila! Optimized polygons that didn't compromise much on geographic accuracy.

Step 2: GeoJSON Conversion

Next up, I converted the original dataset from a Web Feature Service (WFS) to GeoJSON format. This made it way easier to work with the data as standalone files and boosted the CLI tool's data handling capabilities.

Step 3: GeoTiles Grid Data Prep

I downloaded the GeoTiles grid data from fwrite.org, which is crucial for identifying relevant point cloud data segments. Then, I converted the shapefile to GeoJSON to keep everything consistent and user-friendly.

The CLI Magic

Now, let's talk about how this CLI tool actually works its magic:

User Input: The CLI tool starts by receiving parameters from the user, like the city name or geographic coordinates.
Tile Detection: It uses the simplified city polygons and GeoTiles grid data to find overlapping tiles by checking for intersections. This step figures out which point cloud data segments the user needs.
Data Download: Once the relevant tiles are identified, the CLI tool downloads the corresponding LAZ formatted point cloud data from the GeoTiles source.
Data Manipulation: This is where the real fun begins! I created a data manipulation process with numpy and geopandas. It can filter by AHN classification classes, clip to the city polygon, apply user-defined polygonal clips, decimate, and even do radius-based clipping. Talk about flexibility!
Output Generation: Finally, the processed data is saved in the user's preferred format (LAS or LAZ), ready to be used in further GIS analyses or applications.

Challenges and Solutions

Of course, no project is complete without a few hurdles. Here are some of the issues I faced and how I tackled them:

Slow AHN download: Even for the smallest city in the Netherlands, the download took forever! I implemented multi-threading to download data simultaneously, reducing the total download time by around 8x.
Slow point cloud data manipulation: Initially, I simply used laspy and numpy, but it was taking way too long. To speed up polygon intersection test, I employed rasterisation to make time complexity log(O). That results in way faster than before.
Runtime errors galore: Dynamic typing in Python led to many runtime errors. Introducing a type system helped validate the program and improved productivity.
Invalid user parameters: Validating parameters on the fly wasn't user-friendly, so I implemented validation that runs immediately after the CLI is invoked.
Large LAS/LAZ file sizes: To help users deal with large file sizes, I added options for decimation and radius clipping, allowing them to reduce the number of points and focus on specific areas.

Wrapping Up

There you have it, folks! Developing this CLI tool was an incredible learning experience, and I hope sharing my journey has given you some insights and inspiration for your own projects. Remember, every challenge is an opportunity to grow and find creative solutions. Happy coding!

DEV Community: HideBa