(almost) Everything I know about building an open source web application

bwyss — Sun, 29 Dec 2019 02:51:01 +0000

I have been building web mapping apps for some time and thought I'd jot down some notes on the subject.

I'll cover:

Javascript libraries used to create a map on the web
Geospatial data formats
Getting data from point A (server / cloud) to point B (the web app / internet browser)

Software Resources

First you're going to want to select a front end mapping API. There are many available, but I'll just focus on a few.

2D
- MapBox-GL
- Leaflet
- Openlayers
- Deck-GL (2.5D)
3D
- CesiumJs
- Vadstena by Melown tech
- Harp
Geospatial DataBase
- PostGIS an extension of PostgreSQL
- GeoCouch (an extension of CouchDB)
- MongoDB
Middleware
Online Geospatial Tools

2D

Pick Your API

Ok, so you are after a 2D web mapping application, let's get started. Each of the above open source 2D web mapping apis are great and would probably fit the bill for most applications, but there are a few differences that might be worth considering. Leaflet might be the best bet for a basic project, while OpenLayers is probably better suited for advanced applications. I tend to use MapBox-GL for most apps. I'll get into some reasons why a bit later in the post.

Basemap

You are going to need a basemap. A basemap provides a background of geographical information for the map, it can be a satellite and or street map or a hybrid of the two. For most situations it is going to make sense to get a basemap from some kind of mapping provider. A few include: OpenStreetMaps, Bing Maps, some Googling will get you the rest of the way. MapBox studio will allow you to create a custom styled basemap (based on OpenStreetMap data).

Scalability

Before we get too far, we need to think for a moment about the data that you want to render in the map. If a lot of data needs to be rendered in the map, we can run into scalability issues if we are not careful. Most data formats that we will deal with in 2D maps fall into two categories: vector and raster data. Vector data can be points, lines and polygons. Raster data is generally served to the front end mapping environment as tiled images. If your application does not require scaling tricks, you can render vector data directly from GeoJSON. Tip online GeoJson editor is a hand tool for creating and editing GeoJSON.

In the early days of web mapping, it was easy to encounter performance issues when rendering too many vector features in a web browser. A solution was to burn the vector features into a raster on the server side, and then render the images in the web application. This is still a common approach with middleware solutions like GeoServer or MapServer.

Another (more modern) approach is to use MapBox vector tiles (MVT). With MVTs you can render vast amounts of vector data to the web. MVTs can be generated (commonly on the server side) and served by a cloud provider or you can generate and serve them yourself.

MapBox-GL and OpenLayers can render MVT, but Leaflet cannot without an extension. The MapBox-GL API provides integrated client side filtering and styling directly on the MVT spec. This can be a significant advantage if you plan to use MVT extensively in your app. Tip T-Rex, Tegola, Tippecanoe and Geojson-VT can be used to create MVTs.

Navigation

I have never build a navigation app, but there are a lot of navigation tools out there:

OSRM (open source)
Open Source Routing Machine (open source)
Graphhopper (open source)
Valhalla (open source)
MapBox
Google Maps
Bing
ESRI
Mapquest

Data Manipulation

You may want to allow a user to tinker with the data, for example modify the location of a vector point. In this case you cannot modify raster images or MVT data directly from the font end application. One solution is to identify the modification in the front end, then send the change server side and then update the data, you will then need to re-render your data front end, however this could be troublesome, let me explain why:

Suppose you are burning your vector data into rasters, you may want to cache these images (caching is a really good idea, we have not talked about rendering raster tiles in detail, but in a nut shell you will need to render a pyramid of tiles, each layer of the pyramid will be needed for all of the zoom levels available to your web mapping application). Now you have just changed a feature in your data, in order to make the data change visible in the map you will need to blow away your cache and start burning new images, this cache management can be a painful and expensive.

If you are using MVT, a solution could be to first request the GeoJson data for a feature from the server or geospatial middleware (even if you are rending the bulk of the data using raster images or MVT). Then, you can manipulate that GeoJson feature on the front-end with Turf.js, then pass your newly updated GeoJSON feature back to the server for a database update. Next, you will need to update the MVT and manage the cache, be carful, often times the MVT are cached client side. The best solution I have found is to isolate the layers that may get frequent updates from layers that are not often getting updated. Then modify the MVT headers such that the cache refresh rate is very low. This is not a very elegant solution, so if someone has better ideas I'm all ears.

3D

Because 3D web rendering is kind of new, I have only had the opportunity to deep dive into CesiumJS, so I'm going to talk mostly about that.

Raster tiles can be rendered in a 3D environment. Vector formats like GeoJSON can also be rendered in a 3D environment, however, similar to the 2D environment, if you render a lot of vector features you can run into performance issue quickly.

When viewing a map in 2D, you are looking at the data from a top down - birds eye view. Raster and vector formats are well suited for this perspective. However, when we start looking at the map in 3D it is interesting to explore different ways of rendering data. For this reason, Cesium has developed 3D tiles and quantized mesh terrain format formats that can be used to render point-cloud, 3D models and terrain data.

3D Tiles

In CesiumJS or Deck-GL ecosystems 3D tiles can be used to render data in the map. 3D tiles are a spatial data structure that enable hierarchical level of detail (HLOD) so only visible tiles are streamed to the front-end map.

With 3D tiles you can encode a bunch of data formats into 3D tiles and then render 3D tiles in CesiumJS. The formats include:

glTF (.gltf, .glb)
CityGML (.citygml, .xml, .gml)
KML/COLLADA (.kml, .kmz)
LASer (.las, .laz)
COLLADA (.dae)
Wavefront OBJ (.obj)

When the above data formats are wrapped into the 3D tiles spec and rendered in the web, CesiumJS will "define a spatial hierarchy for fast streaming and precision rendering, balancing performance and visual quality at any scale from global to building interiors". This is a necessity when rendering very large data sets like point clouds. It's worth mentioning that ESRI has a similar open source spec called i3s, shoot me a note if you know of an open source i3s web consumer.

Cesium also developed a terrain spec, with it you can take a digital elevation model (DEM) and convert it into a quantized mesh terrain format. You can create and serve 3D tiles and terrain in the cloud via Cesium ion or you can create them using open source tools like Entwine, for point cloud -> 3D tiles and cesium terrain builder docker for DEM -> terrain. Side note if you are after a simple point cloud web render Cesium might be over kill and you might want to check out Potree. Tip lopocs is an open source point-cloud server. Pro-tip check out PDAL for all your point cloud translating and manipulating needs.

Back-End

The data that you want to render on the map is going to be saved somewhere, the two conventional solutions for this are a database or a geospatial cloud service. There are a number of geospatial hosting / serving solutions out there, MapBox, Here, Esri, Carto, MapZen to name a few. This could be a good solution if you want to pay a monthly fee and be done. But this post is on open source solutions so I am going to focus on that. You might need a geospatial database depending on the complexity and size of your data and your application requirements. Tip Geo-Stackexchange is a great resource for geospatial software development questions.

Vector Data

A geospatial database will allow you to geo-index your data, this allows the data to be accessed much more efficiently. Geospatial databases also allow you to do geospatial operations on the data. For example you might want to find all of the points that are within a given distance from another point. The most feature rich open source geospatial database is PostGIS which is an extension of PostgreSQL. If you are looking for a noSQL option then GeoCouch (an extension of CouchDB) may be a good fit, or if you just need to serve up some GeoJSON features MongoDB might be all you need. Pro-tip GDAL is a translator library for raster and vector geospatial data formats.

Raster Data

PostGIS is probably the best option for hosting raster geospatial data. In order to get those raster to the web, you are going to need a geospatial middleware, see below for details. Tips rasterio is a nifty little raster project that can read and write geospatial raster data and QGIS is an open source GIS tool I use for pre-processing geospatial data.

Geospatial Middleware

A geospatial middleware is used to translate or transform data from the database or from a directory into a format that is easily interpreted by the mapping API in the front-end. For example, if you have some vector data in your database or in a file directory, the middleware can generate and serve MVT to the front end web application. Middleware can also generate and serve raster tiles from a raster source file. GeoServer, GRASS GIS and MapServer are a few great middleware tools. A geospatial middleware is also going to help you by moving geospatial data around in an OGC standard format (which is a good idea!). Pro-tip GeoNode is a full-stack open source geospatial content management solution.

Wish List

If I could snap my fingers and change anything in the open source web mapping world, I would love to see a 3D vector tiles spec developed for the CesiumJS ecosystem.

Also, I would change the way that the MapBox-GL API manages the basemap layer. If you render a basemap and then add some raster or vector layers on top of the basemap and then change the basemap for example from a street map to a satellite map, you will need re-order all of your layers to get back to the map state as it was before the basemap was changed.

Community

If you're like me and get hooked on this stuff, you'll also want to check out FOSS4G a yearly open geospatial technology and business conference organized by the Open Source Geospatial Foundation. They also have regional flavors of the conference. Hope to see you there!

DEV Community: bwyss