SamGIS - Segment Anything applied to GIS

SamGIS - Segment Anything applied to Geographic Information Systems (GIS)

This project, composed by a SPA frontend and by a backend (samgis-be), is an attempt to perform machine learning image segmentation on geo-spatial data even without the use of dedicated graphics cards. I recently added the ability to start from natural language text prompts.

I wrote the backend adapting SAM Exporter with the aim to use the machine learning segment anything project for the purpose to improve polygons recognition in GIS web applications.

Starting from version 1.5.1 the backend integrates changes borrowed from sam_onnx_full_export, to support OnnxRuntime 1.17.x and later versions. Please note that on MacOS directly running the project from the command line suffers from memory leaks, making inference operations slower than normal. It's best therefore running the project inside a docker container, unless in case of development or debugging activities.

These are the project resource links:

• a public HuggingFace space demo
• the documentation and AWS lambda OpenAPI specs

Note that the HuggingFace space demo has its static frontend files under the /static folder.

Ask me for a user via linkedin if you would like to test the authenticated demo or an explanation about the code.

LISA and SamGIS

I also added support for image segmentation via natural language text prompts: here are the details about the SamGIS integration with LISA.

SamGIS also replaces SurferDTM

SurferDTM was an attempt to find stratovolcanoes within digital terrain models (DEM/DTM). As a new feature SamGIS now can now use also input DEM images (like the first image in this gallery):

Note that the "terrarium" source provider data encode the image as RGB, so the backend processes it to obtain the elevation data (like the second image in this gallery) using this formula (from the map tile provider page):

(red * 256) + green + (blue / 256) - 32768

I also added a custom elaboration step to create an intermediate RGB image composed by channels:

• RED - normalized DEM image
• GREEN - normalized 2d composed image by
  • normalized DEM image
  • slope
  • curvature
• BLUE - curvature

Segment Anything will finally do the inference on this rgb image. This is the json payload request:

{
   "bbox": {
         "ne": {"lat": 46.203706648934954, "lng": 9.401906739170105},
         "sw": {"lat": 46.12464369105346, "lng": 9.231103669101747}},
   "prompt": [
         {"id": 238, "type": "point", "data": {"lat": 46.158760519552146, "lng": 9.306795638666486}, "label": 1},
         {"id": 250, "type": "point", "data": {"lat": 46.152693546080975, "lng": 9.289288652508679}, "label": 0},
         {"id": 264, "type": "point", "data": {"lat": 46.16613515509541, "lng": 9.319840059725284}, "label": 0},
         {"id": 272, "type": "point", "data": {"lat": 46.161377438862836, "lng": 9.288087192674299}, "label": 0},
         {"id": 285, "type": "point", "data": {"lat": 46.174103408003454, "lng": 9.308340372739261}, "label": 0},
         {"id": 293, "type": "point", "data": {"lat": 46.17005996124035, "lng": 9.298042145587583}, "label": 0}
   ],
   "zoom": 13,
   "source_type": "nextzen.terrarium"
}

Project architecture of AWS demo

1. The user interact with a WebMap (in my example it's a leaflet map that use OpenStreetMap map data).
2. The user send a POST request to an API using
   1. the current map extent
   2. the current map zoom
   3. a prompt array of objects; objects could be
      • an "include" marker position (a couple of float - latitude and longitude)
      • an "exclusionary" marker position (a couple of float - latitude and longitude)
      • a "include" rectangle (two latitude and longitude positions - north-east and south-west)
3. because of CORS constrain it's difficult to directly send requests to remote resources on different domains: to avoid this problems I prepared a simple Cloudflare proxy function that forward the user request to the remote API
4. The API Gateway proxies the request to the serverless backend
5. the serverless Lambda API elaborate the request:
   1. it parses the request and in particular translate the input latitude/longitude prompt into a x/y (pixel coordinates) one
   2. it downloads the geo-referenced tiles then merge and crop them
   3. it instantiates a MobileSam model instance if not already created using OnnxRuntime as runtime
   4. it prepares the image embedding and...
   5. ...it starts the image inference process
   6. it transforms the recognition masks into a single binary mask then it vectorizes the recognized binary mask:
      1. first it converts the mask into a GeoPandas GeoDataframe using rasterio.features.shapes and GeoDataFrame.from_features
      2. then it convert the new Geodataframe into a geojson using GeoDataFrame.to_json
6. the Cloudflare proxy function parses the response and forward it to the frontend
7. the frontend parse the geojson to load within the WebMap and shows some response info like the non-relativistic response duration time:

To avoid abuse the prompt input web map that I host on my personal site is under authentication (I use auth0.com).

Fastapi-based demo architecture

This project variant works almost as this. There are these differences:

• opencv-python is not anymore a SamGIS project dependency
• the server now saves and re-uses image embeddings to avoid resource waste and speed up the inference process
• SPA frontend now is integrated directly within the Fastapi backend

Frontend and backend now are on the same domain. Therefore there is no longer use for a proxy (the AWS demo used a Cloudflare Pages function) to make requests to APIs on a different domain.

Additionally, since I can host this Fastapi demo for free on HuggingFace, I don't need anymore an authentication system.

AWS lambda, a serverless solution: some pros...

• It can work also with docker images for complex or big projects (like this)
• Almost easy, working out-of-the-box integration with other AWS services and third-parts authentication systems
• Pay-as-you-go service

...and some cons

• Setup not always easy (I used Powertools for AWS Lambda (Python) to improve my logger setup)
• The serverless model suggested by AWS not always it's the best solution
• Arm graviton EC2 instances that run AWS lambda are greatly limited (OnnxRuntime at the moment cannot run on Arm AWS lambda and probably will never run).