Muhammed Enes Duran

Posted on Jul 23

Why AI-Generated GIS Code Can Run Successfully and Still Be Wrong

#ai #opensource #showdev #geospatial

Geospatial code has an unusual failure mode: it can execute without errors, produce a clean-looking result, and still be methodologically wrong.

This problem is becoming more important as AI coding agents generate complete GIS workflows from short natural-language instructions.

The issue is not always that an AI agent does not know GeoPandas, rasterio, PostGIS, Earth Engine, or ArcPy. In many cases, it knows the correct API and can produce executable code.

The problem is that API correctness is not the same as geospatial correctness.

A silent CRS failure

Consider a simple buffer operation:

`buffered = gdf.buffer(1000)`

The code looks reasonable. It runs successfully and produces new geometries.

But what does 1000 mean?

When the dataset uses a geographic coordinate reference system such as EPSG:4326, its coordinate units are degrees rather than metres. The operation may still return valid geometries, but they do not represent a 1,000-metre buffer.

A safer workflow makes the spatial assumptions explicit:

if gdf.crs is None:
    raise ValueError("The input dataset has no CRS.")

if gdf.crs.is_geographic:
    projected_crs = gdf.estimate_utm_crs()

    if projected_crs is None:
        raise ValueError("A suitable projected CRS could not be determined.")

    gdf = gdf.to_crs(projected_crs)

buffered = gdf.buffer(1000)

Even this pattern is not universally correct. A locally estimated UTM projection may be unsuitable for datasets that span multiple zones, countries, or continents.

The important point is not one specific code snippet. A defensible workflow must ask:

Is the CRS defined?
Are the coordinate units appropriate?
Is the projection suitable for the geographic extent?
How will the output be verified?

Without those checks, plausible output can hide an invalid spatial operation.

A machine-learning result that looks better than it is

Spatial machine learning has a similar problem.

A common model evaluation starts with a random train/test split:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
)

For independent observations, this can be a reasonable starting point.

Spatial observations, however, are often correlated with nearby observations. A random split can place neighbouring samples in both the training and test sets.

The model then appears to generalize, while it may only be exploiting local spatial similarity.

This can produce an impressive accuracy score that collapses when the model is applied to a genuinely new region.

A more defensible evaluation may require:

spatial blocks or geographic groups
region-level holdouts
distance-based separation
temporal separation for future prediction
maps of residuals and prediction errors
comparison between random and spatial validation

Spatially blocked validation often produces a lower score.

That lower score may be the more honest result.

Rendering is not verification

Maps introduce another form of silent failure.

A map can render correctly while using:

an unsuitable projection
misleading classification boundaries
inconsistent scales across dates
inaccessible colour choices
unreported missing values
geometries silently removed during processing

A change-detection map can look convincing even when the source rasters are misregistered, collected in different seasons, or processed at incompatible levels.

Successful rendering proves that the software produced an image.

It does not prove that the analysis is valid.

What an AI agent needs beyond API knowledge

A general-purpose coding agent may know which function to call. It does not necessarily know when a workflow should stop.

For geospatial tasks, an agent needs explicit operational rules:

Never calculate area or distance in a geographic CRS.
Treat spatial leakage as a default risk in predictive modelling.
Validate geometries and account for input and output row counts.
Preserve georeferencing through raster and deep-learning pipelines.
Report uncertainty when the analytical method supports it.
Verify outputs numerically and visually.
Refuse invalid spatial or temporal comparisons instead of producing a misleading result.

These are not optional improvements added after the analysis.

They are part of the analytical method itself.

Building GeoAI Skills

I built GeoAI Skills to encode these safeguards as reusable Agent Skills.

The current public preview contains 18 skills covering:

geospatial data engineering
PostGIS and spatial SQL
remote sensing
Google Earth Engine
geospatial deep learning
spatial statistics
geostatistics and interpolation
terrain and hydrology
suitability analysis
point clouds and LiDAR
network accessibility
movement trajectories
change detection
cartography and geovisualization
guarded ArcGIS Pro automation
machine-learning experiment standards
software engineering and DevOps practices
multi-stage GeoAI workflow orchestration The objective is not to make an AI agent sound like a GIS expert.

The objective is to make assumptions visible, require verification, and fail loudly when a workflow is not defensible.

A land-suitability request, for example, may involve data preparation, remote sensing, terrain analysis, multi-criteria decision analysis, and cartographic delivery.

GeoAI Skills uses a central orchestrator to route each stage to the relevant specialist skill while maintaining shared safeguards such as CRS checks, leakage prevention, uncertainty reporting, and verification.

Guarded ArcGIS Pro automation

The suite also includes an arcgis-pro-automation skill.

It works with my open-source arcgis-mcp-bridge project to support controlled local ArcPy execution.

*The integration covers workflows involving:
*

ArcGIS Pro projects
file and enterprise geodatabases
vector and raster geoprocessing
spatial and statistical analysis
network analysis
maps and layouts

Because ArcPy operations can modify real projects and datasets, the execution model includes path restrictions and confirmation gates for mutating or destructive actions.

The goal is not unrestricted automation.

It is controlled automation with explicit boundaries.

Testing whether the correct skills activate

A skill is not useful if the agent does not activate it for the right request—or activates it for unrelated requests.

The repository currently contains 131 typed evaluation scenarios across the 18 skills.

These scenarios include:

positive activation cases
negative cases
ambiguous requests
collisions between related skills
artifact-oriented evaluations

For a frozen 17-skill, 120-case routing suite using Claude Code 2.1.214 and Claude Sonnet 5, the published results were:

100% routing precision
92.86% routing recall
92.5% full-route accuracy

These results have an important limitation.

They measure whether the expected skills were selected for that exact runtime, model, and evaluation suite. They do not prove that every generated analysis is correct, and they are not universal compatibility or answer-quality claims.

The eighteenth skill, arcgis-pro-automation, was added after the frozen benchmark and is not included in those headline figures.

Routing evidence, behavioural correctness, and real-world analytical quality should be evaluated separately.

Trying the public preview

The complete suite can be explored and installed interactively with:

npx skills add muend/geoai-skills

A single specialist skill can also be installed independently:

npx skills add muend/geoai-skills --skill postgis-spatial-sql

GitHub repository:

https://github.com/muend/geoai-skills

Skills collection:

https://www.skills.sh/muend/geoai-skills

The project is still an early public preview.