Enrich Elasticsearch Logs With GeoIP at Ingest

An access log full of raw IPs can't answer where a traffic spike came from. Elasticsearch GeoIP fixes that at write time, enriching public-IP events with whatever country, region, city, and coordinate data is available so queries, alerts, and dashboards downstream can use geography directly. This guide sets up that enrichment in an ingest pipeline, fixes the two things that silently break it, and puts your logs on a Kibana map.

TL;DR

• The geoip processor ships with Elasticsearch as a module. No plugin install, and it auto-downloads the free MaxMind GeoLite2 databases.
• Enrichment happens in an ingest pipeline. Test it with the _simulate API before you point real data at it.
• Your Kibana map will be empty unless the location field is mapped as geo_point before you index. Set that in an index template first.
• Private and internal IPs (RFC 1918) return no geo fields by design. Handle them, don't ignore them.
• GeoLite2 is city-level-optimistic and CC BY-SA. When you need better accuracy or security context, swap the database at ingest without changing the pipeline shape.

The short version: turn on a bundled processor, wire it into a pipeline, map one field correctly, and Kibana renders your traffic on a map. The parts that trip people up are the geo_point mapping and the IPs that have no location, both covered below. The core examples below target modern Elasticsearch 8.x and 9.x clusters with security enabled. Third-party ingest plugins have their own Elasticsearch-version compatibility requirements, so check the plugin's supported version before installing one.

What the Elasticsearch geoip processor gives you

The geoip processor is a module distributed with Elasticsearch, so on a current cluster there's nothing to install. By default it uses the GeoLite2 City, GeoLite2 Country, and GeoLite2 ASN databases from MaxMind, shared under CC BY-SA 4.0, and Elasticsearch downloads and updates them automatically from the Elastic GeoIP endpoint.

Feed it a field holding an IP address and it writes a geoip object onto the document. With the default City database, that object can include continent_name, country_iso_code, country_name, region_iso_code, region_name, city_name, and a location object holding lat and lon. You control which of those land with the properties option; the default set is continent_name, country_iso_code, country_name, region_iso_code, region_name, city_name, and location.

Two option names worth knowing before you write anything: target_field (default geoip) is where the enriched object goes, and ignore_missing (default false) decides whether a document with no IP field quietly passes through or errors. We'll use both.

Step 1: Create the ingest pipeline

An ingest pipeline is a named list of processors that runs on a document before it's indexed. Create one with a single geoip processor pointed at your IP field:

# Create a pipeline named "geoip-pipeline" that enriches the "ip" field.
# --cacert points at the CA cert Elasticsearch generated on first start;
# on 8.x/9.x the HTTP layer is TLS-on by default, so http:// will fail.
curl -sS -X PUT "https://localhost:9200/_ingest/pipeline/geoip-pipeline" \
  --cacert /etc/elasticsearch/certs/http_ca.crt \
  -u "elastic:$ELASTIC_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "description": "Add GeoIP fields from the ip field",
    "processors": [
      {
        "geoip": {
          "field": "ip",
          "target_field": "geoip",
          "ignore_missing": true
        }
      }
    ]
  }'

ignore_missing: true matters more than it looks. Real log streams have events without a client IP (health checks, internal jobs). Without it, every one of those throws and the document is rejected. With it, they index cleanly and simply carry no geoip object.

Before you send a single real document, dry-run the pipeline with the _simulate API. This runs your processors against a sample doc and returns the result without indexing anything:

# Dry-run the pipeline against a real public IP.
# 8.8.8.8 is a convenient public test address. The exact GeoLite2 fields and coordinates can change as the database is updated.
curl -sS -X POST "https://localhost:9200/_ingest/pipeline/geoip-pipeline/_simulate" \
  --cacert /etc/elasticsearch/certs/http_ca.crt \
  -u "elastic:$ELASTIC_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "docs": [
      { "_source": { "ip": "8.8.8.8" } }
    ]
  }'

The response wraps the enriched document under doc._source, and you should see a populated geoip object:

{
  "docs": [
    {
      "doc": {
        "_source": {
          "ip": "8.8.8.8",
          "geoip": {
            "continent_name": "North America",
            "country_iso_code": "US",
            "country_name": "United States",
            "location": { "lon": -97.822, "lat": 37.751 }
          }
        }
      }
    }
  ]
}

If _simulate gives you the fields you expect, the pipeline works. The rest is deciding what data flows through it and making sure Kibana can read the output.

Step 2: Map location as geo_point before you index

Here's the failure that fills the Elastic forums: everything ingests fine, the geoip.location field is populated, and the Kibana map is stubbornly empty. The cause is mapping. Elasticsearch's dynamic mapping sees { "lat": ..., "lon": ... } and guesses two float fields. Kibana Maps only plots a field typed as geo_point, and it won't infer that on its own.

Fix it once with an index template so every matching index is born with the right type:

# Any index matching "logs-geo-*" gets geoip.location typed as geo_point.
# Create this BEFORE the first document lands, or the wrong type sticks.
curl -sS -X PUT "https://localhost:9200/_index_template/logs-geo-template" \
  --cacert /etc/elasticsearch/certs/http_ca.crt \
  -u "elastic:$ELASTIC_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "index_patterns": ["logs-geo-*"],
    "template": {
      "settings": {
        "index.default_pipeline": "geoip-pipeline"
      },
      "mappings": {
        "properties": {
          "geoip": {
            "properties": {
              "location": { "type": "geo_point" }
            }
          }
        }
      }
    }
  }'

Two things happened in that template. The mappings block pins geoip.location to geo_point so Kibana will plot it. And index.default_pipeline sets the pipeline to run automatically on anything written to a logs-geo-* index, so you don't have to name it on every request.

Pitfall: mapping type is fixed at field creation. If documents already landed and geoip.location came out as float, you can't change it in place. You create a new index with the correct template and reindex into it. Get the template in before the first write and you never hit this.

Step 3: Wire the pipeline to your log stream

You have two ways to attach the pipeline. The template above already set index.default_pipeline, which is the clean option: write to the index normally and enrichment is automatic. The other option is naming the pipeline per request with ?pipeline=geoip-pipeline, which is handy for one-off backfills but noisy to repeat on every write.

If you ingest with your own code, the bulk helper is the usual path. Here's a minimal, production-shaped Python ingester using the official client:

import os
from elasticsearch import Elasticsearch, helpers

# Never hardcode credentials. Pull them from the environment.
ES_URL = os.environ.get("ES_URL", "https://localhost:9200")
ES_USER = os.environ.get("ES_USER", "elastic")
ES_PASSWORD = os.environ.get("ES_PASSWORD")
CA_CERT = os.environ.get("ES_CA_CERT", "/etc/elasticsearch/certs/http_ca.crt")

client = Elasticsearch(
    ES_URL,
    basic_auth=(ES_USER, ES_PASSWORD),
    ca_certs=CA_CERT,
    request_timeout=10,  # fail fast instead of hanging a worker
)

def log_events():
    # Replace with your real source (a file tail, a queue, etc.).
    yield {"ip": "8.8.8.8", "path": "/checkout", "status": 200}
    yield {"ip": "2001:4860:4860::8888", "path": "/login", "status": 401}

def index_logs():
    # The index name matches logs-geo-*, so the default_pipeline runs
    # and geo_point mapping applies. No per-doc pipeline needed here.
    actions = ({"_index": "logs-geo-000001", "_source": event}
               for event in log_events())
    try:
        success, errors = helpers.bulk(client, actions, raise_on_error=False)
        if errors:
            # Log rejected docs; don't let one bad IP kill the batch.
            for err in errors:
                print(f"index error: {err}")
        return success
    except Exception as exc:
        # Network or auth failure: surface it, don't swallow it.
        print(f"bulk ingest failed: {exc}")
        return 0

if __name__ == "__main__":
    print(f"indexed {index_logs()} events")

raise_on_error=False plus an error loop is deliberate: one document with a malformed IP shouldn't sink a 10,000-event batch. You log the rejects and keep going.

One convention worth adopting if you use the Elastic Common Schema: put the client address in source.ip and target source.geo instead of a bare geoip field. Elastic's own dashboards and detection rules expect ECS field names like source.geo.country_iso_code, source.geo.city_name, and source.geo.location. Set "field": "source.ip" and "target_field": "source.geo" in the processor and your data drops straight into the built-in tooling.

A word on Logstash, since half the tutorials on this topic start there. The Logstash geoip filter still works and still makes sense if Logstash is already in your pipeline doing Grok parsing and transforms. But if Elasticsearch is your ingest point, the ingest-pipeline approach keeps enrichment inside the cluster with one less moving part to run and monitor. Use the filter when you're already in Logstash; use the processor when you're not.

Step 4: Put it on a map in Kibana

With geo_point mapping in place, Kibana Maps takes a couple of clicks:

1. Create a data view over your logs-geo-* indices (Stack Management, then Data Views).
2. Open Maps from the Analytics menu and add a layer from that data view.
3. Because geoip.location (or source.geo.location) is a geo_point, the documents layer plots each event as a point. Style by count to cluster dense areas. If you followed a tutorial that told you to create a "Tile Map" or "Coordinate Map" visualization, that's why it didn't work. Those classic visualization types were retired; the Maps application replaced them, and it reads geo_point fields directly through a data view.

Private and internal IPs return nothing

The geoip processor looks up public IP geolocation. Private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), loopback, and carrier-grade NAT (100.64.0.0/10) have no geographic location, so those documents come back with no geoip object at all. That's correct behavior, but if a chunk of your traffic is internal, a naive setup leaves you with events that silently have no geo and no explanation.

Tag them instead of dropping them. Add a conditional processor that marks internal traffic before the lookup, so a missing geoip object is intentional rather than a mystery:

# Mark RFC 1918 addresses as internal, then geoip-enrich the rest.
# Explicitly match the RFC 1918 private ranges so internal traffic
# is labeled instead of silently appearing without GeoIP data.
curl -sS -X PUT "https://localhost:9200/_ingest/pipeline/geoip-pipeline" \
  --cacert /etc/elasticsearch/certs/http_ca.crt \
  -u "elastic:$ELASTIC_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "description": "Tag internal IPs, geoip-enrich the rest",
    "processors": [
      {
        "set": {
          "field": "network.scope",
          "value": "internal",
          "if": "def ip = ctx.ip; ip != null && (ip.startsWith(\"10.\") || ip.startsWith(\"192.168.\") || ip.startsWith(\"172.16.\") || ip.startsWith(\"172.17.\") || ip.startsWith(\"172.18.\") || ip.startsWith(\"172.19.\") || ip.startsWith(\"172.20.\") || ip.startsWith(\"172.21.\") || ip.startsWith(\"172.22.\") || ip.startsWith(\"172.23.\") || ip.startsWith(\"172.24.\") || ip.startsWith(\"172.25.\") || ip.startsWith(\"172.26.\") || ip.startsWith(\"172.27.\") || ip.startsWith(\"172.28.\") || ip.startsWith(\"172.29.\") || ip.startsWith(\"172.30.\") || ip.startsWith(\"172.31.\"))"
        }
      },
      {
        "geoip": {
          "field": "ip",
          "target_field": "geoip",
          "ignore_missing": true
        }
      }
    ]
  }'

Internal events now carry network.scope: internal, so one common reason for a missing geoip object is explicit instead of silent. Public IPs with no database match can still have no geoip object. No more staring at half-empty maps wondering where the other half went.

Backfilling indices you already have

Enrichment only touches documents at write time, so turning on the pipeline does nothing for the millions of events already sitting in your indices. To geo-tag existing data, run _update_by_query with the pipeline applied:

# Reprocess existing docs through the pipeline.
# Run this against a copy or a low-traffic window first: it rewrites
# every matched document and costs real I/O.
curl -sS -X POST "https://localhost:9200/logs-geo-000001/_update_by_query?pipeline=geoip-pipeline&wait_for_completion=false" \
  --cacert /etc/elasticsearch/certs/http_ca.crt \
  -u "elastic:$ELASTIC_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {
      "bool": {
        "must_not": [
          { "exists": { "field": "geoip" } }
        ]
      }
    }
  }'

The must_not exists geoip filter avoids reprocessing documents that were successfully enriched. Be aware that private IPs and public IPs with no database match also have no geoip field, so they will be selected again on a later run. If you need a strictly resumable backfill, add a separate processing marker and filter on that instead.
wait_for_completion=false returns a task ID immediately so a long backfill doesn't tie up your connection; poll it with the Tasks API.

One caveat: if the old index has geoip.location mapped as float (because it predates your template), _update_by_query writes the fields but Kibana still won't map them. In that case you reindex into a freshly templated index instead. Backfill in place when the mapping is already correct; reindex when it isn't.

When GeoLite2 is not enough

GeoLite2 is free, bundled, and good enough to see which countries your traffic comes from. It is also the lite tier: MaxMind's own positioning is that GeoLite2 is less accurate than paid GeoIP2, city-level results are approximate, and the CC BY-SA 4.0 license carries attribution and share-alike obligations that some commercial deployments can't accept. When you outgrow it, you swap the database, not the pipeline.

You have a few paths, and they nest cleanly into what you already built:

• Point the same processor at a licensed MaxMind database. Elasticsearch can download GeoIP2 directly using your MaxMind license key through the database configuration API, and the geoip processor keeps working unchanged.
• Use IPinfo databases via the ip_location processor. Recent Elasticsearch versions ship an ip_location processor that supports IPinfo databases alongside MaxMind, so you can switch providers without leaving the ingest layer.
• Load a custom database file. Both processors accept a database_file option, so a compatible MaxMind-format database supported by the processor can back the lookup.
• Enrich from a local database that also carries security and timezone context. IPGeolocation publishes an Elasticsearch ingest processor plugin that runs the lookup against a locally installed IPGeolocation database, so alongside country and city you get currency, timezone, and IP-security signals in the same document, with no per-event API call. That last option is worth a concrete look because it combines IPGeolocation's geolocation, timezone, currency, and security data in one local enrichment path. The plugin installs with elasticsearch-plugin install, a setup script downloads the database tier you licensed, and the processor is named ipgeo. Its required options are field (your IP field) and database_version (DB-I through DB-VII, the database tier you licensed); target_field defaults to ipgeo, and include pulls in the extra data classes:

# The ipgeo processor from the IPGeolocation plugin.
# database_version selects the licensed DB tier; include pulls in
# the security and timezone data classes on top of geolocation.
curl -sS -X PUT "https://localhost:9200/_ingest/pipeline/ipgeo-pipeline" \
  --cacert /etc/elasticsearch/certs/http_ca.crt \
  -u "elastic:$ELASTIC_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "description": "Enrich with IPGeolocation databases",
    "processors": [
      {
        "ipgeo": {
          "field": "ip",
          "database_version": "DB-VI",
          "ignore_missing": true,
          "include": "security,timezone"
        }
      }
    ]
  }'

A document enriched by that pipeline carries a richer object than GeoLite2 returns. The relevant part of the response, from the plugin's own docs:

{
  "ipgeo": {
    "country_name": "United States",
    "country_code2": "US",
    "city": "San Francisco",
    "state_prov": "California",
    "zipcode": "94107-2008",
    "location": { "lon": -122.39117, "lat": 37.78229 },
    "currency": { "symbol": "$", "code": "USD", "name": "US Dollar" },
    "time_zone": {
      "name": "America/Los_Angeles",
      "offset": -7.0,
      "is_dst": true,
      "current_time": "2024-05-17 04:52:01.693-0700"
    },
    "security": {
      "is_proxy": false,
      "proxy_type": "",
      "is_tor": false,
      "is_anonymous": false,
      "is_known_attacker": false,
      "is_bot": false,
      "is_spam": false,
      "is_cloud_provider": false,
      "threat_score": 0
    }
  }
}

The trade-off is honest and worth stating plainly: this path costs a plugin install, a licensed database, and an Elasticsearch restart, and the ipgeo.location field still needs the same geo_point mapping as before. What you get back is currency, timezone, and a security block (proxy, Tor, cloud-provider, known-attacker flags, and a threat_score) on every event, computed locally with no lookup latency. If your logs feed fraud rules or security dashboards, that context in the same document is the reason to pay. If you only need a map of where traffic comes from, GeoLite2 is already enough and you can stop at Step 4.

Operational notes

A few things that bite in production and rarely make it into tutorials:

The auto-downloaded databases expire. If Elasticsearch can't reach the Elastic GeoIP endpoint for 30 days, it stops using the stale databases and enrichment quietly goes blank, showing up as _geoip_expired_database tags on your documents. In an air-gapped cluster this is a when, not an if. Plan for a reverse-proxy or custom endpoint, or serve your own databases and turn the downloader off.

To disable the downloader entirely, set ingest.geoip.downloader.enabled: false in the cluster settings. Do this when you run only custom or licensed databases and don't want Elasticsearch reaching out for GeoLite2 updates you aren't using. Leave it enabled if you rely on Elasticsearch's automatically downloaded GeoLite2 databases. Disabling the downloader removes those downloaded databases; the 30-day expiration behavior applies when updates remain enabled but Elasticsearch cannot reach the GeoIP endpoint.

Enrichment runs on ingest nodes and costs CPU per document. On a high-volume cluster, watch ingest node load after you turn on a geo pipeline, and give dedicated ingest nodes room if a lookup runs on every event.

Get the geo_point mapping in before the first write, tag your internal IPs, and decide up front whether the free database is enough or you need licensed accuracy and security context. Those three calls are the difference between a map that just works and a week of debugging why half your events have no location.