DEV Community: Philipp Krenn

Elasticsearch: 15 years of indexing it all, finding what matters

Philipp Krenn — Fri, 05 Sep 2025 10:09:13 +0000

Elasticsearch just turned 15-years-old. It all started back in February 2010 with the announcement blog post (featuring the iconic “You Know, for Search” tagline), first public commit, and the first release, which happened to be 0.4.0.

Let’s take a look back at the last 15 years of indexing and searching, and turn to the next 15 years of relevance.

`GET _cat/stats`

Since its launch, Elasticsearch has been downloaded an average of 3 times per second, totaling over 1.45 billion downloads.

The GitHub stats are equally impressive: More than 83,000 commits from 2,400 unique authors, 38,000 issues, 25,000 forks, and 71,500 stars. And there is no sign of slowing down.

All of this is on top of countless Apache Lucene contributions. We’ll get into those for the 25 year anniversary of Lucene, which is also this year. In the meantime, you can check out the 20 year anniversary page to celebrate one of the top Apache projects.

A search (hi)story

There are too many highlights to list them all, but here are 15 releases and features from the last 15 years that got Elasticsearch to where it is today:

Elasticsearch, the company (2012): The open source project became an open source company, setting the stage for its growth.
ELK Stack (2013): Elasticsearch joined forces with Logstash and Kibana to form the ELK Stack, which is now synonymous with logging and analytics.
Version 1 (2014): The first stable release introduced key features like snapshot/restore, aggregations, circuit breakers, and the _cat API.
Shield and Found (2015): Shield brought security to Elasticsearch clusters in the form of a (paid) plugin. And the acquisition of found.no brought Elasticsearch to the cloud, evolving into what is now Elastic Cloud. As an anecdote, nobody could find Found — SEO can be hard for some keywords.
Version 2 (2015): Introduced pipelined aggregations, security hardening with the Java Security Manager, and performance and resilience improvements.
Version 5 and the Elastic Stack (2016): Skipping two major versions to unify the version numbers of the ELK Stack and turning it into the Elastic Stack after adding Beats. This version also introduced ingest nodes and the scripting language Painless.
Version 6 (2017): Brought zero-downtime upgrades, index sorting, and the removal of types to simplify data modeling.
**Version 7** (2019): Changed the cluster coordination to the more scalable and resilient Zen2, single-shard default settings, built-in JDK, and adaptive replica selection.
Free security (2019): With the 6.8 and 7.1 releases, core security became free to help everyone secure their cluster.
ILM, data tiers, and searchable snapshots (2020): Made time-series data more manageable and cost-effective with Index Lifecycle Management (ILM), tiered storage, and searchable snapshots.
Version 8 (2022): Introduced native dense vector search with HNSW and enabled security by default.
ELSER (2023): Launched Elastic Learned Sparse EncodeR model, bringing sparse vector search for better semantic relevance.
Open source again (2024): Added AGPL as a licensing option to bring back open source Elasticsearch.
Start Local (2024): Made it easier than ever to run Elasticsearch and Kibana: curl -fsSL https://elastic.co/start-local | sh
LogsDB (2024): A new specialized index mode that reduces log storage by up to 65%.

The future of search is bright

Thanks to the rise of AI capabilities, search is more relevant and interesting than ever. So what is next for Elasticsearch? There’s way too much to name, so we’ll stick to three areas and the challenges they address.

Serverless

No shards, nodes, or versions. Elasticsearch Serverless takes care of the operational issues you might have experienced in the past:

15 years in, and someone is still setting number_of_shards: 100 for no reason.
15 years, and we’re still debating refresh_interval: 1s vs 30s like it’s a life-or-death decision.
15 years of major versions, minor heart attacks, and the thrill of migrating to the latest version.

You can try out Elasticsearch Serverless today.

ES|QL

“Cheers to 15 years of Elasticsearch — where the Query DSL is still the most complex part of your day.” But it doesn’t have to be. The new Elasticsearch Piped Query Language (ES|QL) brings a much simpler syntax and a significant investment into a new compute engine with performance in mind. While we’re building out more features, you can already use ES|QL today. Don’t worry; the Query DSL will understand.

AI everywhere

15 years of query tuning, and we’re still just throwing boost: 10 at the problem.
15 years of making your logs searchable while you still have no idea what’s happening in production.
Still the best at finding that one log line… if you remember how you indexed it.

AI is redefining what’s possible — from turning raw logs into actionable insights with the AI Assistant for observability and security, to more relevant search with semantic understanding and intelligent re-ranking.

This is only the beginning. More AI-powered features are on the horizon — bringing smarter search, enhanced observability, and stronger security. The future of Elasticsearch isn’t just about finding data; it’s about understanding it. Stay tuned — the best is yet to come.

Thanks to all of you

Thanks to all contributors, users, and customers over the last 15 years to make Elasticsearch what it is today. We couldn’t have done it without you and are grateful for every query you send to Elasticsearch.

Here’s to the next 15 years. Enjoy!

Elasticsearch Transforms: Calculate the Total Duration from Multiple Status Updates

Philipp Krenn — Mon, 12 Jul 2021 00:00:00 +0000

Elasticsearch Transforms let you

convert existing documents into summarized ones (pivot transforms) or
find the latest document having a specific unique key (latest transforms).

This entity-centric view can be helpful for various kinds of data that consist of multiple documents like user behavior or sessions. For example, the duration of a session or a request in a distributed system is a common scenario. The following post is based on a StackOverflow question that is coming up repeatedly in minor variations — use it as a blueprint.

Sample Data

There are three different entities with the uniqueIDs A, B, and C. Each one of them can have multiple status updates with eventStart.timestamp or eventStop.timestamp:

PUT test/_doc/1
{
  "uniqueID": "A",
  "eventStart": {
    "timestamp": "2020-07-01T13:50:55.000Z"
  }
}
PUT test/_doc/2
{
  "uniqueID": "A",
  "eventStop": {
    "timestamp": "2020-07-01T13:51:00.000Z"
  }
}
PUT test/_doc/3
{
  "uniqueID": "B",
  "eventStart": {
    "timestamp": "2020-07-01T13:52:25.000Z"
  }
}
PUT test/_doc/4
{
  "uniqueID": "B",
  "eventStop": {
    "timestamp": "2020-07-01T13:53:00.000Z"
  }
}
PUT test/_doc/5
{
  "uniqueID": "A",
  "eventStop": {
    "timestamp": "2020-07-01T13:54:55.000Z"
  }
}
PUT test/_doc/6
{
  "uniqueID": "C",
  "eventStart": {
    "timestamp": "2020-07-01T13:54:55.000Z"
  }
}

Relying on the default mapping, the two date and the keyword fields are relevant for calculating the different durations:

# Request
GET test/_mapping

# Response
{
  "test" : {
    "mappings" : {
      "properties" : {
        "eventStart" : {
          "properties" : {
            "timestamp" : {
              "type" : "date"
            }
          }
        },
        "eventStop" : {
          "properties" : {
            "timestamp" : {
              "type" : "date"
            }
          }
        },
        "uniqueID" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}

Transforms API

The approach for the calculation is:

Group by the uniqueID.
Get the earliest eventStart and the last eventStop timestamp.
Calculate the time difference (in seconds).

While Kibana provides a UI on top of the Elasticsearch Transform API to click together a job, this example sticks to the Elasticsearch API, which is easier to follow along and reproduce. One handy API is preview with POST _transform/_preview.

Starting with the first step of grouping and since the aggregations part is mandatory, counting the number of status updates:

# Request
POST _transform/_preview
{
  "source": {
    "index": "test"
  },
  "dest": {
    "index": "test_transformed"
  },
  "pivot": {
    "group_by": {
      "id": {
        "terms": {
          "field": "uniqueID.keyword"
        }
      }
    },
    "aggregations": {
      "event_count": {
        "value_count": {
          "field": "_id"
        }
      }
    }
  }
}

# Response
{
  "preview" : [
    {
      "event_count" : 3,
      "id" : "A"
    },
    {
      "event_count" : 2,
      "id" : "B"
    },
    {
      "event_count" : 1,
      "id" : "C"
    }
  ],
  "generated_dest_index" : {
    "mappings" : {
      "_meta" : {
        "_transform" : {
          "transform" : "transform-preview",
          "version" : {
            "created" : "7.13.1"
          },
          "creation_date_in_millis" : 1626059453830
        },
        "created_by" : "transform"
      },
      "properties" : {
        "event_count" : {
          "type" : "long"
        },
        "id" : {
          "type" : "keyword"
        }
      }
    },
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1"
      }
    },
    "aliases" : { }
  }
}

For the final result it is “just” missing the right aggregation(s): The bucket script aggregation sounds promising.

Transforms with Bucket Script Aggregation

Continuing the previous transformation, this one adds the earliest start timestamp, the latest end timestamp, and the duration between the two:

# Request
POST _transform/_preview
{
  "source": {
    "index": "test"
  },
  "dest": {
    "index": "test_transformed"
  },
  "pivot": {
    "group_by": {
      "id": {
        "terms": {
          "field": "uniqueID.keyword"
        }
      }
    },
    "aggregations": {
      "event_count": {
        "value_count": {
          "field": "_id"
        }
      },
      "start": {
        "min": {
          "field": "eventStart.timestamp"
        }
      },
      "stop": {
        "max": {
          "field": "eventStop.timestamp"
        }
      },
      "duration": {
        "bucket_script": {
          "buckets_path": {
            "start": "start.value",
            "stop": "stop.value"
          },
          "script": """
            return (params.stop - params.start)/1000;
          """
        }
      }
    }
  }
}

# Response
{
  "preview" : [
    {
      "duration" : 240.0,
      "stop" : "2020-07-01T13:54:55.000Z",
      "event_count" : 3,
      "start" : "2020-07-01T13:50:55.000Z",
      "id" : "A"
    },
    {
      "duration" : 35.0,
      "stop" : "2020-07-01T13:53:00.000Z",
      "event_count" : 2,
      "start" : "2020-07-01T13:52:25.000Z",
      "id" : "B"
    },
    {
      "stop" : null,
      "event_count" : 1,
      "start" : "2020-07-01T13:54:55.000Z",
      "id" : "C"
    }
  ],
  ...

The calculation in Painless is surprisingly simple: (params.stop - params.start)/1000:

More complicated datetime APIs aren’t needed. Every date in Elasticsearch is stored as a long since the epoche in milliseconds, so a simple difference is enough.
Moving to seconds is a division by 1,000.
Missing end times are automatically handled.

To create the transform job and not just preview it, you need to adjust the request to the following:

PUT _transform/test_duration
{
  "description": "Calculate the duration of an event from multiple status updates (based on its uniqueID)",
  "frequency": "1m",
  "source": {
    "index": "test"
  },
  "dest": {
    "index": "test_transformed"
  },
  "pivot": {
    "group_by": {
      "id": {
        "terms": {
          "field": "uniqueID.keyword"
        }
      }
    },
    "aggregations": {
      "event_count": {
        "value_count": {
          "field": "_id"
        }
      },
      "start": {
        "min": {
          "field": "eventStart.timestamp"
        }
      },
      "stop": {
        "max": {
          "field": "eventStop.timestamp"
        }
      },
      "duration": {
        "bucket_script": {
          "buckets_path": {
            "start": "start.value",
            "stop": "stop.value"
          },
          "script": """
            return (params.stop - params.start)/1000;
          """
        }
      }
    }
  }
}

With GET _transform/test_duration you can see the transform job. And you must explicitly start it with POST _transform/test_duration/_start — otherwise it won’t do anything.

Finally, the stats API is great to see what the job is or has been up to:

# Request
GET _transform/test_duration/_stats

# Response
{
  "count" : 1,
  "transforms" : [
    {
      "id" : "test_duration",
      "state" : "stopped",
      "stats" : {
        "pages_processed" : 2,
        "documents_processed" : 6,
        "documents_indexed" : 3,
        "documents_deleted" : 0,
        "trigger_count" : 1,
        "index_time_in_ms" : 41,
        "index_total" : 1,
        "index_failures" : 0,
        "search_time_in_ms" : 20,
        "search_total" : 2,
        "search_failures" : 0,
        "processing_time_in_ms" : 2,
        "processing_total" : 2,
        "delete_time_in_ms" : 0,
        "exponential_avg_checkpoint_duration_ms" : 114.0,
        "exponential_avg_documents_indexed" : 3.0,
        "exponential_avg_documents_processed" : 6.0
      },
      "checkpointing" : {
        "last" : {
          "checkpoint" : 1,
          "timestamp_millis" : 1626063846766
        },
        "changes_last_detected_at" : 1626063846766
      }
    }
  ]
}

And Kibana gets to the same result:

Last but not least, these are the generated documents:

# Request
GET test_transformed/_search

# Response
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test_transformed",
        "_type" : "_doc",
        "_id" : "QRRx52klPRvG45a5oLgZ95sAAAAAAAAA",
        "_score" : 1.0,
        "_source" : {
          "duration" : 240.0,
          "stop" : "2020-07-01T13:54:55.000Z",
          "event_count" : 3,
          "start" : "2020-07-01T13:50:55.000Z",
          "id" : "A"
        }
      },
      {
        "_index" : "test_transformed",
        "_type" : "_doc",
        "_id" : "Qq7col5MOHvjTNMiAGonnqAAAAAAAAAA",
        "_score" : 1.0,
        "_source" : {
          "duration" : 35.0,
          "stop" : "2020-07-01T13:53:00.000Z",
          "event_count" : 2,
          "start" : "2020-07-01T13:52:25.000Z",
          "id" : "B"
        }
      },
      {
        "_index" : "test_transformed",
        "_type" : "_doc",
        "_id" : "Q-N5zMGevsgbxCl0WsHH6CIAAAAAAAAA",
        "_score" : 1.0,
        "_source" : {
          "stop" : null,
          "event_count" : 1,
          "start" : "2020-07-01T13:54:55.000Z",
          "id" : "C"
        }
      }
    ]
  }
}

That’s it for calculating the duration 🥳

Aggregation without Transforms

Do you need transforms for getting this result? No.

With some small modifications, you can get the same result with a regular aggregation:

# Request
POST test/_search
{
  "size": 0,
  "aggregations": {
    "group_by": {
      "terms": {
        "field": "uniqueID.keyword"
      },
      "aggregations": {
        "start": {
          "min": {
            "field": "eventStart.timestamp"
          }
        },
        "stop": {
          "max": {
            "field": "eventStop.timestamp"
          }
        },
        "duration": {
          "bucket_script": {
            "buckets_path": {
              "start": "start.value",
              "stop": "stop.value"
            },
            "script": """
              return (params.stop - params.start)/1000;
            """
          }
        }
      }
    }
  }
}

# Response
{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 6,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "group_by" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "A",
          "doc_count" : 3,
          "stop" : {
            "value" : 1.593611695E12,
            "value_as_string" : "2020-07-01T13:54:55.000Z"
          },
          "start" : {
            "value" : 1.593611455E12,
            "value_as_string" : "2020-07-01T13:50:55.000Z"
          },
          "duration" : {
            "value" : 240.0
          }
        },
        {
          "key" : "B",
          "doc_count" : 2,
          "stop" : {
            "value" : 1.59361158E12,
            "value_as_string" : "2020-07-01T13:53:00.000Z"
          },
          "start" : {
            "value" : 1.593611545E12,
            "value_as_string" : "2020-07-01T13:52:25.000Z"
          },
          "duration" : {
            "value" : 35.0
          }
        },
        {
          "key" : "C",
          "doc_count" : 1,
          "stop" : {
            "value" : null
          },
          "start" : {
            "value" : 1.593611695E12,
            "value_as_string" : "2020-07-01T13:54:55.000Z"
          }
        }
      ]
    }
  }
}

While the structure of the result is different, the result is the same — some additional notes:

No need to retrieve the underlying documents, which "size": 0 does.
Inside a terms aggregation the other sub aggregations run.
How many status updates are involved is automatically counted in the doc_count so the value_count isn’t needed.
The bucket_script is the same.

Conclusion

Hopefully, this is a useful blueprint for transforms or equivalent aggregations. Happy calculations now that you know how all the pieces fit together and which traps to avoid (like starting the transform job).

The documentation also describes when (not) to use transforms, which leads to the classic “it depends” discussion:

When Should You Use Transforms?

If you have a large amount of data and the complete aggregation might be too slow or resource-intensive, especially when queried frequently or shown in a large dashboard.
If you want to clean up your data to save some disk space and only care about the entity-centric view and not every status update.
If working with the entity-centric documents is simpler — either through the Elasticsearch API or in Kibana. For example, calculating the average, mean, or 99th percentile of durations in Kibana is straightforward with the transform but a lot more challenging without.

When Should You Not Use Transforms?

If you require up-to-date results and the (configurable) frequency might be an issue.
If you are reading the data so infrequently, the transform job’s overhead isn’t worth it.

Can You Use Runtime Fields Instead?

Runtime fields solve a different problem and only add or override fields within a document. You could run an aggregation on top of runtime fields, but they alone don’t provide any cross-document features.

Range Visualization with Kibana: (Always) Start with Lens

Philipp Krenn — Wed, 23 Jun 2021 00:00:00 +0000

From a recent question on Twitter:

[How can you] create a visualization like the below? Based on access logs, I want to see percentages of requests answered within 1s, 2s, 5s and 10s, over time.

Just pointing me in the right direction would probably help. Should I use Vega, Lens, Visualize,… something else? Or am I thinking in the wrong way?

Excellent question and something pretty common, so here is a quick run-through from the initial approach to the final visualization.

Where Should You Start?

Kibana has accumulated many different visualization types over time. But when you want to create a new one, the menu is already trying to guide you, as shown by the screenshot (Kibana 7.13):

If you don’t know where to start, pick Lens since it’s versatile and beginner-friendly.
Unless you want a map, then go with Maps.
If you want to visualize a time series and need specific features like calculations or annotations that aren’t (yet) available in Lens, select TSVB , which stands for Time Series Visual Builder and is sometimes called Visual Builder for short.
If nothing else does the job and you need an escape hatch, pick Custom visualization , which lets you create Vega visualizations. Tons of options but also complexity — there might be 🐲 waiting for you on this one.
There is also the now mostly redundant Aggregation based visualizations though there should be very few scenarios where you’d have to fall back to those.
Text and Controls both have their place but are much less common and won’t be helpful for the graph above.

Diving into Kibana Lens

So how can you create a similar visualization like the one above? Metricbeat has a good dataset with event.duration in nanoseconds for this visualization:

Example:

{
  "_index": "metricbeat-7.13.1-2021.06.15-000001",
  "_type": "_doc",
  "_id": "cjxfEHoB_nPKlQ3D_niZ",
  "_version": 1,
  "fields": {
    "event.duration": [
      8723623
    ],
  ...
}

The first step is to drop the Records into the visualization area, which puts @timestamp on the horizontal axis and the Count of records on the vertical one. Then switch from Bar vertical stacked to Area percentage.

Break down by with Intervals on the event.duration field with Create custom ranges. Set the ranges from 0 to 1000000 (these are nanoseconds), 1000000 to 2000000,…

Optionally set the Visual options to Curve lines and on the Bottom axis disable Gridlines.

The current state should be pretty close to the desired visualization — done!

And if you want to see the creation step by step, here is a recording of it:

Conclusion

Once you know where to start, it is relatively straightforward: Lens unless you have a reason for something else. It’s still adding new features in every release and will cover more and more use-cases. So if in doubt, start exploring in Lens.

Final note because I did that wrong at first: Be sure to include Infinity in your last interval. Otherwise, your visualization might look different — somewhere between misleading and wrong.

How to Automate Elastic Cloud with Terraform

Philipp Krenn — Thu, 17 Dec 2020 00:00:00 +0000

While it’s nice to click around a UI for exploring, Infrastructure as Code is what you want for production; both for documentation and to (re-) create resources whenever needed because nobody can remember what they configured a couple of months ago.

Also, to minimize drift between environments, you want to have automation in place. Otherwise, production might have some additional surprises for you.

Elastic Cloud hasn’t had a stable public API until May. Since then, you can automate your clusters:

Elastic Cloud Control (ecctl) is the official command-line interface. It covers both the public Elastic Cloud service as well as Elastic Cloud Enterprise, though this post only focuses on the public service.
ecctl, in turn, depends heavily on Elastic’s cloud-sdk-go library, which provides a common ground for all of Elastic Cloud’s programmatic code in Go.

However, for managing infrastructure, Terraform providers are pretty much the standard by now. While ecctl and the SDK have a stable release (1.1.0 at the moment), the Terraform provider for Elastic Cloud has just been released as beta and is now also available on the Terraform Registry. Since it’s also building on top of cloud-sdk-go, it is coming along quite nicely. Though use it at your own risk.

Automate with Terraform

Before you start, you need to have Terraform 0.13+ installed. On macOS, run brew install terraform if you don’t have it already and check your setup with:

$ terraform version
Terraform v0.14.2

Create a file terraform.tf with the content:

terraform {
  required_providers {
    ec = {
      source = "elastic/ec"
      version = "0.1.0-beta"
    }
  }
}

resource "ec_deployment" "terraform-demo" {
  region                 = "eu-west-1"
  version                = "7.9.0"
  deployment_template_id = "aws-io-optimized-v2"

  elasticsearch {
    topology {
      size = "2g"
      zone_count = "1"
    }
  }

  kibana {
    topology {
      size = "1g"
      zone_count = "1"
    }
  }
}

Most of this should be self-explaining, like the memory size per instance or zone_count for availability zones. But region and deployment_template_id require more detail: In the documentation for “Available regions, deployment templates and instance configurations”, you get a list of the available regions on AWS, Azure, and GCP. Each of these regions contains the complete set of available templates and instances. The example above is a good starting point. Still, for more advanced setups in resource usage (compute- or memory-optimized, hot-warm architecture) or use-case (Enterprise Search, Observability, Security), you should use a more specialized setup.

Next up, initialize Terraform — only showing the relevant output:

$ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding elastic/ec versions matching "0.1.0-beta"...
- Installing elastic/ec v0.1.0-beta...
- Installed elastic/ec v0.1.0-beta (unauthenticated)
Terraform has been successfully initialized!

Then try to plan your configuration, which will fail:

$ terraform plan
Error: authwriter: 1 error occurred:
* one of apikey or username and password must be specified

Time to get your API key. Log into Elastic Cloud and head to the API keys page under Elasticsearch Service → Account → API keys to generate a key.

Now you could store the API key in the Terraform file, but this is a bad idea. Don’t share your secrets and don’t check them into source control — this is one of the most common reasons for hijacked accounts or ransomed data.

While you can set the environment variable EC_API_KEY directly, I’m a big fan of envchain. It allows you only to expose the environment variables needed for the current command and also makes working with different environments, accounts, or clients much saner. The following command creates an ec namespace and will ask you for the key:

$ envchain --set ec EC_API_KEY

Now back to the Terraform plan. If you have set EC_API_KEY without using envchain, remove envchain ec from this and all upcoming commands. There will be a fair bit of output, but only the last line with Plan: is really relevant:

$ envchain ec terraform plan
Plan: 1 to add, 0 to change, 0 to destroy

This is what it should look like. Next, apply the template and confirm with yes when asked:

$ envchain ec terraform apply

After a little while, your brand new cluster is up and running. You can log into Kibana with the “Open Kibana” button without needing a password thanks to Single Sign-On (since Elastic Stack 7.7).

Further down the page, you can see an overview of your cluster. Two things are sticking out here:

You can apply an upgrade.
Your single Elasticsearch instance doesn’t provide high availability.

To fix this, you need to apply two changes to your terraform.tf file:

Change to the latest version with version = "7.10.1".
Add a second availability zone in elasticsearch and topology with zone_count = "2".

But you must apply them one at a time. Otherwise, you’ll run into the error:

api error: clusters.topology_and_version_change.prohibited: You must perform a version upgrade separately from changes to the cluster topology (memory, number of zones, dedicated master nodes, etc). The following topology changes have been detected: zone_count changed (resources.elasticsearch[0])

So change one of the two settings, apply the change with envchain ec terraform apply and then do the same for the other setting. For reference, the final terraform.tf file.

After completion, your cluster is production-ready. Note that Elastic Cloud added a tiebreaker node automatically since Elasticsearch uses quorum-based decision making.

Access the Default Credentials

While Kibana is the right tool to dive into your data, you want to use Elasticsearch’s REST API for any application access or automation. Since Elastic Cloud enforces security, what are your credentials?

If you look back at the output of the initial terraform apply, you will see (amongst a lot more):

# ec_deployment.terraform-demo will be created
+ resource "ec_deployment" "terraform-demo" {
+ apm_secret_token = (sensitive value)
+ deployment_template_id = "aws-io-optimized-v2"
+ elasticsearch_password = (sensitive value)
+ elasticsearch_username = (known after apply)
+ id = (known after apply)
+ name = "terraform-demo"
+ region = "eu-west-1"
+ version = "7.9.0"

The Elasticsearch username is elastic, but the password is randomly generated and hidden. You can retrieve it with the following command where terraform-demo is the name of your cluster:

$ echo ec_deployment.terraform-demo.elasticsearch_password | terraform console

And you can fetch the Elasticsearch endpoint with:

$ echo ec_deployment.terraform-demo.elasticsearch[0].https_endpoint | terraform console

Use both outputs to create the following request; note the extra space in front of the command so your credentials won’t become part of your history:

$ curl -u elastic:<password> https://<cluster-name>.eu-west-1.aws.found.io:9243/
{
  "name" : "instance-0000000000",
  "cluster_name" : "22fe1...",
  "cluster_uuid" : "uWHPf...",
  "version" : {
    "number" : "7.10.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "1c34507e66d7db1211f66f3513706fdf548736aa",
    "build_date" : "2020-12-05T01:00:33.671820Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Of course, the default credentials should be the starting point to add more users with just the required set of privileges.

Limit Access to Your Cluster

Speaking of security, Elastic Cloud supports both PrivateLink (on AWS for now) and IP filtering. For extra security of our cluster, you can add an IP filter with Terraform.

First the terraform.tf file needs a new resource with the IP filter — replace 1.2.3.4 with your public IP address:

resource "ec_deployment_traffic_filter" "allow_my_ip" {
  name = "Allow my IP"
  region = "eu-west-1"
  type = "ip"
  rule {
    source = "1.2.3.4"
  }
}

And then reference the resource with traffic_filter = [ec_deployment_traffic_filter.allow_my_ip.id] next to the deployment_template_id. Or download the final terraform.tf file.

After applying the change with envchain ec terraform apply, the previous curl command works just like before. But if you change the IP address to something else and apply that update, the response changes — but this is an intended error:

$ curl -u elastic:<password> https://<cluster-name>.eu-west-1.aws.found.io:9243/
{"ok":false,"message":"Forbidden"}

Add a Dash of ecctl

If you want to use the command-line tool ecctl in addition to Terraform, you can use it as well. First, you need to install it:

$ brew tap elastic/tap
$ brew install elastic/tap/ecctl
$ ecctl version
Version: v1.1.0
Client API Version: 2.8.0-ms48
Go version: go1.15.2
Git commit: a2293c25
Built: Thu 03 Dec 23:21:49 2020
OS/Arch: darwin / amd64

Next, reuse the EC_API_KEY environment variable and list your deployment with ecctl:

$ envchain ec ecctl deployment list
ID NAME ELASTICSEARCH KIBANA APM ENTERPRISE_SEARCH APPSEARCH
9072a... terraform-demo 9072a..... bed04... - - -

To get more details about this deployment, use the ID from the previous command. The output will be quite long — showing you everything in the earlier screenshots as JSON and more:

$ envchain ec ecctl deployment show 32a82...

For more features, see the command reference documentation or take a look at the available commands and flags of ecctl. And with that, you should have all the necessary tools to automate your Elastic Cloud clusters.

Conclusion

One final step before you go. Remove the demo cluster to avoid unnecessary cost with the following command and confirm it with yes:

$ envchain ec terraform destroy
Destroy complete! Resources: 2 destroyed.

And just like that, everything is gone again. Use your powers with care!

That’s it for a quick introduction. Happy automation and for more information see the examples and documentation on GitHub.

PS: When talking about Infrastructure as Code, I have avoided the pets vs cattle topic quite intentionally. While you don’t want to raise datastores by hand 🐮, you will have to take more care of them than a stateless service 🐶.

Custom Favicons for Production and Order

Philipp Krenn — Fri, 31 Jul 2020 00:00:00 +0000

Favicons — the little pictures in your bookmarks and (way too many) open browser tabs — are great to see what is what quickly. Unless it all looks like this:

Those tabs are the Elastic Cloud admin UI, the Elasticsearch documentation, elastic.co, Kibana on Elastic Cloud, and Kibana on localhost.

Why Are All My Icons the Same?

From a company’s perspective, it makes sense to stick to a single consistent icon. Especially casual users might have a hard time remembering a set of icons for a single organization. The same goes for the product view — those are all pieces of the Elastic Stack. What started as a UI for Elasticsearch has evolved into more than just the classic visualizations, which are part of a bigger picture. Kibana has become the general window into the Elastic Stack with different solutions and the overall management.

But none of that helps you as a heavy user if you have multiple tabs open and no easy way to tell what was the documentation and what the production or development Kibana instances. It is a bit of a pain point, but you can fix it with a little hack.

Tab Modifier Plugin to the Rescue

The Tab Modifier plugin for Chrome and compatible browsers can replace favicons based on the URL fragment. While it hasn’t been updated in over two years, it’s doing its job.

After installing the plugin, you can define its rules. For example, the Elastic documentation is under the URL https://www.elastic.co/guide/, which you can target with the following rule:

The only thing missing now is the icon. For this scenario, you can download the Elastic brand assets, which contain both the company as well as the solution and product logos. Picking the intended logo as an SVG (Chrome supports that since version 80) from the download you can convert it to a data URI and paste the result into the icon field. That’s it.

You need to add three more rules to fix the example from the top:

The Elastic Cloud admin UI starts with https://cloud.elastic.co.
Any Kibana instance on Elastic Cloud contains .found.io:9243/.
Any local Kibana installation contains localhost:5601/ — supporting both HTTP and HTTPS but expecting the default Kibana port (so I won’t confuse it with Hugo on localhost:1313/).

Reloading each tab, you will now know what is what right away:

PS: This approach can also “hide” your Twitter, YouTube,… tabs at work, which seems to be one of the intended use-cases.

How Does Elasticsearch Store a float Value into an integer Field?

Philipp Krenn — Thu, 25 Jun 2020 00:00:00 +0000

Today there was a Discuss post on “Elasticsearch data type” that demonstrates one of the more confusing features in Elasticsearch. But if you are familiar with Elasticsearch it is an excellent puzzle — so follow along and test your knowledge.

First, add a document:

PUT users/_doc/1
{
  "user_id": 1
}

This index uses a dynamic mapping, which defaults to what data type for the user_id field?

Default Numeric Type

GET users/_mapping shows the answer:

{
  "users" : {
    "mappings" : {
      "properties" : {
        "user_id" : {
          "type" : "long"
        }
      }
    }
  }
}

So your user_id field is a long. Next, you try to add four more documents:

PUT users/_doc/2
{
  "user_id" : 2
}
PUT users/_doc/3
{
  "user_id" : "3"
}
PUT users/_doc/4
{
  "user_id" : 4.5
}
PUT users/_doc/5
{
  "user_id" : "5.1"
}

The Document with ID two is the same as our first one, so that will work. But what happens if you try to store a string, a float, or even a stringified float value into a long field?

Handling Dirty Data

It still works. But why?

By default, Elasticsearch will coerce data to clean it up. Quoting from its documentation:

Coercion attempts to clean up dirty values to fit the datatype of a field. For instance:

Strings will be coerced to numbers.

Floating points will be truncated for integer values.

Especially for quoted numbers, this makes sense, since some systems err on the side of quoting too much rather than too little. Perl used to be one of the well-known offenders there, and coercing would helpfully clean this up.

Sounds reasonable, but you want to verify this by retrieving the documents with GET users/_search and expect the user_id values 1 2 3 4 5, right? But you actually get the result — focus on the array in hits.hits:

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 5,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "users",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "user_id" : 1
        }
      },
      {
        "_index" : "users",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "user_id" : 2
        }
      },
      {
        "_index" : "users",
        "_type" : "_doc",
        "_id" : "3",
        "_score" : 1.0,
        "_source" : {
          "user_id" : "3"
        }
      },
      {
        "_index" : "users",
        "_type" : "_doc",
        "_id" : "4",
        "_score" : 1.0,
        "_source" : {
          "user_id" : 4.5
        }
      },
      {
        "_index" : "users",
        "_type" : "_doc",
        "_id" : "5",
        "_score" : 1.0,
        "_source" : {
          "user_id" : "5.1"
        }
      }
    ]
  }
}

Is this a bug? How could you store 4.5 in a long? If you recheck the mapping with GET users/_mapping, it’s still returning "type": "long".

`_source` Is Only an Illusion

The final piece in this puzzle is that Elasticsearch never changes the _source. But the stored field user_id is a long as you would expect. You can verify this by running an aggregation on the field:

GET users/_search
{
  "size": 0,
  "aggs": {
    "my_sum": {
      "sum": {
        "field": "user_id"
      }
    }
  }
}

Which gives the correct result for 1 + 2 + 3 + 4 + 5 = 15:

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 5,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : []
  },
  "aggregations" : {
    "my_sum" : {
      "value" : 15.0
    }
  }
}

Which, by the way, now defaults to a floating representation for value which you could change with the extra parameter "format": "0". That would add a "value_as_string" : "10" to the result. But let’s leave it at that.

Conclusion

I hope you are less confused than before or at least enjoyed the puzzle. As a parting note, be aware that coerce might be removed in the future since it is a trappy feature — especially around truncating floating-point numbers 😄.

meetup.com Members Are a Vanity Metric

Philipp Krenn — Sat, 28 Mar 2020 00:00:00 +0000

The most prominent number Meetup shows for groups are their members. Those are nice, big numbers, aren’t they?

For example the Elastic Vienna group has 1,050 members. I should have celebrated the 1,000^th or maybe the 2¹⁰ member. But what does that number really mean?

Elastic Vienna

You can get some statistics by adding stats/members/?range=all to a group’s URL; by default, you need to be a (co-) organizer to have access. This is what it looks like for the Elastic Vienna group:

I know this page is using the old layout, and they stopped generating the data by the end of 2019. Meetup is a very broken place — everybody should probably start thinking about alternatives, but that’s a different story. For the sake of argument, this data is good enough.

There is also Member Joins and Going statistics, but I think Total and Active Members give you the best overview:

You can see the development of members joining leading to the current total number.
My take is that Active Members are a more stable metric than Going, since those might be influenced by outside factors like competing meetups, soccer games,… — all the things you usually want to avoid when scheduling an event but it doesn’t always work out.

In total, we had 30 meetups in this group, and we strive for a quarterly cadence.

For comparison here are some other groups with different patterns:

ViennaDB

We had 47 meetups over the years, and it was my most active group until 2016, capturing some of the NoSQL hype and covering a broad set of topics around anything database related. Though the past years got quieter and quieter.

Papers We Love Vienna

This group had 46 events, and we are striving for a monthly cadence, usually taking a summer break, so around ten meetups per year. Also, this group is a little different from most others since it doesn’t have presentations. Instead, we discuss an academic paper that everybody has read before. We usually are between four and eight attendees with a few regulars and some people only showing up once for a specific paper.

SilverStripe Austria

This is what an inactive group looks like. We had seven meetups, but the last one was in 2014.

Vanity Metrics

So what does that number tell? Not too much since it’s a vanity metric:

The opposite of actionable metrics are vanity metrics (like web hits or number of downloads) which only serve to document the current state of the product but offer no insight into how we got here or what to do next. – Maurya, Ash. “3 Rules to Actionable Metrics in a Lean Startup”

But it’s a “big number” that is only growing. So if that’s what you’re after, it’s pretty much perfect.

The better number and the one that Meetup should be showing instead though would be Active Members because that tells you much more about the current state of a group. If I want to speak at a meetup, I care about the current state and far less about its past buildup.

Conclusion

While the total members are the more impressive number, the more meaningful metric would be active members. Wouldn’t it be great if Meetup would switch to that one or at least show both?

What Do You Do as a Developer Advocate at Elastic?

Philipp Krenn — Tue, 25 Feb 2020 00:00:00 +0000

And by a developer advocate I mostly mean myself, since all of us have slightly different approaches based on their background and region. This includes having advocates in China, Japan, South Korea, or France for more than four years to foster local communities and plant the seeds that you can maybe harvest later, which I find a lovely metaphor for our job.

My personal goal is successful users. That is mostly around education (what is possible, how have best practices evolved, what are common problems, and how can they be solved) and helping them solve their problems. Only when people are successful with your product, they’ll spread the word and potentially become customers. That is a natural extension of their success and not something I would or should be pushing. Otherwise, your perception will switch to sales very quickly, and that isn’t a win for the position.

Generally, I’m good at keeping myself entertained. While we have shared goals and visions, a lot is up to each one of us. This can be both great and terrifying. Depending on your mindset. it gives you all the freedom to pick up relevant and interesting opportunities as you go along, or you’re constantly questioning if you’re doing the right thing. I’m definitely in the great camp — also because:

It’s easier to ask forgiveness than it is to get permission. – https://en.wikiquote.org/wiki/Grace_Hopper

Events

Probably my main activity is speaking at conferences and meetups (no paid slots though) as well as creating talks and demos for them. And sharing materials with colleagues — either slides or providing demo instances, for example, with Strigo. In the past years, I have consistently spoken at 100+ events and traveled more than 200 days, but I’m generally sticking to the EMEA region — from Ireland over the Nordics to Russia and Isreal plus down to South Africa. On top of the content side, this involves a lot of travel arrangements and coordination.

Speaking also includes demos at interested companies. Those are often called lunch & learn or brown bag lunch (BBL), which are particularly common in France where my colleague David is our master of BBLs or Aravind in India.

In Vienna, I’m running the Elastic Vienna, ViennaDB, and Papers We Love Vienna meetup groups. Not all of them are strictly work-related, but they do fit the general ecosystem and to broaden views as well as contacts.

On top of that, I do a couple of Elastic trainings per year. And we’re getting more into webinars and virtual meetups now as well, both to scale and because of the current health situation.

Solving Problems

If you’re asking a (technical) question on the @elastic Twitter account there is a high chance that I’ll be answering. Hopefully — with — a — good — solution — or — two — (or more).

I’m also helping out on our official Discuss forum and Slack group as well as on StackOverflow. And while I’ll never answer more questions on Discuss than my colleagues Alexander, David, or Mark, I’m making up for it on our internal #technical-questions Slack channel. Because that is a great multiplier to help more folks from us to help the community. And this is where you learn and try out things yourself — it is like the fast track to the problems from many other people.

Open Source

Open Source is where Elastic started, and as our founder and CEO Shay keeps saying:

Open source is a distribution model that allows us to build community. It’s a force multiplier.

My three focus areas there are:

Google Summer of Code and I’m very happy that after 2018 we’re one of the mentoring organizations in 2020 again.
An area we’re currently building out is sponsoring of other projects we’re using. Like cURL and quite a few others — more to come.
Bringing back feedback from the open source community to our engineering, product, and sales teams. It is a delicate balance, but someone has to speak for the open source side inside a company as well.

Internal

Working with the community isn’t a one-way street. The point isn’t only to spread the message but also to bring feedback back into your organization. Besides GitHub issues and Slack messages, we also have a very active mailing list for sightings. This is our place of sharing any mention of Elastic products (good and bad) or what competitors are up to. It is great to learn about the field, what is working, what is broken, or about common issues.

If Apple Mail can be trusted, I have recently sent my 1️⃣0️⃣0️⃣0️⃣^th message to that mailing list.

Other Tasks

Especially my Asian colleagues like Jongmin, Martin, or Xiaoguo spend a lot of time on localized content. And it helps a lot in many regions. But I’m less convinced about it in Europe and the German speaking area in specific. That is why it is less of a priority for me:

Recreating existing content in German feels like a rather dull task for me — it already exists, and I would instead explore additional topics. And since our documentation and code will always be in English, there is no way around it anyway once you want to dive a little deeper.
You’ll either end up with a wild mix of Denglisch (Deutsch + English) or create hilarious texts trying to find German words for things like observability, Jaeger intake, ingestion, Operator,… that I find almost impossible to read myself.

There are more, but most of them are infrequent or quick to do on the side:

File GitHub issues or pull requests as things come up in discussions, especially around documentation.
Review content for community members either for an event or a written article.
Give internal feedback on upcoming features, blog posts, or announcements.
Help out at an Elastic{ON} Tour stop.

Less Relevant

Something I find less relevant in that discussion is titles and departments :

Titles are generally abused. Some prefer Community Engineer, Community Advocate, Technical Evangelist,… but I would see it as a personal preference rather than a clear distinction in the end.
The only thing that is constant with a fast-growing company like Elastic is change. I have been all over the organizational chart, and it has never actually changed my job (yet). When I was hired we were a DevRel team directly under the CTO. When our lead Shaunak wanted to move back to development, we were split up, and each advocate joined a different engineering team — mine was Infrastructure since that was my background where Drew had to adopt me. After that we were joined together again in a Community team in Product under Tyler; until Product was folded into Engineering, and we were a Community team there. Only to be then moved to Marketing first lead by Mark now by Kelley. I’m not yet sure where we might be going next.

What counts in the end is that it’s a technical role spending most of the time for and with the community. No title will fix that if your role is to sell directly — regardless of where you are in the organizational structure.

Conclusion

Community is a wide field. Finding the right mix of things to do to keep yourself motivated and to add value — both for the company and your community — is up to each one of us.

PS: If you’re looking for a speaker for your next conference, meetup, or lunch & learn, don’t hesitate to contact me.

Combining date and date_nanos in Elasticsearch and Kibana

Philipp Krenn — Mon, 26 Aug 2019 00:00:00 +0000

The Elasticsearch 7.0 release added support for nanosecond timestamps. So besides the date datatype there’s now also date_nanos. One of the more confusing things is when you combine the two datatypes, which this post is exploring.

Mapping of `date_nanos`

Before diving into combining datatypes, there’s one other cause of confusion: date_nanos is not automatically picked up by dynamic mapping — when you don’t provide a mapping and Elasticsearch tries to guess the datatype of each field based on the value of the first document you index.

For illustration, create two new indices with dynamic mapping and retrieve them:

PUT timestamp-millis/_doc/1
{
  "timestamp": "2019-01-01T12:10:30.124Z"
}
PUT timestamp-nanos/_doc/1
{
  "timestamp": "2019-01-01T12:10:30.124456789Z"
}
GET timestamp-*/_mapping

Both have the same mapping, which is probably not what you intended:

{
  "timestamp-nanos" : {
    "mappings" : {
      "properties" : {
        "timestamp" : {
          "type" : "date"
        }
      }
    }
  },
  "timestamp-millis" : {
    "mappings" : {
      "properties" : {
        "timestamp" : {
          "type" : "date"
        }
      }
    }
  }
}

Combining `date` and `date_nanos` in Elasticsearch

Now to the combining of datatypes. Delete the existing indices and start over with the explicit mapping:

DELETE timestamp-*
PUT timestamp-millis
{
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date"
      }
    }
  }
}
PUT timestamp-nanos
{
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date_nanos"
      }
    }
  }
}

Also, add four sample documents close to each other:

PUT timestamp-millis/_doc/1
{
  "timestamp": "2019-01-01T12:10:30.123Z"
}
PUT timestamp-millis/_doc/2
{
  "timestamp": "2019-01-01T12:10:30.124Z"
}
PUT timestamp-nanos/_doc/3
{
  "timestamp": "2019-01-01T12:10:30.123456789Z"
}
PUT timestamp-nanos/_doc/4
{
  "timestamp": "2019-01-01T12:10:30.123498765Z"
}

How does this behave when you search over both indices and want to sort on the field timestamp?

GET timestamp-*/_search
{
  "sort": "timestamp"
}

Probably not the way you would expect (only including the relevant fields of hits):

{
  "_index" : "timestamp-millis",
  "_id" : "1",
  "_source" : {
    "timestamp" : "2019-01-01T12:10:30.123Z"
  },
  "sort" : [1546344630123]
},
{
  "_index" : "timestamp-millis",
  "_id" : "2",
  "_source" : {
    "timestamp" : "2019-01-01T12:10:30.124Z"
  },
  "sort" : [1546344630124]
},
{
  "_index" : "timestamp-nanos",
  "_id" : "3",
  "_source" : {
    "timestamp" : "2019-01-01T12:10:30.123456789Z"
  },
  "sort" : [1546344630123456789]
},
{
  "_index" : "timestamp-nanos",
  "_id" : "4",
  "_source" : {
    "timestamp" : "2019-01-01T12:10:30.123498765Z"
  },
  "sort" : [1546344630123498765]
}

Or actually, if you look at the sort field and read the documentation, it does make sense:

[date_nanos] are still stored as a long representing nanoseconds since the epoch.

But this is still not what you want. Luckily the feature was added in 7.2: “Add date and date_nanos conversion to the numeric_type sort option.”

So the query you want to use is:

GET timestamp-*/_search
{
  "sort": {
    "timestamp": {
      "numeric_type": "date_nanos"
    }
  }
}

And it works by casting all the timestamp fields to nanosecond precision:

{
  "_index" : "timestamp-millis",
  "_id" : "1",
  "_source" : {
    "timestamp" : "2019-01-01T12:10:30.123Z"
  },
  "sort" : [1546344630123000000]
},
{
  "_index" : "timestamp-nanos",
  "_id" : "3",
  "_source" : {
    "timestamp" : "2019-01-01T12:10:30.123456789Z"
  },
  "sort" : [1546344630123456789]
},
{
  "_index" : "timestamp-nanos",
  "_id" : "4",
  "_source" : {
    "timestamp" : "2019-01-01T12:10:30.123498765Z"
  },
  "sort" : [1546344630123498765]
},
{
  "_index" : "timestamp-millis",
  "_id" : "2",
  "_source" : {
    "timestamp" : "2019-01-01T12:10:30.124Z"
  },
  "sort" : [1546344630124000000]
}

Does This Work for Kibana?

Kibana 7.3 added general date_nanos support though there are some limitations, which are described in the Github issue for that feature:

We can’t fully support date_nanos in Kibana, due to the technical limitations of JavaScript. JavaScript stores all numeric values in a 64bit floating point number. Thus we only can represent integer (non decimal numbers) to a precision of 52 bit (the mantisse in that floating point number).

To try it out with mixed datatypes, you need to create the index pattern first and pick the field timestamp for the Time Filter field name on the following screen:

And it almost works. Displaying works — this shows a 10ms time window — but the sort order is not correct:

When you edit the index pattern and change the Format to Date Nanos:

Then Discover displays the nanosecond precision of the timestamp, but it still does not fix the sorting issue:

Why is sorting not working correctly? Because the underlying query is not using "numeric_type": "date_nanos". You can check that through the Inspect button to see both the request and the response:

I have raised an issue in Kibana for a fix.

Update in 7.5

The release of Kibana 7.5.0 fixed the remaining issue. You don’t have to change the Format of the field anymore, but it just works now:

Conclusion

Once you are avoiding traps like dynamic mapping and watch out for "numeric_type": "date_nanos", combining date and date_nanos for sorting work in Elasticsearch and ~~almost in~~ Kibana.

PS: Thanks for the Stack Overflow question that led me down this rabbit hole.

DEV Community: Philipp Krenn

Elasticsearch: 15 years of indexing it all, finding what matters

GET _cat/stats

A search (hi)story

The future of search is bright

Serverless

ES|QL

AI everywhere

Thanks to all of you

Elasticsearch Transforms: Calculate the Total Duration from Multiple Status Updates

Sample Data

Transforms API

Transforms with Bucket Script Aggregation

Aggregation without Transforms

Conclusion

Range Visualization with Kibana: (Always) Start with Lens

Where Should You Start?

Diving into Kibana Lens

Conclusion

How to Automate Elastic Cloud with Terraform

Automate with Terraform

Access the Default Credentials

Limit Access to Your Cluster

Add a Dash of ecctl

Conclusion

Custom Favicons for Production and Order

Why Are All My Icons the Same?

Tab Modifier Plugin to the Rescue

How Does Elasticsearch Store a float Value into an integer Field?

Default Numeric Type

Handling Dirty Data

_source Is Only an Illusion

Conclusion

meetup.com Members Are a Vanity Metric

Elastic Vienna

ViennaDB

Papers We Love Vienna

SilverStripe Austria

Vanity Metrics

Conclusion

What Do You Do as a Developer Advocate at Elastic?

Events

Solving Problems

Open Source

Internal

Other Tasks

Less Relevant

Conclusion

Combining date and date_nanos in Elasticsearch and Kibana

Mapping of date_nanos

Combining date and date_nanos in Elasticsearch

Does This Work for Kibana?

Update in 7.5

Conclusion

`GET _cat/stats`

`_source` Is Only an Illusion

Mapping of `date_nanos`

Combining `date` and `date_nanos` in Elasticsearch