DEV Community: Elastic

What is Context Engineering?

Carly Richmond — Tue, 09 Sep 2025 14:22:24 +0000

With the fast-paces evolving nature of AI, new terms and techniques appear all the time. One of the latest discussions is around context engineering. If you are not sure what context engineering is, why it's important, or what techniques you can use to optimize the context your agentic systems use, read on to find out.

What is context engineering?

Context engineering refers to a collection of practices that can be combined to provide the right information to Large Language Models (or LLMs) to help them accomplish the desired task. It's important to ensure that the LLMs we use in agents and MCP tools have the right information sources to ensure they provide accurate results, and don't hallucinate or fail to give the desired answer. My high school maths teacher always talked about the notion of "rubbish in, rubbish out" in terms of the inputs we provided to our calculations and proofs.

The same goes for LLMs. We can't expect LLMs to provide the answers and automations we need accurately without providing them with the right information. As you can see from the above example leveraging ChatGPT, it can only pick up information on which it was trained, or that is provided in the context via the components discussed in subsequent sections.

Components

The below visualization showcases the key components of context that we can use to improve the responses of LLMs invoked by AI agents and called by MCP tools:

As Dexter Horthy outlines as his 3rd principal of 12-Factor Agents, it's important to own your context to ensure LLMs generate the best outputs possible.

RAG

RAG is an architectural pattern where data sourced from an information retrieval system, such as Elasticsearch, is provided to an LLM to ground and enhance the results that they generate. We've covered RAG in many Elasticsearch Labs blogs including this one which provides an overview of the construct, and this tutorial which covers building a single RAG chatbot with Python, Langchain, React and Elasticsearch.

Although some suggest that the ever expanding context window size of newer LLMs mean that "RAG is dead", in fact may find their LLM suffers from context confusion, as covered by Drew Breunig. Context confusion refers to the issue where surplus information provided to the LLM leads to a sub-optimal response. RAG helps direct LLMS to the desired result as it addresses common limitations of general LLMs, including:

Lack of specific domain knowledge for jargon-heavy disciplines such as financial services or engineering.
Newer information or events that have happened after the model has been trained.
Hallucinations, where the LLM generates incorrect answers.

RAG typically involves pulling relevant documents from a data store and passing via the prompt, or through dedicated AI tools invoked by an LLM. A simple example from the AI SDK Travel Planner leveraging the Elasticsearch JavaScript client is given below:

import { tool as createTool } from 'ai';
import { z } from 'zod';

import { Client } from '@elastic/elasticsearch';
import { SearchResponseBody } from '@elastic/elasticsearch/lib/api/types';

import { Flight } from '../model/flight.model';

const index: string = "upcoming-flight-data";
const client: Client = new Client({
  node: process.env.ELASTIC_ENDPOINT,
  auth: {
    apiKey: process.env.ELASTIC_API_KEY || "",
  },
});

function extractFlights(response: SearchResponseBody<Flight>): (Flight | undefined)[] {
    return response.hits.hits.map(hit => { return hit._source})
}

export const flightTool = createTool({
  description:
    "Get flight information for a given destination from Elasticsearch, both outbound and return journeys",
  parameters: z.object({
    destination: z.string().describe("The destination we are flying to"),
    origin: z
      .string()
      .describe(
        "The origin we are flying from (defaults to London if not specified)"
      ),
  }),
  execute: async function ({ destination, origin }) {
    try {
      const responses = await client.msearch({
        searches: [
          { index: index },
          {
            query: {
              bool: {
                must: [
                  {
                    match: {
                      origin: origin,
                    },
                  },
                  {
                    match: {
                      destination: destination,
                    },
                  },
                ],
              },
            },
          },

          // Return leg
          { index: index },
          {
            query: {
              bool: {
                must: [
                  {
                    match: {
                      origin: destination,
                    },
                  },
                  {
                    match: {
                      destination: origin,
                    },
                  },
                ],
              },
            },
          },
        ],
      });

      if (responses.responses.length < 2) {
        throw new Error("Unable to obtain flight data");
      }

      return {
        outbound: extractFlights(responses.responses[0] as SearchResponseBody<Flight>),
        inbound: extractFlights(responses.responses[1] as SearchResponseBody<Flight>)
      };
    } catch (e) {
      console.error(e);
      return {
        message: "Unable to obtain flight information",
        location: location,
      };
    }
  },
});

Retrieving relevant information from sources such as Elasticsearch or others, and even utilising techniques such as LLM summarization or data aggregation as my colleague Alex achieved when building an MCP data to summarize and query his health data, can ensure that the LLM has the precise data it needs to provide the answer. This context can then be passed using emerging protocols such as Model Context Protocol (MCP) or Agent2Agent Protocol (known as A2A).

Prompts

Perhaps considered a more established practice, but still very much a subset of context engineering, prompt engineering refers to the practice of refining and crafting effective inputs (or prompts) to an LLM to produce the result we want. Although commonly structured as simple text, prompts can consist of other media sources such as images and sounds. Sander Schulhoff et al. in their survey of prompt engineering define the following components of a prompt:

Directive: the instruction or question serving as the main intent of the request.
Exemplars: demonstrable examples to guide the LLM to accomplish the task.
Output Formatting: the format the output information is expected to be returned, such as JSON or unstructured text. This is important as depending on the source of data the LLM may need to translate (for example from the structured JSON of an Elasticsearch query response to another format compared to returning the result directly).
Style instructions: guidance on how to alter the structure of the output. This is considered a specific type of output formatting.
Role: the persona the LLM needs to emulate to achieve the task (for example a travel agent).
Additional information: other useful details needed to complete the task, including context from other sources.

Specifically, the below example showcases each of these elements for a prompt for a travel planning agent:

All of these elements can be tweaked and evaluated to ensure the optimal result is obtained by the LLM. In addition to these elements, there are numerous techniques that can be used to structure and optimize prompts to gain the answer you need. For example, Wei et al. in their 2023 paper found that standard zero-shot prompts where we ask an LLM a simple structured question fair less effectively compared to chain-of-thought prompting techniques for arithmetic and reasoning tasks. The differences are summarized in the example below:

When considering the format of the prompt to provide you need to consider several factors, including:

The type of task (for example simple recall or translation compared to complex arithmetic reasoning).
Task complexity and ambiguity. Ambiguous requests may lead to unpredictable results.
The inputs you are providing as context, along with the format.
The output required.
The capabilities of your chosen LLM.
The persona you would like the LLM to emulate.

Memory

Much like humans, AI applications rely on both short and long-term memory to recall information. Within context engineering:

Short-term memory, often referred to as state or chat history, refers to the messages exchanged in the current conversation between the user and the model. This includes the initial and follow-up questions presented by the user.
Long-term memory, simply referred to as memory, refers to information shared across conversations. Key examples would be relevant common information or recent prior conversations.

Taking our Travel Planner Agent example, the short-term memory would include the travel dates and location, along with any follow-up messages if the user changes their mind and wants to explore another destination. The long term memory in this place could contain profile information about the user's travel preferences, along with past trips that could be used to inform the suggestions of what activities to include in a new itinerary (such as wine tasting opportunities for those who have taken part in those activities on prior vacations).

Most AI frameworks provide the ability to manage both chat history and memory as it's important to ensure the history is managed to ensure it fits within the context window alongside the other elements of context. Taking LangGraph as an example, short term memory is managed as part of agent state using a checkpointer, meanwhile long term memory is persisted to long term stores:

As we build multi-agent architectures, we need to also be mindful about the segregation of memory and context. When splitting tasks among sub-agents in a larger flow, each agent may need knowledge of the other agent's results to remain in sync. However, over time these additions result in a context window overflow:

It is important that the context stored in both types of memory to ensure relevant and up to date context is provided to LLMs. Failure to do so can result in context poisoning. This can be carried out through malicious intent, as we see in prompt injection and data poisoning attacks as per the OWASP LLM Application Top 10. But it can also occur via innocent reasons such as the build up of history that can distract the model, or even contradictory information that results in clashes. In the Gemini 2.5 report, researchers found that a Pokémon-playing Gemini agent showed a tendency to repeat actions from its history instead of forming novel approaches, meaning the growing context became more of a hindrance to solving the problem. For these reasons practices such as chat history trimming, summarization and relevant retrieved information should be managed.

Structured Outputs

As we move to complex AI agent architectures, there is a need to ensure that the outputs emitted by LLMs adhere to a schema or contract that makes it easier to parse and integrate with other systems and workflows.

We are all used to freeform text results, but these formats can be difficult to integrate into dependent systems and agents. Much like designing a set of REST endpoints not just adhering to best practices such as the OpenAPI standard but to a contract compatible with other components, we need to specify the output format and schema that we expect the LLM to return. The below example represents specifying a schema to generate an object adhering to a particular schema using AI SDK:

import { generateObject } from 'ai';
import { z } from 'zod';

const { object } = await generateObject({
  model: 'openai/gpt-4.1',
  schemaName: 'Travel Itinerary',
  schemaDescription: 'Sample travel itinerary for a trip',
  schema: z.object({
    title: z.string(),
    location: z.string(),
    hotel: z.object({name: z.string(), roomType: z.string(), amount: z.number(), checkin: z.iso.date(), checkout: z.iso.date()}),
    flights: z.array(z.object({carrier: z.string(), flightNo: z.string().max(8), origin: z.string(), destination: z.string(), date: z.iso.datetime()})),
    excursions: z.array(z.object({ name: z.string(), amount: z.string(), date: z.iso.datetime()}))
  }),
  prompt: 'Generate a travel itinerary based on the specified location',
});

The introduction of JSON structured output for LLM outputs makes sense. It's common to need to balance processing structured and unstructured data, just as Elasticsearch does internally. For this reason there is emerging support in some models for generating outputs adhering to a provided JSON schema, including through the Structured Outputs feature available in the OpenAI platform. When combined with function calling, this allows us to define standard contracts for the passing of information between tools. However, given LLMs can generate JSON with syntax issues it's important to handle potential errors gracefully when processing results.

Available Tools

The final element of context that we can use within context engineering is the tools that we provide to LLMs to provide data. Tools allow us to perform actions such as automating operations such as booking the trip defined by our itinerary, retrieving data using RAG as discussed previously, or providing information from other sources of information.

We have shown an example of a RAG tool above with our flightTool, but tools can be used to pull in other sources of information, for example the below weather tool built with AI SDK:

import { tool as createTool } from 'ai';
import { z } from 'zod';

import { WeatherResponse } from '../model/weather.model';

export const weatherTool = createTool({
  description: 
  'Display the weather for a holiday location',
  parameters: z.object({
    location: z.string().describe('The location to get the weather for')
  }),
  execute: async function ({ location }) {
    // While a historical forecast may be better, this example gets the next 3 days
    const url = `https://api.weatherapi.com/v1/forecast.json?q=${location}&days=3&key=${process.env.WEATHER_API_KEY}`;

    try {
      const response = await fetch(url);
      const weather : WeatherResponse = await response.json();
      return { 
        location: location, 
        condition: weather.current.condition.text, 
        condition_image: weather.current.condition.icon,
        temperature: Math.round(weather.current.temp_c),
        feels_like_temperature: Math.round(weather.current.feelslike_c),
        humidity: weather.current.humidity
      };
    } catch(e) {
      console.error(e);
      return { 
        message: 'Unable to obtain weather information', 
        location: location
      };
    }
  }
});

Irrespective of the framework used, a tool comprises of:

A description of what the tool does to inform the LLM.
The parameters expected by the function, along with defined data types. Here we define these using the Typescript validation library zod.
The function to be invoked by the LLM when the tool is used.

If a LLM supports tool calling, it can choose to call (potentially) one or many tools to solve the problem. I have discussed my experiences of model choice before while building my own multi-tool AI agent. It's important when choosing a model to investigate the level of tool calling support using resources such as the Hugging Face Open LLM Leaderboard or Berkeley Function-Calling Leaderboard. The problem is that given the LLM decides which tools are relevant to the objective, it is possible that can be confused by tool many tools and call irrelevant tools as discussed by Drew Breunig. This idea of tool confusion is also discussed in the 2024 paper by Paramanayakam et al. where they found the performance of Llama 3.1 8b improved when provided with less tools (19 compared with 46).

Optimizing the number of tools available is an open area of research. Experiments to apply RAG architectures to combat tool confusion, such as retrieving tool descriptions to optimize tool selection in MCP so that providing relevant tool descriptions to the LLM results in more accurate results.

Conclusion

Here we have explained what context engineering is, and gave an overview of the key components of context. If you are interested in learning more check out the below resources.

Resources

Elasticsearch: 15 years of indexing it all, finding what matters

Philipp Krenn — Fri, 05 Sep 2025 10:09:13 +0000

Elasticsearch just turned 15-years-old. It all started back in February 2010 with the announcement blog post (featuring the iconic “You Know, for Search” tagline), first public commit, and the first release, which happened to be 0.4.0.

Let’s take a look back at the last 15 years of indexing and searching, and turn to the next 15 years of relevance.

`GET _cat/stats`

Since its launch, Elasticsearch has been downloaded an average of 3 times per second, totaling over 1.45 billion downloads.

The GitHub stats are equally impressive: More than 83,000 commits from 2,400 unique authors, 38,000 issues, 25,000 forks, and 71,500 stars. And there is no sign of slowing down.

All of this is on top of countless Apache Lucene contributions. We’ll get into those for the 25 year anniversary of Lucene, which is also this year. In the meantime, you can check out the 20 year anniversary page to celebrate one of the top Apache projects.

A search (hi)story

There are too many highlights to list them all, but here are 15 releases and features from the last 15 years that got Elasticsearch to where it is today:

Elasticsearch, the company (2012): The open source project became an open source company, setting the stage for its growth.
ELK Stack (2013): Elasticsearch joined forces with Logstash and Kibana to form the ELK Stack, which is now synonymous with logging and analytics.
Version 1 (2014): The first stable release introduced key features like snapshot/restore, aggregations, circuit breakers, and the _cat API.
Shield and Found (2015): Shield brought security to Elasticsearch clusters in the form of a (paid) plugin. And the acquisition of found.no brought Elasticsearch to the cloud, evolving into what is now Elastic Cloud. As an anecdote, nobody could find Found — SEO can be hard for some keywords.
Version 2 (2015): Introduced pipelined aggregations, security hardening with the Java Security Manager, and performance and resilience improvements.
Version 5 and the Elastic Stack (2016): Skipping two major versions to unify the version numbers of the ELK Stack and turning it into the Elastic Stack after adding Beats. This version also introduced ingest nodes and the scripting language Painless.
Version 6 (2017): Brought zero-downtime upgrades, index sorting, and the removal of types to simplify data modeling.
**Version 7** (2019): Changed the cluster coordination to the more scalable and resilient Zen2, single-shard default settings, built-in JDK, and adaptive replica selection.
Free security (2019): With the 6.8 and 7.1 releases, core security became free to help everyone secure their cluster.
ILM, data tiers, and searchable snapshots (2020): Made time-series data more manageable and cost-effective with Index Lifecycle Management (ILM), tiered storage, and searchable snapshots.
Version 8 (2022): Introduced native dense vector search with HNSW and enabled security by default.
ELSER (2023): Launched Elastic Learned Sparse EncodeR model, bringing sparse vector search for better semantic relevance.
Open source again (2024): Added AGPL as a licensing option to bring back open source Elasticsearch.
Start Local (2024): Made it easier than ever to run Elasticsearch and Kibana: curl -fsSL https://elastic.co/start-local | sh
LogsDB (2024): A new specialized index mode that reduces log storage by up to 65%.

The future of search is bright

Thanks to the rise of AI capabilities, search is more relevant and interesting than ever. So what is next for Elasticsearch? There’s way too much to name, so we’ll stick to three areas and the challenges they address.

Serverless

No shards, nodes, or versions. Elasticsearch Serverless takes care of the operational issues you might have experienced in the past:

15 years in, and someone is still setting number_of_shards: 100 for no reason.
15 years, and we’re still debating refresh_interval: 1s vs 30s like it’s a life-or-death decision.
15 years of major versions, minor heart attacks, and the thrill of migrating to the latest version.

You can try out Elasticsearch Serverless today.

ES|QL

“Cheers to 15 years of Elasticsearch — where the Query DSL is still the most complex part of your day.” But it doesn’t have to be. The new Elasticsearch Piped Query Language (ES|QL) brings a much simpler syntax and a significant investment into a new compute engine with performance in mind. While we’re building out more features, you can already use ES|QL today. Don’t worry; the Query DSL will understand.

AI everywhere

15 years of query tuning, and we’re still just throwing boost: 10 at the problem.
15 years of making your logs searchable while you still have no idea what’s happening in production.
Still the best at finding that one log line… if you remember how you indexed it.

AI is redefining what’s possible — from turning raw logs into actionable insights with the AI Assistant for observability and security, to more relevant search with semantic understanding and intelligent re-ranking.

This is only the beginning. More AI-powered features are on the horizon — bringing smarter search, enhanced observability, and stronger security. The future of Elasticsearch isn’t just about finding data; it’s about understanding it. Stay tuned — the best is yet to come.

Thanks to all of you

Thanks to all contributors, users, and customers over the last 15 years to make Elasticsearch what it is today. We couldn’t have done it without you and are grateful for every query you send to Elasticsearch.

Here’s to the next 15 years. Enjoy!

NLP and Elastic: Getting started

Priscilla Parodi — Thu, 02 Jun 2022 21:13:34 +0000

| Menu | Next Post: NLP HandsOn |

Natural language processing (NLP) is the branch of artificial intelligence (AI) that focuses on understanding human language as closely as possible to human interpretation, combining computational linguistics with statistical, machine learning and deep learning models.

Some examples of NLP tasks:

Named entity recognition is a type of information extraction, identifying words or phrases as entities.

(model used)

Sentiment analysis is a type of text classification, attempting to extract subjective emotions from text.

(model used)

There are more examples that can be used according to your use case.

BERT

In 2018, Google sourced a new technique for pre-training NLP called BERT.

BERT uses “transfer learning”, which is the method of pre-training linguistic representations. Pre-training refers to how BERT was first trained using unsupervised learning on a large source of plain text extracted from a collection of samples (800 million words) and Wikipedia documents (2,500 million words). Earlier models required manual labeling.

BERT was pretrained on two tasks: language modeling (15% of tokens were masked and BERT was trained to predict them from context) and next sentence prediction (BERT was trained to predict if a chosen next sentence was probable or not given the first sentence). With this understanding, BERT can be adapted to many other types of NLP tasks very easily.

Knowing the intent and context and not just the keywords, it is possible to go further in understanding in a way that is even closer to the way humans understand.

NLP with Elastic

To support models that use the same tokenizer as BERT, Elastic is supporting the PyTorch library, one of the most popular machine learning libraries that supports neural networks like the Transformer architecture that BERT uses, enabling NLP tasks.

In general, any trained model that has a supported architecture is deployable in Elasticsearch, including BERT and variants.

These models are listed by NLP task. Currently, these are the tasks supported:

Extract information

Named entity recognition
Fill-mask
Question answering

Classify text

Language identification
Text classification
Zero-shot text classification

Search and compare text

Text embedding
Text similarity

As in the cases of classification and regression, when a trained model is imported you can use it to make predictions (inference).

Note: For NLP tasks you must choose and deploy a third-party NLP model. If you choose to perform language identification, as an option we have a trained model lang_ident_model_1 provided in the cluster.

NLP with Elastic Solutions

There are many possible use cases to add NLP capabilities to your Elastic project and here are some examples:

Security

Spam detection: Text classification capabilities are useful for scanning emails for language that often indicates spam, allowing content to be blocked or deleted and preventing malware emails.

PUT spam-detection/_doc/1
{
  "email subject": "Camera - You are awarded a SiPix Digital Camera! Call 09061221066. Delivery within 28 days.",
  "is_spam": true
}

Enterprise Search

Analysis of unstructured text: Entity recognition is useful for structuring text data, adding new field types to your documents and allowing you to analyze more data and obtain even more valuable insights.

PUT /source-index
{
  "mappings": {
    "properties": {
      "input":    { "type": "text" }
    }
  }
}

PUT /new-index
{
  "mappings": {
    "properties": {
      "input":    { "type": "text" },  
      "organization":  { "type": "keyword"  }, 
      "location":   { "type": "keyword"  }     
    }
  }
}

Observability

Service request and incident data: Extracting meaning from operational data, including ticket resolution comments, allows you to not only generate alerts during incidents, but also go further by observing your application, predicting behavior, and having more data to improve ticket resolution time.

...
  "_source": {
    "support_ticket_id": 119237,
    "customer_id": 283823,
    "timestamp": "2021-06-06T17:23:02.770Z",
    "text_field": "Response to the case was fast and problem was solved after first response, did not need to provide any additional info.",
    "ml": {
      "inference": {
        "predicted_value": "positive",
        "prediction_probability": 0.9499962712516151,
        "model_id": "heBERT_sentiment_analysis"
      }
    }
  }
...

NLP HandsOn

Now, let's proceed with an end-to-end example! To prepare for the NLP HandsOn, we will need an Elasticsearch cluster running at least version 8.0 with an ML node. If you haven't created your Elastic Cloud Trial yet, now is the time.

| Menu | Next Post: NLP HandsOn |

This post is part of a series that covers Artificial Intelligence with a focus on Elastic's (Creators of Elasticsearch, Kibana, Logstash and Beats) Machine Learning solution, aiming to introduce and exemplify the possibilities and options available, in addition to addressing the context and usability.

NLP HandsOn

Priscilla Parodi — Thu, 02 Jun 2022 21:13:24 +0000

| Menu |

Note: This HandsOn assumes that you have already followed the step-by-step Setup of your Elastic Cloud Trial account, and also that you have read the blog NLP and Elastic: Getting started.
Config: To prepare for the NLP HandsOn, we will need an Elasticsearch cluster running at least version 8.0 with an ML node.

To start using NLP in your Stack you will need to import your model. The first thing we need to do is upload your model into a cluster.

In our eland library, a Python Elasticsearch client for exploring and analyzing data in Elasticsearch, we have some simple methods and scripts that allow you to upload models from local disk, or to pull models down from the Hugging Face model hub.

Once models are uploaded into the cluster, you’ll be able to allocate those models to specific ML nodes. Once model allocation is complete, we’re ready for inference.

Eland can be installed from PyPI via pip.

Before you go any further, make sure you have Python installed.

You can check this by running:

Unix/macOS
python3 --version

You should get some output like:
Python 3.8.8

Additionally, you’ll need to make sure you have pip available.

You can check this by running:

Unix/macOS
python3 -m pip --version

You should get some output like:
pip 21.0.1 from …

If you installed Python from source, with an installer from python.org, or via Homebrew you should already have pip.

If you don't have Python and pip installed, install it first.

With that, Eland can be installed from PyPI via pip:

$ python3 -m pip install eland

Getting started

To interact with your cluster through the API, we will need to use your Elasticsearch cluster endpoint information.

The endpoint looks like:
https://<user>:<password>@<hostname>:<port>

Open your deployment settings to find your endpoint information and click on the gear icon.

Copy your Elasticsearch endpoint as in the image below.

Note: If you want to try out examples with your own cluster, remember to include your endpoint URLs and authentication details.

Now add the username and password so your request can be authenticated, your endpoint will look like this:

https://elastic:123456789@00c1f8.es.uscentral1.gcp.cloud.es.io:9243

username: elastic is a built-in superuser. Grants full access to cluster management and data indices.

password: If you don't have your password, you will need to reset it and generate a new password.

Copy your endpoint, you'll need it later.

In parallel, let's proceed locating the first model to be imported.

We will import the model from Hugging Face, an AI community to build, train and deploy open source machine learning models.

In this demo we will use a random sentiment analysis model but feel free to import the model you want to use. You can read more details about this model on the Hugging Face webpage.

Copy the model name as in the image below.

Now that we have all the necessary information (elasticsearch cluster endpoint information and the name of the model we want to import) let's proceed by importing the model:

Open your terminal and update the following command with your endpoint and model name:

eland_import_hub_model --url https://<user>:<password>@<hostname>:<port> \
--hub-model-id <model_name> \
--task-type <task_type>

In this case we are importing the bhadresh-savani/distilbert-base-uncased-emotion model to run the text_classification task.

In Huggning Face filters you will be able to see the task of each model. Supported values are fill_mask, ner, question_answering, text_classification, text_embedding, and zero_shot_classification.

eland_import_hub_model --url https://elastic:<password>@<hostname>:<port> \
--hub-model-id bhadresh-savani/distilbert-base-uncased-emotion \
--task-type text_classification

You will see that the Hugging Face model will be loaded directly from the model hub and then your model will be imported into Elasticsearch.

Wait for the process to end.

Let's check if the model was imported.

Click Machine Learning in your Kibana menu.

Under model management click Trained Models:

Your model needs to be on this list as shown in the image below, if it is not on this list check if there was any error message in the previous process.

If your model is on this list it means it was imported but now you need to start the deployment. To do this, in the last column under Actions click Start deployment.

After deploying, the State column will have the value started and under Actions the Start deployment option will be disabled, which means that the deploy has been done.

Let's test our model!

Copy your model ID:

In Kibana's menu, click Dev Tools.

In this UI you will have a console to interact with the REST API of Elasticsearch.

We will to use the inference processor to evaluate this model.

POST _ml/trained_models/<model_id>/deployment/_infer
{
  "docs": { "text_field": "<input>"}
}

This POST method contains a docs array with a field matching your configured trained model input, typically the field name is text_field. The text_field value is the input you want to infer.

In our case it will be:

POST _ml/trained_models/bhadresh-savani__distilbert-base-uncased-emotion/deployment/_infer
{
  "docs": { "text_field": "Elastic is the perfect platform for knowledgebase NLP applications"}
}

Where the model_id is bhadresh-savani__distilbert-base-uncased-emotion and the value that I am using as a test is Elastic is the perfect platform for knowledgebase NLP applications.

Clicking the play button you can send the request:

In this case the predicted sentiment is "joy".

That's it, the model is working. 🚀

Note: You can run more tests to determine if this model works for what you need.

To get all the statistics of your model you can use the _stats request:

GET _ml/trained_models/<model_id>/_stats

Let's continue with part 2, How to run this model on data being ingested?

To do this, let's start by importing a .csv file into Elasticsearch. So we can run the model while importing data.

I think it's interesting to run an analysis on random texts and tweets are good use cases.

Recently Elon Musk announced his interest in buying Twitter, but before that he was famously active on the platform. As we have a sentiment analysis model, let's proceed with analyzing a sample of Elon's tweets.

I found this database on Kaggle, this is a good website for locating datasets.

Note: We don't have a huge amount of data, 172Kb between November 16, 2012 and September 29, 2017. But as this is not a research paper this is not a problem.

Feel free to use whatever data you prefer, or even the twitter API.

Let's download this file:

And import into Elasticsearch.

There are different ways to do this, but since this is a small .csv file, we can use the Upload a file integration.

In the Kibana menu, click Integrations, you will see a list of integrations we have for collecting data.

Search for Upload a file as in the image below:

And then click Select or drag and drop a file and choose your csv file, in our case data_elonmusk.csv that you downloaded earlier.

You will see something similar to the image below:

Click Override settings to rename the Tweet column to text_field. As explained before, there needs to be a field that matches your configured trained model input which is typically called text_field. With this, the model will be able to identify the field to be analyzed.

Rename the Tweet column/field to text_field. Click Apply.

After the page loads, click Import.

And then click Advanced to edit the import process settings.

The import process has several steps:

Processing file - Turning the data into NDJSON documents so they can be ingested using the bulk api

Creating index - Creating the index using the settings and mappings objects

Creating ingest pipeline - Creating the ingest pipeline using the ingest pipeline object

Uploading data - Loading data into the new Elasticsearch index

Creating a data view (Index pattern) - Create a Kibana index pattern (if the user has opted to)

As you can see the CSV processor is being used in the ingest pipeline to import your document.

Feel free to edit the mapping or ingest pipeline.

In our case we need to edit the ingest pipeline to add our previously trained and imported model.

Add the model that will infer the data being ingested into the processor as in the image below:

  {
       "inference": {
       "model_id": "bhadresh-savani__distilbert-base-uncased-emotion"
        }
    }

After that add your index name and click Import. If for some reason it doesn't work, repeat the process and check if you typed something incorrectly.

Note: What we are doing is adding your model for inference in the ingest pipeline, it doesn't need to be a .csv. Read more about it here.

When it finishes loading, your screen will look like mine, click View index in Discover.

If you didn't disable Create data view when you were importing data you should be able to locate your index by the name you used. Now you can explore your index data.

Next to the word Documents, click Field statistics, so far this is a beta feature but excellent for exploring your data. As we can see, Elon was feeling Joyful in 70% of the analyzed tweets considering this sentiment analysis model. The second most popular sentiment in Elon's tweets was anger and then fear.

Let's click on the lens button on the right side of the screen to open Kibana Lens and explore this data.

When the screen loads. Click and drag the Time field to explore this data considering the date of each tweet.

Considering time, some suggestions will appear, I liked one of them, but instead of every 30 days I edited it for an annual review. Also try filtering only by prediction probability between 0.90 and 1 for better accuracy. Here you can have fun with the analysis you want to run.

Apparently anger has increased over time, but joy remains the most common in Elon's Tweets. Fear increased until the beginning of 2016 but decreased in 2017.

Well, there are several interpretations for data, we always need to take into account the model used, accuracy, the quality of our data, the information we seek, the type of analysis and our interpretation, context and knowledge, but I believe that now it is possible to see how useful it is to analyze language.

For example, try running a classification model with the inference data (which is now a new field) to predict sentiment in addition to checking for influencers. Also try importing other models and using other datasets.

I also imported a NER model to identify entities in the same dataset so we can start to correlate text topics (keywords) with sentiment. The year Elon talked about Tesla the most in this dataset was 2015, which coincides with the year with the greatest increase in joy.

This news is from 2015 and Elon was really positive about Tesla even with the company reporting losses.

Again, these are not necessarily facts. But my goal is to show a little bit of what we can do with NLP analysis and correlation (which does not imply causation 😅).

Let's proceed with the last part, How to run this model on an existing index?

If your data is already indexed and you want to infer your model considering this data but without changing the index content, this is possible. If this is your case, let's proceed with this test.

In the Kibana menu click Ingest Pipeline and then Create pipeline and New pipeline.

Give your pipeline a name and click Add a processor.

The first step is to rename the field that will be inferred to text_field.

For that add the Rename processor, in the message field add the field to be renamed and in the target field add text_field. And then click Add.

Now we will add the Inference processor, for that click again Add processor and then under Model ID add your Model ID, in our case: bhadresh-savani__distilbert-base-uncased-emotion

Click Add.

Click Create pipeline and copy the name of your pipeline, you will need it later.

Now open Dev Tools and run the following request (adding your source index, dest index and pipeline name):

POST _reindex
{
  "source": {
    "index": "<your-source-index-name>"
  },
  "dest": {
    "index": "<your-ml-dest-index-name>",
    "pipeline": "<your-pipeline-name>"
  }
}

This copies documents from a source to a destination. You can copy all documents to the destination index, or reindex a subset of the documents, you can also use source filtering to reindex a subset of the fields in the original documents.

This will take some time, wait for the successful response as in the image below:

For this new index you don't have the Data View yet, you need it to access the Elasticsearch data that you want to explore, to do that click Stack Management in the Kibana menu and then click Data Views.

Click Create new data view and then for the Name field add the name of your new index, in my case it is elon-output-ml. Click Create data view.

Now open Discover and select the new index.

That's it, without making changes to your current index you have a new index with the result of this model.

I hope you enjoy using NLP with the Elastic Stack! Feedback is always welcome.

This post is part of a series that covers Artificial Intelligence with a focus on Elastic's (Creators of Elasticsearch) Machine Learning solution, aiming to introduce and exemplify the possibilities and options available, in addition to addressing the context and usability.

Troubleshooting beginner level Elasticsearch errors

Lisa Jung — Fri, 20 Aug 2021 14:22:25 +0000

Throughout my blog posts, we have learned about CRUD operations, fine tuning the relevance of your search, queries, aggregations, and mapping.

As you continue your journey with Elasticsearch, you will inevitably encounter some common errors associated with the topics we have covered in the blogs.

Learn how to troubleshoot these pesky errors so you can get unstuck!

Prerequisite work

Watch this video from time stamp 14:28-21:46. This video will show you how to complete steps 1 and 2.

Set up Elasticsearch and Kibana*
Add the news headline dataset and e-commerce dataset to Elasticsearch*
Open the Kibana console(AKA Dev Tools)
Keep two windows open side by side(this blog and the Kibana console)

We will be sending requests from Kibana to Elasticsearch to learn how to troubleshoot errors!

Notes
1) If you would rather download Elasticsearch and Kibana on your own machine, follow the steps outlined in Downloading Elasticsearch and Kibana(macOS/Linux and Windows).

2) The video will show you how to add the news headlines dataset.

Add the news headlines data first then follow the same steps to add the e-commerce dataset.

Create an index called ecommerce_original_data and add the e-commerce data to that index.

Additional Resources

Interested in beginner friendly workshops on Elasticsearch and Kibana? Check out my Beginner's Crash Course to Elastic Stack series!

1) Part 6 Workshop Recording
This blog is a complementary blog to Part 6 of the Beginner's Crash Course to Elastic Stack. If you prefer learning by watching videos instead, check out the recording!

2) Beginner's Crash Course to Elastic Stack Table of Contents
This table of contents includes repos from all workshops in the series. Each repo includes resources shared during the workshop including the video recording, presentation slides, related blogs, Elasticsearch requests and more!

Want To Troubleshoot Your Errors? Follow The Clues!

Whenever you perform an action with Elasticsearch and Kibana, Elasticsearch responds with an HTTP status(red box) and a response body(blue box).

The request below asks Elasticsearch to index a document and assign it an id of 1.

The HTTP status of 201-success(red box) indicates that the document has been successfully created.

The response body(blue box) indicates that a document with an assigned id of 1 has been created in the beginners_crash_course index.

As we work with Elasticsearch, we will inevitably encounter error messages like the one below.

When this happens, the HTTP status and the response body will provide valuable clues about why the request failed!

When you are a beginner, it is hard to even understand what the error message is saying. Often it is hard to know where to even start.

I find that just the act of seeing different types of error messages and learning how to troubleshoot makes troubleshooting process less intimidating.

To keep things simple, we are going to limit our scope. We will focus on errors you might see while working with requests covered in previous blogs.

We are going to learn how to interpret these error messages and how we should approach troubleshooting these errors.

Common Errors

Here are some common errors that you may encounter as you work with Elasticsearch.

Unable to connect

The cluster may be down or it may be a network issue. Check the network status and cluster health to identify the problem.

Connection unexpectedly closed

The node may have died or it may be a network issue. Retry your request.

5XX Errors

Errors with an HTTP status starting with 5 stems from internal server error in Elasticsearch. When you see this error, take a look at the Elasticsearch log and identify the problem.

4XX Errors

Errors with an HTTP status starting with 4 stems from client errors. When you see this error, correct the request before retrying.

As beginners, we are still familiarizing ourselves with the rules and syntax required to communicate with Elasticsearch. Majority of the error messages we encounter are likely to have been caused by the mistakes we make while writing our requests(4XX errors). When we see this error, we need to fix the request before trying it again.

To strengthen our understanding of the requests we have learned throughout the previous blogs, we will only focus on 4XX errors during this blog.

Thought Process For Troubleshooting Errors

Whenever we see an error message, it is really helpful to go through the thought process for troubleshooting errors.

Follow the thought process to narrow down the cause and find the resources to help you fix the error.

What number does the HTTP status start with(4XX? 5XX?)
What does the response say? Always read the full message!
Use the Elasticsearch documentation as your guide. Compare your request with the example from the documentation. Identify the mistake and make appropriate changes.

At times, you will encounter error messages that are not very helpful. We will go over a couple of these and see how we can troubleshoot these types of errors.

Trip Down Memory Lane

In my previous blogs, we learned how to send requests related to the following topics:

We will revisit each topic and troubleshoot common errors you may encounter as you explore each topic.

Errors Associated With CRUD Operations

Error 1: 404 No such index[x]

In the Beginner's guide to performing CRUD operations with Elasticsearch and Kibana, we learned how to perform CRUD operations.

Let's say we have sent the following request to retrieve a document with an id of 1 from the common_errors index.

Request sent:

GET common_errors/_doc/1

Copy and paste the request into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch returns a 404-error(red box) along with the cause of the error(blue lines) in the response body.

The HTTP status starts with a 4, meaning that there was a client error with the request sent.

When you look at the response body, Elasticsearch lists the reason(line 6) as "no such index [common_errors]".

The two possible explanations for this error are:

The index common_errors truly does not exist or was deleted
We do not have the correct index name

Cause of Error 1

In our example, the cause of the error is quite clear! We have not created an index called common_errors and we were trying to retrieve a document from an index that does not exist.

Let's create an index called common_errors.

Syntax:

PUT Name-of-the-Index

The following example asks Elasticsearch to create a new index called common_errors.

Example:

PUT common_errors

Copy and paste the example into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch returns a 200-success HTTP status(red box) acknowledging that the index common_errors has been successfully created(blue box).

Error 2: 405 Incorrect HTTP method for uri, allowed: [x]

Now that we have created the index common_errors, let's index a document!

Suppose you have remembered that you could use the HTTP verb PUT to index a document and send the following request:

PUT common_errors/_doc
{
  "source_of_error": "Using the wrong syntax for PUT or POST indexing request"
}

Copy and paste the request into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch returns a 405-error(red box) along with the cause of the error(blue lines) in the response body.

This HTTP status starts with a 4, meaning that there was a client error with the request sent.

If you look at the response, Elasticsearch lists the reason as "Incorrect HTTP method for uri... and method: [PUT], allowed:[POST]".

Cause of Error 2

This error message suggests that we used the wrong HTTP verb to index this document.

You can use either PUT or POST HTTP verb to index a document. Each HTTP verb serves a different purpose and requires a different syntax.

We learned about the difference between the two verbs in the Beginner's guide to performing CRUD operations with Elasticsearch and Kibana under the Index a document section.

The following are excerpts from the blog.

When indexing a document, HTTP verb PUT or POST can be used.

The HTTP verb PUT is used when you want to assign a specific id to your document

Syntax:

PUT name-of-the-Index/_doc/id-you-want-to-assign-to-this-document
{
  field_name: "value"
}

Let's compare the syntax to the request we just sent:

PUT common_errors/_doc
{
  "source_of_error": "Using the wrong syntax for PUT or POST indexing request"
}

You will see that our request uses the HTTP verb PUT. However, it does not include the document id we want to assign to this document.

If you add the id of the document to the request as seen below, you will see that the request is carried out without a hitch!

Correct example for PUT indexing request:

PUT common_errors/_doc/1
{
  "source_of_error": "Using the wrong syntax for PUT or POST indexing request"
}

Copy and paste the correct example into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch returns a 201-success HTTP status(red box) acknowledging that document 1 has been successfully created.

The HTTP verb POST is used when you want Elasticsearch to autogenerate an id for the document.

If this is the option you wanted, you could fix the error message by replacing the verb PUT with POST and not including the document id after the document endpoint(_doc).

Syntax:

POST Name-of-the-Index/_doc
{
  field_name: "value"
}

Correct example for POST indexing request:

POST common_errors/_doc
{
  "source_of_error": "Using the wrong syntax for PUT or POST indexing request"
}

Copy and paste the correct example into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch returns a 201-success HTTP status(red box) and autogenerates an id(line 4) for the document that was indexed.

Error 3: 400 Unexpected Character: was expecting a comma to separate Object entries at [Source: ...] line: x

Suppose you wanted to update document 1 by adding the fields "error" and "solution" as seen in the example.

Example:

POST common_errors/_update/1
{
  "doc": {
    "error": "405 Method Not Allowed"
    "solution": "Look up the syntax of PUT and POST indexing requests and use the correct syntax."
  }
}

Expected response from Elasticsearch:
Elasticsearch returns a 400-error(red box) along with the cause of the error in the response body.

This HTTP error starts with a 4, meaning that there was a client error with the request sent.

Cause of Error 3

If you look at the response, Elasticsearch lists the error type(line 12) as "json_parse_exception" and the reason(line 13) as "...was expecting comma to separate Object entries at ... line: 4]".

In Elasticsearch, if you have multiple fields("errors" and "solution") in an object("doc"), you must separate each field with a comma. The error message tells us that we need to add a comma between the fields "error" and "solution".

Add the comma as shown below and send the following request:

POST common_errors/_update/1
{
  "doc": {
    "error": "405 Method Not Allowed",
    "solution": "Look up the syntax of PUT and POST indexing requests and use the correct syntax."
  }
}

Expected response from Elasticsearch:
You will see that the document with an id of 1 has been successfully updated.

Errors Associated With Sending Queries

In blogs fine tuning the relevance of your search and queries, we learned how to send queries about news headlines in our index.

As a prerequisite part of these workshops, we added the news headlines dataset to an index we named news_headlines.

Afterwards, we sent various queries to retrieve documents that match the specific criteria.

Let's go over common errors you may encounter while working with these queries.

Error 4: 400 [x] query does not support [y]

Suppose you want to use the range query to pull up news headlines published within a specific date range.

You have sent the following request:

GET news_headlines/_search
{
  "query": {
    "range": {
      "date": 
        "gte": "2015-06-20",
        "lte": "2015-09-22"
    }
  }
}

Expected response from Elasticsearch:
Elasticsearch returns a 400-error(red box) along with the cause of the error in the response body.

This HTTP status starts with a 4, meaning that there was a client error with the request sent.

If you look at the response, Elasticsearch lists the error type(line 5) as "parsing_exception" and the reason(line 6) as "[range] query does not support [date]".

This error message is misleading as the range query should be able to retrieve documents that contain terms within a provided range. It should not matter that you have requested to run a range query against the field "date".

Let's check out the screenshots from the Elastic documentation on the range query to see what is going on.

Pay attention to the syntax of the range query line by line.

Screenshot from the documentation:

Compare this syntax to the request we have sent earlier:

GET news_headlines/_search
{
  "query": {
    "range": {
      "date": 
        "gte": "2015-06-20",
        "lte": "2015-09-22"
    }
  }
}

Cause of Error 4

The culprit of this error is the range query syntax!

Our request is missing curly brackets around the inner fields("gte" and "lte") of the field "date".

Let's add the curly brackets as shown below and send the following request:

GET news_headlines/_search
{
  "query": {
    "range": {
      "date": {
        "gte": "2015-06-20",
        "lte": "2015-09-22"
      }
    }
  }
}

Expected response from Elasticsearch:
Elasticsearch returns a 200-success HTTP status(red box) and retrieves news headlines that were published between the specified date range.

Error 5: 400 Unexpected character...: was expecting double-quote to start field name.

In the queries blog, we learned about the multi_match query. This query allows you to search for the same search terms in multiple fields at one time.

Suppose you wanted to search for the phrase "party planning" in the fields "headline" and "short_description" as shown below:

GET news_headlines/_search
{
  "query": {
    "multi_match": {
      "query": "party planning",
      "fields": [
        "headline",
        "short_description"
      ]
    },
    "type": "phrase"
  }
}

Copy and paste the request into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch returns a 400-error(red box) along with the cause of the error in the response body.

This HTTP status starts with a 4, meaning that something is not quite right with the request sent.

If you look at the response, Elasticsearch lists the error type(line 5) as "parsing_exception" and the reason(line 6) as "[multi_match] malformed query, expected [END_OBJECT] but found [FIELD_NAME]".

Cause of Error 5

This is a vague error message that does not really tell you what went wrong.

However, we do know that the error is coming from somewhere around line 10, which suggests that the error may have something to do with the "type" parameter(line 11).

When you check the opening and closing brackets from the outside in, you will realize that the "type" parameter is placed outside of the multi_match query.

Move the "type" parameter up a line and move the comma from line 10 to line 9 as shown below and send the request:

GET news_headlines/_search
{
  "query": {
    "multi_match": {
      "query": "party planning",
      "fields": [
        "headline",
        "short_description"
      ],
      "type": "phrase"
    }
  }
}

Expected response from Elasticsearch:
Elastcsearch returns a 200-success(red box) response.

All hits contain the phrase "party planning" in either the field "headline" or "short description" or both!

Error 6: 400 parsing_exception

When we search for something, we often ask a multi-faceted question. For example, you may want to retrieve entertainment news headlines published on "2018-04-12".

This question actually requires sending multiple queries in one request:

A query that retrieves documents from the "ENTERTAINMENT" category
A query that retrieves documents that were published on "2018-04-12"

Let's say you are most familiar with the match query so you write the following request:

GET news_headlines/_search
{
  "query": {
    "match": {
      "category": "ENTERTAINMENT",
      "date":"2018-04-12"
    }
  }
}

Copy and paste the request into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch returns a 400-error(red box) along with the cause of the error in the response body.

This HTTP status starts with a 4, meaning that something is off with the query syntax.

If you look at the response, Elasticsearch lists the error type(line 5) as "parsing_exception" and the reason(line 6) as "[match] query doesn't support multiple fields, found [category] and [date]".

Cause of Error 6

Elasticsearch throws an error because a match query can query documents from only one field. Our request tried to query multiple fields using only one match query.

Bool Query
In the queries blog, we learned how to combine multiple queries into one request by using the bool query.

With the bool query, you can combine multiple queries into one request and you can further specify boolean clauses to narrow down your search results.

This query offers four clauses that you can choose from:

must
must_not
should
filter

You can mix and match any of these clauses to get the relevant search results you want.

In our use case, we have two queries:

A query that retrieves documents from the "ENTERTAINMENT" category
A query that retrieves documents that were published on "2018-04-12"

The news headlines we want could be filtered into a yes or no category:

Is the news headline from the "ENTERTAINMENT" category? yes or no
Was the news headline published on "2018-04-12"? yes or no

When documents could be filtered into either a yes or no category, we can use the filter clause and include two match queries within it:

GET news_headlines/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "match": {
            "category": "ENTERTAINMENT"
          }
        },
        {
          "match": {
            "date": "2018-04-12"
          }
        }
      ]
    }
  }
}

Copy and paste the request into the Kibana console and send the request.

Expected response from Elastcsearch:
Elasticsearch returns a 200-success HTTP status(red box) and shows the top 10 hits whose "category" field contains the value "ENTERTAINMENT" and the "date" field contains the value of "2018-04-12".

Errors Associated With Aggregations and Mapping

Suppose you want to get the summary of categories that exist in our dataset.

Since this requires summarizing your data, you decide to send the following aggregations request:

GET news_headlines/_search
{
  "aggs": {
    "by_category": {
      "terms": {
        "field": "category"
      }
    }
  }
}

Copy and paste the request into the Kibana console and send the request.

Expected response from Elasticsearch:
By default, Elasticsearch returns both the top 10 search hits and aggregations results. Notice that the top 10 search hits take up lines 16-168.

Error 7: 400 Aggregation definition for [x], expected a [y]

Let's say you are only interested in aggregations results.

You remember that you can add a "size" parameter and set it equal to 0 to avoid fetching the hits.

You send the following request to accomplish this task:

GET news_headlines/_search
{
  "aggs": {
    "size": 0,
    "by_category": {
      "terms": {
        "field": "category"
      }
    }
  }
}

Copy and paste the request into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch returns a 400-error(red box) along with the cause of the error in the response body.

This HTTP status starts with a 4, meaning that there was a client error with the request sent.

If you look at the response, Elasticsearch lists the error type(line 5) as "parsing_exception" and the reason(line 6) as "Aggregation definition for [size starts with a [VALUE_NUMBER], expected a [START_OBJECT]".

Something is off with our aggregations request syntax.

Let's take a look at the screenshots from the Elastic documentation on aggregations and see what we missed.

Screenshot from the documentation:
Pay close attention to the syntax of the aggregations request.

Cause of Error 7

This error is occurring because the "size" parameter was placed in a spot where Elasticsearch is expecting the name of the aggregations("my-agg-name").

If you scroll down to the Return only aggregation results section in the documentation, you will see that the "size" parameter is placed outside of the aggregations request as shown below.

Screenshot from the documentation:

Place the "size" parameter outside of the aggregations request and set it equal to 0 as shown below.

Send the following request:

GET news_headlines/_search
{
  "size": 0,
  "aggs": {
    "by_category": {
      "terms": {
        "field": "category"
      }
    }
  }
}

Expected response from Elasticsearch:
As intended, Elasticsearch does not retrieve the top 10 hits(line 16).

You can see the aggregations results(an array of categories) without having to scroll through the hits.

Error 8: 400 Field [x] of type [y] is not supported for z type of aggregation

The next two errors(error 8 & 9) are related to the requests we have learned in aggregations and mapping blogs.

In these blogs, we have worked with the e-commerce dataset.

In the aggregations blog, we have added the e-commerce dataset to Elasticsearch and named the index ecommerce_original_data.

Then, we had to follow additional steps in Set up data within Elasticsearch section in the aggregations blog.

Screenshot from the aggregations blog:
To set up data within Elasticsearch, we have implemented the following steps.

We never covered why we had to go through these steps. It was all because of the the error message we are about to see next!

From this point on, imagine that you had just added the e-commerce dataset into the ecommerce_original_data index. We have not completed steps 1 and 2.

In the aggregations blog, we learned how to group data into buckets based on time interval.

This type of aggregations request is called the date_histogram aggregation.

Suppose we wanted to group our data into 8 hour buckets so we have sent the request below:

GET ecommerce_original_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_by_8_hrs": {
      "date_histogram": {
        "field": "InvoiceDate",
        "fixed_interval": "8h"
      }
    }
  }
}

Copy and paste the request into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch returns a 400-error(red box) along with the cause of the error in the response body.

This HTTP status starts with a 4, meaning that there was a client error with the request sent.

If you look at the response, Elasticsearch lists the error type(line 5) as "illegal_argument_exception" and the reason(line 6) as "Field [InvoiceDate] of type [keyword] is not supported for aggregation [date_histogram]".

This error is different from the syntax error messages we have gone over thus far.

It says that the field type "keyword" is not supported for the date_histogram aggregation, which suggests that this error may have something to do with the mapping.

Let's check the mapping of the ecommerce_original_data index:

GET ecommerce_original_data/_mapping

Copy and paste the request into the Kibana console and send the request.

Expected response from Elasticsearch:
You will see that the field "InvoiceDate" is typed as "keyword"(blue box).

Cause of Error 8

Let's take a look at the screenshots from the Elastic documentation on date_histogram aggregation.

Screenshot from the documentation:

The first sentence gives us a valuable clue on why this error occurred!

The date_histogram aggregation cannot be performed on a field typed as "keyword".

To perform a date_histogram aggregation on the "InvoiceDate" field, the "InvoiceDate" field must be mapped as field type "date".

But the mapping for the field "date" already exists. What are we going to do?!

Remember, you cannot change the mapping of the existing field!

The only way you can accomplish this is to:
Step 1: Create a new index with the desired mapping
Step 2: Reindex data from the original index to the new one
Step 3: Send the date_histogram aggregation request to the new index

In the aggregations blog, this is why we carried out steps 1 and 2.

Let's go over step 1!

Step 1: Create a new index(ecommerce_data) with the following mapping

The following request creates a new index called ecommerce_data with the desired mapping. Notice that the field "InvoiceDate" is typed as "date" and the "format" of the date has been specified here as well.

PUT ecommerce_data
{
  "mappings": {
    "properties": {
      "Country": {
        "type": "keyword"
      },
      "CustomerID": {
        "type": "long"
      },
      "Description": {
        "type": "text"
      },
      "InvoiceDate": {
        "type": "date",
        "format": "M/d/yyyy H:m"
      },
      "InvoiceNo": {
        "type": "keyword"
      },
      "Quantity": {
        "type": "long"
      },
      "StockCode": {
        "type": "keyword"
      },
      "UnitPrice": {
        "type": "double"
      }
    }
  }
}

Copy and paste the request into the Kibana console and send the request.

Expected response from Elasticsearch:
You will get a 200-success HTTP status(red box) acknowledging that the index ecommerce_data with the desired mapping has been successfully created.

Let's check the mapping of the ecommerce_data index by sending the following request:

GET ecommerce_data/_mapping

Expected response from Elasticsearch:
Elasticsearch retrieves the mapping of the ecommerce_data index where the field "InvoiceDate" is typed as "date" and the date "format" is specified as "M/d/yyyy H:m".

Why did we have to specify the "format" of the field "InvoiceDate"?
It has to do with the format of the field "InvoiceDate" in our e-commerce dataset.

We have not put any data into the ecommerce_data index. However, we still have e-commerce data in the ecommerce_original_data index.

Let's send the following request to retrieve documents from this index so we can see the date format of the field "InvoiceDate":

GET ecommerce_original_data/_search

Expected response from Elasticsearch:

The format of the field "InvoiceDate" is "M/d/yyyy H:m".

By default, Elasticsearch is configured to recognize iso8601 date format(ex. 2021-07-16T17:12:56.123Z).

If the date format in your dataset differs from the iso8601 format, Elasticsearch will not recognize it and throw an error.

In order to prevent this from happening, we specify the date format of the "InvoiceDate" field("format": "M/d/yyyy H:m") within the mapping.

The symbols used in the date format was formed using this documentation.

We have covered a LOT! Let's do a recap on why we are carrying out these steps in the first place.

In the aggregations blog, we added the e-commerce dataset to the ecommerce_original_data index where the field "InvoiceDate" was dynamically typed as "keyword".

When we tried to run a date_histogram aggregation on the field "InvoiceDate", Elasticsearch threw an error saying that it can only perform the date_histogram aggregation on a field typed as "date".

Since we could not change the mapping of an existing field "InvoiceDate", we had to carry out step 1 where we created a new index called ecommerce_data with the desired mapping for the field "InvoiceDate".

Step 2: Reindex the data from original index(source) to the one you just created(destination)
At this point, we have a new index called ecommerce_data with the desired mapping. However, there is no data in this index.

To correct that, we will send the following request to reindex the data from the ecommerce_original_data index to the ecommerce_data index:

POST _reindex
{
  "source": {
    "index": "ecommerce_original_data"
  },
  "dest": {
    "index": "ecommerce_data"
  }
}

Copy and paste the request into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch successfully reindexes the e-commerce dataset from the ecommerce_original_data index to the ecommerce_data index.

Step 3: Send the date_histogram aggregation request to the new index(ecommerce_data)
Now that the data has been reindexed to the new index, let’s send the date_histogram aggregation request we sent earlier.

The following is almost identical to the original request except that the index name has been changed to the new index(ecommerce_data).

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_by_8_hrs": {
      "date_histogram": {
        "field": "InvoiceDate",
        "fixed_interval": "8h"
      }
    }
  }
}

Expected response from Elasticsearch:
Elasticsearch returns a 200-success(red box) response. It divides the dataset into 8 hour buckets and returns them in the response.

Let's go over the last error!

Error 9: 400 Found two aggregation type definitions in [x]: y and z

One of the cool things about Elasticsearch is that you can build any combination of aggregations to answer more complex questions.

For example, let's say we want to get the daily revenue and the number of unique customers per day.

This requires grouping data into daily buckets.

Within each bucket, we calculate the daily revenue and the number of unique customers per day.

Let's say we wrote the following request to accomplish this task:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_per_day": {
      "date_histogram": {
        "field": "InvoiceDate",
        "calendar_interval": "day"
      },
      "daily_revenue": {
        "sum": {
          "script": {
            "source": "doc['UnitPrice'].value * doc['Quantity'].value"
          }
        }
      },
      "number_of_unique_customers_per_day": {
        "cardinality": {
          "field": "CustomerID"
        }
      }
    }
  }
}

Copy and paste the request above into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch returns a 400-error(red box) along with the cause of the error in the response body.

This HTTP error starts with a 4, meaning that there was a client error with the request sent.

Cause of Error 9

This error is occurring because the structure of the aggregations request is incorrect.

In order to accomplish our goals, we first group data into daily buckets. Within each bucket, we calculate the daily revenue and the unique number of customers per day.

Therefore, our request contains an aggregation(pink brackets) within an aggregation(blue brackets).

The following demonstrates the correct aggregations request structure. Note the sub-aggregations that encloses the "daily_revenue" and the "number_of_unique_customers_per_day":

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_per_day": {
      "date_histogram": {
        "field": "InvoiceDate",
        "calendar_interval": "day"
      },
      "aggs": {
        "daily_revenue": {
          "sum": {
            "script": {
              "source": "doc['UnitPrice'].value * doc['Quantity'].value"
            }
          }
        },
        "number_of_unique_customers_per_day": {
          "cardinality": {
            "field": "CustomerID"
          }
        }
      }
    }
  }
}

Copy and paste the request into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch returns a 200-success HTTP status(red box).

It groups the dataset into daily buckets. Within each bucket, the number of unique customers per day as well as the daily revenue are calculated.

Congratulations. You have mastered how to troubleshoot beginner level Elasticsearch errors!

Next time you see an error, breathe easy and apply the thought process of troubleshooting errors covered in this blog!

Understanding mapping with Elasticsearch and Kibana

Lisa Jung — Fri, 13 Aug 2021 18:20:06 +0000

Have you ever encountered the error “Field type is not supported for [whatever you are trying to do with Elasticsearch]”?

The most likely culprit of this error is the mapping of your index!

Mapping is the process of defining how a document and its fields are indexed and stored. It defines the type and format of the fields in the documents. As a result, mapping can significantly affect how Elasticsearch searches and stores data.

Understanding how mapping works will help you define mapping that best serves your use case.

By the end of this blog, you will be able to define what a mapping is and customize your own mapping to make indexing and searching more efficient.

Prerequisite work

Watch this video from time stamp 14:28-19:00. This video will show you how to complete steps 1 and 2.

Set up Elasticsearch and Kibana*
Open the Kibana console(AKA Dev Tools)
Keep two windows open side by side(this blog and the Kibana console)

We will be sending requests from Kibana to Elasticsearch to learn how mapping works!

Note
If you would rather download Elasticsearch and Kibana on your own machine, follow the steps outlined in Downloading Elasticsearch and Kibana(macOS/Linux and Windows).

Additional Resources

Interested in beginner friendly workshops on Elasticsearch and Kibana? Check out my Beginner's Crash Course to Elastic Stack series!

1) Part 5 Workshop Recording

This blog is a complementary blog to Part 5 of the Beginner's Crash Course to Elastic Stack. If you prefer learning by watching videos instead, check out the recording!

2) Beginner's Crash Course to Elastic Stack Table of Contents

This table of contents includes repos from all workshops in the series. Each repo includes resources shared during the workshop including the video recording, presentation slides, related blogs, Elasticsearch requests and more!

What is a Mapping?

Imagine you are building an app that requires you to store and search data. Naturally, you will want to store data using the smallest disk space while maximizing your search performance.

This is where mapping comes into play!

Mapping defines how a document and its fields are indexed and stored.

It does that by defining field types. Depending on its type, the fields are stored and indexed accordingly.

Learning how to define your own mapping will help you:

optimize the performance of Elasticsearch
save disk space

Before we delve into mapping, let's review a few concepts!

Review from previous blogs

In the previous blogs, we learned how to query and aggregate data to gain insights.

But before we could run queries or aggregations, we had to first add data to Elasticsearch.

Let’s say we are creating an app for a produce warehouse. You want to store data about produce in Elasticsearch so you can search for it.

In Elasticsearch, data is stored as documents. A document is a JSON object that contains whatever data you want to store in Elasticsearch.

Take a look at the following document about a produce item. In a JSON object, it contains a list of fields such as "name", "botanical_name", "produce_type" and etc.

When you take a closer look at the fields, you will see that each field is of a different JSON data type.

For example, for the field "name", the JSON data type is string. For the field "quantity", the JSON data type is integer. For the field "preferred_vendor", the JSON data type is boolean.

Indexing a Document

Let’s say we want to index the document above. We would send the following request.

For requests we will go over, the syntax is included for you so you can customize this for your own use case.

Syntax:

POST Enter-name-of-the-index/_doc
{
  "field": "value"
}

But for our tutorial, we will use the example instead.

The following example asks Elasticsearch to create a new index called the temp_index, then index the following document into it.

Example:

POST temp_index/_doc
{
  "name": "Pineapple",
  "botanical_name": "Ananas comosus",
  "produce_type": "Fruit",
  "country_of_origin": "New Zealand",
  "date_purchased": "2020-06-02T12:15:35",
  "quantity": 200,
  "unit_price": 3.11,
  "description": "a large juicy tropical fruit consisting of aromatic edible yellow flesh surrounded by a tough segmented skin and topped with a tuft of stiff leaves.These pineapples are sourced from New Zealand.",
  "vendor_details": {
    "vendor": "Tropical Fruit Growers of New Zealand",
    "main_contact": "Hugh Rose",
    "vendor_location": "Whangarei, New Zealand",
    "preferred_vendor": true
  }
}

Copy and paste the example into the console and send the request.

Expected response from Elasticsearch:
Elasticsearch will confirm that this document has been successfully indexed into the temp_index.

What we have covered so far is a review from previous blogs. What we have not gone over is what actually goes on behind the scenes when you index a document.

This is where mapping comes into play!

Mapping Explained

Mapping determines how a document and its fields are indexed and stored by defining the type of each field.

The mapping of an index look something like the following:

It contains a list of the names and types of fields in an index. Depending on its type, each field is indexed and stored differently in Elasticsearch.

Therefore, mapping plays an important role in how Elasticsearch stores and searches for data.

Dynamic Mapping

In the previous request we sent, we had created a new index called the temp_index, then we indexed a document into it.

Mapping determines how a document and its fields should be indexed and stored. But we did not define the mapping ahead of time.

So how did this document get indexed?

When a user does not define the mapping in advance, Elasticsearch creates or updates the mapping as needed by default. This is known as dynamic mapping.

Take a look at the following diagram. It illustrates what happens when a user asks Elasticsearch to create a new index without defining the mapping ahead of time.

When a user does not define the mapping in advance, Elasticsearch creates or updates the mapping as needed by default. This is known as dynamic mapping.

With dynamic mapping, Elasticsearch looks at each field and tries to infer the data type from the field content. Then, it assigns a type to each field and creates a list of field names and types known as mapping.

Depending on the assigned field type, each field is indexed and primed for different types of requests(full text search, aggregations, sorting). This is why mapping plays an important role in how Elasticsearch stores and searches for data.

View the Mapping

Now that we have indexed a document without defining the mapping in advance, let's take a look at the dyanmic mapping that Elasticsearch has created for us.

To view the mapping of an index, you send the following request.

Syntax:

GET Enter_name_of_the_index_here/_mapping

The following example asks Elasticsearch to retrieve the mapping of the temp_index.

Example:

GET temp_index/_mapping

Copy and paste the example into the Kibana console and send the request.

Expected response from Elasticsearch:
Elasticsearch returns the mapping of the temp_index.

It lists all the fields of the document in an alphabetical order and lists the type of each field(text, keyword, long, float, date, boolean and etc). These are a few of many field types that are recognized by Elasticsearch. For the list of all field types, click here!

At a first glance, this mapping may look really complicated. Rest assured. We are going to break this down into bite sized pieces!

Depending on what your use case is, the mapping can be customized to make storage and indexing more efficient.

The rest of the blog will cover what type of mapping is best for different types of requests. Then, we will learn how to define our own mapping!

Indexing Strings

Let's take a look at our document again.

Many of these fields such as "botanical_name", "produce_type", and "country_of_origin"(teal box) contain strings.

Let's examine the mapping of the string fields "botanical_name","country_of_origin", and "description" below.

You will see that these string fields(orange lines) have been mapped as both text and keyword(green lines) by default.

There are two kinds of string data types:

Text
Keyword

By default, every string gets mapped twice as a text field and as a keyword multi-field. Each field type is primed for different types of requests.

Text field type is designed for full-text searches. One of the benefits of full text search is that you can search for individual terms in a non-case sensitive manner.

Keyword field type is designed for exact searches, aggregations, and sorting. This field type becomes handy when you are searching for original strings.

Depending on what type of request you want to run on each field, you can assign the field type as either text or keyword or both!

Let's delve into text field type first.

Text Field Type

Text Analysis

Ever notice that when you search in Elasticsearch, it is not case sensitive or the punctuation does not seem to matter? This is because text analysis occurs when your fields are indexed.

The diagram below illustrates how the string "These pineapples are sourced from New Zealand." would be analyzed in Elasticsearch.

By default, strings are analyzed when it is indexed. When the string is analyzed, the string is broken up into individual words also known as tokens. The analyzer further lowercases each token and removes punctuations.

Inverted Index
Take a look at the following request shown in the diagram. It asks Elasticsearch to index a document with the field "description" and assign the document an id of 1.

When this request is sent, Elasticsearch will look at the field "description" and see that this field contains a string. Therefore, it will map the field "description" as both text and keyword by default.

When a field is mapped as type text, the content of the field passes through an analyzer.

Once the string is analyzed, the individual tokens are stored in a sorted list known as the inverted index(see table above). Each unique token is stored in an inverted index with its associated ID(red numbers in the table above).

The same process occurs every time you index a new document.

Let's say we indexed a new document with identical content as document 1. But we gave this document an id of 2(pink box).

The content of the field "description" will go through the same process where it is split into individual tokens. However, if these tokens already exist in the inverted index, only the document IDs are updated(red numbers in the table above).

Look at the diagram below. If we add another document with an id of 3(pink box) with one new token(red box), then the new token(caledonia)is added to the inverted index. Also, the doc ids are updated for existing tokens(red numbers in the table below).

This is what happens in the background when you index fields that are typed as text.

Fields that are assigned the type text are optimal for full text search.

Take a look at this diagram below. The client is asking to fetch all documents containing the terms "new" or "zealand".

When this request is sent, Elasticsearch does not read through every document for the search terms. It goes straight to the inverted index to look up the search terms, finds the matching document ids and sends the ids back to the user.

The inverted index is the reason Elasticsearch is able to search with speed.

Moreover, since all the terms in the inverted index are lowercase and your search queries are also analyzed, you can search for things in a non-case sensitive manner, and still get the same results.

Let's do another review.

Take a look at the diagram below. We send three requests to index three documents.

Each document has a field called "country" with a string value. By default, the field "country" is mapped twice as text(teal arrow) and keyword(yellow arrow).

Text fields(teal arrow) are analyzed and the tokens are stored in an inverted index(teal table). This is great for full text search but it is not optimized for performing aggregations, sorting or exact searches.

For the types of actions mentioned above, Elasticsearch depends on the keyword field(yellow arrow).

When a keyword field(yellow arrow) is created, the content of the field is not analyzed and is not stored in an inverted index.

Instead, keyword field uses a data structure called doc values(yellow table) to store data.

Keyword Field Type

Keyword field type is used for aggregations, sorting, and exact searches. These actions look up the document ID to find the values it has in its fields.

Keyword field is suited to perform these actions because it uses a data structure called doc values to store data.

Take a look at the diagram below that illustrates doc values.

For each document, the document id along with the field value(the original string) are added to a table. This data structure(doc values) is designed for actions that require looking up the document ID to find the values it has in its fields. Therefore, the fields that are typed as keyword are best suited for actions such as aggregations, sorting, and exact searches.

When Elasticsearch dynamically creates a mapping for you, it does not know what you want to use a string for. As a result, Elasticsearch maps all strings to both field types(text and keyword).

In cases where you do not need both field types, the default setting is wasteful. Since both field types require creating either an inverted index or doc values, creating both field types for unnecessary fields will slow down indexing and take up more disk space.

This is why we define our own mapping as it helps us store and search data more efficiently.

Doing so takes more planning because we need to decide what type of requests we want to run on these fields. The decisions we make will allow us to designate which string fields will:

only be full text searchable or
only be used in aggregation or
be able to support both options

Mapping Exercise

Now that we understand the field types text and keyword, let’s go over how we can optimize our mapping.

To do so, we are going to do an exercise!

Project: Build an app for a client who manages a produce warehouse

This app must enable users to:

search for produce name, country of origin and description
identify top countries of origin with the most frequent purchase history
sort produce by produce type(fruit or vegetable)
get the summary of monthly expense

Goals

Figure out the optimal mapping for desired features
Create an index with the optimal mapping and index documents into it
Learn how you should approach a situation that requires changing the mapping of an existing field
Learn how to use the runtime field

The following is a sample document we will be working with for this exercise. Pay attention to the field names as these field names will come up frequently during this exercise!

Sample data

{
  "name": "Pineapple",
  "botanical_name": "Ananas comosus",
  "produce_type": "Fruit",
  "country_of_origin": "New Zealand",
  "date_purchased": "2020-06-02T12:15:35",
  "quantity": 200,
  "unit_price": 3.11,
  "description": "a large juicy tropical fruit consisting of aromatic edible yellow flesh surrounded by a tough segmented skin and topped with a tuft of stiff leaves.These pineapples are sourced from New Zealand.",
  "vendor_details": {
    "vendor": "Tropical Fruit Growers of New Zealand",
    "main_contact": "Hugh Rose",
    "vendor_location": "Whangarei, New Zealand",
    "preferred_vendor": true
  }
}

Plan of Action
The figures below outline the plan of action for each feature requested by the client. Explanations regarding each bullet point are provided in the blog.

The first feature allows the user to search for produce name, country of origin and description.

Let's take a look at the sample document shared with you earlier.

Sample data

{
  "name": "Pineapple",
  "botanical_name": "Ananas comosus",
  "produce_type": "Fruit",
  "country_of_origin": "New Zealand",
  "date_purchased": "2020-06-02T12:15:35",
  "quantity": 200,
  "unit_price": 3.11,
  "description": "a large juicy tropical fruit consisting of aromatic edible yellow flesh surrounded by a tough segmented skin and topped with a tuft of stiff leaves.These pineapples are sourced from New Zealand.",
  "vendor_details": {
    "vendor": "Tropical Fruit Growers of New Zealand",
    "main_contact": "Hugh Rose",
    "vendor_location": "Whangarei, New Zealand",
    "preferred_vendor": true
  }
}

The first feature involves working with the fields "name", "country_of_origin", and "description"(first bullet point in the figure below).

All of these fields contain strings. By default, these fields will get mapped twice as text and keyword.

Let’s see if we need both field types.

Our client wants to search on these fields. But it is unlikely that the user will send search terms that are exactly the way it is written in our documents.

So the user should be able to run a search for individual terms in a non-case sensitive manner. Therefore, the fields "name", "country_of_origin" and "description" should be full text searchable(second bullet point in the figure below).

Let’s say our client does not want to run aggregation, sorting, or exact searches on the fields "name" and "description"(third bullet point in the figure above).

For these fields, we do not need the keyword type. To avoid mapping these fields twice, we will specify that we only want the text type for these fields(blue bullet point in the figure above).

At this point, we know that the field "country_of_origin" should be full text searchable(text). But it is unclear whether we will need aggregation, sorting, or exact searches to be performed on this field(fourth bullet point in the figure above).

Let’s leave that for now.

If you look at the second feature, the user should be able to identify top countries of origin with the most frequent purchase history.

This feature involves the field "country_of_origin"(first bullet point in the figure above).

For this type of request, we need to run the terms aggregations on this field(second bullet point in the figure above), which means that we need a keyword field.

Since we need to perform full text search(first feature) and aggregations(second feature) on this field, we will map this field twice as text and keyword(third bullet point in the figure above).

The third feature allows the user to sort by produce type. This feature involves the field "produce_type" which contains a string(first bullet point in the figure above).

Since this feature requires sorting(second bullet point in the figure above), we need a keyword type for this field.

Let’s say the client does not want to run a full text search for this field.

As a result, we will need to map this field as keyword type only(third bullet point in the figure above).

The fourth feature allows the user to get the summary of monthly expense.

This feature involves the fields "date_purchased", "quantity", and "unit_price"(first bullet point in the figure above).

So let’s break this down.

This feature requires splitting the data into monthly buckets.

Then, calculating the monthly revenue for each bucket by adding up the total of each document in the bucket.

The next diagram shows you the April bucket. The documents of all produce purchased during April are included in this bucket.

Let's zoom in on one of the documents to look at its fields(json object highlighted with pink lines).

You will notice that our documents do not have a field called "total".

However, we do have fields called "quantity" and "unit_price"(pink box).

In order to get the total for each document, we need to multiply the values of the fields "quantity" by "unit_price". Then, add up the total of all documents in each bucket. This will yield the monthly expense.

The following figure shows the summary of optimal mapping for all string fields that we have just discussed.

Take a look at the first bullet point in the figure above. It states that for the fourth feature, we need to calculate the total for each produce so it can be used to calculate the monthly expense.

The second bullet point in the figure outlines the additional step we are going to take to make our mapping more efficient.

The following is the reason why we are disabling the field "botanical_name" and the object "vendor_details"(second bullet point).

Our document contains multiple fields.

Sample data

{
  "name": "Pineapple",
  "botanical_name": "Ananas comosus",
  "produce_type": "Fruit",
  "country_of_origin": "New Zealand",
  "date_purchased": "2020-06-02T12:15:35",
  "quantity": 200,
  "unit_price": 3.11,
  "description": "a large juicy tropical fruit consisting of aromatic edible yellow flesh surrounded by a tough segmented skin and topped with a tuft of stiff leaves.These pineapples are sourced from New Zealand.",
  "vendor_details": {
    "vendor": "Tropical Fruit Growers of New Zealand",
    "main_contact": "Hugh Rose",
    "vendor_location": "Whangarei, New Zealand",
    "preferred_vendor": true
  }
}

After going through each feature, we know that the string field "botanical_name" and the object "vendor_details" will not be used.

Since we do not need the inverted index or the doc values of these fields, we are going to disable these fields. This in turn prevents the inverted index and the doc values of these fields from being created.

Disabling these fields will help us save disk space and minimize the number of fields in the mapping.

Defining your own mapping

Now that we have thought through the optimal mapping, let’s define our own!

Before we get to that, there are a few rules about mapping you need to know.

Rules

If you do not define a mapping ahead of time, Elasticsearch dynamically creates a mapping for you.
If you do decide to define your own mapping, you can do so at index creation.
ONE mapping is defined per index. Once the index has been created, we can only add new fields to a mapping. We CANNOT change the mapping of an existing field.
If you must change the type of an existing field, you must create a new index with the desired mapping, then reindex all documents into the new index.

We will go over how these rules work as we go over the steps of defining our own mapping!

Step 1: Index a sample document into a test index.
Syntax:

POST Name-of-test-index/_doc
{
  "field": "value"
}

The first thing we are going to do is to create a new index called test_index and index a sample document.

The document in the following example is the same sample document we have seen earlier. The sample document must contain the fields that you want to define. These fields must also contain values that map closely to the field types you want.

Copy and paste the following example into the Kibana console and send the request.

Example:

POST test_index/_doc
{
  "name": "Pineapple",
  "botanical_name": "Ananas comosus",
  "produce_type": "Fruit",
  "country_of_origin": "New Zealand",
  "date_purchased": "2020-06-02T12:15:35",
  "quantity": 200,
  "unit_price": 3.11,
  "description": "a large juicy tropical fruit consisting of aromatic edible yellow flesh surrounded by a tough segmented skin and topped with a tuft of stiff leaves.These pineapples are sourced from New Zealand.",
  "vendor_details": {
    "vendor": "Tropical Fruit Growers of New Zealand",
    "main_contact": "Hugh Rose",
    "vendor_location": "Whangarei, New Zealand",
    "preferred_vendor": true
  }
}

Expected response from Elasticsearch:
The test_index is successfully created.

By default, Elasticsearch will create a dynamic mapping based on the sample document.

What is the point of step 1?

Earlier in the blog, we have indexed the sample document and viewed the dynamic mapping. If you remember, the mapping for our sample document was pretty lengthy.

Since we do not want to write our optimized mapping from scratch, we are indexing a sample document so that Elasticsearch will create the dynamic mapping.

We will use the dynamic mapping as a template and make changes to it to avoid writing out the whole mapping for our index!

Step 2: View the dynamic mapping
Syntax:

GET Name-the-index-whose-mapping-you-want-to-view/_mapping

Let's view the mapping of the test_index by sending the following request.

Example:

GET test_index/_mapping

Expected response from Elasticsearch:
Elasticsearch will display the dynamic mapping it has created. It lists the fields in an alphabetical order. The sample document is identical to the one we previously indexed into thetemp_index.

Step 3: Edit the mapping
Copy and paste the entire mapping from step 2 into the Kibana console. Then, make the changes specified below.

From the pasted results, remove the "test_index" along with its opening and closing brackets. Then, edit the mapping to satisfy the requirements outlined in the diagram below.

Your optimized mapping should look like the following:

You will see that the field "country_of_origin"(2) has been typed as both text and keyword.

The fields "description"(3) and "name"(4) have been typed as text only.

The field field "produce_type"(5) has been typed as keyword only.

In the field "botanical_name"(1), a parameter called "enabled" was added and was set to "false". This prevents the inverted index and doc values from being created for this field.

The same has been done for the object "vendor_details"(6).

The mapping above defines the optimal mapping for all of our desired features except for the one marked a red x in the figure below.

Don't worry about the unaddressed feature yet as we are saving this for later!

Step 4: Create a new index with the optimized mapping from step 3.
Syntax:

PUT Name-of-your-final-index
{
  copy and paste your optimized mapping here
}

Next, we will create a new index called produce_index with the optimized mapping we just worked on.

If you still have the mapping from step 3 in the Kibana console, delete the mapping from the Kibana console.

Then, copy and paste the following example into the Kibana console and send the request!

Example:

PUT produce_index
{
  "mappings": {
    "properties": {
      "botanical_name": {
        "enabled": false
      },
      "country_of_origin": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      },
      "date_purchased": {
        "type": "date"
      },
      "description": {
        "type": "text"
      },
      "name": {
        "type": "text"
      },
      "produce_type": {
        "type": "keyword"
      },
      "quantity": {
        "type": "long"
      },
      "unit_price": {
        "type": "float"
      },
      "vendor_details": {
        "enabled": false
      }
    }
  }
}

Expected response from Elasticsearch:
Elasticsearch creates the produce_index with the optimized mapping we defined above!

Step 5: Check the mapping of the new index to make sure the all the fields have been mapped correctly

Syntax:

GET Name-of-test-index/_mapping

Let's check the mapping of the produce_index to make sure that all the fields have been mapped correctly.

Copy and paste the following example into the Kibana console and send the request!

Example:

GET produce_index/_mapping

Expected response from Elasticsearch:
Compared to the dynamic mapping, our optimized mapping looks more simple and concise!

The current mapping satisfies the requirements that are marked with green check marks in the figure below.

Just by defining our own mapping, we prevented unnecessary inverted index and doc values from being created which saved a lot of disk space.

The type of each field has been customized to serve specific client requests. As a result, we also optimized the performance of Elasticsearch as well!

Step 6: Index your dataset into the new index
Now that we have created a new index(produce_index) with the optimal mapping, it is time to add data to the produce_index.

For simplicity's sake, we will index two documents by sending the following requests.

Index the first document

POST produce_index/_doc
{
  "name": "Pineapple",
  "botanical_name": "Ananas comosus",
  "produce_type": "Fruit",
  "country_of_origin": "New Zealand",
  "date_purchased": "2020-06-02T12:15:35",
  "quantity": 200,
  "unit_price": 3.11,
  "description": "a large juicy tropical fruit consisting of aromatic edible yellow flesh surrounded by a tough segmented skin and topped with a tuft of stiff leaves.These pineapples are sourced from New Zealand.",
  "vendor_details": {
    "vendor": "Tropical Fruit Growers of New Zealand",
    "main_contact": "Hugh Rose",
    "vendor_location": "Whangarei, New Zealand",
    "preferred_vendor": true
  }
}

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
Elasticsearch successfully indexes the first document.

Index the second document
The second document has almost identical fields as the first document except that it has an extra field called "organic" set to true!

POST produce_index/_doc
{
  "name": "Mango",
  "botanical_name": "Harum Manis",
  "produce_type": "Fruit",
  "country_of_origin": "Indonesia",
  "organic": true,
  "date_purchased": "2020-05-02T07:15:35",
  "quantity": 500,
  "unit_price": 1.5,
  "description": "Mango Arumanis or Harum Manis is originated from East Java. Arumanis means harum dan manis or fragrant and sweet just like its taste. The ripe Mango Arumanis has dark green skin coated with thin grayish natural wax. The flesh is deep yellow, thick, and soft with little to no fiber. Mango Arumanis is best eaten when ripe.",
  "vendor_details": {
    "vendor": "Ayra Shezan Trading",
    "main_contact": "Suharto",
    "vendor_location": "Binjai, Indonesia",
    "preferred_vendor": true
  }
}

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
Elasticsearch successfully indexes the second document.

Let's see what happens to the mapping by sending this request below:

GET produce_index/_mapping

Expected response from Elasticsearch:
The new field("organic") and its field type(boolean) have been added to the mapping(orange box). This is in line with the rules of mapping we discussed earlier since you can add new fields to the mapping. We just cannot change the mapping of an existing field!

What if you do need to make changes to the existing field type?

Let's say you have added a huge produce dataset to the produce_index.

Then, your client asks for an additional feature where the user can run a full text search on the field "botanical_name".

This is a dilemma since we disabled the field "botanical_name" when we created the produce_index.

What are we to do?

Remember, you CANNOT change the mapping of an existing field. If you do need to make changes to the existing field, you must create a new index with the desired mapping, then reindex all documents into the new index.

STEP 1: Create a new index(produce_v2) with the latest mapping.
This step is very similar to what we just did with the test_index and the produce_index.

We are just creating a new index(produce_v2) with a new mapping where we remove the "enabled" parameter from the field "botanical_name" and change its type to "text".

Example:

PUT produce_v2
{
  "mappings": {
    "properties": {
      "botanical_name": {
        "type": "text"
      },
      "country_of_origin": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "date_purchased": {
        "type": "date"
      },
      "description": {
        "type": "text"
      },
      "name": {
        "type": "text"
      },
      "organic": {
        "type": "boolean"
      },
      "produce_type": {
        "type": "keyword"
      },
      "quantity": {
        "type": "long"
      },
      "unit_price": {
        "type": "float"
      },
      "vendor_details": {
        "type": "object",
        "enabled": false
      }
    }
  }
}

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
Elasticsearch creates a new index(produce_v2) with the latest mapping.

Check the mapping of the produce_v2 index by sending the following request.

View the mapping of produce_v2:

GET produce_v2/_mapping

Expected response from Elasticsearch:
You will see that the field "botanical_name" has been typed as text(orange box).

STEP 2: Reindex the data from the original index(produce_index) to the one you just created(produce_v2).

We just created a new index(produce_v2) with the desired mapping. The next step is to reindex the data from the produce_index to the produce_v2 index.

To do so, we send the following request:

POST _reindex
{
  "source": {
    "index": "produce_index"
  },
  "dest": {
    "index": "produce_v2"
  }
}

The field "source" refers to the original index that contains the dataset we want to reindex. The field "dest"(destination) refers to the new index with the optimized mapping. The example requests for the data to be reindexed to the new index.

Copy and paste the request above into the Kibana console and send the request!

Expected response form Elasticsearch:
This request reindexes data from the produce_index to the produce_v2 index. The produce_v2 index can now be used to run the requests that the client has specified.

Runtime Field

We have one last feature to work on!

Our client wants to be able to get the summary of monthly expense.

This feature requires splitting the data into monthly buckets.

Then, calculating the monthly revenue for each bucket by adding up the total of each document in the bucket.

The problem is that our documents do not have a field called "total". However, we do have fields called "quantity" and "unit_price".

In order to get the total for each document, we need to multiply the values of the field "quantity" by "unit_price". Then, add up the total of all documents in each bucket. This will yield the monthly expense.

At this point, you might be thinking we will have to calculate the total and store this in a new field called "total". Then, add the field "total" to the existing documents and run the sum aggregation on this field.

This can get kinda hairy because it involves adding a new field to the existing documents in your index. This requires reindexing your data and starting all over again.

But there is a new feature that helps you get around this issue. It is called the runtime field!

What is runtime?
A runtime is that moment in time when Elasticsearch is executing your requests. During runtime, you can actually create a temporary field and calculate a value within it. This field then can be used to run whatever request that was sent during runtime.

This may sound very abstract to you at this point so let’s break this down.

For our last feature, the missing ingredient is the field "total". We must calculate the total by multiplying the values of the fields "quantity" and "unit_price".

Right now the field "total" does not exist in our dataset so we cannot aggregate on this field.

What we are going to do is to create a runtime field called "total" and add it to the mapping of the existing index.

Step 1: Create a runtime field and add it to the mapping of the existing index.

Syntax:

PUT Enter-name-of-index/_mapping
{
  "runtime": {
    "Name-your-runtime-field-here": {
      "type": "Specify-field-type-here",
      "script": {
        "source": "Specify the formula you want executed"
      }
    }
  }
}

The following request asks Elasticsearch to create a runtime field called "total" and add it to the mapping of produce_v2 index.

The runtime field(1) is created only when a user runs a request against this field.

We know that the field "total"(2) will contain decimal points so we will set the field "type" to "double"(3).

In the "script"(4), we write out the formula to calculate the value for the field "total". The "script" instructs Elasticsearch to do the following(5): For each document, multiply the value of the field "unit_price" by the value of the field "quantity".

Copy and paste the following example into the Kibana console and send the request!

Example:

PUT produce_v2/_mapping
{
  "runtime": {
    "total": {
      "type": "double",
      "script": {
        "source": "emit(doc['unit_price'].value* doc['quantity'].value)"
      }
    }
  }
}

Expected response from Elasticsearch:
Elasticsearch successfully adds the runtime field to the mapping.

Step 2: Check the mapping:
Let's check the mapping of the produce_v2 index by sending the following request.

GET produce_v2/_mapping

Expected response from Elasticsearch:
Elasticsearch adds a runtime field to the mapping(red box).

Note that the runtime field is not listed under "properties" object which includes the fields in our documents. This is because the runtime field "total" is not indexed!

The runtime field is only created and calculated at runtime as you execute your request!

Remember, our indexed documents do not have a field "total". But by adding a runtime field to our mapping, we are leaving some instructions for Elasticsearch.

The mapping tells Elasticsearch that if it were to receive a request on the field "total", it should create a temporary field called "total" for each document and calculate its value by running the "script". Then, run the request on the field "total" and send the results to the user.

Step 3: Run a request on the runtime field to see it perform its magic!
Let's put runtime field to the test.

We are going to send the following request to perform a sum aggregation on the runtime field "total".

Note that the following request does not aggregate the monthly expense here. We are running a simple aggregation request to demonstrate how the runtime field works!

Syntax:

GET Enter_name_of_the_index_here/_search
{
  "size": 0,
  "aggs": {
    "Name your aggregations here": {
      "Specify the aggregation type here": {
        "field": "Name the field you want to aggregate on here"
      }
    }
  }
}

The following request runs a sum aggregation against the runtime field "total" over all documents in our index.

Example:

GET produce_v2/_search
{
  "size": 0,
  "aggs": {
    "total_expense": {
      "sum": {
        "field": "total"
      }
    }
  }
}

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
When this request is sent, a runtime field called "total" is created and calculated for documents within the scope of our request(entire index). Then, the sum aggregation is ran on the field "total" over all documents in our index.

What are some of the benefits of using the runtime field?
The runtime field is only created and calculated when a request made on the runtime field is being executed. Runtime fields are not indexed so these do not take up disk space.

We also did not have to reindex in order to add a new field to existing documents. For more information on runtime fields, check out this blog!

There you have it. You have now mastered the basics of mapping!

Play around with mapping on your own and come up with the optimal mapping for your use case!

Running aggregations with Elasticsearch and Kibana

Lisa Jung — Thu, 05 Aug 2021 17:59:39 +0000

There are two main ways to search in Elasticsearch:
1) Queries retrieve documents that match the specified criteria.
2) Aggregations present the summary of your data as metrics, statistics, and other analytics.

In my previous blog, we learned how to retrieve documents by sending queries.

This blog will cover how you can summarize your data as metrics, statistics, or other analytics by sending aggregations requests!

By the end of this blog, you will be able to run:

metric aggregations
bucket aggregations
combined aggregations

Prerequisite work

Watch this video from time stamp 15:00-21:46. This video will show you how to complete steps 1-3.

Set up Elasticsearch and Kibana*
Add e-commerce dataset to Elasticsearch*
Open the Kibana console(AKA Dev Tools)
Keep two windows open side by side(this blog and the Kibana console)

We will be sending aggregations requests from Kibana to Elasticsearch to learn how aggregations work!

Notes
1) If you would rather download Elasticsearch and Kibana on your own machine, follow the steps outlined in Downloading Elasticsearch and Kibana(macOS/Linux and Windows).

2) The video will show you how to add the news headline dataset. Follow the same steps but add the e-commerce dataset linked above. Create an index called ecommerce_original and add the data to that index.

Additional Resources

Interested in beginner friendly workshops on Elasticsearch and Kibana? Check out my Beginner's Crash Course to Elastic Stack series!

1) Part 4 Workshop Recording

This blog is a complementary blog to Part 4 of the Beginner's Crash Course to Elastic Stack. If you prefer learning by watching videos instead, check out the recording!

2) Beginner's Crash Course to Elastic Stack Table of Contents

This table of contents includes repos of all workshops in the series. Each repo includes resources shared during the workshop including the video recording, presentation, related blogs, Elasticsearch requests and more!

Set up data within Elasticsearch

Now that you have completed the prerequisite steps, it is time to set up data within Elasticsearch.

Often times, the dataset is not optimal for running requests in its original state.

For example, the type of a field may not be recognized by Elasticsearch or the dataset may contain a value that was accidentally included in the wrong field and etc.

These are exact problems that I ran into while working with this dataset. The following are the requests that I sent to Elasticsearch to yield the results included in the blog.

Copy and paste these requests into the Kibana console(Dev Tools) and run these requests in the order shown below.

STEP 1: Create a new index(ecommerce_data) with the following mapping.

PUT ecommerce_data
{
  "mappings": {
    "properties": {
      "Country": {
        "type": "keyword"
      },
      "CustomerID": {
        "type": "long"
      },
      "Description": {
        "type": "text"
      },
      "InvoiceDate": {
        "type": "date",
        "format": "M/d/yyyy H:m"
      },
      "InvoiceNo": {
        "type": "keyword"
      },
      "Quantity": {
        "type": "long"
      },
      "StockCode": {
        "type": "keyword"
      },
      "UnitPrice": {
        "type": "double"
      }
    }
  }
}

STEP 2: Reindex the data from the original index(source) to the one you just created(destination).

POST _reindex
{
  "source": {
    "index": "name of your original index when you added the data to Elasticsearch"
  },
  "dest": {
    "index": "ecommerce_data"
  }
}

STEP 3: Remove the negative values from the field "UnitPrice".
When you explore the minimum unit price in this dataset, you will see that the minimum unit price value is -11062.06.

To keep our data simple, I used the delete_by_query API to remove all unit prices less than 0.

POST ecommerce_data/_delete_by_query
{
  "query": {
    "range": {
      "UnitPrice": {
        "lte": 0
      }
    }
  }
}

STEP 4: Remove values greater than 500 from the field "UnitPrice".
When you explore the maximum unit price in this dataset, you will see that the maximum unit price value is 38,970.

When the data is manually examined, the majority of the unit prices are less than 500. The max value of 38,970 would skew the average.

To simplify our demo, I used the delete_by_query API to remove all unit prices greater than 500.

POST ecommerce_data/_delete_by_query
{
  "query": {
    "range": {
      "UnitPrice": {
        "gte": 500
      }
    }
  }
}

Get information about documents in an index

Before we can run aggregations requests, we need to know what information is included in our dataset.

This will help us figure out what type of questions we could ask and identify the appropriate fields to run aggregations on to get the answers.

The following query will retrieve information about documents in the ecommerce_data index. This query is a great way to explore the structure and content of your document.

Syntax:

GET Enter_name_of_the_index_here/_search

Example:

GET ecommerce_data/_search

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
Elasticsearch displays a number of hits(line 12) and a sample of 10 search results by default(lines 16+).

The first hit(a document) is shown on lines 17-31. The field "_source"(line 22) lists all the fields(the content) of the document.

The ecommerce_data index contains transaction data from a company that operates in multiple countries.

Each document is a transaction of an item and it contains the following fields:

Description
Quantity
InvoiceNo
CustomerID
UnitPrice
Country
InvoiceDate
StockCode

As you can see, running this query helps us understand what types of questions we could ask about our dataset and which fields we need to aggregate on to get the answers.

Aggregations Requests

The basic syntax of an aggregations request looks like the following.

Syntax:

GET Enter_name_of_the_index_here/_search
{
  "aggs": {
    "Name your aggregations here": {
      "Specify the aggregation type here": {
        "field": "Name the field you want to aggregate on here"
      }
    }
  }
}

There are various types of aggregations that you can run with Elasticsearch.

In the prerequisite steps, we added the e-commerce dataset and set up data within Elasticsearch.

To learn how these different types of aggregations work, we will pretend that we own an e-commerce app and that we have added our transaction data to the ecommerce_data index.

We will be sending various aggregations requests to get insights about transactions of items sold on our e-commerce app!

Metric Aggregations

Metric aggregations are used to compute numeric values based on your dataset. It can be used to calculate the values of sum,min, max, avg, unique count(cardinality) and etc.

When you are running an e-commerce app, it is important to know how your business is performing. A great way to measure that is to compute these metrics we mentioned above.

Let's calculate these values!

Metric aggregations can only be performed on fields that contain numeric values.

Take a look at the image below. This is an example of a document in our index(lines 22-31).

The fields "Quantity" and "UnitPrice" contain numeric values. Metric aggregations can be performed on these fields.

Compute the sum of all unit prices in the index

Let's say we want to sum up the values of the field "UnitPrice" over all documents in the ecommerce_data index.

Syntax:

GET Enter_name_of_the_index_here/_search
{
  "aggs": {
    "Name your aggregations here": {
      "sum": {
        "field": "Name the field you want to aggregate on here"
      }
    }
  }
}

The following example follows the aggregations syntax.

Example:

GET ecommerce_data/_search
{
  "aggs": {
    "sum_unit_price": {
      "sum": {
        "field": "UnitPrice"
      }
    }
  }
}

This aggregations request is named as "sum_unit_price". It instructs Elasticsearch to perform sum aggregations on the field "UnitPrice" over all documents in the ecommerce_data index

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
By default, Elasticsearch returns top 10 hits(lines 16+).

When you minimize hits(red box- line 10), you will see the results of aggregations we named "sum_unit_price"(image below). It displays the sum of all unit prices present in our index.

If the sole purpose of running aggregations requests is to get the aggregations results, you can add a size parameter and set it to 0 as shown below.

This parameter prevents Elasticsearch from fetching the top 10 hits so that the aggregations results are shown at the top of the response.

Using a size parameter

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "sum_unit_price": {
      "sum": {
        "field": "UnitPrice"
      }
    }
  }
}

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
We no longer need to minimize the hits to get access to the aggregations results!

We will be setting the size parameter to 0 in all requests from this point on.

Compute the lowest(min) unit price of an item

What if we wanted to calculate the lowest(min) unit price of an item?

Syntax:

GET Enter_name_of_the_index_here/_search
{
  "size": 0,
  "aggs": {
    "Name your aggregations here": {
      "min": {
        "field": "Name the field you want to aggregate on here"
      }
    }
  }
}

The following example is very similar to the last aggregations request we sent.

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "lowest_unit_price": {
      "min": {
        "field": "UnitPrice"
      }
    }
  }
}

The differences are that:

we are naming the aggregations as "lowest_unit_price".
we set the aggregations type to min which is short for minimum.

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
The lowest unit price of an item is 1.01.

Compute the highest(max) unit price of an item

What if we were interested in the highest(max) unit price of an item?

Syntax:

GET Enter_name_of_the_index_here/_search
{
  "size": 0,
  "aggs": {
    "Name your aggregations here": {
      "max": {
        "field": "Name the field you want to aggregate on here"
      }
    }
  }
}

The following example is very similar to the last aggregations request we sent.

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "highest_unit_price": {
      "max": {
        "field": "UnitPrice"
      }
    }
  }
}

The differences are that:

we are naming the aggregations as "highest_unit_price".
we set the aggregations type to max which is short for maximum.

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
The highest unit price of an item is 498.79.

Compute the average unit price of items

What if we wanted to calculate the average unit price of items?

Syntax:

GET Enter_name_of_the_index_here/_search
{
  "size": 0,
  "aggs": {
    "Name your aggregations here": {
      "avg": {
        "field": "Name the field you want to aggregate on here"
      }
    }
  }
}

The following example is very similar to the last aggregations request we sent.

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "average_unit_price": {
      "avg": {
        "field": "UnitPrice"
      }
    }
  }
}

The differences are that:

we are naming the aggregations as "average_unit_price".
we set the aggregations type to type to avg which is short for average.

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
The average unit price of an item is ~4.39.

Compute the count, min, max, avg, sum in one go

Calculating the count, min, max, avg, and sum individually can be a tedious task. The stats aggregations can calculate all of these in one go!

Syntax:

GET Enter_name_of_the_index_here/_search
{
  "size": 0,
  "aggs": {
    "Name your aggregations here": {
      "stats": {
        "field": "Name the field you want to aggregate on here"
      }
    }
  }
}

The following example is very similar to the last aggregations request we sent.

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "all_stats_unit_price": {
      "stats": {
        "field": "UnitPrice"
      }
    }
  }
}

The differences are that:

we are naming the aggregations as "all_stats_unit_price".
we set the aggregations type to stats which is short for statistics.

Let's copy and paste this example into the Kibana console and send the request!

Expected Response from Elasticsearch:
The stats aggregation will yield the values of count(the number of unit prices aggregation was performed on), min, max, avg, and sum(sum of all unit prices in the index).

Cardinality Aggregation

What if you want the number of unique customers whom have bought from the app?

You would run the cardinality aggregation, which computes the count of unique values for a given field!

Syntax:

GET Enter_name_of_the_index_here
{
  "size": 0,
  "aggs": {
    "Name your aggregations here": {
      "cardinality": {
        "field": "Name the field you want to aggregate on here"
      }
    }
  }
}

The following example is very similar to the last aggregations request we sent.

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "number_unique_customers": {
      "cardinality": {
        "field": "CustomerID"
      }
    }
  }
}

The differences are that:

we are naming the aggregations as "number_unique_customers".
we set the aggregations type to cardinality.

We perform cardinality aggregations on the field "CustomerID" as each customer is given a unique customer ID. By identifying unique customer IDs, we are able to get the number of unique customers in our transaction data.

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
Approximately, there are 4325 unique number of customers in our dataset.

Limiting the scope of an `aggregation`

In previous examples, aggregations were performed on all documents in the ecommerce_data index.

What if you want to run aggregations on a subset of the documents?

For example, our index contains e-commerce data from multiple countries. Let's say you want to calculate the average unit price of items sold in Germany.

To limit the scope of the aggregations, you can add a query clause to the aggregations request.

The query clause defines the subset of documents that aggregations should be performed on.

The syntax of combined query and aggregations request look like the following.

Syntax:

GET Enter_name_of_the_index_here/_search
{
  "size": 0,
  "query": {
    "Enter match or match_phrase here": {
      "Enter the name of the field": "Enter the value you are looking for"
    }
  },
  "aggregations": {
    "Name your aggregations here": {
      "Specify aggregations type here": {
        "field": "Name the field you want to aggregate here"
      }
    }
  }
}

Let's take a look at the following example which calculates the average unit price of items sold in Germany.

Example:

This request instructs Elasticsearch to query(1) all documents that "match"(2) the value "Germany" in the field "Country"(3).

Elasticsearch is then instructed to run aggregations(4) on the queried data. We name the aggregations "germany_average_unit_price"(5). Then, we tell Elasticsearch to get the average(6) of the values in the field "UnitPrice"(7) over all queried documents.

This in turn will tell us the average unit price of items sold in Germany.

Let's copy and paste the following example into the Kibana console and send the request!

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "query": {
    "match": {
      "Country": "Germany"
    }
  },
  "aggs": {
    "germany_average_unit_price": {
      "avg": {
        "field": "UnitPrice"
      }
    }
  }
}

Expected response from Elasticsearch:
The average of unit price of items sold in Germany is ~4.58.

The combination of query and aggregations request allowed us to perform aggregations on a subset of documents.

What if we wanted to perform aggregations on several subsets of documents?

This is where bucket aggregations come into play!

Bucket Aggregations

When you want to aggregate on several subsets of documents, bucket aggregations will come in handy.

Bucket aggregations group documents into several subsets of documents called buckets. All documents in a bucket share a common criteria.

The following diagram illustrates a bucket aggregations request that splits documents into monthly buckets.

There are various ways you can group documents into buckets. These are:

Date_histogram aggregation
Histogram aggregation
Range aggregation
Terms aggregation

Date_histogram Aggregation

When you are looking to group data by time interval, the date_histogram aggregation will prove very useful!

Our ecommerce_data index contains transaction data that has been collected over time(from the year 2010 to 2011).

If we are looking to get insights about transactions over time, our first instinct should be to run the date_histogram aggregation.

There are two ways to define a time interval with date_histogram aggregation. These are Fixed_interval and Calendar_interval.

Fixed_interval
With the fixed_interval, the interval is always constant.

Let's say we wanted to create a bucket for every 8 hour interval.

Syntax:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "Name your aggregations here": {
      "date_histogram": {
        "field":"Name the field you want to aggregate on here",
        "fixed_interval": "Specify the interval here"
      }
    }
  }
}

The following example is similar to the previous aggregations request we have sent.

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_by_8_hrs": {
      "date_histogram": {
        "field": "InvoiceDate",
        "fixed_interval": "8h"
      }
    }
  }
}

We name our aggregations "transactions_by_8_hrs". Then, we set the type of aggregation to "date_histogram".

We instruct Elasticsearch to perform this aggregation on the field "InvoiceDate" and to split the documents into buckets at a "fixed_ interval" of 8 hours.

Let's copy and paste this example into the Kibana console and send the request!

Expected response from Elasticsearch:
Elasticsearch creates a bucket for every 8 hours("key_as_string") and shows the number of documents("doc_count") grouped into each bucket.

Another way we can define the time interval is through the calendar_interval.

Calendar_interval
With the calendar_interval, the interval may vary.

For example, we could choose a time interval of day, month or year. But daylight savings can change the length of specific days, months can have different number of days, and leap seconds can be tacked onto a particular year.

So the time interval of day, month, or leap seconds could vary!

A scenario where you might use the calendar_interval is when you want to calculate the monthly revenue.

Syntax:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "Name your aggregations here": {
      "date_histogram": {
        "field":"Name the field you want to aggregate on here",
        "calendar_interval": "Specify the interval here"
      }
    }
  }
}

The following example is similar to the previous aggregations request we have sent.

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_by_month": {
      "date_histogram": {
        "field": "InvoiceDate",
        "calendar_interval": "1M"
      }
    }
  }
}

However, we name the aggregations "transactions_by_month". Then, we set the type of aggregation to "date_histogram".

We instruct Elasticsearch to perform date_histogram aggregation on the field "InvoiceDate" and to split the documents into buckets at a "calendar_interval" of 1 month.

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
Elasticsearch creates monthly buckets. Within each bucket, the starting date and time of each monthly bucket is included in the field "key_as_string".

The field "key" shows the date and time represented as a timestamp.

The field "doc_count" shows the number of documents that fall within the time interval.

Bucket sorting for date_histogram aggregation
Take a look at the response above. You will notice that these buckets were sorted in ascending order of dates.

The field "key" shows the date and time represented as timestamps.

By default, the date_histogram aggregation sorts buckets based on the "key"
values in ascending order.

To reverse this order, you can add an order parameter to the aggregations as shown below. Then, specify that you want to sort buckets based on the "_key" values in descending(desc) order.

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_by_month": {
      "date_histogram": {
        "field": "InvoiceDate",
        "calendar_interval": "1M",
        "order": {
          "_key": "desc"
        }
      }
    }
  }
}

Expected response from Elasticsearch:
You will see that buckets are now sorted to return the most recent interval first.

Histogram Aggregation

With the date_histogram aggregation, we were able to create buckets based on time intervals.

With the histogram aggregation, we can create buckets based on any numerical interval.

For example, let's say we wanted to create buckets based on price interval that increases in increments of 10.

Syntax:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "Name your aggregations here": {
      "histogram": {
        "field":"Name the field you want to aggregate on here",
        "interval": Specify the interval here
      }
    }
  }
}

The following example is similar to the last request.

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_per_price_interval": {
      "histogram": {
        "field": "UnitPrice",
        "interval": 10
      }
    }
  }
}

The differences are that we are naming the aggregations "transactions_per_price_interval".

We instruct Elasticsearch to run a histogram aggregation on the field "UnitPrice" and configure the price interval to increase in increments of 10.

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
Elasticsearch returns an array of buckets where each bucket represents a price interval("key").

Each interval increases in increments of 10 in unit price. It also includes the number of documents placed in each bucket("doc_count").

In the first price interval, there are more than 400,000 transactions for items priced within this interval.

In the next price interval, there are over 20,000 transactions.

It seems like the higher we go up in price interval, the number of transactions decreases. Could this be a pattern?!

This might be something worth exploring if we are looking to improve our sales strategy.

Bucket sorting for histogram aggregation
Similar to the date_histogram aggregation, the histogram aggregation sorts the buckets based on the "key" values as well.

By default, the histogram aggregation sorts buckets based on the "key" values in ascending order.

But what if we wanted to sort by descending order?

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_per_price_interval": {
      "histogram": {
        "field": "UnitPrice",
        "interval": 10,
        "order": {
          "_key": "desc"
        }
      }
    }
  }
}

All you have to do is to add the order parameter as shown above. Then, specify that you want to sort by "_key" values in descending("desc") order.

This way, the highest price interval is listed first!

Copy and paste the example into the console and send the request!

Expected response from Elasticsearch:
You will see that the buckets are now sorted to return the price intervals in descending order.

Range Aggregation

The range aggregation is similar to the histogram aggregation in that it can create buckets based on any numerical interval.

The difference is that the range aggregation allows you to define intervals of varying sizes so you can customize it to your use case.

For example, what if you wanted to know the number of transactions for items from varying price ranges(between 0 and $50, between $50-$200, and between $200 and up)?

Syntax:

GET Enter_name_of_the_index_here/_search
{
  "size": 0,
  "aggs": {
   "Name your aggregations here": {
      "range": {
        "field": "Name the field you want to aggregate on here",
        "ranges": [
          {
            "to": x
          },
          {
            "from": x,
            "to": y
          },
          {
            "from": z
          }
        ]
      }
    }
  }
}

The following example is similar to the last request.
Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_per_custom_price_ranges": {
      "range": {
        "field": "UnitPrice",
        "ranges": [
          {
            "to": 50
          },
          {
            "from": 50,
            "to": 200
          },
          {
            "from": 200
          }
        ]
      }
    }
  }
}

The differences are that we are naming the aggregations "transactions_per_custom_price_ranges".

We run the range aggregation on the field "UnitPrice". Then, we provide the ranges below(0 to 50, 50 to 200, 200 and up).

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
Elasticsearch returns an array of buckets where each bucket represents a customized price interval("key"). It also includes the number of documents("doc_count") placed in each bucket.

We see that over 400,000 transactions have occurred for items priced between 0 to 50.

855 transactions have occurred for items priced between 50-200.

307 transactions have occurred for items priced from 200 and up.

At this point you might be wondering if you can sort the range aggregation.

Bucket sorting for range aggregation
The range aggregation is sorted based on the input ranges you specify and it cannot be sorted any other way!

Terms Aggregation

The terms aggregation creates a new bucket for every unique term it encounters for the specified field. It is often used to find the most frequently found terms in a specified field of a document.

Syntax:

GET Enter_name_of_the_index_here/_search
{
  "aggs": {
    "Name your aggregations here": {
      "terms": {
        "field": "Name the field you want to aggregate on here",
        "size": State how many top results you want returned here
      }
    }
  }
}

For example, let's say you want to identify 5 customers with the highest number of transactions(documents).

Each document is a transaction and each transaction includes a customer ID.

Therefore, if we find the 5 most frequently occurring customer IDs, we will find our top 5 customers.

To do this, we send the following example request.

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "top_5_customers": {
      "terms": {
        "field": "CustomerID",
        "size": 5
      }
    }
  }
}

We name the aggregations as "top_5_customers". We specify that we want to perform terms aggregation on the field "CustomerID".

Since we only want the top 5 customers, we set the size parameter within the terms aggregation to 5.

Copy and paste the example into the Kibana console and send the request!

Expected response from Elasticsearch:
Elasticsearch will return 5 customer IDs("key") with the highest number of transactions("doc_count").

Bucket sorting for terms aggregation
By default, the terms aggregation sorts buckets based on the "doc_count"
values in descending order.

But what if you want to sort the results in ascending order?

You would send the following request!
Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "5_customers_with_lowest_number_of_transactions": {
      "terms": {
        "field": "CustomerID",
        "size": 5,
        "order": {
          "_count": "asc"
        }
      }
    }
  }
}

When we ask Elasticsearch to sort the buckets in ascending order, it will display customers IDs with the lowest number of documents. In other words, customers with the lowest number of transactions.

To account for that, we name our aggregations "5_customers_with_lowest_number_of_transactions". Then, we instruct Elasticsearch to perform "terms" aggregations on the field "CustomerID" and that we want 5 buckets returned("size": 5).

Then, we add an "order" parameter to the "terms" aggregation and specify that we want to sort buckets based on the "_count" values in ascending("asc") order!

Copy and paste this example into the Kibana console and send the request!

Expected response from Elasticsearch:
You will see that the buckets are now sorted in ascending order of "doc_count", showing buckets with the lowest "doc_count" first.

Combined Aggregations

So far, we have ran metric aggregations or bucket aggregations to answer simple questions.

There will be times when we will ask more complex questions that require running combinations of these aggregations.

For example, let's say we wanted to know the sum of revenue per day.

To get the answer, we need to first split our data into daily buckets(date_histogram aggregation).

Within each bucket, we need to perform metric aggregations to calculate the daily revenue.

The combined aggregations request looks like the following.

Calculate the daily revenue

Let's break this down.

We let Elasticsearch know that we are sending an aggregations request(1).

In order to calculate daily revenue, we first need to split documents into daily buckets. So we name this aggregations request "transactions_per_day"(2).

Since we are creating buckets based on time intervals, we run a date_histogram aggregation(3) on the field "InvoiceDate"(4).

Then, we set the "calendar_interval" to a "day"(5).

Thus far, we have split the documents into daily buckets.

Within the "transactions_per_day" aggregations, we create a sub-aggregations(6) called "daily_revenue"(7). This will calculate the total revenue generated each day.

You will notice this aggregation looks a little different.

It uses the "script"(9) to calculate the daily revenue. What is that all about?

Let’s take a look at the fields of our transaction data again.

When we look at our transaction document(lines 22-31), you will see that it lists the "Quantity"(line 24) of an item sold and the "UnitPrice"(line 27) of that item.

But we do not see the total revenue from this transaction. So to get the total revenue, we need to multiply the "Quantity" of item sold by its "UnitPrice".

This is where the script(9) comes in.

Script is used to dynamically create something in Elasticsearch.

In our context, it is used to dynamically calculate the total revenue per transaction.

In the "source" field(10), we instruct Elasticsearch that for each document(doc) in the daily bucket, get the value of the field "UnitPrice" and multiply that by the value of the field "Quantity".

That will calculate the total revenue of each transaction in our bucket. Then, we tell Elasticsearch to "sum"(8) up the total revenue from all the transactions in our bucket to calculate the daily_revenue(7).

Copy and paste the following example into the Kibana console and send the request!

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_per_day": {
      "date_histogram": {
        "field": "InvoiceDate",
        "calendar_interval": "day"
      },
      "aggs": {
        "daily_revenue": {
          "sum": {
            "script": {
              "source": "doc['UnitPrice'].value * doc['Quantity'].value"
            }
          }
        }
      }
    }
  }
}

Expected Response from Elasticsearch:
Elasticsearch returns an array of daily buckets.

Within each bucket, it shows the number of documents("doc_count") within each bucket as well as the revenue generated from each day("daily_revenue").

Calculating multiple metrics per bucket

You can also calculate multiple metrics per bucket.

For example, let's say you wanted to calculate the daily revenue and the number of unique customers per day in one go. To do this, you can add multiple metric aggregations per bucket as shown below!

The request above is almost identical to the last request except that we added an additional metric aggregations called "number_of_unique_customers_per_day"(2) to the sub-aggregations(1).

To calculate this value, we instruct Elasticsearch to perform cardinality aggregations(3) on the field "CustomerID"(4).

Copy and paste the following example into the Kibana console and send the request!

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_per_day": {
      "date_histogram": {
        "field": "InvoiceDate",
        "calendar_interval": "day"
      },
      "aggs": {
        "daily_revenue": {
          "sum": {
            "script": {
              "source": "doc['UnitPrice'].value * doc['Quantity'].value"
            }
          }
        },
        "number_of_unique_customers_per_day": {
          "cardinality": {
            "field": "CustomerID"
          }
        }
      }
    }
  }
}

Expected Response from Elasticsearch:
Elasticsearch returns an array of daily buckets.

Within each bucket, you will see that the "number_of_unique_customers_per_day" and the "daily_revenue" have been calculated for each day!

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_per_day": {
      "date_histogram": {
        "field": "InvoiceDate",
        "calendar_interval": "day"
      },
      "aggs": {
        "daily_revenue": {
          "sum": {
            "script": {
              "source": "doc['UnitPrice'].value * doc['Quantity'].value"
            }
          }
        },
        "number_of_unique_customers_per_day": {
          "cardinality": {
            "field": "CustomerID"
          }
        }
      }
    }
  }
}

Sorting by metric value of a sub-aggregation
You do not always need to sort by time interval, numerical interval, or by doc_count! You can also sort by metric value of sub-aggregations.

Let's take a look at the request below. Within the sub-aggregation, metric values "daily_revenue"(3) and "number_of_unique_customers_per_day"(4) are calculated.

Let's say you wanted to find which day had the highest daily revenue to date!

All you need to do is to add the "order" parameter(1) and sort buckets based on the metric value of "daily_revenue" in descending("desc") order(2)!

Copy and paste the following example into the console and send the request!

Example:

GET ecommerce_data/_search
{
  "size": 0,
  "aggs": {
    "transactions_per_day": {
      "date_histogram": {
        "field": "InvoiceDate",
        "calendar_interval": "day",
        "order": {
          "daily_revenue": "desc"
        }
      },
      "aggs": {
        "daily_revenue": {
          "sum": {
            "script": {
              "source": "doc['UnitPrice'].value * doc['Quantity'].value"
            }
          }
        },
        "number_of_unique_customers_per_day": {
          "cardinality": {
            "field": "CustomerID"
          }
        }
      }
    }
  }
}

Expected response from Elasticsearch:
You will see that the response is no longer sorted by date. The buckets are now sorted to return the highest daily revenue first!

Great job! You have mastered the basics of running aggregations requests with Elasticsearch and Kibana.

Try to run various metric aggregations, bucket aggregations, and combined aggregations on your own and see what type of insights you can find!

Elastic Anomaly Detection - Learning Process and Anomaly Score

Priscilla Parodi — Mon, 02 Aug 2021 14:39:44 +0000

| Menu | Next Post: Elastic Anomaly Detection - Categorization |

As the name suggests, the algorithm needs to identify anomalies in the data.

But how does the model identify anomalies?

How do we identify anomalies?

For example, considering the image bellow.

What is abnormal in this image?

What if I add something to this image? Now, considering the updated image below, what is abnormal?

It was probably easier with the second image because the cat is not a dog, making the cat the anomaly in this image, for most people. In this process, you are identifying patterns.

Identifying patterns is an essential part of our learning process, but the answers are not necessarily obvious, because you know what a cat is and what a dog is, not from the pictures I showed you, I never told you this, but because you learned it during your life.

We must always remember that the algorithms will only process the data that you choose to share.

In the case of a child who is still learning the difference between a dog and a cow, for example, we might receive the answer that all animals in the image belong to the same category: animals. This answer is not incorrect; it simply applies different criteria based on similar characteristics observed in the available data.

If we are seeking a more specific answer, considering all possible details, variables, and behavior, we need to ensure that all data that could contribute to the answer is analyzed over time. The more data we have, the better our understanding will be.

In the case of a child, for them to identify the cat as 'abnormal' they would need more examples, more data would need to be “analyzed” over time. The conclusion is the same for the algorithms.

Based on this information, you already know that the question 'What is abnormal?' is answered by taking into account what is considered normal (which can vary), and to determine what is normal, the algorithm identifies patterns over time.

There are multiple types of Anomaly Detection analyses available in Elastic's ML solution, including:

Single Metric analysis, for jobs that analyze a single time series;
Multi-Metric analysis, to split a single time series into multiple time series;
Population analysis, to identify abnormal behaviors in a homogeneous "population" over a period of time;
Categorization analysis, which is a machine learning process that tokenizes a text field, clusters similar to data together, and classifies it into categories;

The Anomaly Detection feature analyzes the input stream of data, models its behavior using techniques to construct a model that best matches your data, and performs analysis based on the detectors you defined in your job, considering possible rules and dates you want to ignore or disqualify from being modeled.

The blue line in the chart represents the actual data values, while the shaded blue area represents the bounds for the expected values. Initially, the range of expected values is wide due to a limited amount of data in the analyzed time period. Consequently, the model fails to capture the periodicity in the data.

After processing more data, a model is built with coefficients that result in expected values close to the actual values. This leads to the shaded blue area being close to the blue line. By comparing the values to this area, we can determine if they fall outside of it and monitor the anomaly score to indicate the severity of potential anomalies.

Anomaly Score

The anomaly score (severity) is a value from 0 to 100, which indicates the significance of the observed anomaly compared to previously seen anomalies. Highly anomalous values are shown in red.

In order to provide a sensible view of the results, an anomaly score is calculated for each bucket time interval (we use the concept of a bucket to divide up a continuous stream of data into batches, between 10 minutes and 1 hour, for processing).

When you review your machine learning results, there is a multi_bucket_impact property that indicates how strongly the final anomaly score is influenced by multi-bucket analysis; anomalies with medium or high impact on multiple buckets are represented with a cross symbol instead of a circle.

| Menu | Next Post: Elastic Anomaly Detection - Categorization |

This post is part of a series that covers Artificial Intelligence with a focus on Elastic's (Creators of Elasticsearch) Machine Learning solution, aiming to introduce and exemplify the possibilities and options available, in addition to addressing the context and usability.

Elastic Anomaly Detection - Categorization

Priscilla Parodi — Mon, 02 Aug 2021 14:39:32 +0000

| Menu | Next Post: Elastic Anomaly Detection and Data Visualizer HandsOn|

For categorization analysis, the learning process is the same, but there are other steps to process the text.

The input data must be a text field, typically containing repeated elements such as log messages because it's not a natural language processing (NLP) and it works best on machine-written messages.

When you create a categorization anomaly detection job, the machine learning model processes the input text into different categories, identifying patterns over time, as you can see in this example:

Input text

Log message:

Jul 20 15:02:19 localhost sshd[8903]: Invalid user admin from 58.218.92.41 port 26062
Jul 20 15:02:19 localhost sshd[8903]: input_userauth_request: invalid user admin [preauth]
Jul 20 15:02:20 localhost sshd[8903]: Connection closed by 58.218.92.41 port 26062 [preauth]
Jul 20 17:10:23 localhost sshd[2074]: Received disconnect from 41.43.112.199 port 41805:11: disconnected by user
Jul 20 17:10:23 localhost sshd[2074]: Disconnected from 41.43.112.199 port 26062
Jul 20 17:10:23 localhost sshd[2072]: pam_unix (sshd:session): session closed for user ec2-user
Jul 20 19:14:55 localhost sshd[8944]: pam_unix (sshd:session): session closed for user ec2-user by (uid=0)
Jul 20 19:17:22 localhost runner: pam_unix(runuser-1:session): session closed for user ec2-user 
Jul 20 19:17:22 localhost runner: pam_unix(runuser-1:session): session opened for user ec2-user by (uid=0)
Jul 20 19:17:23 localhost runner: pam_unix(runuser-1:session): session closed for user ec2-user

Step 1 - Remove mutable text

Mutable texts are not taken into account to not identify an anomaly or a pattern where there is no relevance as the value is always changing, e.g, date and time.

localhost sshd: Invalid user from port
localhost sshd: input_userauth_request: invalid user [preauth]
localhost sshd: Connection closed by port [preauth]
localhost sshd: Received disconnect from port disconnected by user
localhost sshd: Disconnected from port
localhost sshd: pam_unix session: session closed for user ec2-user
localhost sshd[8944]: pam_unix session: session closed for user ec2-user by (uid=0)
localhost runner: pam_unix session: session closed for user ec2-user 
localhost runner: pam_unix session: session opened for user ec2-user by (uid=0)
localhost runner: pam_unix session: session closed for user ec2-user

Step 2 - cluster similar messages together

Which can mean a line or several lines that are part of a task, for example, and that are respecting a pattern.

->mlcategory:1
localhost sshd: Invalid user from port

->mlcategory:2
localhost sshd: input_userauth_request: invalid user [preauth]

->mlcategory:3
localhost sshd: Connection closed by port [preauth]

->mlcategory:4
localhost sshd: Received disconnect from port disconnected by user

->mlcategory:5
localhost sshd: Disconnected from port

->mlcategory:6
localhost sshd: pam_unix session: session closed for user ec2-user localhost sshd[8944]: pam_unix session: session closed for user ec2-user by (uid=0) localhost runner: pam_unix session: session closed for user ec2-user localhost runner: pam_unix session: session opened for user ec2-user by (uid=0) localhost runner: pam_unix session: session closed for user ec2-user

Step 3 - Count per time bucket

By processing analyzing time buckets, the behavior in a cluster can be better and easily identified for anomaly checking.

In the image below you can see an example of the graphic behavior of each ml category over time for a further time bucket analysis:

As an example, at a specific time bucket, we could see an mlcategory:1 followed by an mlcategory:4, twice:

mlcategory:1 -> mlcategory:4 -> mlcategory:1 -> mlcategory:4.

We could call it bucket 1, as a reference, and so on, bucket 2...

| Menu | Next Post: Elastic Anomaly Detection and Data Visualizer HandsOn|

This post is part of a series that covers Artificial Intelligence with a focus on Elastic's (Creators of Elasticsearch) Machine Learning solution, aiming to introduce and exemplify the possibilities and options available, in addition to addressing the context and usability.

Elastic Anomaly Detection and Data Visualizer HandsOn

Priscilla Parodi — Mon, 02 Aug 2021 14:39:16 +0000

| Menu | Next Post: Elastic Data Frame - Outlier Detection |

Note: This HandsOn assumes that you have already followed the step-by-step Setup of your Elastic Cloud account and added the Samples available there to replicate the analysis mentioned here. If not, please, follow the steps mentioned there.

We've added a Sample containing data you don't know anything about.

As mentioned here, it is very important to know the data and type of data we have in order to know what kind of analysis we can do.

So, before proceeding with Anomaly Detection we can, for example, use Data Visualizer Kibana>Machine Learning>Data visualizer to understand more about the available data.

In a real use case you may prefer to change some field, mapping or wrong/empty data, but in this case we will use the available data exactly as it is.

Let's select the Index Pattern [eCommerce] Orders. When the page loads you usually won't be able to see any data because the time interval initially set is too short (15 minutes) for this type of data that is not continuously being updated, for you to access all available data click on Use full kibana_sample_data_ecommerce_data.

Now we have some important information, the data type (text/keyword/geo_point/number...), % of documents containing that data type, distintic values, distributions and the possibility to visualize data in a graph Actions, something we won't cover at this point. This can help us with the most important part before analyzing data, which is: What answer don't we have? What is this data not telling us that it would be important to know based on the needs that I/my company have?

Let's pretend we are an Ecommerce owner and we want to analyze our data, the first thing we can notice is that we only have 1 month data May 26, 2021 -> Jun 26, 2021 which may not be positive for a complete analysis because we have holidays and events that can momentarily change customer behavior.

Maybe it's our first month with the company, we could do a full analysis after a few years and possibly map and add rules to skip holidays and momentary events, but now what we want is to start an analysis that will be useful at this point, clearly, with limited possibilities.

Something that can make sense is to understand who our customers are, after all, in just 1 month we already have a total of 4675 events with 3321 distinct values for customer_full_name.keyword, which means that we have a good amount of unique customers.

Another thing that stands out is that these customers are from different continents and countries, we are a global ecommerce company. But how does this population spend their money? Do they behave similarly?

Let's run an Anomaly Detection analysis: Machine Learning>Anomaly Detection>Create job and select [eCommerce] Orders then, click Use a Wizard>Population

Again, let’s use the full sample, click on Use full kibana_sample_data_ecommerce_data>Next

To define a population we need a relationship between the data, not necessarily with the same value for all, but we need at least one field that is common, which characterizes the data as belonging to the same population.

For the Population field, let's add some location data, let’s use the geoip.region_name.

Now, considering a population characterized by groups from different regions, we want to know how these people spend their money, so as a metric to identify abnormal behavior, let's add the sum of taxful_total_price, to understand if there are abnormal behaviors in the sum of the total amount spent over time.

Your screen should look like this image:

We can also add the Bucket Span on the left side, usually the Bucket Span is 15 minutes, but you can click Estimate bucket span and based on your data automatically set a good interval for time series analysis. In my case the estimated time was 3h.

And on the right side you will see the Influencers, if you want to see the influence of other fields on the result, and as you can see, the region will already be there.

Click Next and then define a name for the Job ID, I used the name: pop_price_region. At this time we are not going to add additional or advanced settings. After that, click Next and your webpage should look like this image:

Click Next one more time if your webpage looks like this, otherwise check the error message. Finally, click Create Job.

After loading, click on View Results, in this case we don't want the Job running in real time, otherwise you could do that.

A new page will load and as you can see, we don't have an Anomaly Score > 75, which means we don't have one high-severity event, but we do have two anomalies > 50, in orange.

The two events with severity>50, taking into account the population and not just a single metric, came from New York on June 17th 2021 (Current: $998.88 / Typical: $117.59, 8x higher, Probability: 0.00148...), and from Cairo Governorate on June 21st 2021 (Current: $885.97 / Typical: $118.45, 7x higher, Probability: 0.00234), although all this detailed information is important it is worth remembering that the severity value is a normalized value from 0-100 of all these data considering the behavior of the population in the analyzed period of time, which means that only one data, alone, is not necessarily relevant, it is possible to find other purchases with a value also 7x higher with less relevance, for example.

If you want to leave this job running, as mentioned above, you just need to click Machine Learning>Anomaly Detection>(your job)>Start datafeed, you will set the start date and set the end time, select no end time to search in real time and then click Start.

You can also create alerts based on severity and connect to services like Email, IBM Resilient, Jira, Microsoft Teams, Slack or even write to an index or create a Webhook connector. There are also APIs to perform machine learning anomaly detection activities.

| Menu | Next Post: Elastic Data Frame - Outlier Detection |

This post is part of a series that covers Artificial Intelligence with a focus on Elastic's (Creators of Elasticsearch) Machine Learning solution, aiming to introduce and exemplify the possibilities and options available, in addition to addressing the context and usability.

HandsOn Setup - Elastic Cloud

Priscilla Parodi — Mon, 02 Aug 2021 14:39:00 +0000

| Menu |

For HandsOn posts we will use Elastic Cloud. If you don't use Elastic Cloud yet, on this link you can access a 30-day trial. Just add your email to start using your free trial as you can see in the image below.

After creating your account you will not have any deployment available, to create a new one click on Create Deployment as in the image below.

Feel free to choose the settings you prefer but before creating your deployment you need to customize to add the ML node.

Then you can finally create your deployment and open Kibana. At this point don't worry about the settings beyond that.

When Kibana opens you will see a message like the message in image 1 below, suggesting that you start adding data. To proceed implementing the data analysis demonstrated here you will need to add all available samples.

That's it. Now you can proceed with the data analysis examples.

This post is part of a series that covers Artificial Intelligence with a focus on Elastic's (Creators of Elasticsearch, Kibana, Logstash and Beats) Machine Learning solution, aiming to introduce and exemplify the possibilities and options available, in addition to addressing the context and usability.

Elastic Data Frame - Outlier Detection

Priscilla Parodi — Mon, 02 Aug 2021 14:38:34 +0000

| Menu | Next Post: Elastic Data Frame - Regression Analysis |

Unlike the Anomaly Detection models, this is a multi-variate analysis, it enables a better understanding of complex behaviors that are described by many features. For this analysis we have 3 models with different algorithms and learning types (Outlier, Regression and Classification) and in this post we'll talk about Outlier Detection.

Outlier detection identifies unusual data points in the dataset (Unsupervised ML).

When we talk about time series modeling and population anomaly detection, we look for outliers but basing it on how far the metric is from the normal model.

With Outlier Detection we are looking at clusters of data and evaluating density and distance using multi-variate analysis. We are not interested in tracking evolution of this dataset over time like we do in population anomaly detection and there are no buckets.

Evaluation of the Outlier detection

Outliers may denote errors or unusual behavior. In the Elastic Stack, we use an ensemble of four different distance and density based outlier detection methods, based on this approach, a metric is computed called local outlier factor for each data point. The higher the local outlier factor, the more outlying is the data point.

| Menu | Next Post: Elastic Data Frame - Regression Analysis |

This post is part of a series that covers Artificial Intelligence with a focus on Elastic's (Creators of Elasticsearch) Machine Learning solution, aiming to introduce and exemplify the possibilities and options available, in addition to addressing the context and usability.