Elasticsearch Tokenizers – Partial Word Tokenizers

https://grokonez.com/elasticsearch/elasticsearch-tokenizers-partial-word-tokenizers

Elasticsearch Tokenizers – Partial Word Tokenizers

In this tutorial, we're gonna look at 2 tokenizers that can break up text or words into small fragments, for partial word matching: N-Gram Tokenizer and Edge N-Gram Tokenizer.

I. N-Gram Tokenizer

ngram tokenizer does 2 things:

• break up text into words when it encounters specified characters (whitespace, punctuation...)
• emit N-grams of each word of the specified length (quick with length = 2 -> [qu, ui, ic, ck] )

=> N-grams are like a sliding window of continuous letters.

For example:

POST _analyze
{
  "tokenizer": "ngram",
  "text": "Spring 5"
}

It will generate terms with a sliding (1 char min-width, 2 chars max-width) window:

[ "S", "Sp", "p", "pr", "r", "ri", "i", "in", "n", "ng", "g", "g ", " ", " 5", "5" ]

Configuration

• min_gram: minimum length of characters in a gram (min-width of the sliding window). Defaults to 1.
• max_gram: maximum length of characters in a gram (max-width of the sliding window). Defaults to 2.
• token_chars: character classes that will be included in a token. Elasticsearch will split on characters that don’t belong to:
• letter (a, b, ...)
• digit (1, 2, ...)
• whitespace (" ", "\n", ...)
• punctuation (!, ", ...)
• symbol ($, %, ...)

Defaults to [] (keep all characters).

For example, we will create a tokenizer with sliding window (width = 3) and character classes: only letter & digit.

PUT jsa_index_n-gram
{
  "settings": {
    "analysis": {
      "analyzer": {
        "jsa_analyzer": {
          "tokenizer": "jsa_tokenizer"
        }
      },
      "tokenizer": {
        "jsa_tokenizer": {
          "type": "ngram",
          "min_gram": 3,
          "max_gram": 3,
          "token_chars": [
            "letter",
            "digit"
          ]
        }
      }
    }
  }
}

POST jsa_index_n-gram/_analyze
{
  "analyzer": "jsa_analyzer",
  "text": "Tut101: Spring 5"
}

Terms:

More at:

https://grokonez.com/elasticsearch/elasticsearch-tokenizers-partial-word-tokenizers

Elasticsearch Tokenizers – Partial Word Tokenizers