DEV Community: Nico Orfanos

Understanding Elasticsearch Analyzers

Nico Orfanos — Mon, 12 Jun 2023 20:01:29 +0000

If you want to truly understand the analysis process in Elasticsearch, you need to get familiar with analyzers. It's what sets Elasticsearch apart from NoSQL databases like MongoDB.

An analyzer in Elasticsearch is a pipeline. You feed it text, and it gives you back a bunch of tokens.

The analyzer pipeline consists of three steps:

Analyzer
├── 1. Char filters
├── 2. Tokenizer
└── 3. Token filters

Think of them as different stages through which your text flows.

Char filters

First up, we have character filters. These filters preprocess the text before it gets split into tokens by the tokenizer.

For example, it can transform emojis like :) into the word _happy.”

Tokenizer

Next, we have the Tokenizer. The Tokenizer splits the text into smaller units called tokens.

For instance, if we use the whitespace tokenizer on the phrase "Hello World," it would intelligently split it into two tokens: Hello and World.

Token filters

Now that our text is split into tokens, the token filters come to play. They are responsible for applying changes to the generated tokens.

One popular use case is stemming, where the token went is stemmed to go.

Analysis

Bringing it all together, this entire process is referred to as analysis.

Note that analyzers can be customized by configuring different combinations of character filters, tokenizers, and token filters based on your requirements.

Understanding analyzers is like holding the key to the relevancy capabilities in Elasticsearch. It allows you to fine-tune your search queries and ultimately enhance the overall user experience.

Elasticsearch Indices and Documents

Nico Orfanos — Tue, 06 Jun 2023 07:35:53 +0000

To use Elasticsearch effectively, it's essential to understand key concepts like indexes and documents. In this blog post, we'll explore what indexes and documents are in the context of Elasticsearch.

What is an Index?

Imagine a kid's room with a drawer full of toys. In Elasticsearch, an index serves a similar purpose—it acts as a repository for related data. Think of it as a collection of documents that are grouped together for easy searching and retrieval.

To visualize this, let's consider a simple representation of a movies index:

movies
├─ Document 1
├─ Document 2
├─ Document 3
├─ ...

Each individual piece of data within an index is called a Document.

You can create an Index using Sigmie like this:

$index = $sigmie->newIndex('movies')->create();

What is a Document?

A Document is just a JSON stored in an Index.

Document = JSON

In Elasticsearch, a Document is represented as a JSON object and stored within an Index. It contains the actual data that you want to search, analyze, or retrieve.

Here's an example of what a document might look like:

{
   "title": "The Shawshank Redemption"
}

The document contains a single field, title, with the value The Shawshank Redemption.

The JSON structure is allowing us to represent various data types and structures. The Document structure is the most important part of achieving what you want using Elasticsearch.

Here’s how you can create an Instance of an Elasticsearch Document in Sigmie:

use Sigmie\Document\Document;

$document = new Document(['name' => 'The Shawshank Redemption']),

What is Indexing

Indexing is the simple act of adding Documents into an Index. It doesn’t matter what are using Elasticsearch for, indexing is important simply because you can’t do anything without Documents in your index.

And this is the way to index a Document with Sigmie:

$index->collect()->add($document);

An Example

Here's an example demonstrating how to add three movie documents to our "movies" index:

$documents = [
    new Document(['title' => 'The Shawshank Redemption']),
    new Document(['title' => 'Inception']),
    new Document(['title' => 'Pulp Fiction']),
];

$index->collect()->merge($documents);

In the above code, we create an array of Document objects, each representing a movie. By invoking the merge method on the collected index and passing the documents, we add them to the index

Here is what the Index looks like once we merge the Documents.

movies
├─ "The Shawshank Redemption"
├─ "Inception"
├─ "Pulp Fiction"

Run Elasticsearch on your local machine for development

Nico Orfanos — Mon, 05 Jun 2023 07:41:03 +0000

The Open Source version of Elasticsearch is the best solution out there for many things. We can use it in our daily developer lives as a Search Engine to create a fast and relevant search experience for our users or as analytics storage to craft some nice dashboards.

If you want to try and work with Elasticsearch locally this post is for you. I will show you all you need to know to run and manage Elasticsearch on your local machine.

Docker

The easiest way to run and manage Elasticsearch locally is Docker. If you haven't installed Docker yet, follow these steps:

Run this command to download the Docker installation script.

$ curl -fsSL https://get.docker.com -o install-docker.sh

Once you downloaded the installation script run this command to install docker on your system.

$ sudo sh install-docker.sh

Running Elasticsearch

Assuming Docker is up and running, use the following command to run Elasticsearch:

$ docker run -p 9200:9200 --name elasticsearch -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2-amd64

This command pulls the Elasticsearch Open Source (OSS) image tagged with version 7.10.2, binds the ports 9200 to your localhost, and sets the discovery type to "single-node."

Let’s see what all this means.

Elasticsearch operates on port 9200 using the flag -p 9200:9200 we are binding this port from the docker container to our local machine. Making Elasticsearch available on http://localhost:9200.
Using —name elasticsearch we name our docker container so that we can reference it easily later on.
When Elasticsearch is running in a production environment, it’s usually a Cluster of multiple Elasticsearch nodes running. These nodes communicate with each other over the network and are always ready to discover new nodes that want to join the cluster. Since we are running Elasticsearch on our local machine for development purposes it makes sense to disable this behavior. Passing the environmental variable discovery.type=single-node we are telling Elasticsearch that it shouldn’t look for other Elasticsearch nodes in the network.
And last we are pulling the OSS image of Elasticsearch (docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2-amd64). OSS stands for “Open Source Software.” and is the open-source version of Elasticsearch that is more lightweight than the full version and includes all necessary functionalities of the search and analytics engine.

The OOS version is Licensed under the Apache 2.0 license giving us the freedom to use, modify, and distribute Elasticsearch without any licensing restrictions.

Here you can find all the official Elasticsearch docker images provided by Elastic the company behind Elasticsearch.

After the docker run command finishes running you can check if Elasticsearch is also running with curl.

$ curl -X GET http://localhost:9200

If everything is correct you will see this output in your console

{
  "name" : "0a25fd395139",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "nBknoxw0QyG-fHS8j9dW8w",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "docker",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Starting, Stopping, and Restarting Elasticsearch:

Once you have Elasticsearch running as a Docker container, you can control its lifecycle using the following commands:

To stop Elasticsearch:

$ docker stop elasticsearch

This command stops the running Elasticsearch container gracefully.

To start Elasticsearch:

$ docker start elasticsearch

This command starts the previously created Elasticsearch container.

To restart Elasticsearch:

$ docker restart elasticsearch

This command restarts the Elasticsearch container and applies any configuration changes or recovers from crashes.

Dealing with Elasticsearch Crashes

If your Elasticsearch crashes and docker restart elasticsearch will not fix the problem, you can always wipe all Elasticsearch data away and start new.

Elasticsearch stores its data in the /usr/share/elasticsearch/data/nodes folder in the docker container. If you delete this folder you will have a fresh Elasticsearch to work with.

Use the following command to delete the nodes folder within the docker container.

$ docker exec -it elasticsearch rm -rf /usr/share/elasticsearch/data/nodes

This command removes the Elasticsearch data stored in the container’s /usr/share/elasticsearch/data/nodes directory. However, keep in mind that executing this command will result in data loss and you shouldn’t use it if you have data in Elasticsearch that you don’t want to lose.

After deleting the nodes folder, it is essential to restart Elasticsearch to recreate the necessary data structures. Use the docker restart elasticsearch command to do this.

Conclusion:

Running Elasticsearch with Docker provides a convenient way to manage Elasticsearch. By following the steps outlined you can quickly run Elasticsearch using Docker and start playing with it.

Now that you have Elasticsearch running, in a future post, I will show you how you can use Elasticsearch to create a powerful search for your application.

Building and Searching an Elasticsearch Index with Sigmie in PHP

Nico Orfanos — Sun, 04 Jun 2023 11:12:08 +0000

Elasticsearch is a powerful search engine that allows you to index and search through large volumes of data quickly and efficiently. In this blog post, we will explore how to create an Elasticsearch index, populate it with documents, and perform searches using the Sigmie Library in PHP.

FInd the instruction to install Sigmie here.

Connection

Before we can interact with Elasticsearch using Sigmie, we need to set up the Sigmie client. We need to connect to Elasticsearch and create an instance of the Sigmie class.

use Sigmie\Base\Http\ElasticsearchConnection;
use Sigmie\Http\JSONClient;
use Sigmie\Sigmie;

$http = JSONClient::create(['localhost:9200']);

$connection = new ElasticsearchConnection($http);

$sigmie = new Sigmie($connection);

Creating an Index

To begin, we need to set up an Elasticsearch, let’s use the movies Index name. We also need to define the index properties. To do this we create a new instance of the Sigmie\Mappings\NewProperties builder class and specifying two text field, the name and description fields. Also we enable lowercasing that will improve our Search relevance.

use Sigmie\Mappings\NewProperties;

$properties = new NewProperties;
$properties->text('name');
$properties->text('description');

$index = $sigmie->newIndex('movies')
    ->properties($properties)
    ->lowercase()
    ->create();

Populating the Index

Once the index is created, we can populate it with documents. We create an array of the Sigmie\Document\Document class and fill the name and description keys with values.

In this example, we add three documents representing fictional movies.

$index->merge([
    new Document([
        'name' => 'Mickey',
        'description' => 'Adventure in the woods',
    ]),
    new Document([
        'name' => 'Goofy',
        'description' => 'Mickey and his friends',
    ]),
    new Document([
        'name' => 'Donald',
        'description' => 'Chasing Goofy',
    ]),
]);

Performing a Search

After populating the index, we can search it. We create a new search instance, specifying the index name and passing the same properties that we defined earlier. We set the search query string to mickey and specify the fields we want to search and to retrieve. In our example we search only the name field, but we retrieve both name and description. Finally, we execute the search and retrieve the hits as a JSON response.

$search = $sigmie->newSearch('movies')
    ->properties($props)
    ->queryString('mickey')
    ->fields(['name'])
    ->retrieve(['name', 'description'])
    ->get();

$hits = $search->json('hits');

Conclusion
We have covered a simple approach to searching in Elasticsearch using Sigmie. We have seen how to set up an index, populate it with documents, and perform basic search.

However, it is important to note that search is more than just finding a single word in a sea of documents. In future posts, we will explore dive deeper and explore various techniques to enhance your search.