DEV Community: Lisa Jung

Beginner's guide to running Elasticsearch and Kibana v8+ Locally (macOS/Linux and Windows)

Lisa Jung — Wed, 30 Nov 2022 16:29:50 +0000

As of 8.0, Elastic Stack security is on by default for self-managed clusters!

When you download and run Elasticsearch and Kibana v8+ for the first time, you will notice that credentials for the elastic superuser and a unique enrollment token are automatically generated for you.

This blog will walk you through:

using the enrollment token to configure Kibana and connect Kibana to Elasticsearch
logging in as a superuser to start using your new secure deployment.

Resources

Download pages for Elasticsearch and Kibana

The directions for running Elasticsearch and Kibana using macOS/Linux and Windows are slightly different and are linked below for your convenience!

macOS and Linux
Windows

If you want more in depth overview of Elasticsearch and Kibana, check out my blog on Beginner's Crash Course to Elastic Stack Series.

For macOS and Linux

Elasticsearch

Step 1: Download Elasticsearch

Go to the Elasticsearch download page.

Choose the appropriate platform (green box) and click on the download option (orange box) for Elasticsearch.

Elasticsearch will begin to download (red box).

Step 2: Unzip Elasticsearch

For this tutorial, I dragged and dropped Elasticsearch to my desktop.

Go to the desktop and unzip Elasticsearch by double clicking on it (blue box).

Once Elasticsearch is unzipped, you will see a blue folder named elasticsearch-version # on your desktop.

Step 3: Start Elasticsearch

Using the command line, we will cd into the unzipped Elasticsearch folder on the desktop and start Elasticsearch!

Open a terminal of your choice. From your home directory, cd into the Desktop by executing the following command.

# In the terminal of your choice 
cd Desktop

Change into the unzipped Elasticsearch directory by executing the following command.

#In the terminal
cd elasticsearch-8.5.2

You will see that you are now in the elasticsearch directory (orange box).

Heads up
At the time of this blog creation, the latest version of Elasticsearch was 8.5.2 (yellow box).

Note that by the time you come across this blog, a newer version of Elasticsearch may have been released. Therefore, your Elasticsearch directory may reflect a different version number. Make sure to cd into the correct directory.

In the terminal, execute the following command to start Elasticsearch.

# In the terminal
bin/elasticsearch

You will see the cursor blinking for a while before Elasticsearch starts running!

Keep this terminal open to continue running Elasticsearch for the rest of the tutorial.

Step 4: Find and save the credentials for Elasticsearch and Kibana

In the same terminal, either scroll down to the end or use "command + f" to find the term password.

Save the generated password for the elastic user and the enrollment token for Kibana in a secure location. These values are shown only once when you start Elasticsearch for the first time.

Also, note that the enrollment token for Kibana is only valid for the next 30 min!

Kibana

Next, we will download Kibana. Using the enrollment token, we will configure Kibana and connect it to Elasticsearch.

Step 1: Download Kibana

Go to the Kibana download page.

Choose the appropriate platform (green box) and click on the download button (orange) for Kibana.

Kibana will begin to download (red box).

Step 2: Unzip Kibana

For this tutorial, I dragged and dropped Kibana to my desktop.

Go to the desktop and unzip Kibana by double clicking on it (blue box).

Once Kibana is unzipped, you will see a blue folder named kibana-version # on your desktop.

Step 3: Start Kibana

By using the command line, we will cd into the unzipped Kibana directory on the desktop and start Kibana!

Open a new tab in your terminal. From your home directory, execute the following command to cd into to the Desktop.

# In a new tab of the terminal
cd Desktop

Change into the unzipped Kibana directory by executing the following command.

# In the terminal
cd kibana-8.5.2

You will see that you are now in the Kibana directory (orange box).

Heads up
At the time of this blog creation, the latest version of Kibana was 8.5.2.

Note that by the time you come across this blog, a newer version of Kibana may have been released. Therefore, your Kibana directory may reflect a different version number. Make sure to cd into the correct directory!

In the terminal, execute the following command start Kibana.

# In the terminal
bin/kibana

You will see the cursor blinking for a while before Kibana starts running!

When you look at the log, you will see the message "Kibana has not been configured" and it will point to a URL that will help you configure Kibana and connect it to Elasticsearch (red box).

Copy and paste the URL (red box) into a browser and hit enter.

You will see the following page.

Paste the enrollment token you had saved earlier and click on the "Configure Elastic" option (blue button).

Kibana will automatically set up everything and will connect over TLS to Elasticsearch.

When the set up completes, it will display the login page.

Use the username "elastic" and enter the unique password you had saved from the Elasticsearch section.

When you log in, you will see the following page. Click on the "Explore on my own" option (pink box).

It will take you to the Kibana home page. You are now ready to start using your new secure deployment!

Step 4: Start exploring what you can do with Elasticsearch and Kibana

The step by step guide is covered in my Beginner's Crash Course to Elastic Stack Series.

Part 1
This workshop will show you how you can perform CRUD (Create, Read, Update, and Delete) operations with Elasticsearch and Kibana.

Part 1 GitHub repo contains:

the video recording of the workshop. Timestamp 25:03 will take you directly to the CRUD operations section.
the requests needed to perform CRUD operations with Elasticsearch and Kibana.

Part 2
This workshop will show you how you can send search queries from Kibana to Elasticsearch and how you can fine-tune the relevance of your search results.

Part 2 GitHub repo contains the video recording of this workshops as well as all the queries and aggregations requests covered during the workshop.

Check out the following links to explore these topics after setting up Elasticsearch and Kibana:

There you have it. Now, go explore what you can do!

For Windows

Elasticsearch

Step 1: Download Elasticsearch

Go to the Elasticsearch download page.

Choose the appropriate platform (green box) and click on the download option (orange box) for Elasticsearch.

Elasticsearch will begin to download (red box).

Step 2: Unzip Elasticsearch

For this tutorial, I dragged and dropped Elasticsearch to my desktop.

Right click on the Elasticsearch directory to display a drop down menu and select the "Extract All" option (red box) .

Click on the "Extract" option (blue box).

Once extraction is complete, you will see a pop up window with an unzipped elasticsearch directory (orange box).

Right click on it to display the drop down menu and click on the "Open in Terminal" option (green box).

You will see the Elasticsearch directory open in your terminal.

In the terminal, execute the following command to start Elasticsearch.

bin\elasticsearch.bat

You will see the cursor blinking for a while before Elasticsearch starts running!

Keep this terminal open to continue running Elasticsearch for the rest of the tutorial.

Step 4: Find and save the credentials for Elasticsearch and Kibana

In the same terminal, scroll down to the end and you will see the password and enrollment token displayed.

Save the generated password for the elastic user and the enrollment token for Kibana in a secure location. These values are shown only once when you start Elasticsearch for the first time.

Also, note that the enrollment token for Kibana is only valid for the next 30 min!

Kibana

Next, we will download Kibana. Using the enrollment token, we will configure Kibana and connect it to Elasticsearch.

Step 1: Download Kibana

Go to the Kibana download page.

Choose the appropriate platform (green box) and click on the download button (orange) for Kibana.

Kibana will begin to download (red box).

Step 2: Unzip Kibana

For this tutorial, I dragged and dropped Kibana to my desktop.

Right click on the Kibana directory to display the drop down menu and select the "Extract All" option (red box) .

Click on the "Extract" option (blue box).

Step 3: Start Kibana

Once the extraction is complete, you will see a pop up window with an unzipped Kibana directory (orange box).

Right click on it to display the drop down menu and click on "Open in Terminal" option (green box).

You will see the terminal open on your screen.

In the terminal, execute the following command to start Kibana.

bin\kibana.bat

You will see the cursor blinking for a while before Kibana starts running!

When you look at the log, you will see the message "Kibana has not been configured" and it will point to a URL that will help you configure Kibana and connect it to Elasticsearch (red box).

Copy and paste the URL (red box) into a browser and hit enter.

You will see the following page.

Paste the enrollment token you had saved from Elasticsearch section and click on the "Configure Elastic" option (blue button).

Kibana will automatically set up everything and will connect over TLS to Elasticsearch.

When the set up completes, it will display the login page.

Use the username "elastic" and enter the unique password you saved from the Elasticsearch step.

When you log in, you will see the following page. Click on the "Explore on my own" option (pink box).

It will take you to the Kibana home page. You are now ready to start using your new secure deployment!

Step 4: Start exploring what you can do with Elasticsearch and Kibana

The step by step guide is covered in my Beginner's Crash Course to Elastic Stack Series.

Part 1
This workshop will show you how you can perform CRUD(Create, Read, Update, and Delete) operations with Elasticsearch and Kibana.

Part 1 GitHub repo contains:

the video recording of the workshop. Timestamp 25:03 will take you directly to the CRUD operations section.
the requests needed to perform CRUD operations with Elasticsearch and Kibana.

Part 2
This workshop will show you how you can send search queries from Kibana to Elasticsearch and how you can fine-tune the relevance of your search results.

Part 2 GitHub repo contains the video recording of this workshops as well as all the queries and aggregations requests covered during the workshop.

Check out the following links to explore these topics after setting up Elasticsearch and Kibana:

There you have it. Now, go explore what you can do!

Part 10: Visualize data using Kibana Lens

Lisa Jung — Wed, 07 Sep 2022 15:15:21 +0000

Table of Content

In this blog series, we have created a full stack app (Node.js and React)that enables users to search for earthquake data stored in Elasticsearch.

Searching for data is one way to learn about our dataset.

But we can learn so much more when we can visualize our data.

To wrap up this series, we will learn how to use Kibana Lens to analyze and visualize earthquake data stored in Elasticsearch!

Resources

Would you rather watch a video to learn this content? Click on the link below!

Episode 10: Visualize data with Kibana Lens

What are we creating today?

We will create a Kibana dashboard that contains panels of various data visualizations.

Table

In the upper left corner(red box), we have a table that lists 10 locations with the highest frequency of earthquakes.

Heat Map

To the right(orange box), we have an earthquake heat map. This allows you to view the world map and visualize the location and the frequency of earthquakes by color.

Metrics

In the row below(yellow box), we have high level metrics about our earthquake data.

It shows you the number of earthquakes in our dataset, the maximum magnitude, and the average depth of earthquakes in our data.

Histogram

In the next row, we have the earthquake magnitude histogram(green box). This graph displays different levels magnitude that exist in our dataset and how many earthquakes have exhibited this magnitude.

Donut Chart

To the right, we have a donut chart(blue box) that shows the types and percentage of quakes present in our dataset.

Creating the Dashboard

Let’s go over how we can create this dashboard with Kibana.

Step 1: Open the Kibana home page

From the home page, click on the menu icon(red box). From the drop down menu, click on the Dashboard option(blue box).

Step 2: Create a dashboard

Kibana dashboard lets you easily create views that pull together charts, maps, and filters to display the full picture of your Elasticsearch data(excerpt from Kibana Dashboard page).

Click on the Create a dashboard option(pink box).

It will take you to the dashboard where you can add and edit visualizations.

Step 3: Create a table

The first visualization we will create is a table that lists 10 locations with the highest frequency of earthquakes.

From the dashboard page, click on the Select type option(blue box).

It will display a drop down menu with various visualization tools.

From the drop down menu, choose the Lens option(red box).

Kibana Lens is an intuitive UI that helps you create data visualization with ease!

Whenever possible, start with Lens as you can create majority of the visualizations with this tool.

First thing we must do is to select the correct data source and the time range in which we want to visualize the data. All of the panels in the dashboard will reflect earthquake data that falls within the chosen time range.

Data Source
You will see that the dashboard has already chosen the earthquakes data view(red box) we have created in part 7.

If you have multiple data views, make sure the correct data source is selected.

Time Range
Click on the calendar icon(red box) to display the drop down menu.

In part 7, we have retrieved USGS earthquake data from the past 30 days.

Select the Last 30 days option(green box).

Table
Click on the downward arrow(red box) to display the menu of visualization type. Select the Table option(blue box).

We are creating a table that lists 10 locations with the highest frequency of earthquakes. Since location data is stored in the field place, drag and drop the field place.keyword (red box) into the editor(blue box).

It will automatically create a table for you with two columns(red and blue boxes).

The column Top 5 values of place.keyword(red box) lists 5 locations with the highest document(i.e. earthquake) counts.

The column Count of Records(blue box) lists the number of documents(i.e. earthquakes) that have occurred in that particular location.

Let's look at the table that we want to create.

Our final table contains two columns called Location by frequency and # of earthquakes.

It lists 10 locations with the highest frequency of earthquakes in descending order.

We will change our current table to mirror the final table.

Column: Top 5 values of place.keyword
From the table page, click on the Top 5 values of place.keyword option(blue box).

It will display a pop up menu where you can customize the rows of the table.

Red box
You will see that the rows reflect data from the field place.keyword.

Orange box
The number of values displayed is set to 5.

Change this value to 10 to display 10 locations with the highest frequency of earthquakes.

Yellow and green boxes
The rows are ranked by count of records in descending order.

Blue box
The Name is set to Top 5 values of place.keyword.

Change this value to Location by frequency.

The changes you make will be auto saved. You will also see that the table is auto updated as well.

Next, we will update the column Count of records.

Click on the close option(red box).

Column: Count of records
Click on the Count of records option(red box).

From the pop up menu, change the Name to # of earthquakes(red box).

As you make these changes, you will see that the changes are auto saved and the table is auto updated(blue box).

Click on the Save and Return button(purple box) in the upper right corner.

You will see that the table has been added to the dashboard.

Let's add a title to this table.

Click on the [No Title] section(red box).

In the pop up menu, type in "10 locations with the highest frequency of earthquakes"(red box). Click on the Save button(green box).

You will see that the title(red box) has been added to the table panel.

Step 4: Save and name the dashboard

From the dashboard page, click on the Save button(green box) in the upper right corner.

Name the dashboard to whatever it makes sense to you. I named mine "earthquakes_data_visualization"(red box). Click on the Save button(green box).

You will see that our dashboard is now saved and has been named as earthquakes_data_visualization(red box).

Step 5: Create a heat map

With heat maps, you can view the world map and visualize the location and the frequency of earthquakes by color.

It is an effective way visualize earthquake hotspots around the world and easily detect patterns.

From the dashboard page, click on the select type option(red box) and click on Maps(blue box).

The page will display a world map.

Click on the Add layer button(red box).

Click on the Heat map option(blue box).

Click on the downward arrow(red box) to select the earthquakes data view(blue box) we created in part 7 .

You will see that Lens automatically selected the field coordinates(blue box below). This field contains inner fields lat(latitude) and lon(longtude). These fields were mapped to a geo_point to mark the location of earthquakes.

Click on the Add layer button(red box).

It will display a pop up menu where you can further customize the heat map.

To create the heat map shown in the beginning of this blog, set the resolution to low(blue box) and click on the Save & close option(red box).

You will see that the earthquake locations are now marked with different colors on the map.

Warm colors are associated with the maximum earthquake activity, whereas the cool colors are associated with minimum earthquake activity.

By looking at our heat map, we can see that the states California and Utah seem to have frequent occurrences of quakes.

Click on the Save and return button(red box) in the upper right corner.

You will see that the heat map has been added to our earthquakes_data_visualization dashboard.

Let's give this panel a title.

Click on the [No Title] section(red box).

In the pop up menu, type in "Earthquake heat map"(blue box) and click on the Save button(red box).*

You will see that the panel title(red box) has been added.

Step 6: Create metrics panels

Let's calculate and display high level metrics such as the number of earthquakes, maximum magnitude, and average depth.

From the dashboard, click on the Select type option(red box) and click on the Lens option(blue box).

In the next page, click on the downward arrow(red box) to reveal the Visualization type drop down menu. Scroll down on the menu until you see the Metric option(blue box) then click on it.

Number of earthquakes
The first panel we will create is Number of earthquakes.

The number of documents(i.e. earthquakes) that fall within the chosen time range(last 30 days) will give us this value.

To get this value, drag and drop Records(red box) into the editor(orange box).

You will see that we have a total of 8,181 documents that fall within the last 30 days time range.

Let's rename "Count of records"(red box) to "Number of earthquakes" as shown in the final dashboard.

To make the changes, click on "Count of records"(orange box above) to display the pop up menu.

Under the Name section(green box), type in "Number of earthquakes". Click on the Save and return button(pink box).

You will see that the panel for Number of earthquakes has been added to the dashboard.

Max magnitudes
Let's create a panel that displays the maximum magnitude of earthquake that exist in our data.

From the dashboard, click on the Select type option(red box) and click on the Lens option(blue box).

In the next page, click on the downward arrow(red box) and select the Metric option(blue box).

We are creating a panel for maximum magnitude of earthquakes found in our data set.

Drag the field mag(red box) into the editor(orange box).

You will see that it automatically displays "Median of mag"(red box).

However, we want to display the maximum value of magnitude found in our data set.

Click on "Median of mag"(orange box) in the edit menu to change this setting.

In the pop up menu, click on the function Maximum(green box) to get the Maximum magnitude.

In the Name section(orange box), update the name to "Maximum magnitude".

Click on the Save and return button(pink box) in the upper right corner to add this panel to the dashboard.

You will see that the panel Maximum magnitude has been added to the dashboard and that the maximum magnitude of earthquake in our data set is 7!

Average Depth
Let's create a panel for Average depth of earthquakes in our data set.

From the dashboard, click on the Select type option(red box) and click on the Lens option(blue box).

Click on the downward arrow(red box) and select the Metric option(blue box).

We are creating a panel for average depth of earthquakes in our data.

Drag the field depth(red box) into the editor(orange box).

You will see that it automatically displays the "Median of depth"(red box).

However, we want to display the average depth of earthquakes found in our data set.

Click on "Median of depth"(orange box) in the edit menu to change this setting.

In the pop up menu, click on the function Average(green box) to get the average depth of earthquakes.

In the Name section(orange box), update this to "Average depth (km)".

Click on the Save and return button(pink box) in the upper right corner to add this panel to the dashboard.

You will see that the panel Average depth (km) of earthquakes has been added to the dashboard and that the average depth of earthquake in our data set is 20.46 km!

In our final dashboard, we see that these three metrics panels take up a row.

One of the cool things about the dashboard is that you can change the size of these panels and reorganize the panels in any way you want.

You can adjust the size of the panel by grabbing the lower right corner of the panel(green box) and adjusting the size of the panel.

You can move the panel by clicking the upper part of the panel(blue box) then by holding and dragging the panel wherever you wish.

After resizing and reorganizing your metrics panels, your dashboard should look like the following.

Step 7: Create a magnitude histogram

Next, we will create an earthquake magnitude histogram.

This panel displays different levels magnitude that exist in our dataset and how many earthquakes have exhibited this magnitude.

To create this histogram, click on the Select type option(red box) and select the Lens option(blue box).

Click on the downward arrow(red box) then select the Bar vertical option(blue box).

You will see the following page where you can add a field to a horizontal axis(red box) and vertical axis(blue box).

Let's take a look at the magnitude histogram in the final dashboard.

On the horizontal axis, the levels of magnitude are displayed. On the vertical axis, we have the number of earthquakes that exhibit the corresponding magnitude.

Drag the field mag(green box) into the Horizontal axis section(red box).

Drag Records(orange box) into the Vertical axis section(blue box).

You will see that it automatically creates a histogram for you with Count of records(purple box) on the vertical axis and mag(green box) on the horizontal axis.

Let's rename the vertical axis label Count of records to Number of earthquakes.

Click on Count of records(red box) from the edit menu.

In the Name section(purple box), type in "Number of earthquakes".

Let's also change the color of the histogram to pink as shown in the final dashboard.

From the edit menu. click on the Series color option(orange box).

It will display the color options you can apply to the histogram.

Click on the pink color(black arrow).

You will see that the color of the histogram has been changed to pink.

Let's rename the horizontal axis label mag to Magnitude.

Click on the close option(blue box).

Click on mag(orange box) from the edit menu.

*In the Name section(purple box), type in "Magnitude".

Click on the Save and return button(green box) in the upper right corner.*

You will see that the histogram has been added to the dashboard.

You see many earthquakes exhibiting magnitude level that fall between 0.8-1.2, which are pretty mild earthquakes that are not usually felt.

We also observe fewer number of earthquakes exhibiting higher magnitude levels that cause different levels of damage.

For more information on earthquake magnitude level and its effects, check out this resource!

Let's give this panel a title!

Click on the [No Title]section(orange box).

In the pop up menu, type in "Earthquake magnitude histogram"(green box) then click on the Save button(purple box).

You will see that the panel title has been added.

Step 8: Create a donut chart

Lastly, let's create a donut chart that shows the types and percentage of quakes present in our dataset.

From the dashboard page, click on the Select type option(red box) and click on the Lens option(blue box).

In the next page, click on the downward arrow(red box). Scroll down until you see the Donut option(blue box) and click on it.

Let's look at the final Donut chart.

The donut is divided into types of quakes. Each type is represented by a slice of the donut.

Drag the field type(orange box) and drop it into the Slice by section(blue box).

The size of the slice is determined by how many earthquakes(Records) of each type exists in our data set.

Drag and drop Records(red box) into the Slice by section(green box).

You will see that Lens has automatically created a donut chart that shows the type and percentage of quakes that exists in our data.

As you can see, the majority of the quakes present in our data is of type earthquake(98.22%).

Ice quake makes up 0.01% of our data.

Some of the quakes are man made:

quarry blast(1.22%)
explosion(0.54%)
chemical explosion(0.01%)

Let's create a legend for types of quakes and place it in the upper right side of the donut chart panel.

Click on the legend logo(green box).

It will display a drop down menu where you can edit the legend.

In the Display section, click on the Show option(orange box).

You will see a legend(blue box) show up in the upper right corner.

The Alignment section(purple box) allows you to specify where in the panel the legend should be displayed. Since we want the legend in the upper right corner, leave the default option(right facing arrow).

Click on the Save and return button(red box) in the upper right corner to add this panel to the dashboard.

You will see that the donut chart has been added to the dashboard.

If you hover over a slice(red box), it shows the quake type(quarry blast) this slice represents, how many documents(i.e. quakes) are of this type(100), and what percentage of documents are of this type(1.22%).

Let's give this panel a title.

Click on the [No Title] section(blue box).

In the pop up menu, type in "Quake types"(blue box) and click on the Save button(red box).

You will see that the panel title has been added(blue box).

Save the dashboard by clicking on the Save button(red box) in the upper right corner.

Click on the Switch to view mode(orange box).

You will see the dashboard in presentation mode as shown in the final dashboard.

Congratulations! You have completed season 2 of Mini Beginner's Crash Course to Elasticsearch and Kibana!

In this series, you learned how to:

Build a full stack JavaScript app
Connect the app to Elasticsearch
Retrieve API data
Transform the retrieved data
Ingest transformed data into Elasticsearch
Set up the app for search
Display search results for the app user
Visualize data with Kibana Lens

Now that you’ve got the basics under your belt, build on what you've learned and see what you can create with Elasticsearch and Kibana!

Part 9: Set up the Node.js server to handle Elasticsearch requests

Lisa Jung — Wed, 07 Sep 2022 15:15:03 +0000

Table of Content | Read Next: Part 10 - Visualize data with Kibana Lens

Review

In part 8, we have created the client side of the app.

Using the client, the user can search for earthquakes that match the specified criteria(type, magnitude, location, date range).

We have set up our client to capture the user input and send the input to the server.

In this blog, we will set up our server to:

receive the user input from the client and pass the user input into a Elasticsearch request
send the request to Elasticsearch and receive relevant documents from Elasticsearch
send the received documents to the client so the search results could be displayed to the user

Resources

Would you rather watch a video to learn this content? Click on the link below!

Episode 9: Set up the Node.js server to handle Elasticsearch request

Want the code covered in this blog? Click on the link below to access it!

GitHub repo for part 9

Set up the server to handle Elasticsearch request

Using your code editor, open the earthquake_app directory. Within the server directory, locate the server.js file.

In server.js, replace the existing code with the following code.

//in server/server.js'
const { Client } = require('@elastic/elasticsearch');
const client = require('./elasticsearch/client');
const express = require('express');
const cors = require('cors');

const app = express();

const data = require('./data_management/retrieve_and_ingest_data');

app.use('/ingest_data', data);

app.use(cors());

app.get('/results', (req, res) => {
  const passedType = req.query.type;
  const passedMag = req.query.mag;
  const passedLocation = req.query.location;
  const passedDateRange = req.query.dateRange;
  const passedSortOption = req.query.sortOption;

  async function sendESRequest() {
    const body = await client.search({
      index: 'earthquakes',
      body: {
        sort: [
          {
            mag: {
              order: passedSortOption,
            },
          },
        ],
        size: 300,
        query: {
          bool: {
            filter: [
              {
                term: { type: passedType },
              },
              {
                range: {
                  mag: {
                    gte: passedMag,
                  },
                },
              },
              {
                match: { place: passedLocation },
              },
              // for those who use prettier, make sure there is no whitespace.
              {
                range: {
                  '@timestamp': {
                    gte: `now-${passedDateRange}d/d`,
                    lt: 'now/d',
                  },
                },
              },
            ],
          },
        },
      },
    });
    res.json(body.hits.hits);
  }
  sendESRequest();
});

const PORT = process.env.PORT || 3001;

app.listen(PORT, () => console.group(`Server started on ${PORT}`));

Let's go over the newly added code!

Heads up!

For reference purposes only, I have included screenshots of code that I will be explaining.

If you need to copy and paste the code, please refer to the code snippet above or the GitHub repo for part 9.

Line 4
We import cors dependency we have installed in part 2.

We will be using cors to allow our server(http://localhost:3000) and client(http://localhost:3001) of different origin to exchange information without encountering a CORS error.

Line 12
We use app.use(cors()) to enable all CORS requests.

Lines 14-66
We create an endpoint called /results to handle HTTP requests from the client.

Review from part 8

Using the client, the user can search for earthquakes based on the criteria they select(type, magnitude, location, and date range).

The user can also sort the results by descending or ascending order of magnitude.

When the user specifies the criteria and clicks on the search button, the user input is captured and sent to the server.

Lines 15-19 create constants for the user input received from the client(type, mag, location, dateRange and sortOption).

Lines 21-64
We define the sendESRequest() function. This function is designed to send a search request to Elasticsearch.

This search request retrieves earthquake documents that match the user's selected criteria(type, magnitude, location, and time range).

Line 22

We create a constant called body and set it equal to the client.search() method.

Within the client.search() method, we leave instructions for Elasticsearch on what documents we wish to retrieve.

Line 23
We instruct Elasticsearch to search against the index earthquakes.

Line 24
In the request body(lines 24-61), we include the search criteria.

Sort by Magnitude (Lines 25-31)
Using the client, a user can sort the search results by descending or ascending order of magnitude.

To accomplish this, we add the sort parameter(line 25) in the request body. We specify that the search results must be ordered by the value of the field mag(line 27).

The sorting order(desc or asc) is determined by the user input passedSortOption(line 28).

Line 32
We instruct Elasticsearch to retrieve up to 300 matching documents.

Lines 33-60
Our app is designed to retrieve earthquake documents that match the user's chosen criteria(type, mag, location, and dateRange).

The documents must match all four of the user's criteria to be considered as a hit(search result). To retrieve such documents, we will write four queries and combine them into one request.

The bool query becomes super handy when we are trying to accomplish such a task!

We covered this query in season 1 of Mini Beginner's Crash Course to Elasticsearch and Kibana. Check out these videos(part 1 & part 2) if you need a refresher!

In a bool query(lines 33-60), we use a combination of one or more clauses to narrow down our search results.

In our case, documents in the earthquakes index either match or do not match user input(yes or no).

Therefore, we use the filter clause(line 35) to retrieve documents that match the user's input of type, magnitude, location, and date range.

Quake Type (lines 36-38)
Using our client, a user can select the type of quake from the drop down menu.

When a user selects an option, the user input is sent to the server and saved as a constant called passedType(line 15).

passedType contains the option value the user has selected from the dropdown menu(earthquake, quarry blast, ice quake, or explosion).

Since we are looking for a specific term(earthquake, quarry blast, ice quake, or explosion) within the document, we use the term query.

We instruct Elasticsearch to look for the term contained in the constant passedType in the field type(line 37).

Magnitude (lines 39-45)
Using our client, a user can select the level of magnitude from the drop down menu.

These options allow the user to search for earthquakes with magnitudes that is greater than or equal to 2.5, 5.5, 6.1, 7, and 8.

When a user selects an option, the user input is sent to the server and saved as a constant called passedMag(line 16).

passedMag contains the value of the chosen option (2.5, 5.5, 6.1, 7, or 8).

Since we are looking for documents that reflect a range of magnitudes, we use the range query for this task.

We run the range query(line 40) on the field mag(line 41). We look for documents that contain values that are greater than or equal(gte) to the value of passedMag(line 42).

Location (lines 46-48)
Using our client, a user can type in the location in which they want to search for earthquakes.

The user input is sent to the server and saved as a constant called passedLocation(line 17).

passedLocation contains whatever text the user has typed in.

To search the field place for the typed user input, full text search should be performed. Therefore, we use the match query to look for passedLocation in the field place(line 47).

Date Range (lines 50-57)
Using our client, a user can choose the date range of earthquake occurrence.

This user input is sent to the server and saved as a constant called passedDateRange(line 18).

passedDateRange contains the date range option the user has selected(past 7 days, past 14 days, past 21 days, and past 30 days).

Since we are looking for documents that reflect a range of dates, we need to use the range query for this task.

We run the range query(line 51) on the field @timestamp(line 52).

Lines 54-55
We instruct Elasticsearch to look for documents whose timestamp falls within the past x days that the user has specified.

At first glance, these lines of code may look complicated so let's break this down.

The term gte in line 53 stands for "greater than or equal to".

The term lt in line 54 stands for "less than".

We use these terms to specify the date range that a document must fall into to be considered as a search result.

The term now/d represents the current timestamp(the time when your request reaches Elasticsearch).

Therefore, lines 54-55 are telling Elasticsearch to find the documents that fall between now and now minus PassedDateRange which is the number of days specified by the user.

As a result, Elasticsearch will look for documents whose timestamp falls within the past x days that the user has specified.

Line 63

When the request is sent to Elasticsearch, Elasticsearch will retrieve relevant documents and send the documents to the server.

Once the documents have been received by the server, the server is instructed to send the documents to the client so it can display the search results.

Line 65
We call the sendESRequest() function so it would execute when the server receives an http request to the /results endpoint.

Final product

Now the moment we have all been waiting for! Let's check out what we have built!

Go back to your terminal.

cd into the earthquake_app directory. Start the server by executing the following command.

//in terminal within the earthquake_app directory
npm start

Screenshot of the terminal:

Add a new tab to your terminal.

cd into the earthquake directory then into the client directory. Start the client by executing the following command.

//in terminal earthquake_app/client
npm start

Screenshot of the terminal:

It should automatically display the home page of the app(http://localhost:3000)!

In the browser, specify the criteria of earthquakes you are looking for and hit the search button.

If Elasticsearch contains the documents you seek, you will see the search results displayed in the form of cards.

You should see that the search results are sorted by the sorting order you have specified!

If Elasticsearch does not have the documents you seek, you will see the following message displayed on the page.

There you have it! You have successfully created a full stack JavaScript app that can search for earthquake data stored in Elasticsearch!

Next, we are going to explore our data even further.

Move on to Part 10 to visualize data with Kibana Lens!

Part 8: Build the client side of the app with React

Lisa Jung — Wed, 07 Sep 2022 15:14:58 +0000

Table of Content | Read Next: Part 9: Set up the Node.js server to send search request to Elasticsearch and receive results

In this blog, we will be creating the client side of our Earthquakes app with React.

Review

The following is what we will build.

Our client allows the user to search for earthquakes based on quake type, magnitude, location, and date range. It also allows the user to sort the search results by ascending or descending order of magnitude.

When the user hits the search button, the user input is sent to the server via HTTP request.

The server passes the user input into a Elasticsearch request and sends the request to Elasticsearch.

Elasticsearch retrieves relevant documents and sends the documents to the server. The server sends the documents to the client.

Upon receiving the documents, the client displays the results in the form of cards. Each card contains information about one earthquake.

Resources

Would you rather watch a video to learn this content? Click on the link below!

Episode 8: Build the client with React

Want the code covered in this blog? Click on the link below to access it!

GitHub repo for part 8

Building the client

Throughout the blog series, we have been working with the earthquake_app directory. In this directory, we have added the server side code under the server directory.

In this blog, we will be adding the client side code to the earthquake_app directory.

Step 1: Create a React app

Open a new tab in the terminal.

cd into the earthquake_app directory and execute the following command.

//in terminal within the earthquake_app directory
npx create-react-app client

Screenshot of the terminal:

This command will create a React app called client within the earthquake_app directory.

Step 2: Install Axios

Once the react app is created, cd into the client directory.

//in terminal within the earthquake_app directory
cd client

Screenshot of the terminal:

We will be using a library called axios to send HTTP requests to our server.

Install axios by executing the following command.

//in terminal earthquake_app/client
npm i axios

Screenshot of the terminal:

From the code editor, expand the client directory to locate the package.json file.

You will see that the library called axios has been installed(line 9).

Step 3: Set up the proxy to our server.

In the same file, we will add the "proxy" key and point the proxy at localhost:3001 where our Node.js server is running.

Add the following code between the "scripts" object(line 20) and the "eslintConfig" object(line 21).

//in client/package.json
"proxy": "http://localhost:3001",

Your package.json file should look like this(line 21):

Step 4: Build the client

When you look within the client directory, you will see the src directory. Expand this directory to access the App.js file.

Replace the default content of App.js with the following snippet:

//in client/src/App.js
import axios from 'axios';
import { useState } from 'react';
import './App.css';

const App = () => {
  const [chosenType, setChosenType] = useState(null);
  const [chosenMag, setChosenMag] = useState(null);
  const [chosenLocation, setChosenLocation] = useState(null);
  const [chosenDateRange, setChosenDateRange] = useState(null);
  const [chosenSortOption, setchosenSortOption] = useState(null);
  const [documents, setDocuments] = useState(null);

  const sendSearchRequest = () => {
    const results = {
      method: 'GET',
      url: 'http://localhost:3001/results',
      params: {
        type: chosenType,
        mag: chosenMag,
        location: chosenLocation,
        dateRange: chosenDateRange,
        sortOption: chosenSortOption,
      },
    };
    axios
      .request(results)
      .then((response) => {
        console.log(response.data);
        setDocuments(response.data);
      })
      .catch((error) => {
        console.error(error);
      });
  };

  return (
    <div className='app'>
      <nav>
        <ul className='nav-bar'>
          <li>Earthquake Watch</li>
        </ul>
      </nav>
      <p className='directions'>
        {' '}
        Search for earthquakes using the following criteria:
      </p>
      <div className='main'>
        <div className='type-selector'>
          <ul>
            <li>
              <select
                name='types'
                id='types'
                value={chosenType}
                onChange={(e) => setChosenType(e.target.value)}
              >
                <option value={null}>Select a Type</option>
                <option value='earthquake'>Earthquake</option>
                <option value='quarry blast'>Quarry Blast</option>
                <option value='ice quake'>Ice Quake</option>
                <option value='explosion'>Explosion</option>
              </select>
            </li>
            <li>
              <select
                name='mag'
                id='mag'
                value={chosenMag}
                onChange={(e) => setChosenMag(e.target.value)}
              >
                <option value={null}>Select magnitude level</option>
                <option value='2.5'>2.5+</option>
                <option value='5.5'>5.5+</option>
                <option value='6.1'>6.1+</option>
                <option value='7'>7+</option>
                <option value='8'>8+</option>
              </select>
            </li>
            <li>
              <form>
                <label>
                  <input
                    className='form'
                    type='text'
                    placeholder='Enter city, state, country'
                    value={chosenLocation}
                    onChange={(e) => setChosenLocation(e.target.value)}
                  />
                </label>
              </form>
            </li>
            <li>
              <select
                name='dateRange'
                id='dateRange'
                value={chosenDateRange}
                onChange={(e) => setChosenDateRange(e.target.value)}
              >
                <option value={null}>Select date range</option>
                <option value='7'>Past 7 Days</option>
                <option value='14'>Past 14 Days</option>
                <option value='21'>Past 21 Days</option>
                <option value='30'>Past 30 Days</option>
              </select>
            </li>
            <li>
              <select
                name='sortOption'
                id='sortOption'
                value={chosenSortOption}
                onChange={(e) => setchosenSortOption(e.target.value)}
              >
                <option value={null}>Sort by</option>
                <option value='desc'>Largest Magnitude First</option>
                <option value='asc'>Smallest Magnitude First</option>
              </select>
            </li>
            <li>
              <button onClick={sendSearchRequest}>Search</button>
            </li>
          </ul>
        </div>
        {documents && (
          <div className='search-results'>
            {documents.length > 0 ? (
              <p> Number of hits: {documents.length}</p>
            ) : (
              <p> No results found. Try broadening your search criteria.</p>
            )}
            {documents.map((document) => (
              <div className='results-card'>
                <div className='results-text'>
                  <p>Type: {document._source.type}</p>
                  <p>Time: {document._source['@timestamp']}</p>
                  <p>Location: {document._source.place}</p>
                  <p>Latitude: {document._source.coordinates.lat}</p>
                  <p>Longitude: {document._source.coordinates.lon}</p>
                  <p>Magnitude: {document._source.mag}</p>
                  <p>Depth: {document._source.depth}</p>
                  <p>Significance: {document._source.sig}</p>
                  <p>Event URL: {document._source.url}</p>
                </div>
              </div>
            ))}
          </div>
        )}
      </div>
    </div>
  );
};

export default App;

Let's go over the code.

Heads up!

For reference purposes only, I have included screenshots of code that I will be explaining.

If you need to copy and paste the code, please refer to the code snippet above or the GitHub repo for part 8.

Lines 1-3
We import all the dependencies that we will need to carry out desired tasks.

Line 1
We import axios to send HTTP request to the server.

Line 2
We import useState hook to manage state.

Line 3
We import App.css to apply styling to the page.

To explain the code in a logical manner, I may skip around a little bit so be sure to pay attention to the line numbers.

Set up the client to display a list of search options and send user input to the server

Lines 36-149
These lines of code are responsible for what the users see when they first interact with our app.

Let's take a look at lines 38-46.

Line 40 renders the app name displayed on the page(red box).

Line 45 renders the directions for users(orange box).

Quake Type
Lines 51-62 create the drop down menu for quake types.

The user gets to choose from four options.

Earthquake(line 58)
Quarry Blast(line 59)
Ice Quake(line 60)
Explosion(line 61)

Each option has a value associated with it(red box).

Lines 54-55
When a user chooses an option from the select control, its value(red box above) is set as state for the chosenType variable.

The state variable for chosenType is defined in line 6.

Magnitude
The next drop down menu allows the user to select the magnitude level.

Lines 65-77 create this drop down menu.

The user gets to choose from five options.

2.5+ (line 72)
5.5+ (line 73)
6.1+ (line 74)
7+ (line 75)
8+ (line 76)

Lines 68-69
When a user chooses an option from the select control, its value is set as state for the chosenMag variable.

The state variable for chosenMag is defined in line 7.

Location
The user can also specify the location of earthquakes they are interested in.

Lines 80-90 create a form where the user can type in the city, state, or country of interest.

Lines 86-87
When a user types in the location, user input is set as state for the chosenLocation variable.

The state variable for chosenLocation is defined in line 8.

Date range
The next drop down menu allows the user to select the date range.

Lines 93-104 create this drop down menu.

The user gets to choose from four options.

Past 7 Days (line 100)
Past 14 Days (line 101)
Past 21 Days(line 102)
Past 30 Days (line 103)

Lines 96-97
When a user chooses an option from the select control, its value is set as state for the chosenDateRange variable.

The state variable for chosenDateRange is defined in line 9.

Sort by Magnitude
The next drop down menu allows the user to sort the search results by descending or ascending level of magnitude.

Lines 107-116 create this drop down menu.

The user gets to choose from two options.

Largest Magnitude First (line 114)
Smallest Magnitude First (line 115)

Lines 110-111
When a user chooses an option from the select control, its value is set as state for the chosenSortOption variable.

The state variable for chosenSortOption is defined in line 10.

Line 119
This line of code creates the search button.

We set it up so that when the button is clicked, it calls the sendSearchRequest function.

Lines 13-34
The function sendSearchRequest is defined here.

Lines 14-24

In line 14, we create a constant called results.

We specify that we want to send a get(line 15) request to our server(localhost:3001) with a url ending in /results(line 16).

Lines 17-23
Within this request, we pass the params(the user input) that were collected.

Lines 25-33

We use axios(line 25) to send the get request to our server(line 26).

Remember, when the client sends the user input to the server, the server will pass the user input into a Elasticsearch request and send the request to Elasticsearch.

Elasticsearch will retrieve relevant documents and send these documents to the server. Upon receiving these documents, the server will send the results to the client.

When the client receives a response (i.e. search results) from the server, the client will print the response to the console(line 28).

Line 11
In line 11, we defined the state variable documents.

Line 29
The client will update the state variable documents to include the documents received from the server.

Lines 31-33
If an error were to occur, we print the error in the console.

So far, we have discussed how we can display a list of search options for the user and how to capture the user input and send it to the server.

Next, we are going to focus on how the client should handle the documents that were retrieved from Elasticsearch.

Set up the client to handle documents received from the server

Lines 123-146
These lines of code handle documents received from the server.

These lines of code are only rendered when documents are received from the server.

Lines 125-129
We set up a ternary operator to specify that when the number of documents is greater than 0, the number of documents is displayed on the screen.

If the number of documents returned are 0, then display the message "No results found. Try broadening your search criteria."

Lines 130-144
The client runs through the documents(line 130) and creates a card(line 131 -143) for each document in the array. Each card displays the following information about each earthquake.

Type(line 133)
Time(line 134)
Location(line 135)
Latitude(line 136)
Longitude(line 137)
Magnitude(line 138)
Depth(line 139)
Significance(line 140)
Event URL(line 141)

The information is accessed by document._source.name_of_the_field.

Step 5: Add styling to the page.

Open the App.css file located within the src directory.

Replace the default content with the following code snippet.

//in client/src/App.css
@import url('https://fonts.googleapis.com/css2?family=Open+Sans+Condensed:wght@300&display=swap');

* {
  font-family: 'Open Sans Condensed', sans-serif;
}

body {
  background-image: url(https://i.imgur.com/lx59qEN.jpg);
  background-position-x: center;
  background-position-y: center;
  background-size: 100% auto;
  background-repeat: no-repeat;
  background-attachment: fixed;
  background-origin: initial;
  background-clip: initial;
  margin: 0;
}

div {
  color: white;
}

.nav {
  width: 100%;
  float: left;
}

.nav-bar {
  margin: 0;
  padding: 8px;
  height: 50px;
  background-color: rgb(28, 28, 28);
}

.nav-bar li {
  display: inline;
  margin-top: 0px;
  font-size: 2.5em;
}

.directions {
  margin-left: 14px;
  font-size: 1.5em;
}

.main {
  display: flex;
  flex-direction: column;
  font-family: inherit;
}
.type-selector {
  margin-left: -30px;
}

.type-selector li {
  float: left;
  display: inline;
  margin-left: 4px;
  margin-top: -19px;
  font-style: inherit;
}

.type-selector li:focus,
.type-selector li:hover {
  outline: none;
  border: 1px solid #bbbbbb;
}

.form {
  width: 200px;
  height: 16px;
}

.form::placeholder {
  color: black;
  opacity: 1;
}

.type-selector button:hover {
  color: #be3400;
  border: 0.1rem #404040 solid; 
}

.type-selector button {
  height: 23px;
}

.search-results {
  display: block;
  width: 100%;
  margin-left: 13px;
  font-family: inherit;
}

.results-card {
  background-color: rgb(25, 129, 67, 0.7);
  flex: 0 1 29rem;
  height: 23rem;
  border-radius: 10px;
  font-weight: bold;
  border: 1px;
  float: left;
  margin: 1px;
  margin-top: 5px;
  margin-right: 10px;
  padding: 10px epx;
  justify-content: space-between;
  max-width: 400px;
}

.results-text {
  margin-left: 2rem;
  margin-right: 2rem;
}

This code snippet will create the styling seen in the UI. Since this is Elasticsearch and Kibana tutorial, we will not go over this in detail.

Step 6: Start the client
Locate the tab of the terminal where we created the client.

Start your client by executing the following command in the terminal.

//in terminal within the client directory
npm start

Screenshot of terminal:

You will see that our client is shown on the browser.

When you click on each select option, it will render the corresponding drop down menu. It will also display a form where a user can type in the location of interest.

Summary
In this blog, we have created a UI, set up our client to capture user input and send the input to the server.

The client is also set up to receive search results from the server and display the results in the form of cards.

What's next?

In the next blog, we will set up our server to pass the user input received from the client into a Elasticsearch request and send it to Elasticsearch.

Elasticsearch will then retrieve the relevant documents and send the documents to the server.

We will set up our server to receive the documents and send the documents to the client so the results could be displayed to the user.

Move on to Part 9 to learn how to do just that!

Part 7 : Set up the Node.js server to retrieve API data and send the data to Elasticsearch

Lisa Jung — Wed, 07 Sep 2022 15:14:51 +0000

Table of Content | Read Next: Part 8: Build the client side of the app with React

In part 6, we created an ingest pipeline(earthquake_data_pipeline) to transform the data retrieved from the USGS API. We also created an index called earthquakes with the desired mapping.

In this blog, we are going to set up the server to:

retrieve earthquake data from the USGS API
send the retrieved data to Elasticsearch earthquake_data_pipeline for data transformation
instruct Elasticsearch to ingest the transformed data into the earthquakes index

The following diagram summarizes the tasks we will perform in this blog.

Resources

Would you rather watch a video to learn this content? Click on the link below!

Episode 7: Set up the Node.js server to retrieve API data and send the data to Elasticsearch

Want the code covered in this blog? Click on the link below to access it!

GitHub repo for part 7

Recognition

I want to give a huge shout out to Mark Mayfield!

When I was first trying to figure out how to retrieve and ingest API data into Elasticsearch, I came across his blog.

His blog was instrumental in helping me write the part 7 blog. Thank you Mark!

I have modified the code shared in Mark's blog to fit our use case.

I have also broken down the code line by line for easier understanding.

API Data Retrieval

Step 1: Set up your server to retrieve earthquake data from the USGS API

Open the earthquake_app using the code editor of your choice.

Within the server directory, create a directory called data_management. Within the data_management directory, create a file called retrieve_and_ingest_data.js(red box).

retrieve_and_ingest_data.js file will instruct the server to execute three things:

Upon receiving an http request from the browser, retrieve data from the USGS API
send the retrieved data to Elasticsearch earthquake_data_pipeline for data transformation
instruct Elasticsearch to ingest the transformed data into the earthquakes index

In the retrieve_and_ingest_data.js, copy and paste the following snippet.

//in server/data_management/retrieve_and_ingest_data.js
const express = require('express');
const router = express.Router();
const axios = require('axios');
const client = require('../elasticsearch/client');
require('log-timestamp');

const URL = `https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson`;

router.get('/earthquakes', async function (req, res) {
  console.log('Loading Application...');
  res.json('Running Application...');

  indexData = async () => {
    try {
      console.log('Retrieving data from the USGS API');

      const EARTHQUAKES = await axios.get(`${URL}`, {
        headers: {
          'Content-Type': ['application/json', 'charset=utf-8'],
        },
      });

      console.log('Data retrieved!');

      results = EARTHQUAKES.data.features;

      console.log('Indexing data...');

      results.map(
        async (results) => (
          (earthquakeObject = {
            place: results.properties.place,
            time: results.properties.time,
            tz: results.properties.tz,
            url: results.properties.url,
            detail: results.properties.detail,
            felt: results.properties.felt,
            cdi: results.properties.cdi,
            alert: results.properties.alert,
            status: results.properties.status,
            tsunami: results.properties.tsunami,
            sig: results.properties.sig,
            net: results.properties.net,
            code: results.properties.code,
            sources: results.properties.sources,
            nst: results.properties.nst,
            dmin: results.properties.dmin,
            rms: results.properties.rms,
            mag: results.properties.mag,
            magType: results.properties.magType,
            type: results.properties.type,
            longitude: results.geometry.coordinates[0],
            latitude: results.geometry.coordinates[1],
            depth: results.geometry.coordinates[2],
          }),
          await client.index({
            index: 'earthquakes',
            id: results.id,
            body: earthquakeObject,
            pipeline: 'earthquake_data_pipeline',
          })
        )
      );

      if (EARTHQUAKES.data.length) {
        indexData();
      } else {
        console.log('Data has been indexed successfully!');
      }
    } catch (err) {
      console.log(err);
    }

    console.log('Preparing for the next round of indexing...');
  };
  indexData();
});

module.exports = router;

Let's go over this code snippet.

Heads up!

For reference purposes only, I have included screenshots of code that I will be explaining.

If you need to copy and paste the code, please refer to the code snippet above or the GitHub repo for part 7.

Lines 1-3 require dependencies necessary to receive and send HTTP requests.

Line 4
We require the Elasticsearch Client we created in part 4.

Line 5
We require a dependency called log-timestamp we installed in part 2. It prepends timestamps to the messages displayed in the terminal via the console.log() method.

You will see this dependency come into play when we retrieve data from the USGS API.

Line 7
We create a constant called URL and set it equal to the URL of the all earthquake data from the past 30 days API. We will be retrieving data from this API.

We use lines 9-77 to define what our server should do when it receives a HTTP request to the /earthquakes end point

Line 9
We define a route for /earthquakes.

Line 10
When the server receives a HTTP request to the/earthquakes end point, it displays the message "Loading application..." in the terminal.

Line 11
In the browser(client), it displays the message "Running Application..."

We use lines 13-77 to:

retrieve data from the USGS API
create an object for each earthquake
send the objects to Elasticsearch earthquake_data_pipeline for data transformation
instruct Elasticsearch to ingest transformed objects into the earthquakes index

Line 13

We define a function called indexData().

Line 15
When this function is called, it prints the message "Retrieving data from the USGS API" in the terminal.

Lines 17-21
We send a GET request to the USGS API.

Line 23
We display the message "Data retrieved!" in the terminal.

Line 25
We create a variable called results. This variable is set equal to the retrieved earthquake data from the API.

Remember the USGS API data structure covered in part 5?

EARTHQUAKES.data.features gives you access to the features array(green box).

This array contains earthquake objects. Each object contains info about one earthquake.

Within each object, the fields properties(orange box) and geometry(blue box) contain the information that we want(pink box).

Line 27
We display the message "Indexing data" in the terminal.

Lines 29-55

These lines of code run through an array of earthquake objects. For each earthquake object in the array, it creates a json object that will be indexed later as documents.

Lines 56-61

We use our instance of Elasticsearch Client to:

send retrieved API data to Elasticsearch earthquake_data_pipeline for data transformation
instruct Elasticsearch to ingest transformed data into the earthquakes index

Line 56
We use the client.index() method to index the transformed data.

Line 57
We specify that the data should be indexed into the earthquakes index.

Line 58
We give each earthquake document an id identical to the id of the earthquake object retrieved from the API. We do this to keep the earthquake id from the USGS API and document id consistent.

Line 59
We set the body equal to earthquakeObject we defined in lines 31-55. The body represents a document of one earthquake.

Line 60
We instruct Elasticsearch client to send the retrieved data to Elasticsearch earthquake_data_pipeline.

The earthquake_data_pipeline will transform the data in the manner we specified in part 6.

Then, Elasticsearch will index the transformed data into the earthquakes index.

Lines 65-72
We set up an if, else, catch statement.

Lines 65-67
If there is still data left that has not been indexed, then keep calling the indexData() function.

Lines 67-69
If all data has been indexed, then print "Data has been indexed successfully!" in the terminal.

Lines 70-72
If an error occurs during indexing, then print the error in the terminal.

Line 74
After data ingestion has been completed, print the message "Preparing for the next round of indexing..." in the terminal.

Line 76
We call the indexData() function.

Line 79
We expose the router via Node.js module exports as this will be used in server.js.

Add the data ingestion route to `server.js`

Step 1: open `server.js` within the `server` directory

Copy and paste the following into lines 8 and 10.

//in server/server.js
const data = require('./data_management/retrieve_and_ingest_data');

app.use('/ingest_data', data);

Your server.js should look like the following:

Let's go over this snippet.

Line 8
We create a constant called data and require the retrieve_and_ingest_data.js file from our data_management directory.

Line 10
When the server receives an http request from a url path that starts with /ingest_data, the function we defined in retrieve_and_ingest_data.js is executed.

Let's test to see if this works!

Using your terminal, cd into the earthquake_app directory. Start the server by executing the following command.

//in terminal within the earthquake_app directory
npm start

You should see something similar to the screenshot in the terminal.

Open a new browser. Arrange the the terminal and the browser side by side as shown below.

Copy and paste the following url in the address bar of your browser and hit enter:

//in the address bar of your browser
http://localhost:3001/ingest_data/earthquakes

You will see the following in the terminal acknowledging that the data is being retrieved, transformed, and ingested into Elasticsearch. You will also see that a timestamp has been prepended to the messages printed on the console!

You will also see the message "Running Application..." displayed on the browser.

Evaluate the outcome of data transformation and data ingestion

Let's take a look to see if the data has been properly transformed and ingested into Elasticsearch.

One of the ways to view data stored in Elasticsearch is via Kibana Discover tool.

With Discover, you can quickly search and filter your data, get information about the structure of the fields, and display your findings in a visualization. You can also customize and save your searches and place them on a dashboard(excerpt from Kibana documentation).

In order to use Discover, we must first specify where it can find the data we want to explore.

We do so by creating a data view.

A data view allows you to specify the data source you want to explore so that Kibana could find this data.

From the Kibana home page, click on the Stack Management option(red box).

From the Stack Management page, click on the Data Views option(red box).

Click on the Create data view button(red box).

You will see that the index earthquakes(red box) is listed as one of our data sources.

Under the Name section, type in the name of the index earthquakes(red box).

Discover also allows you to explore data in various time ranges.

Since our earthquake data is in time series, we will be able to use the time range feature of Discover.

Since earthquake data has a timestamp field called @timestamp, Kibana will automatically update the Timestamp field with the field @timestamp(blue box).

Click on the Create data view option(green box).

You will see that data view called earthquakes have been created. The Time field has been set to field @timestamp.

Click on the menu icon(red box) and click on the Discover option(blue box).

You will see that the data view earthquakes(red box) has been already selected in the Discover page.

The message "No results match your search criteria"(orange box) is also displayed on the screen.

Let's see why we are encountering this message.

Take a look at the time range highlighted with a green box.

Discover is currently being instructed to display data that has been collected within the Last 15 minutes. Since our index does not contain such data, we encounter a message that no results match our search criteria.

Let's adjust our timeline to Last 30 days to view the results.

Click on the calendar icon(blue box). You will see a drop down menu where you can select the time range in which you want to view the data. Select the Last 30 days option(red box).

You will see that in the index earthquakes, there are 9,108 documents(red box) that have been indexed from the USGS API.

The table (yellow box) displays documents within the earthquakes index. Each row displays information about a document(an earthquake).

Let's take a look at a document to see if data was properly transformed before being ingested into Elasticsearch.

Click on the arrow(red box) to expand the document information(blue box).

You will see that the document contains the desired fields and the timestamp format we have set up in part 6.

In this blog, we have successfully:

retrieved earthquake data from the USGS API and sent the data to Elasticsearch
transformed the retrieved data using the ingest pipeline(earthquake_data_pipeline)
ingested the transformed data into the earthquakes index

Now that we have data in Elasticsearch, it is time set up our app for search.

Move on to part 8 to build the client with React so the users could specify the earthquake data that they are searching for!

Part 6: Set up Elasticsearch for data transformation and data ingestion

Lisa Jung — Wed, 07 Sep 2022 15:14:45 +0000

Table of Content | Read Next: Part 7: Set up the Node.js server to retrieve API data and send the data to Elasticsearch

Resources

Would you rather watch a video to learn this content? Click on the link below!

Episode 6: Set up Elasticsearch for data transformation and data ingestion

Set up Elasticsearch for data transformation and data ingestion

In part 5, we figured out:

a plan for transforming data before ingesting data into Elasticsearch
a desired mapping for our data

In this blog, we will accomplish two tasks:

create an ingest pipeline to transform the retrieved data
create an index called earthquakes with the desired mapping

Before we get started, let’s talk about the data journey for our app.

We are building an app where users can search for earthquake data stored in Elasticsearch.

In part 7, we will set up our server to retrieve the data from the USGS API and send the data to Elasticsearch ingest pipeline.

Ingest pipeline is used for data transformation.

It consists of a series of configurable tasks called processors. Each processor performs a specialized task. For example, it can remove fields, extract values from text, enrich your data and etc.

Each processor runs in the order you set them up and they make specific changes to the incoming documents.

After the processors have run, Elasticsearch will add the transformed documents to the earthquake index we will create.

If you want to delve deeper into ingest pipelines, check out this documentation!

Set up Elasticsearch for data transformation and data ingestion

Step 1: Review the data transformation requirements

Our ingest pipeline will be used to transform the data retrieved from the USGS API.

Before we create an ingest pipeline, let's review what changes we want to make to the data.

remove the unnecessary info from the retrieved data
change the Unix epoch time in the field time to human readable timestamp
create fields coordinates.lat and coordinates.lon as shown below

Step 2: Create an `ingest pipeline`

Ingest pipelines can be created and managed via Kibana's Ingest Pipelines feature or the ingest APIs.

We will be using Kibana to create this pipeline.

From the Kibana home page, click on the Stack Management option(red box).

From the Stack Management page, click on the Ingest Pipelines option(red box).

Click on the Create pipeline option(red box) and select the New pipeline option from the drop down menu(blue box).

Name your pipeline to whatever it makes sense to you.

For this project, I named mine earthquake_data_pipeline(red box).

Step 3: Add the desired processors to the pipeline

Click on the Add a processor option(red box).

You should see the following pop up menu.

Task 1: Remove the fields that we do not need from the retrieved data

Here is an example of an earthquake object from the USGS earthquake API:

   {
     "type":"Feature",
     "properties":{
        "mag":1.13,
        "place":"11km ENE of Coachella, CA",
        "time":1650316843970,
        "updated":1650317059011,
        "tz":null, 
        "url":"https://earthquake.usgs.gov/earthquakes/eventpage/ci40240408",
        "detail":"https://earthquake.usgs.gov/earthquakes/feed/v1.0/detail/ci40240408.geojson",                
        "felt":null,
        "cdi":null,
        "mmi":null,
        "alert":null,
        "status":"automatic",
        "tsunami":0,
        "sig":20,
        "net":"ci",
        "code":"40240408",
        "ids":",ci40240408,",
        "sources":",ci,",
        "types":",nearby-cities,origin,phase-data,scitech-link,",
        "nst":37,
        "dmin":0.07687,
        "rms":0.26,
        "gap":48,
        "magType":"ml",
        "type":"earthquake",
        "title":"M 1.1 - 11km ENE of Coachella, CA"
     },
     "geometry":{
        "type":"Point",
        "coordinates":[
          -116.0736667,
          33.7276667,
          2.09
        ]

    },
     "id":"ci40240408"
   }

The following is a sample document with the desired fields we want to store in Elasticsearch.

{
  "mag": 1.13,
  "place": "11km ENE of Coachella, CA",
  "time": 2022-05-02T20:07:53.266Z,
  "url": "https://earthquake.usgs.gov/earthquakes/eventpage/ci40240408",
  "sig": 20,
  "type": "earthquake",
  "depth": 2.09,
  "coordinates": {
    "lat": 33.7276667,
    "lon": -116.0736667
  }
}

As you can see, API earthquake object has additional info we do not need.

By comparing the two, we can identify the fields that we want to remove from the retrieved data.

Here is the list of fields that we do not need.

updated
tz
detail
felt
cdi
mmi
alert
status
tsunami
net
code
ids
sources
types
nst
dmin
rms
gap
magType
title

To remove these fields, we can use the Remove processor.

Under the processor section, type Remove in the search bar. Click on the Remove processor(red box).

Elasticsearch will display the following menu. In the Fields section(red box), we will add the names of the fields that we wish to remove.

Type in the name of the field you wish to remove then press enter. You will see that the field name has been added.

In the same box, repeat the same process to specify the names of all the fields we wish to remove.

Your screen should look like the image below(red box).

Activate the Ignore missing option(red box).

Click on the Add button(red box).

You will see that the Remove processor has been added to the earthquake_data_pipeline(red box).

Task 2: Change the Unix epoch time in the field time to human readable timestamp

In order to make this change, you should use the date processor.

The date processor converts time from one format to another.

Click on the Add a processor option(red box).

Under the Processor section, type in Date and click on the Date option from the drop down menu(red box).

You should see the following drop down menu.

In the Field section(red box), type the name of the field we wish to convert (time).

In the Formats section(red box), we will specify the desired date formats. The format shown in the results card is called UNIX_MS.

Type it into this section and hit enter. Then, click on the Add button(green box) to add the date processor to the ingest pipeline.

You will see that the Date processor has been added to the earthquake_data_pipeline(red box).

When the data goes through the date processor, the content of the field time will be converted to the UNIX_MS format then stored in a new field called @timestamp.

After this process is finished, we do not need the original field time. Therefore, we will remove the field time after the data goes through the date processor.

From the Create pipeline page, click on the Add a processor option(red box).

Under the Processor section, type in Remove and hit enter(red box).

Under the Fields section, type in time and hit enter(red box). Activate the ignore missing option(blue box). Then, click on the Add button(green box).

You will see that the Remove processor for the field time has been added(red box).

Task 3: Create fields called coordinates.lat and coordinates.lon

From the same page, click on the Add a processor option(red box).

Under the processor section, type in Rename and hit enter(red box).

Under the Field section, type in latitude(red box). Under the Target field section, type in coordinates.lat(blue box).

This step will rename the field latitude in the incoming data to coordinates.lat.

Activate the Ignore missing option(yellow box) then click on the Add button(green box) to add this processor to the earthquake_data_pipeline.

You will see that the Rename processor for the field latitude has been added(red box).

Next, we will repeat the same process to add a Rename processor to rename the field longitude from the incoming data to coordinates.lon.

This process is identical to the steps you have performed for the field latitude.

Once this process is done, you will see that a Rename processor for the field longitude has been added to the earthquake_data_pipeline(red box).

We have added all the necessary processors to transform our data.

Before creating the earthquake_data_pipeline, make sure the order of the processors are listed in the order you want them to run.

These processors run sequentially!

Why does the order matter?

Let's say you accidentally reversed the order of date processor for the field time with the Remove processor for the field time.

When the data goes through the ingest pipeline, the data will go through the remove processor before the date processor. As a result, the field time will be removed before it gets to the date processor.

The date processor will have nothing to work with!

This is why we double check the order of the processors before creating the ingest_pipeline!

Next, we will create the earthquake_data_pipeline by clicking on the Create pipeline button(blue box).

You will see that the earthquake_data_pipeline(red box) has been created. If you scroll down on the Processors section(blue box), you will see the list of processors we have added to this pipeline.

Step 4: Create an index called `earthquakes` with the desired mapping

We will accomplish this step using Kibana Dev Tools.

Click on the menu icon(red box).

Scroll down on the drop down menu and click on Dev Tools(red box).

You should see Dev Tools(also known as Kibana console) on the screen.

In the left panel of the Kibana console, copy and paste the following.

PUT earthquakes
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "coordinates": {
        "type": "geo_point"
      },
      "depth": {
        "type": "float"
      },
      "mag": {
        "type": "float"
      },
      "place": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      },
      "sig": {
        "type": "short"
      },
      "type": {
        "type": "keyword"
      },
      "url": {
        "enabled": false
      }
    }
  }
}

Your Kibana console should look the following:

Click on the green arrow to send the request.

Expected output from Elasticsearch:
Elasticsearch will create an index called earthquakes with the desired mapping we defined above!

Summary

In this blog, we have created:

an ingest pipeline(earthquake_data_pipeline) to transform the retrieved data from the USGS API
an index called earthquakes with the desired mapping

In the next blog, we will set up the server to retrieve earthquake data from the USGS API and send the data to the earthquake_data_pipeline in Elasticsearch.

Once the data transformation is complete, the transformed data will be ingested into the earthquakes index.

Move on to Part 7 to set up the Node.js server to retrieve API data and send the data to Elasticsearch for data transformation and data ingestion!

Part 5: Plan for efficient data storage and search performance in Elasticsearch

Lisa Jung — Wed, 07 Sep 2022 15:14:39 +0000

Table of Content | Read Next: Part 6 - Set up Elasticsearch for data transformation and data ingestion

Now that our server is connected to Elasticsearch hosted on Elastic Cloud, it is time to think about the data that we want to ingest into Elasticsearch.

Review from part 1

For our project, we will be retrieving data from the USGS API and ingesting it into Elasticsearch.

Before ingesting data into Elasticsearch, assessing the data structure and planning for desired mapping are essential in ensuring efficient storage and search of data.

In this blog, we will assess the data to:

determine what data we need
discover if we need to transform the data to fit our use case
decide on the desired mapping for efficient storage and search of data

Resources

Would you rather watch a video to learn this content? Click on the link below!

Episode 5: Plan for efficient data storage and search performance in Elasticsearch

The following USGS Earthquake API home page contains the link to the API that contains all earthquake data from the past 30 Days(red arrow). We will be retrieving data from this API!

USGS Earthquake API Home Page

The following documentation contains explanations of the terms(field names) included in the API. Refer to this if you need clarifications on the acronyms or want details about certain fields.

Earthquake Catalog Documentation

The current blog builds upon concepts covered in the blog and video above. Refer to these links if some of the jargons don't make sense to you or if you need a little refresher.

We will be referring to the following Elasticsearch documentations on field data types and numeric field types while coming up with the desired mapping for our data.

You will find these resources helpful when assigning field types to your data.

Now that you have access to all resources, let's get to work!

Step 1: Review the final outcome

Before we examine the data structure, let's review the final outcome of the app we are building.

Our app allows the user to search for earthquakes using the following criteria.

type
magnitude
location
date range

When the user clicks on the search button, the search results are displayed as cards.

Each card displays following information about one earthquake.

The information outlined above is what we need from the USGS API.

We will store this information in Elasticsearch in the form of documents. Each document will contain information about one earthquake.

Step 2: Examine the data structure of earthquake API

Go to the USGS Earthquake Home Page and scroll down to the Output section(red box).

The output shows the data structure of a typical earthquake object found in the API.

Scroll down on the output to view the field features(green box).

The features field contains an array of objects. Each object contains information about one earthquake. It lists the name and data type of fields contained in a typical earthquake object.

In step 1, we determined what information we need to store in Elasticsearch.

You will see that the object properties(orange box) and geometry(blue box) contain the information we seek(underlined in pink).

If you need clarification on the acronyms or want details about certain fields, refer to the Earthquake Catalog Documentation.

Take a look at the image below.

You will see that the earthquake object from the API contains more information than we need.

To save storage, we will only index the fields mag, place, time, url, sig(significance), type, and coordinates array which includes longitude, latitude, and depth in that order.

The API fields that correspond to the info on the card are highlighted in same colors.

When you compare the two, for majority of these, the info from the API is identical to the info displayed in the search results.

However, there are a few that are not the same.

Step 3: Determine whether we need to transform any data before ingestion

Data transformation task 1: time
Let's compare the field time in the API earthquake object with the field Time on our card.

You will see that the field time in API earthquake object is in Unix epoch time(1651522073266). However, Time on the card displays a human readable timestamp(2022-05-02T20:07:53.266Z).

To achieve this outcome, we will convert the Unix epoch time in the API field time to human readable timestamp. Then, store transformed information in the field @timestamp in Elasticsearch(more on that later).

Data transformation task 2: coordinates

You will see that the search results card has fields called latitude, longitude, and depth(pink boxes).

In the API earthquake object, the values of these fields are contained in an array called coordinates(pink box) and are not labeled as such.

To make it easier to identify this information, we will create fields for lat(latitude), lon(longitude), and depth in Elasticsearch.

Then, we will store the corresponding info from the API's coordinates array into its respective fields.

Then, we will store lat and lon into an object called coordinates to keep this information together as a pair as shown below.

Note that in Elasticsearch, the abbreviation lat should be used for latitude and lon should be used for longitude.

Step 4: Determine the desired mapping

We just figured out how we should transform the retrieved data from the API before we store it in Elasticsearch.

Next, we will figure out how to store this data using the smallest disk space while maximizing our search performance.

This is when customizing your mapping come into play!

Mapping defines how a document and its fields are indexed and stored.

It does that by assigning types to fields being indexed. Depending on the assigned field type, each field is indexed and primed for different types of requests(full text search, exact searches, aggregations, sorting & etc).

This is why mapping plays an important role in how Elasticsearch stores and searches for data.

We will be glossing over a lot of the concepts we have covered in Understanding mapping with Elasticsearch and Kibana. Check out this resource if you need more in depth review of these concepts!

Take a look at the table below.

To make this process easier, I have created a table that displays(pink box) the name of the field, description of the field, typical values contained in the field, the purpose this field will serve, and the desired mapping I have chosen for the field.

Let's go through each field and determine why I chose certain field type for each field.

When you take a look at the Typical values column in the table, you will see that these fields either contain numeric(green box) or string(orange box) values.

Let's take a look at the fields that contain numeric values first.

Numeric field types

There are various field types that can be assigned to numeric fields. The field type you choose will depend on the value type a field contains and for what purpose you will be using the field.

Coordinates

In step 3, we decided that we want to store the fields lat(latitude) and lon(longitude) within an object called coordinates.

We are planning on using the fields lat and lon for two tasks:

to display this information on a search results card
to use these coordinates to mark the location of earthquakes on a heat map(part 10).

The second task requires running geo-based queries.

Therefore, the field coordinates should be typed as geo_point in order for this to work.

If you want to learn more about the field type geo_point, check out this documentation.

depth and mag

The typical values for the fields depth and mag are in decimals.

As the values of these fields will only be displayed in the search result cards, we will assign the field type float for these fields.

sig

When you look at the typical values for the field sig, it consists of integers that range between 0 to 1000.

The value of this field will be displayed in the search results card.

We want to choose the field type that will store integers using the smallest disk space.

If you take a look at the documentation for numeric field types, the field type that will allow us to store this data using the smallest disk space is short.

time

The value of this field will be displayed in the search results card. It will also be used to search for earthquakes that have occurred within a chosen date range.

To do so, we will run range queries on this field so we will assign the field type date.

Text field types

Let's go over the fields that contain string data types(place, type, and url).

By default, every field that contains string data type gets mapped twice as a text field and as a keyword multi-field.

Each field type is primed for different types of requests.

Text field type is used for full text search.

keyword field type is used for aggregations, sorting, and exact searches.

In scenarios where you do not need both field types, the default setting is wasteful. It will slow down indexing and use up more disk space.

When deciding on a string field type, make sure you know for what purpose this field will be serving so you can choose the correct field type.

place

The field place will be used for three purposes.

The value of this field will be displayed on the search results card.
The field will be used for full text search(when a user types in the location, the user input will be searched against this field to retrieve relevant data)
Aggregation will be performed on this field to yield a table of 10 locations with the highest frequency of earthquakes (part 10)

Since we need to run both full text search and aggregations on the field place, we will assign both field types text and keyword.

type

The value of this field will be displayed on the search results card.

This field will also be used for exact searches.

When a user searches for a specific type of quake, the user input is searched against the field type to retrieve relevant search results.

The user is prompted to select a type from a list of options. Therefore, we can perform exact searches on this field and will map this field as keyword.

url

The value of the field url is only displayed on a card and it is not used for search.

Therefore, there is no need to create search data structures(inverted index or doc values) for this field so we will disable this field(enabled:false).

Summary
In this blog, we figured out:

how we want to transform the retrieved data before ingesting it into Elasticsearch
the desired mapping to efficiently store and search data in Elasticsearch

Move on to Part 6 to set up Elasticsearch for data transformation and data ingestion!

Part 4: Securely connect Node.js server to Elastic Cloud

Lisa Jung — Wed, 07 Sep 2022 15:14:33 +0000

Table of Content | Read Next: Part 5: Plan for efficient data storage and search performance in Elasticsearch

Resources

Would you rather watch a video to learn this content? Click on the link below!

Episode 4: Securely connect Node.js server to Elastic Cloud

The following are the project directories shared in the blog. Click on the link below to access it!

Connect Node.js Server to Elastic Cloud

In part 3, we signed up for a free trial of Elastic Cloud and created a deployment.

This deployment consists of Elasticsearch cluster, Kibana, Fleet, three Elastic Solutions(Enterprise Search, Observability, Security) and more!

Connecting to this deployment will give us access to everything mentioned above, including our Elasticsearch cluster.

We can connect our server to Elastic Cloud deployment via two different authentication mechanisms:

basic authentication
API key

API key is the best practice for production because it gives users flexibility in creating different API keys for different systems or applications.

In order to generate an API key, we need to log into Elastic Cloud as a user first(basic authentication).

Let's get started!

Step 1: Pull up the earthquake_app directory with your code editor

In part 2, we created a project directory called the earthquake_appand we built a Node.js server within it.

Next, we will create a file where we will store Elastic Cloud access credentials.

Step 2: Set up the config directory

In part 2, we have installed a dependency called config.

The config dependency allows you to store node configuration in an external configuration file. It can be customized based on different environments(local/dev/qa/prod/etc).

In our case, we will be using config to store access credentials such as Elastic cloud ID, username, password, and API key.

Within the earthquake_app directory, create a config directory. Within the config directory, create a default.json file as shown below.

Step 3: Enter the Elastic Cloud access credentials in the default.json file

Copy and paste the following code into the default.json file.

//in config/default.json
{
    "elastic": {
      "cloudID": "DEPLOYMENT_NAME:CLOUD_ID_DETAILS",
      "username": "elastic",
      "password": "LONGPASSWORD"
    }
}

Your default.json file should look like the following screenshot.

We will replace "DEPLOYMENT_NAME:CLOUD_ID_DETAILS"(line 3) and "LONGPASSWORD"(line 5) with our Elastic Cloud deployment access credentials.

In part 3, we have downloaded and saved the access credentials.

Find and open the saved credentials.

You should see something like this.

Copy your password and paste it into line 5 "password" as shown below.

Next, let's find our "DEPLOYMENT NAME:CLOUD_ID_DETAILS".

In part 3, we learned how to get to the Kibana home page.

From the Kibana home page, click on the menu icon(red box).

It will display a drop down menu as shown below.

From the drop down menu, click on the Manage this deployment option(pink box).

You will see the following on your screen.

Look at the region highlighted with a red box. This is where your deployment name and cloud id information are displayed.

Click on the copy button(blue box) then paste the info into line 3 of the default.json file.

Your default.json file should look like the following screenshot with your credentials pasted into line 3 and line 5.

Step 4: Connect to Elastic Cloud using basic authentication

Within the server directory, create a elasticsearch directory. Within the elasticsearch directory, create a client.js file as shown below.

At this point, you might be wondering how we will get our server to communicate with Elastic Cloud so we can establish a secure connection?

It's actually quite simple!

Elasticsearch Client is the simplest and easiest way to communicate with your Elasticsearch cluster from your app, in a programming language of your choice.

Elasticsearch Client allows you to easily access every endpoint of Elasticsearch API to perform a variety of tasks.

Since we are building a server using Node.js with Express, we will be using the Elasticsearch Javascript Client to connect to our Elasticsearch cluster!

In client.js, we will create an instance of Elasticsearch Client. Elasticsearch Client will contain our Elastic Cloud access credentials that point to our Elasticsearch cluster.

Then, we will set up our server to connect to Elastic Cloud and display a message in the terminal of its connection status.

In the client.js file, copy and paste the following code.

//in server/elasticsearch/client.js
const { Client } = require('@elastic/elasticsearch');
const config = require('config');

const elasticConfig = config.get('elastic');

const client = new Client({
  cloud: {
    id: elasticConfig.cloudID,
  },
  auth: {
    username: elasticConfig.username,
    password: elasticConfig.password
  },
});

client.ping()
  .then(response => console.log("You are connected to Elasticsearch!"))
  .catch(error => console.error("Elasticsearch is not connected."))

module.exports = client;

Let's go over the code!

Heads up!
For reference purposes only, I have included screenshots of code that I will be explaining.

If you need to copy and paste the code, please refer to the code above or the GitHub repo for connecting to Elastic Cloud with basic authentication.

Line 1

We require @elastic/elasticsearch library we installed in part 2 and set this equal to a constant Client.

This library is the official Node.js client for Elasticsearch and it comes with a lot of functionalities. You will see this coming into play in line 6 when we create a new instance of Elasticsearch Client.

Line 2
We create a constant called config and require the config dependency we installed in part 2. You will see this come into play in line 4 when you use this dependency to get access credentials stored in the config directory.

Line 4
The access credentials were stored under 'elastic' in config/default.json as shown below.

We use config to get credentials stored under 'elastic' and set it equal to a constant elasticConfig.

This allows us to securely store and access sensitive data and not store these in the source code.

Lines 6-14

We create a new instance of Elasticsearch Client with our access credentials(cloudID, username, and password) that points to our Elasticsearch cluster. We set this equal to a constant called client.

Lines 16-18

We use the ping() method on our Elasticsearch Client to check if the Elasticsearch cluster is up and available to process requests.

We have it set up so that if we receive a response(meaning we are connected to Elasticsearch), our terminal will display the message "You are connected to Elasticsearch!".

If an error occurs during this process, our terminal should display the message "Elasticsearch is not connected".

Line 20
We expose our Elasticsearch Client via Node.js module exports as this will be used in server.js.

Go to the server.js file within the server directory. In line 2, add the following snippet.

//in server/server.js
const client = require('./elasticsearch/client');

Your server.js should look like the following screenshot.

In line 2, we require client.js from the elasticsearch directory. We created an instance of Elasticsearch Client in this file.

We do this as we will need to access our Elasticsearch Client from server.js.

Let's test if we are able to connect to Elastic Cloud as a user(basic authentication)!

Start your server by running the following command in the terminal.

//in terminal earthquake_app directory
npm start

You will see that the server is listening on port 3001 and that you are connected to Elasticsearch!

Step 5: Generate an API key
Now that we have logged into Elastic Cloud as a user, we can generate an API key.

From this point on, the API key will be used to connect to Elastic Cloud.

Within the server directory, create a file called create-api-key.js.

In create-api-key.js, copy and paste the following code.

//in server/create-api-key.js
const client = require('./elasticsearch/client');

async function generateApiKeys(opts) {
  const body = await client.security.createApiKey({
    body: {
      name: 'earthquake_app',
      role_descriptors: {
        earthquakes_example_writer: {
          cluster: ['monitor'],
          index: [
            {
              names: ['earthquakes'],
              privileges: ['create_index', 'write', 'read', 'manage'],
            },
          ],
        },
      },
    },
  });
  return Buffer.from(`${body.id}:${body.api_key}`).toString('base64');
}

generateApiKeys()
  .then(console.log)
  .catch((err) => {
    console.error(err);
    process.exit(1);
  });

Your create-api-key.js should look like the following.

Let's go over the code!

Heads up!
For reference purposes only, I have included screenshots of code that I will be explaining.

If you need to copy and paste the code, please refer to the code above or the GitHub repo for connecting to Elastic Cloud with API key.

Line 1
We require the instance of Elasticsearch Client we created in the previous step.

Lines 3-21
We define the generateApiKeys() function. This function creates an API key that gives access to our Elastic Cloud deployment.

Line 4
We define a constant body and set it equal to client.security.createAPIKey() method.

Lines 5-18
Within this method, we define what Elasticsearch cluster privileges are granted to the server connecting with the generated API key.

Line 6
We specify the name of the server using this API key.

Line 7
Under role_descriptors, we define a role(access privileges) granted to this server.

Line 8
We name this role as earthquakes_example_writer. You could name this role any way you see fit.

Line 9
We specify that this API key comes with cluster monitor privilege, which gives read-only access for determining cluster state.

Lines 12-13
This key also grants additional privileges such as create_index, write, read, and manage operations for an index named earthquakes.

Line 20
The security.createApiKey function(line 4) returns an id and api_key value which are concatenated and encoded in base64.

Lines 23-28
We call the generateApiKeys() function here which prints the API key in the terminal. If an error occurs, it prints the error instead.

Let's generate an API key by running the create-api-key.js file.

In the terminal, create a separate tab. cd into the earthquake_app directory and run the following command.

//in terminal within the earthquake_app directory
node server/create-api-key.js

You will see the following message in the terminal along with your API key.

Step 6: Connect to Elastic Cloud via API key

We will start by storing the API key in the default.json file of our config directory.

Go to default.json file and add a comma at the end of *line 5. In line 6, copy and paste the following.*

//in config/default.json
"apiKey": "paste your generated API key here"

Copy the API key from your terminal and paste it into *line 6 in default.json.

Your default.json file should look like this.

Open the client.js file in the elasticsearch directory.

Replace username and password(lines 11 and 12) with the following code.

//in elasticsearch/client.js
apiKey: elasticConfig.apiKey

Your client.js file should now look like the following screenshot(lines 11).

Go back to the terminal tab where you are running the server.

You will see that the server is connected to Elasticsearch except that this time we are connected to Elastic Cloud deployment via the API key!

Now that our server is connected to our Elasticsearch cluster, it is time to think about the data we want to store in Elasticsearch.

Move on to Part 5 to plan for efficient data storage and search performance in Elasticsearch!

Part 3: Create an Elastic Cloud deployment

Lisa Jung — Wed, 07 Sep 2022 15:14:25 +0000

Table of Content | Read Next: Part 4 - Securely connect Node.js server to Elastic Cloud

Resources

Would you rather watch a video to learn this content? Click on the link below!

Episode 3: Create an Elastic Cloud deployment

Create an Elastic Cloud deployment

Elastic Cloud hosts Elastic's products. It handles the heavy lifting of provisioning the host, managing security, data and much more with just a few clicks on the UI!

I find that for beginners, it is easier and faster to get started with Elastic Cloud.

In this blog series, we will be connecting our server to Elasticsearch and Kibana hosted on Elastic Cloud.

To do so, we must create an Elastic Cloud account.

Step 1: Go to the Elastic Cloud Free Trial Page

Click on the Free Elastic Cloud Trial Link.

This page will give you access to a longer trial period(30 vs 14 days) for Elastic Cloud!

There is no credit card required and the trial will expire on its own.

Step 2: Click on the Start free trial option(red box)

Step 3: Create an Elastic Cloud account

If you don't have an Elastic Cloud account, enter your email and password to create an account(green box).

Step 4: Click on the Start your free trial option(red box)

Step 5: Create an Elastic Cloud deployment
Upon login, you will see the following on your screen.

Name your deployment to whatever it makes sense to you(green box)!

I named mine Mini Beginner's Crash Course to Elasticsearch and Kibana.

Then, click on the Edit settings option(purple box).

You should see the following options.

First, choose the Cloud provider of your choice(red box).

Let’s say you have an app and you want to connect it to Elasticsearch.

But if your app is running on Google Cloud, you don’t want Elasticsearch running in a different Cloud provider because that will cause latency issues.

But for what we are about to do, it doesn’t matter which one you choose. Just select one.

Select the region closest to you(orange box).

Select your hardware profile(yellow box).

Elasticsearch offers several deployment templates for different use cases and workload.

Each template selects the appropriate cloud hardware configuration for different needs. But if you are just getting started or don’t quite know your needs yet, then go with the storage optimized option.

Select the latest version of the Elastic Stack(at the time of blog creation, the latest version was 8.3.3).

Once you have configured your setting, click on the Create deployment option(purple box).

Creation of deployment should take about 5 min. Meanwhile, it will display the deployment credentials.

We will be using these to connect our server to Elastic Cloud.

Download the credentials(yellow box) somewhere as these are shown only once.
We will be using these credentials to connect our server to Elastic Cloud!

Once your deployment is ready, click on the continue button(orange box).

Click on the Explore on my own option(pink box).

You will see the Kibana home page.

We successfully created an Elastic Cloud deployment.

Move on to Part 4 to securely connect Elastic Cloud to Node.js server!

Part 2: Build a server using Node.js with Express

Lisa Jung — Wed, 07 Sep 2022 15:14:18 +0000

Table of Content | Read Next: Part 3 - Create an Elastic Cloud deployment

Let's take the first step in building a full stack app with Elasticsearch!

In this blog, we will be building a server using Node.js with Express.

Resources:

If you are relatively new to Node.js or Express, some of the terms or concepts mentioned in this blog may seem confusing to you.

Check out my blogs if you need some background info on Node.js or Express!

Would you rather watch a video to learn this content? Click on the link below!

Episode 2: Build a server using Node.js with Express

Want the code covered in this blog? Click on the link below to access it!

GitHub repo for part 2

Prerequisite installation:

iTerm2(for Mac only) or other terminal emulator of your choice
VS Code or other code editor of your choice
Node.js

Build a Node.js server

Step 1: Create an earthquake_app project directory

This project directory will eventually include both the server side and the client side code.

We will focus on the server side code for now!

Open up your terminal emulator. From this point on, I will use the term terminal and terminal emulator interchangeably.

In the location of your choosing, execute the following command in the terminal.

//in terminal
mkdir earthquake_app

Side Note: I keep all my projects in a directory called mini_crash_course. You will see this reflected in the screenshots of the terminal shared in this blog.

Step 2: Change the current working directory to earthquake_app

Execute the following command in the terminal.

//in terminal
cd earthquake_app

Step 3: Open the earthquake_app with the code editor of your choice

I use VS code. I have previously set up a code . command to open up VS Code straight from my terminal.

//in terminal
code .

When you execute steps 1-3 in the terminal, the terminal should look similar to the screenshot below.

If you do not use VS Code or have not set up a code . command, ignore the code . command.

Open your project directory with your code editor as you normally would.

When you do so, you will see the earthquake_app directory open on your code editor.

Step 4: Set up a Node.js development environment

As a prerequisite step, you have installed Node.js.

Let's initialize a new Node.js project so we can start building our server!

Go back to your terminal.

Within the earthquake_app directory, execute the following command.

//in terminal within the earthquake_app directory
npm init -y

This command accepts all default options in the terminal dialogue. It creates a package.json file in the root directory. This file keeps track of our app scripts and manages any dependencies our server needs.

In the terminal, you should see something similar to the following screenshot.

Go to your code editor to view the earthquake_app directory. You should see a package.json file within it.

Step 5: Install dependencies

For our server, we need to install the following dependencies.

@elastic/elasticsearch
axios
config
cors
express
log-timestamp
nodemon

Copy and paste the following command and execute it in the terminal.

//in terminal within the earthquake_app directory
npm i @elastic/elasticsearch axios config cors express log-timestamp nodemon

If you have no idea what these dependencies are used for, don't worry! We will go over these when the time comes.

You will see something like this in the terminal.

After executing this command in the terminal, take a look at your code editor. You will see that package.json and package-lock.json files have been automatically created.

If you look at the package.json file, you will see that it lists all the dependencies we have installed.

Step 6: Set up a server directory

Using your code editor, create a directory called server within the earthquake_app directory.

Within the server directory, create a server.js file. This file will be the entry point to the server and handle the server startup, routing, and other functions of our server.

Let's make sure our package.json knows this.

Open the package.json file then go to line 5.

You will see that "main" is set to "index.js"(line 5).

Replace "index.js" with "server.js" as shown below.

This will let package.json know that server.js will be the entry point to the server.

Next, let's specify what command we can use to execute server.js.

Within the package.json file and scroll down to line 6 "scripts". We will add an npm start script here.

Add a comma at the end of line 7 and add the following in line 8.

//in package.json line 8
"start": "nodemon server/server.js"

This step will allow us to execute the server.js file in the server directory by executing npm start in the terminal.

What is nodemon in front of server/server.js you ask?

In step 5, we installed a dependency called nodemon. Without this dependency, the server must be manually restarted every time you make changes to your code.

nodemon automatically restarts the server every time we make changes to our code so we don't have to do it ourselves!

We are now able to start the server by using the command npm start in the terminal.

Step 7: Create an Express app

Within the server.js file, copy and paste the following code.

//in server/server.js
const express = require('express');

const app = express();

const port = 3001;

app.listen(port, () => console.log(`Server listening at http://localhost:${port}`));

Let's go over this code snippet!

Heads up!

For reference purposes only, I have included screenshots of code that I will be explaining.

If you need to copy and paste the code, please refer to the code snippet above or the GitHub repo for part 2.

Line 1
In order to create a server with Express, we need to require the express library to access all the functionalities that come with it.

Line 3
Among these functionalities, express() is used to create an Express app. We set express() equal to a constant app here.

Line 5
We set the constant port to 3001.

Line 7
We instruct the server to listen for HTTP requests on port 3001 and print a message to the console explaining what browser URL you can use to test the server.

Step 8: Start the server

In the terminal, execute the following command to start the server.

//in terminal within the earthquake_app directory
npm start

In the terminal, you will see that your app is executing server.js in the server directory and that the server is listening at http://localhost:3001.

We now have a server that can be connected to Elastic Cloud!

Move on to Part 3 to create an Elastic Cloud deployment.

Part 1: What are we building and learning about in season 2?

Lisa Jung — Wed, 07 Sep 2022 15:14:10 +0000

Table of Content | Read Next: Part 2 - Build a server using Node.js with Express

Resources

Would you rather watch a video to learn this content? Click on the link below!

Episode 1: What's in store for season 2?

Do you want access to the GitHub repo that contains the code covered in this series?

GitHub repo for season 2

What are we building?

We will be building an Earthquake Watch app that enables the users to search for earthquakes.

By specifying the quake type, magnitude, location, and date range, the user can retrieve earthquakes that match the user input. The search results could be also sorted by ascending or descending order of magnitude.

Upon sending the request, the results are displayed in the form of cards containing detailed information about each earthquake.

App Architecture

The following diagram illustrates our app architecture:

We will build a full stack web app using Node.js with Express(server) and React(client). Next, we will connect Elastic Cloud to the server.

Data Ingestion

Elastic Cloud hosts Elastic products including Elasticsearch and Kibana.

Once the server is connected to Elastic Cloud, we will ingest data into Elasticsearch hosted on Elastic Cloud.

We will be ingesting global earthquake data from the USGS API.

The server will be in charge of fetching the data from the USGS API. Upon fetching the data, the server will send the data to Elasticsearch. Elasticsearch in turn will transform the data and ingest the data into an index.

Set up the app for search

After data is ingested into Elasticsearch, we will set up our app for search!

The client will display a set of criteria a user can select to search for earthquakes.

Once the user selects the criteria and clicks on the search button, the client will send the user input to the server. The server in turn will pass the user input into a Elasticsearch request and send the request to Elasticsearch.

Upon receiving the request, Elasticsearch will retrieve relevant results and send the results to the server. The server will send the results to the client which will display the results for the user.

Data Visualization

Learning how to connect Elasticsearch to our app and searching for information is a great way to explore our data.

In this series, we are going to explore our data even further by using Kibana to visualize earthquake data in Elasticsearch!

We will learn how to create these visualizations using Kibana Lens!

Move on to Part 2 to build a server using Node.js with Express!

Beginner's guide to building a full stack app (Node.js & React) with Elasticsearch

Lisa Jung — Wed, 07 Sep 2022 15:14:00 +0000

Want to build a full stack app (Node.js and React) with Elasticsearch but have no idea how to get started?

Season 2 of Mini Beginner's Crash Course to Elasticsearch and Kibana will teach you how to do just that!

Two Ways to Learn

We all have preferred method of learning so choose the format that works for you:

1. Video format(YouTube playlist)
Season 2 video titles start with S2 and have a thumbnail background similar to the following!

2. Blog format (Dev.to)
If you prefer to learn by reading instead, use the following table of content and go through the blogs in the following order!

Complementary GitHub Repo to the Series

Table of Content

Ready to get started?

Kickstart your journey with Part 1 - What are we building and learning about in season 2?

DEV Community: Lisa Jung

Beginner's guide to running Elasticsearch and Kibana v8+ Locally (macOS/Linux and Windows)

Resources

Download pages for Elasticsearch and Kibana

For macOS and Linux

Elasticsearch

Kibana

For Windows

Elasticsearch

Kibana

Part 10: Visualize data using Kibana Lens

Resources

What are we creating today?

Creating the Dashboard

Step 1: Open the Kibana home page

Step 2: Create a dashboard

Step 3: Create a table

Step 4: Save and name the dashboard

Step 5: Create a heat map

Step 6: Create metrics panels

Step 7: Create a magnitude histogram

Step 8: Create a donut chart

Part 9: Set up the Node.js server to handle Elasticsearch requests

Review

Resources

Set up the server to handle Elasticsearch request

Review from part 8

Final product

Part 8: Build the client side of the app with React

Review

Resources

Building the client

Step 1: Create a React app

Step 2: Install Axios

Step 3: Set up the proxy to our server.

Step 4: Build the client

Set up the client to display a list of search options and send user input to the server

Set up the client to handle documents received from the server

Step 5: Add styling to the page.

Part 7 : Set up the Node.js server to retrieve API data and send the data to Elasticsearch

Resources

Recognition

API Data Retrieval

Step 1: Set up your server to retrieve earthquake data from the USGS API

Add the data ingestion route to server.js

Step 1: open server.js within the server directory

Evaluate the outcome of data transformation and data ingestion

Part 6: Set up Elasticsearch for data transformation and data ingestion

Resources

Set up Elasticsearch for data transformation and data ingestion

Set up Elasticsearch for data transformation and data ingestion

Step 1: Review the data transformation requirements

Step 2: Create an ingest pipeline

Step 3: Add the desired processors to the pipeline

Step 4: Create an index called earthquakes with the desired mapping

Part 5: Plan for efficient data storage and search performance in Elasticsearch

Review from part 1

Resources

Numeric field types

Text field types

Part 4: Securely connect Node.js server to Elastic Cloud

Resources

Connect Node.js Server to Elastic Cloud

Part 3: Create an Elastic Cloud deployment

Resources

Create an Elastic Cloud deployment

Part 2: Build a server using Node.js with Express

Resources:

Prerequisite installation:

Build a Node.js server

Part 1: What are we building and learning about in season 2?

Resources

What are we building?

App Architecture

Data Ingestion

Set up the app for search

Data Visualization

Beginner's guide to building a full stack app (Node.js & React) with Elasticsearch

Two Ways to Learn

Table of Content

Add the data ingestion route to `server.js`

Step 1: open `server.js` within the `server` directory

Step 2: Create an `ingest pipeline`

Step 4: Create an index called `earthquakes` with the desired mapping