Elasticsearch and Kibana: A Roller Coaster Full of Exciting Changes: The Rolling Upgrade Adventure.

Emre — Wed, 31 May 2023 21:33:09 +0000

Elasticsearch Rolling Upgrade is a method of gradually transitioning from an old version of Elasticsearch to a new version. In this approach, Elasticsearch nodes are updated one by one, and after each update process is completed, the cluster remains operational. This allows for updating the Elasticsearch version without any interruptions in the system.

This document will describe the process of performing a rolling upgrade to 8.7.0. Before proceeding with the upgrade, it is essential to read the release officail documentation and verify if there are any breaking changes that might affect the services.

Important Notes

Make sure to take a backup of the elasticsearch.yml, kibana.yml file.
Before the upgrade, be sure to take snapshot from all nodes.
Start the process by upgrading the nodes which are not master eligible, and then proceed to upgrade the master eligible nodes, be carefull to update the asterisk (*) marked one must be last.

To avoid any performance issues during the upgrade, please execute the following commands in Kibana DevTools:

Disable shard allocation

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

Stop non-essential indexing and perform a flush

POST /_flush

Then connect the node you want to upgrade and run following commands to upgrade 8.7.0 version.

apt update

systemctl stop elasticsearch

sudo apt-get --only-upgrade install elasticsearch=8.7.0

If you want to keep your settings as they are (elasticsearch.yml), you can proceed with the default option, which is "no" (N).

systemctl daemon-reload
systemctl start elasticsearch

After these commands, you can check if the Elasticsearch service is up and running using the following command:

systemctl status elasticsearch

If the service is active, it means there are no issues. However, if there are problems with the service, you can check the relevant logs under the /var/log/elasticsearch directory. These logs contain important information about the Elasticsearch service, such as error messages, warnings, and other diagnostic details. Examining these logs can help you identify and troubleshoot any issues that may have occurred during or after the upgrade process.

Once you see that the node has rejoined the cluster using the "GET _cat/nodes?v" command in Kibana, you can wait for the unassigned shards to be allocated by running the following command.

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}

When the cluster becomes green again, you can repeat the process for other nodes.

Kibana

To ensure compatibility, it is indeed recommended to update Kibana along with Elasticsearch to the compatible version. Before performing the Kibana upgrade, it is advisable to take a snapshot of the data and also create a backup of the kibana.yml configuration file.

Additionally, it is crucial to review the release notes
of the target Kibana version to check for any critical changes or considerations.

By following these recommendations, you can minimize the risk of version incompatibilities and ensure a smooth upgrade of both Elasticsearch and Kibana while preserving your data and configurations.

After performing the necessary checks, you can proceed with updating Kibana using the following commands:

apt update

systemctl stop kibana

sudo apt-get --only-upgrade install kibana=8.7.0

systemctl restart kibana

Important Note!

By default, Kibana uses port 5601. After executing the previous commands, Kibana will start up and become accessible after a while.

However, in some cases, the port used by Kibana may be different. For example, if Kibana is configured to use HTTPS, it may attempt to start on port 443. In such cases, it is necessary to grant permissions for Kibana to use the specific port.

If you face such a situation, you can use the following commands to grant permissions:

setcap cap_net_bind_service=+epi /usr/share/kibana/bin/kibana
setcap cap_net_bind_service=+epi /usr/share/kibana/bin/kibana-plugin
setcap cap_net_bind_service=+epi /usr/share/kibana/bin/kibana-keystore
setcap cap_net_bind_service=+epi /usr/share/kibana/node/bin/node

systemctl restart kibana