Deploy Elasticsearch 8.5 on Kubernetes with Okteto Cloud free plan

Guide to deploy Elasticsearch 8.5 cluster on Okteto Cloud, for free and with basic security settings

📢 Intro

Elasticsearch (ES) is probably the most common and famous search engine and has introduced a lot of new features with the 8.0 release, like the dense vector field type and the kNN search, that both combined allow Elasticsearch to be used for vector search for many machine-learning applications.

Okteto is an application that allows you to develop inside a container, along with many features it permit the user to start a development environment and provide an automatic SSL Endpoints for k8s.

Unfortunately the new security system introduced by ES 8.0 has produced problems with the official helm chart, so we cannot use the standard Okteto Chart deploy system.
In this article we will see how deploy ES 8.x into kubernetes (k8s) using the Okteto Cloud as platform.

✨ Features

• Elasticsearch 8.5.0 version
• Cluster composed of 3 nodes
• Deployable under the Okteto Cloud free tier
• Protected by Elasticsearch password, internode TLS and HTTPS connection
• Okteto development environment based on [busybox-curl](https://hub.docker.com/r/yauritux/busybox-curl) image

🚀 Steps

• Create an Okteto account, install and configure the Okteto CLI
• Clone the okteto-elasticsearch repo
• Generate the ES certificates:
  • Start Docker and run $ bash scripts/certgen-launcher.sh
• Deploy on Okteto
  • Run $ okteto deploy --build
  • Check the created endpoint from the previous output

• Call the ES endpoint:

  • Note: if not configured [1], <your-password> value is changeme

  $ curl -XGET -u elastic:<your-password> https://<your-endpoint-created>.cloud.okteto.net/_cat/nodes\\?v
  
  # Example:
  $ curl -XGET -u elastic:changeme <https://es01-http-mynamespace.cloud.okteto.net/_cat/nodes\\?v>
  ip          heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
  10.8.38.167            7          62  32    1.69    1.41     0.93 cdfhilmrstw *      es02
  10.8.38.166           10          60  27    1.69    1.41     0.93 cdfhilmrstw -      es01
  10.8.38.168           11          62  36    1.69    1.41     0.93 cdfhilmrstw -      es03
  
  

• Enjoy your cluster!

  • Do you want to use Kibana? see [2]
  • Don't waste free resources, if you don't need the cluster tear down everything with $ okteto destroy -v

✍️ Notes

• Security is provided by:
  • TLS internode communication with user-generated certificates
  • HTTPS endpoint with Okteto managed certificates
• Kubernetes
  • Instead of declaring directly the GKE ingress, we will use the Okteto provided auto SSL
    • Through the dev.okteto.com/auto-ingress: "true" annotation
  • We will create one ClusterIP for each note for port 9300
    • Because ES uses that as the default port for internode communication

🔧 How to

• [1] Change the default Elasticsearch password:
  • Generate the base64 new password
    • $ echo "NEW_PASSWORD" | tr -d \\\\n | base64 -w 0
  • Open the the k8s/elasticsearch.yml file
    • Use the generated value to replace the ELASTIC_PASSWORD value of the Secret component
• [2] Run Kibana locally
  • 🚧 Currently WIP, waiting this ES issue will be resolved
  • Run kibana locally and connect with Okteto cluster:
    • We run the docker locally to don't waste the okteto cloud resources

⚒️ Okteto

Development environment

• We could test the internode network thanks to Okteto development environment
  

  # Start the busybox-curl pod
  $ okteto up
  
  # The pod is mounted with all the local files, including the certificates:
  > ls -l /okteto/
  Dockerfile  README.md   certs       k8s         okteto.yml  scripts
  
  # The pod is deployed into the cluster and could use the certificates:
  > curl -u elastic:changeme es-http:9200
  {
    "name" : "es01",
    "cluster_name" : "okteto-cluster",
  ...
  
  > nc -vz es01 9300
  es01 (10.153.19.186:9300) open
  
  

Sleeping system

• Q: "How can I restart a sleeping development environment?" - link
  • A: Visit any of the public endpoints of your development environment

Okteto useful commands

# Log into the cluster
$ okteto init

# Deploy the local `okteto.yml` - wait 5/10m
$ okteto deploy --wait

# Activate a development container
# > <https://www.okteto.com/docs/reference/cli/#up>
$ okteto up

# Create kubectl context to Okteto cloud
$ okteto kubeconfig
$ kubectl get po

🛂 Disclaimer

This repository is built for side-project purposes and no warranties are provided.
Activities to keep in mind before using in production environments includes but are not limited to:

• We will arbitrarily expose the es01 node as API server:
  • So we don't have load balancing between the API requests
  • There is no guarantee that es01 isn't chosen as the master node
• Create a more robust ES architecture with dedicated ES master nodes
• Fine-tune the ES nodes' roles and HW requirements
• All the points listed in the "TODO" section

🔗References

• Github repository: https://github.com/pistocop/okteto-elasticsearch
• Article on my blog: https://www.pistocop.dev/posts/deploy_es_on_okteto/
• Okteto documentation: https://www.okteto.com/docs/welcome/overview/
• Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html