DEV Community: pistocop

🛠️ A gentle introduction to GKE private cluster deployment

pistocop — Thu, 02 Mar 2023 07:33:25 +0000

🧭 Study how to deploy GKE private cluster using terraform and expose an echo server

🔗 Repo: https://github.com/pistocop/gke-basic-cluster-deployment
⚠️ For production deployment use Terraform Kubernetes Engine Module
📧 Found an error or have a question? write to me

📢 Intro

Kubernetes (k8s), although don’t require my introduction, is the most famous and widely adopted container manager in the world. Hosting by yourself is undoubtedly a very advanced and expert topic, so the major of companies choose a provider that provides it as managed service.

There are multiple famous k8s hosting services (e.g. GKE, EKS, AKS, Okteto), but is no doubt that one of the leading is Google Kubernetes Engine (GKE), a product provided by Google Cloud Platform (GCP).

So can we simply open GKE, start a cluster and we are ready to go? Well… yes but no, it maybe could work until something further structured and production ready is required. So then the GKE settings and tweaks start to emerge and require to be addressed. Take for example the official https://github.com/terraform-google-modules/terraform-google-kubernetes-engine terraform deploy: there are a lot of parameters that can really change how GKE will be deployed and employed.

So, simple things first: we will don’t go through all the possible parameters but we will, here, see a basic deployment of GKE with a description of the main settings that a deployment should address.
For better understanding - and code reusage - we deploy the system using [terraform](https://www.terraform.io/), this will give us the prospect to easily explain all the components with the unique clarity that belong to the code.

🚀 Deploy

Prerequisites
- [terraform](https://www.terraform.io/)
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
- [gcloud](https://cloud.google.com/sdk/gcloud)
- [tfswitch](https://tfswitch.warrensbox.com/) - optional
Download the repo https://github.com/pistocop/gke-basic-cluster-deployment
Enable the following GCP APIs using the GCP console [1]
- Open GCP console -> APIs & Services -> enable APIs and services
- Enable:
  - Compute Engine API
  - Kubernetes Engine API

Compile the input variables:

    $ cp iac/variables.tfvars.example iac/variables.tfvars
    $ vi iac/variables.tfvars # replace with your data

Deploy the cluster:

Replace\set the variables with your data

    # configure gcloud to the desired project
    $ gcloud config set project $PROJECT_ID

    # configure terraform
    $ cd iac
    $ tfswitch
    $ terraform init

    # deploy the GKE pre-requisites
    $ terraform plan -out out.plan -var-file="./variables.tfvars" -var  deploy_cluster=false
    $ terraform apply out.plan

    # deploy GKE - can take more than 20 minutes
    $ terraform plan -out out.plan -var-file="./variables.tfvars" -var deploy_cluster=true
    $ terraform apply out.plan

Deploy the services into k8s

Replace\set the variables with your data

    # set kubectl context
    $ gcloud container clusters get-credentials gkedeploy-cluster --zone $PROJECT_REGION --project $PROJECT_ID

    # create common resources
    $ kubectl apply -f k8s/common

    # deploy the server
    $ kubectl apply -f k8s/gechoserver/

    # wait that the ADDRESS will be displayed - can take more than 10 minutes
    $ kubectl -n dev get ingress -o wide
    NAME          CLASS    HOSTS   ADDRESS          PORTS   AGE
    gechoserver   <none>   *       34.120.114.207   80      67s

    # query the server from internet - can take  more than 10 minutes
    # replace "34.120.114.207" with your address:
    $ curl -XPOST -v <http://34.120.114.207/> -d "foo=bar"

    # ~ Congratulation, your server on GKE is up and running! ~

Destroy the cluster:

    $ cd iac
    $ terraform destroy -auto-approve -var-file="./variables.tfvars" -var deploy_cluster=true

🏗️ Architecture

Terraform

The k8s cluster provider is GKE from Google Cloud Platform (GCP)
The terraform state is stored only locally (e.g. no backend on GCS)

GKE

Deployed as private cluster, so it depends only on internal IPs
Deployed in VPC-native mode:
- The traffic to a specific pod will be routed directly to the correct node thanks to the container native LB
- We deploy the Ingress back-end as ClusterIP instead of NodePort to leverage the container native load balancing
The terraform variable deploy_cluster steers the cluster creation, set to false to create only the networks components
Use a Service Account named gkedeploy-sa

Network

No NAT will be deployed
- Therefore the system cannot pull images from public container ◦ registries like Docker Hub, read more under Tips and Takeaways
VPC doesn't create a subnet for each zone
Two subnetworks are provided:
- gkedeploy-subnet: with the range 10.10.0.0/24 is the subnetwork where GKE nodes will be deployed
  - Instances within this network can access Google APIs and services by using Private Google Access
- gkedeploy-lb-proxy-only-subnet: with the range 10.14.0.0/23 is a proxy only subnet and is required by GCP to reserve a range of IPs used to deploy the Load Balancers
VPC-Native cluster alias IP range could be checked under "Console -> VPC network details -> secondary IPv4 ranges"
- Under that field we find both the cluster_ipv4_cidr_block (for pods - 10.11.0.0/21) and services_ipv4_cidr_block (for services - 10.12.0.0/21) values,
GKE hosted (by Google) master's nodes will use the 10.13.0.0/28 range, see master_ipv4_cidr_block parameter

🥪 Tips and Takeaways

Every component and setting described here is reported on the terraform code, with also more insights, read the code to grasp those concepts
We deploy GKE after other resources (this is why we had 2 terraform plans) because otherwise, sometimes the GKE deployment remains infinitely stuck during the health check process, and terraform returns the error Error: Error waiting for creating GKE cluster: All cluster resources were brought up, but [...]
- Maybe the error is due from SA not yet deployed/up&running, so try to deploy firstly all the resources using var deploy_cluster=false and then deploy the cluster using terraform with var deploy_cluster=true
To pull docker images without adding the NAT component we have two choices (memo: we have Private Google Access enabled):
Enable the Artifact Registry service on your GCP project and upload/mirror the desired images
Chose the images to use from the public Google Container Registry that has a Google allowed IP (e.g. the k8s/gechoserver deployment)
By default, you cannot reach GCE vm using -tunnel-through-iap because the firewalls block that connection
- We add fw-iap firewall rule to terraform in order to use this GCP functionality, named IAP for TCP forwarding
[1] We can write Terraform code to enable the GCP APIs, but is opinionated that we should not
On terraform, under the GKE section, why master_ipv4_cidr_block is required?
- because the k8s master(s) are managed by Google and a peering connection will be created from the Google network with the GKE network
- due to this connection, Google needs to know a free IP range used to assign IPs to the master's components

When deploying a k8s Service object, pay attention when defining UDP/TCP ports: wrong usages fail silently

Example:

Start 2 pods (A and B), declare ClusterIP on port 80 for TCP connection

Run the following code:

# ~ With TCP:
# Client A:
$ nc -l -p 8080

# Client B:
$ nc network-multitool 8080
hello in TCP

# Client A:
hello in TCP # <-- msg received

# ~ With UDP:
# Client A:
$ nc -l -u -p 8081

# Client B:
$ nc -u network-multitool 8081
hello in UDP

# Client A:
<nothing>

🔗 Resources

[1] Can I automatically enable APIs when using GCP cloud with terraform? - so
[2] Best managed kubernetes platform - reddit
Learn Terraform - Provision a GKE Cluster - gh
Official GCP Terraform provider - doc
GKE Ingress for HTTP(S) Load Balancing - doc
Network overview - doc
VPC-native clusters - doc
DNS on GKE: Everything you need to know - medium
A trip with Google Global Load Balancers - medium

Deploy Elasticsearch 8.5 on Kubernetes with Okteto Cloud free plan

pistocop — Fri, 18 Nov 2022 19:21:11 +0000

Guide to deploy Elasticsearch 8.5 cluster on Okteto Cloud, for free and with basic security settings

📢 Intro

Elasticsearch (ES) is probably the most common and famous search engine and has introduced a lot of new features with the 8.0 release, like the dense vector field type and the kNN search, that both combined allow Elasticsearch to be used for vector search for many machine-learning applications.

Okteto is an application that allows you to develop inside a container, along with many features it permit the user to start a development environment and provide an automatic SSL Endpoints for k8s.

Unfortunately the new security system introduced by ES 8.0 has produced problems with the official helm chart, so we cannot use the standard Okteto Chart deploy system.
In this article we will see how deploy ES 8.x into kubernetes (k8s) using the Okteto Cloud as platform.

✨ Features

Elasticsearch 8.5.0 version
Cluster composed of 3 nodes
Deployable under the Okteto Cloud free tier
Protected by Elasticsearch password, internode TLS and HTTPS connection
Okteto development environment based on [busybox-curl](https://hub.docker.com/r/yauritux/busybox-curl) image

🚀 Steps

Create an Okteto account, install and configure the Okteto CLI
Clone the okteto-elasticsearch repo
Generate the ES certificates:
- Start Docker and run $ bash scripts/certgen-launcher.sh
Deploy on Okteto
- Run $ okteto deploy --build
- Check the created endpoint from the previous output

Call the ES endpoint:

Note: if not configured [1], <your-password> value is changeme

$ curl -XGET -u elastic:<your-password> https://<your-endpoint-created>.cloud.okteto.net/_cat/nodes\\?v

# Example:
$ curl -XGET -u elastic:changeme <https://es01-http-mynamespace.cloud.okteto.net/_cat/nodes\\?v>
ip          heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
10.8.38.167            7          62  32    1.69    1.41     0.93 cdfhilmrstw *      es02
10.8.38.166           10          60  27    1.69    1.41     0.93 cdfhilmrstw -      es01
10.8.38.168           11          62  36    1.69    1.41     0.93 cdfhilmrstw -      es03

Enjoy your cluster!
- Do you want to use Kibana? see [2]
- Don't waste free resources, if you don't need the cluster tear down everything with $ okteto destroy -v

✍️ Notes

Security is provided by:
- TLS internode communication with user-generated certificates
- HTTPS endpoint with Okteto managed certificates
Kubernetes
- Instead of declaring directly the GKE ingress, we will use the Okteto provided auto SSL
  - Through the dev.okteto.com/auto-ingress: "true" annotation
- We will create one ClusterIP for each note for port 9300
  - Because ES uses that as the default port for internode communication

🔧 How to

[1] Change the default Elasticsearch password:
- Generate the base64 new password
  - $ echo "NEW_PASSWORD" | tr -d \\\\n | base64 -w 0
- Open the the k8s/elasticsearch.yml file
  - Use the generated value to replace the ELASTIC_PASSWORD value of the Secret component
[2] Run Kibana locally
- 🚧 Currently WIP, waiting this ES issue will be resolved
- Run kibana locally and connect with Okteto cluster:
  - We run the docker locally to don't waste the okteto cloud resources

⚒️ Okteto

Development environment

We could test the internode network thanks to Okteto development environment

# Start the busybox-curl pod
$ okteto up

# The pod is mounted with all the local files, including the certificates:
> ls -l /okteto/
Dockerfile  README.md   certs       k8s         okteto.yml  scripts

# The pod is deployed into the cluster and could use the certificates:
> curl -u elastic:changeme es-http:9200
{
  "name" : "es01",
  "cluster_name" : "okteto-cluster",
...

> nc -vz es01 9300
es01 (10.153.19.186:9300) open

Sleeping system

Q: "How can I restart a sleeping development environment?" - link
- A: Visit any of the public endpoints of your development environment

Okteto useful commands

# Log into the cluster
$ okteto init

# Deploy the local `okteto.yml` - wait 5/10m
$ okteto deploy --wait

# Activate a development container
# > <https://www.okteto.com/docs/reference/cli/#up>
$ okteto up

# Create kubectl context to Okteto cloud
$ okteto kubeconfig
$ kubectl get po

🛂 Disclaimer

This repository is built for side-project purposes and no warranties are provided.
Activities to keep in mind before using in production environments includes but are not limited to:

We will arbitrarily expose the es01 node as API server:
- So we don't have load balancing between the API requests
- There is no guarantee that es01 isn't chosen as the master node
Create a more robust ES architecture with dedicated ES master nodes
Fine-tune the ES nodes' roles and HW requirements
All the points listed in the "TODO" section

🔗References

Github repository: https://github.com/pistocop/okteto-elasticsearch
Article on my blog: https://www.pistocop.dev/posts/deploy_es_on_okteto/
Okteto documentation: https://www.okteto.com/docs/welcome/overview/
Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html