DEV Community: Yasitha Bogamuwa

Analyze AWS Application Load Balancer logs using Amazon Athena

Yasitha Bogamuwa — Sat, 09 Mar 2024 21:31:24 +0000

1. Introduction

The AWS Application Load Balancer logs show details about the requests sent to your ALB, like where they came from, what was requested, and if it was successful. These logs help with fixing problems and understanding how well your system is working.

I recently wanted to analyze ALB access logs to identify suspicious activity. However, the logs are stored in gzip format in an S3 bucket with hundreds of thousands of files, making manual analysis impractical. To analyze these logs, you can use Amazon Athena, a tool that lets you query data stored in Amazon S3 using regular SQL commands.

2. What is Amazon Athena

Amazon Athena is an interactive query service provided by AWS that allows you to analyze and query data stored in Amazon S3 using standard SQL. It enables you to run ad-hoc queries on data in S3 without the need for complex ETL processes or data movement.

Since Athena is serverless, you don't need to handle any infrastructure, and you are charged solely based on the queries you execute.

3. Analyzing ALB Access Logs with Athena

3.1. Make sure to enable Application Load Balancer access logs as described here so that the access logs can be saved to your Amazon S3 bucket.

3.2. Open the Athena console and click Launch query editor.

3.3. Create an Athena database and table for Application Load Balancer logs. To create an Athena database, please run the following command in Query Editor. It's recommended to create the database in the same AWS Region as the Amazon S3 bucket.

CREATE DATABASE <DATABASE_NAME>

3.4. Then, select the database from the dropdown and create an alb_logs table for the ALB logs. Make sure to replace the <YOUR-ALB-LOGS-DIRECTORY>, <ACCOUNT-ID>, and <REGION> with the correct values.

CREATE EXTERNAL TABLE IF NOT EXISTS alb_logs (
            type string,
            time string,
            elb string,
            client_ip string,
            client_port int,
            target_ip string,
            target_port int,
            request_processing_time double,
            target_processing_time double,
            response_processing_time double,
            elb_status_code int,
            target_status_code string,
            received_bytes bigint,
            sent_bytes bigint,
            request_verb string,
            request_url string,
            request_proto string,
            user_agent string,
            ssl_cipher string,
            ssl_protocol string,
            target_group_arn string,
            trace_id string,
            domain_name string,
            chosen_cert_arn string,
            matched_rule_priority string,
            request_creation_time string,
            actions_executed string,
            redirect_url string,
            lambda_error_reason string,
            target_port_list string,
            target_status_code_list string,
            classification string,
            classification_reason string
            )
            ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
            WITH SERDEPROPERTIES (
            'serialization.format' = '1',
            'input.regex' = 
        '([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*):([0-9]*) ([^ ]*)[:-]([0-9]*) ([-.0-9]*) ([-.0-9]*) ([-.0-9]*) (|[-0-9]*) (-|[-0-9]*) ([-0-9]*) ([-0-9]*) \"([^ ]*) (.*) (- |[^ ]*)\" \"([^\"]*)\" ([A-Z0-9-_]+) ([A-Za-z0-9.-]*) ([^ ]*) \"([^\"]*)\" \"([^\"]*)\" \"([^\"]*)\" ([-.0-9]*) ([^ ]*) \"([^\"]*)\" \"([^\"]*)\" \"([^ ]*)\" \"([^\s]+?)\" \"([^\s]+)\" \"([^ ]*)\" \"([^ ]*)\"')
            LOCATION 's3://<YOUR-ALB-LOGS-DIRECTORY>/AWSLogs/<ACCOUNT-ID>/elasticloadbalancing/<REGION>/';

3.5. In the Query Editor settings, choose an S3 bucket to store the results of your Athena queries.

3.6. Now you can use SQL syntax to query the access logs.

The following SQL query in counts the occurrences of different request verbs for requests containing 'hs' in the URL from the 'alb_logs' table. It groups the results by request verb, client IP, and request URL, and limits the output to the first 100 results.

SELECT COUNT(request_verb) AS
 count,
 request_verb,
 client_ip,
 request_url
FROM alb_logs
WHERE request_url LIKE '%hs%'
GROUP BY request_verb, client_ip, request_url
LIMIT 100;

4. References

4.1. Querying Application Load Balancer logs
4.2. Analyzing ALB Access Logs with Amazon Athena
4.3. How do I use Amazon Athena to analyze ALB access logs

Unveiling the Speed Mystery: Investigating Slow S3 Uploads from AWS EKS Pods

Yasitha Bogamuwa — Sun, 30 Jul 2023 13:06:03 +0000

The Problem

At my current workplace, we have been using AWS EKS unmanaged clusters. Recently, our engineering teams have noticed an alarming issue with unreasonably slow S3 file uploads from their applications.

To investigate further, they conducted a comparison by running the same application on their local computers. Surprisingly, they observed that uploading a 10KB file to S3 took approximately 5 seconds on our EKS cluster. However, on their local machines, the identical upload process took approximately 1 second. Consistent testing revealed an average slowness factor of 5 when compared to the local computer uploads.

This noticeable increase in latency raised significant concerns, urging us to uncover the underlying root cause to identify possible solutions.

Investigation

To identify the issue and locate the bottleneck, we came up with the following plan:

1. Create a S3 bucket in the region where our EKS cluster is running.

2. Deploy a debug pod in the EKS cluster using the latest AWS CLI docker image.

# Debug pod configurations
apiVersion: apps/v1
kind: Deployment
metadata:
  name: debug
spec:
  selector:
    matchLabels:
      app: debug
  replicas: 1
  template:
    metadata:
      labels:
        app: debug
    spec:
      containers:
      - name: debug
        image: amazon/aws-cli
        command: ["sleep", "infinity"]

3. Locate the EKS node where the debug pod is running.

kubectl get pods -l=app=debug -o wide

4. Create a 10KB file and upload it to the S3 bucket using AWS CLI from both the debug pod and EKS node where the debug pod is deployed. Measure the time taken for each upload.

# AWS CLI upload command
time aws s3 --debug cp 10kb.png s3://yasithab-debug-bucket/10kb.png

Identifying the Issue

Based on the test conducted, it was discovered that uploading the 10KB file from the EKS node takes around 1 second, which is similar to uploads from a local computer. However, uploading the same file from the pod running on the same EKS node takes approximately 5 seconds. This indicates that there is something in between causing this issue, but the exact cause is still unknown.

# AWS CLI upload results from the EKS node where the debug pod is running
real    0m1.460s
user    0m0.908s
sys     0m0.085s

# AWS CLI upload results from the debug pod
real    0m5.612s
user    0m0.956s
sys     0m0.073s

As we have AWS enterprise support, we reached out to them with the test results and debug logs. They recommended provisioning a new EKS cluster and conducting the same test to isolate any other dependencies. However, we decided against this approach as we consider it unnecessary to set up a new cluster solely for debugging this issue. Therefore, we are currently on our own in resolving this matter, unfortunately.

With no clear direction at this stage, a possible course of action is to compare the AWS CLI debug logs for both node and pod uploads. This can be achieved by utilizing a file comparison tool. Initially, the focus was on investigating potential network delays by comparing the duration of the upload process.

The upload process began when the CLI displayed the following lines.

MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
MainThread - botocore.utils - DEBUG - Registering S3 region redirector handler

The upload process ended when the CLI displayed the following lines.

upload: ./10kb.png to s3://yasithab-debug-bucket/10kb.png
Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

Then, the duration from the beginning of the upload to the end of the upload event was calculated.

In the node upload process, the start time was recorded as 2023-07-13 11:03:14,047, and the end time was recorded as 2023-07-13 11:03:14,161. Hence, the upload process took 114 milliseconds.
In the pod upload process, the start time was recorded as 2023-07-13 10:57:54,740, and the end time was recorded as 2023-07-13 10:57:54,828. Hence, the upload process took 88 milliseconds.

The almost identical upload times indicate that this issue is not caused by network latency.

Upon reviewing the logs further, it was observed that in the pod upload debug log, most of the time was spent on establishing the connection to the Instance Metadata Service (IMDS) endpoint. This process took around 1 second and was attempted three times, totaling approximately 4 seconds. On the other hand, the initialization of the connection to the IMDS endpoint from the EKS node where the debug pod is deployed, only took 190 milliseconds.

# From the EKS node where the debug pod is deployed - 190 milliseconds
2023-07-13 11:03:13,857 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2023-07-13 11:03:14,047 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)

# From the debug pod - 4 seconds and 259 milliseconds
2023-07-13 10:57:50,481 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2023-07-13 10:57:54,740 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)

When comparing the total time taken from the debug pod (5 seconds and 612 milliseconds) with the time taken for establishing the connection to the Instance Metadata Service (IMDS) endpoint from the debug pod (4 seconds and 259 milliseconds), a difference of 1 second and 353 milliseconds is observed. Remarkably, this difference closely matches the time taken from the EKS node upload results.

Hooray! The Instance Metadata Service (IMDS) has been identified as the bottleneck. However, the next step is to understand why this is happening and determine how to resolve it.

What is IMDS and What it does here?

The AWS Instance Metadata Service (IMDS) is a service that allows EC2 instances to access information about themselves. This information includes details like the instance ID, instance type, network configurations, identity credentials, security groups, and more. Applications running on EC2 instances can use IMDS to retrieve dynamic information about the instance. IMDS has two versions available.

Instance Metadata Service Version 1 (IMDSv1) – a request/response method.
Instance Metadata Service Version 2 (IMDSv2) – a session-oriented method.

By default, you have the option to use either IMDSv1 or IMDSv2, or both. The instance metadata service differentiates between IMDSv1 and IMDSv2 requests based on the presence of specific PUT or GET headers, which are exclusive to IMDSv2. If you configure IMDSv2 to be used in the MetadataOptions, IMDSv1 will no longer function.

To retrieve the MetadataOptions for an EKS node, use the following command. Replace <EKS_NODE_INSTANCE_ID> with the specific instance ID of the EKS node you wish to retrieve the MetadataOptions for.

aws ec2 describe-instances --instance-ids <EKS_NODE_INSTANCE_ID> --query "Reservations[].Instances[].MetadataOptions"

The above command will give an output similar to the following:

[
    {
        "State": "applied",
        "HttpTokens": "optional",
        "HttpPutResponseHopLimit": 1,
        "HttpEndpoint": "enabled",
        "HttpProtocolIpv6": "disabled",
        "InstanceMetadataTags": "disabled"
    }
]

In the output, it shows that the HttpTokens attribute is set to optional.

When the HttpTokens is set to optional, it sets the use of IMDSv2 to optional. You can retrieve instance metadata with or without a session token. Without a token, IMDSv1 role credentials are returned, and with a token, IMDSv2 role credentials are returned.

On the other hand, when HttpTokens is set to required, IMDSv2 becomes mandatory. You must include a session token in your instance metadata retrieval requests. In this case, only IMDSv2 credentials are available; IMDSv1 credentials cannot be accessed.

IMDSv2 is an improved version of instance metadata access that adds an extra layer of security against unauthorized access. To use IMDSv2, a PUT request is required to establish a session with the instance metadata service and obtain a token. By default, the response to PUT requests has a response hop limit of 1 at the IP protocol level.

However, this limit is not suitable for containerized applications on Kubernetes that operate in a separate network namespace from the instance. EKS managed node groups that are newly launched or updated will have a metadata token response hop limit of 2. However, in the case of self-managed nodes like ours, the default hop limit is set to 1. Therefore, setting the HttpPutResponseHopLimit to 2 in our EKS is mandatory for retrieving InstanceMetadata over IMDSv2.

How does the upload work with HttpPutResponseHopLimit set to 1 despite the slow upload speed?

When the HttpTokens attribute is set to optional, the AWS CLI/SDK automatically use IMDSv2 calls by default. If the initial IMDSv2 call does not receive a response, the CLI/SDK retries the call and, if still unsuccessful, fallback to using IMDSv1. This fallback process can introduce delays, especially in a container environment.

In a container environment, if the hop limit is set to 1, the IMDSv2 response does not return because accessing the container is considered an additional network hop. To avoid the fallback to IMDSv1 and the resulting delay, it is recommended to set the hop limit to 2 in a container environment.

Resolve the issue

According to AWS documentation, containerized applications running on EKS require the EKS node HttpPutResponseHopLimit in MetadataOptions to be set to 2. To configure this, you can use the following command, replacing <EKS_NODE_INSTANCE_ID> with the instance ID of the specific EKS node you want to retrieve MetadataOptions for.

aws ec2 modify-instance-metadata-options --instance-id <EKS_NODE_INSTANCE_ID> --http-endpoint enabled --http-put-response-hop-limit 1

In many cases, EKS nodes are part of AWS auto scaling groups, making manual modification of instance metadata difficult. To configure Instance Metadata Service (IMDS) for an auto scaling group, you need to modify its associated launch configurations.

To set the HttpPutResponseHopLimit for all instances associated with the launch configuration, go to the Additional configuration section under Advanced details of the template. Set the Metadata response hop limit to 2. If no value is specified, the default is set to 1.

When using Terraform to provision your infrastructure, you can utilize the aws_launch_template resource to modify instance metadata in the following manner.

resource "aws_launch_template" "foo" {
  name = "foo"

  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "optional"
    instance_metadata_tags      = "disabled"
    http_put_response_hop_limit = 2
  }
}

After implementing the HttpPutResponseHopLimit modifications, we conducted the same tests as in the investigation phase. The results of both the EKS node and debug pod uploads were nearly identical.

# AWS CLI upload results from the EKS node where the debug pod is running
real    0m1.310s
user    0m0.912s
sys     0m0.078s

# AWS CLI upload results from the debug pod
real    0m1.298s
user    0m0.931s
sys     0m0.082s

Conclusion

In conclusion, our investigation into the significantly slow S3 file uploads from applications running on AWS EKS clusters revealed that the Instance Metadata Service (IMDS) was the root cause of the issue. By comparing upload times from EKS nodes and pods, we identified a delay in establishing the connection to the IMDS endpoint from within the pods. This delay was not present when uploading directly from the EKS nodes.

To resolve this issue, it is necessary to set the HttpPutResponseHopLimit in the MetadataOptions to 2 for EKS nodes. This can be achieved by modifying the associated launch configurations for auto scaling groups or using the aws_launch_template resource in Terraform.

By setting up the HttpPutResponseHopLimit to 2, we can eliminate the latency and significantly improve the speed of file uploads to S3 from AWS EKS pods. This resolution will enhance the overall performance and efficiency of our applications on the EKS cluster.

References

Multi-Master Kubernetes Cluster Setup with CRI-O and vSphere Storage on Rocky Linux 8

Yasitha Bogamuwa — Thu, 16 Jun 2022 16:15:58 +0000

Overview

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community.

1. Architecture Diagram

2. System Requirements

Before you start installing Kubernetes, you must understand your capacity requirements and provision resources accordingly. The following resource allocations are based entirely on my needs, and to certain extent, you may need to downgrade or upgrade resources.

2.1. Master Nodes

Component	Description
Number of VMs	3
CPU	2 Cores
Memory	8 GB
Disk Size	150 GB SSD
Storage Type	Thin Provision
Operating System	Rocky Linux 8 x64
File System	XFS
Privileges	ROOT access prefered

2.2. Worker Nodes

Component	Description
Number of VMs	3
CPU	4 Cores
Memory	16 GB
Disk Size	500 GB SSD
Storage Type	Thin Provision
Operating System	Rocky Linux 8 x64
File System	XFS
Privileges	ROOT access prefered

2.3. Nginx Load Balancers

Component	Description
Number of VMs	2
CPU	2 Cores
Memory	4 GB
Disk Size	20 GB SSD
Storage Type	Thin Provision
Operating System	Rocky Linux 8 x64
File System	XFS
Privileges	ROOT access prefered

2.4. IP Allocations

Component	Description
Load Balancer Virtual IP	192.168.16.80
VM IPs	192.168.16.100 - 192.168.16.108
MetalLB IP Pool	192.168.16.200 - 192.168.16.250

2.5. DNS Entries

IP	Hostname	FQDN
192.168.16.80	N/A	kube-api.example.local
192.168.16.100	kubelb01	kubelb01.example.local
192.168.16.101	kubelb02	kubelb02.example.local
192.168.16.102	kubemaster01	kubemaster01.example.local
192.168.16.103	kubemaster02	kubemaster02.example.local
192.168.16.104	kubemaster03	kubemaster03.example.local
192.168.16.105	kubeworker01	kubeworker01.example.local
192.168.16.106	kubeworker02	kubeworker02.example.local
192.168.16.107	kubeworker03	kubeworker03.example.local

2.6. VMware Roles and Service Accounts

2.6.1. In order to create a VMware role, navigate to Menu -> Administration -> Roles in the vSphere Client. The roles and permissions required for dynamic provisioning can be found here. For this example I will create a role called manage-kubernetes-node-vms-and-volumes with the following permissions.

2.6.2. Navigate to Menu -> Administration -> Single Sign On -> Users and Groups in the vSphere Client and create a new user. For this example I will create a user called "kubernetes@vsphere.local" with the password "KuB3VMwar3". For now, please make sure NOT to use special characters in the password.

2.6.3. Navigate to Menu -> Administration -> Global Permissions in the vSphere Client and click add permission (+) icon. Then select newly created role and map it to kubernetes user. Please make sure to select Propagate to children option.

3. Configure Nginx Load Balancers

🔵 Important

Verify the MAC address and product_uuid are unique for every node. You can get the MAC address of the network interfaces using the below command.
ip link | grep link/ether
The product_uuid can be checked by using the following command.
cat /sys/class/dmi/id/product_uuid

3.1. Set server hostname.



# Example:
# hostnamectl set-hostname kubelb01

hostnamectl set-hostname <hostname>

3.2. Install prerequisites.



# Clean YUM repository cache
dnf clean all

# Update packages
dnf update -y

# Install prerequisites
dnf install -y vim net-tools chrony ntpstat keepalived nginx policycoreutils-python-utils

3.3. Synchronize server time with Google NTP server.



# Add Google NTP Server
sed -i '/^pool/c\pool time.google.com iburst' /etc/chrony.conf

# Set timezone to Asia/Colombo
timedatectl set-timezone Asia/Colombo

# Enable NTP time synchronization
timedatectl set-ntp true

3.4. Start and enable chronyd service.



# Start and enable chronyd service
systemctl enable --now chronyd

# Check if the chronyd service is running
systemctl status chronyd

3.5. Display time synchronization status.



# Verify synchronisation state
ntpstat

# Check Chrony Source Statistics
chronyc sourcestats -v

3.6. Permanently disable SELinux.



# Permanently disable SELinux
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config

3.7. Disable IPv6 on network interface.



# Disable IPv6 on ens192 interface
nmcli connection modify ens192 ipv6.method ignore

3.8. Execute the following commands to turn off all swap devices and files.



# Permanently disable swapping
sed -e '/swap/ s/^#*/#/g' -i /etc/fstab

# Disable all existing swaps from /proc/swaps
swapoff -a

3.9. Disable File Access Time Logging and enable Combat Fragmentation to enhance XFS file system performance. Add noatime,nodiratime,allocsize=64m to all XFS volumes under /etc/fstab.



# Edit /etc/fstab
vim /etc/fstab

# Modify XFS volume entries as follows
# Example:
UUID="03c97344-9b3d-45e2-9140-cbbd57b6f085"  /  xfs  defaults,noatime,nodiratime,allocsize=64m  0 0

3.10. Tweaking the system for high concurrancy and security.



cat <<'EOF' | sudo tee /etc/sysctl.d/00-sysctl.conf > /dev/null
################################################################################################
# Tweak virtual memory
################################################################################################

# Default: 30
# 0 - Never swap under any circumstances.
# 1 - Do not swap unless there is an out-of-memory (OOM) condition.
vm.swappiness = 1

# vm.dirty_background_ratio is used to adjust how the kernel handles dirty pages that must be flushed to disk.
# Default value is 10.
# The value is a percentage of the total amount of system memory, and setting this value to 5 is appropriate in many situations.
# This setting should not be set to zero.
vm.dirty_background_ratio = 5

# The total number of dirty pages that are allowed before the kernel forces synchronous operations to flush them to disk
# can also be increased by changing the value of vm.dirty_ratio, increasing it to above the default of 30 (also a percentage of total system memory)
# vm.dirty_ratio value in-between 60 and 80 is a reasonable number.
vm.dirty_ratio = 60

# vm.max_map_count will calculate the current number of memory mapped files.
# The minimum value for mmap limit (vm.max_map_count) is the number of open files ulimit (cat /proc/sys/fs/file-max).
# map_count should be around 1 per 128 KB of system memory. Therefore, max_map_count will be 262144 on a 32 GB system.
# Default: 65530
vm.max_map_count = 2097152

################################################################################################
# Tweak file handles
################################################################################################

# Increases the size of file handles and inode cache and restricts core dumps.
fs.file-max = 2097152
fs.suid_dumpable = 0

################################################################################################
# Tweak network settings
################################################################################################

# Default amount of memory allocated for the send and receive buffers for each socket.
# This will significantly increase performance for large transfers.
net.core.wmem_default = 25165824
net.core.rmem_default = 25165824

# Maximum amount of memory allocated for the send and receive buffers for each socket.
# This will significantly increase performance for large transfers.
net.core.wmem_max = 25165824
net.core.rmem_max = 25165824

# In addition to the socket settings, the send and receive buffer sizes for
# TCP sockets must be set separately using the net.ipv4.tcp_wmem and net.ipv4.tcp_rmem parameters.
# These are set using three space-separated integers that specify the minimum, default, and maximum sizes, respectively.
# The maximum size cannot be larger than the values specified for all sockets using net.core.wmem_max and net.core.rmem_max.
# A reasonable setting is a 4 KiB minimum, 64 KiB default, and 2 MiB maximum buffer.
net.ipv4.tcp_wmem = 20480 12582912 25165824
net.ipv4.tcp_rmem = 20480 12582912 25165824

# Increase the maximum total buffer-space allocatable
# This is measured in units of pages (4096 bytes)
net.ipv4.tcp_mem = 65536 25165824 262144
net.ipv4.udp_mem = 65536 25165824 262144

# Minimum amount of memory allocated for the send and receive buffers for each socket.
net.ipv4.udp_wmem_min = 16384
net.ipv4.udp_rmem_min = 16384

# Enabling TCP window scaling by setting net.ipv4.tcp_window_scaling to 1 will allow
# clients to transfer data more efficiently, and allow that data to be buffered on the broker side.
net.ipv4.tcp_window_scaling = 1

# Increasing the value of net.ipv4.tcp_max_syn_backlog above the default of 1024 will allow
# a greater number of simultaneous connections to be accepted.
net.ipv4.tcp_max_syn_backlog = 10240

# Increasing the value of net.core.netdev_max_backlog to greater than the default of 1000
# can assist with bursts of network traffic, specifically when using multigigabit network connection speeds,
# by allowing more packets to be queued for the kernel to process them.
net.core.netdev_max_backlog = 65536

# Increase the maximum amount of option memory buffers
net.core.optmem_max = 25165824

# Number of times SYNACKs for passive TCP connection.
net.ipv4.tcp_synack_retries = 2

# Allowed local port range.
net.ipv4.ip_local_port_range = 2048 65535

# Protect Against TCP Time-Wait
# Default: net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rfc1337 = 1

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# The maximum number of backlogged sockets.
# Default is 128.
net.core.somaxconn = 4096

# Turn on syncookies for SYN flood attack protection.
net.ipv4.tcp_syncookies = 1

# Avoid a smurf attack
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Turn on protection for bad icmp error messages
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Enable automatic window scaling.
# This will allow the TCP buffer to grow beyond its usual maximum of 64K if the latency justifies it.
net.ipv4.tcp_window_scaling = 1

# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1

# Tells the kernel how many TCP sockets that are not attached to any
# user file handle to maintain. In case this number is exceeded,
# orphaned connections are immediately reset and a warning is printed.
# Default: net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_max_orphans = 65536

# Do not cache metrics on closing connections
net.ipv4.tcp_no_metrics_save = 1

# Enable timestamps as defined in RFC1323:
# Default: net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_timestamps = 1

# Enable select acknowledgments.
# Default: net.ipv4.tcp_sack = 1
net.ipv4.tcp_sack = 1

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks.
# net.ipv4.tcp_tw_recycle has been removed from Linux 4.12. Use net.ipv4.tcp_tw_reuse instead.
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_reuse = 1

# The accept_source_route option causes network interfaces to accept packets with the Strict Source Route (SSR) or Loose Source Routing (LSR) option set. 
# The following setting will drop packets with the SSR or LSR option set.
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Turn on reverse path filtering
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP redirect acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0

# Disables sending of all IPv4 ICMP redirected packets.
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Disable IP forwarding.
# IP forwarding is the ability for an operating system to accept incoming network packets on one interface, 
# recognize that it is not meant for the system itself, but that it should be passed on to another network, and then forwards it accordingly.
net.ipv4.ip_forward = 0

# Disable IPv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

################################################################################################
# Tweak kernel parameters
################################################################################################

# Address Space Layout Randomization (ASLR) is a memory-protection process for operating systems that guards against buffer-overflow attacks.
# It helps to ensure that the memory addresses associated with running processes on systems are not predictable,
# thus flaws or vulnerabilities associated with these processes will be more difficult to exploit.
# Accepted values: 0 = Disabled, 1 = Conservative Randomization, 2 = Full Randomization
kernel.randomize_va_space = 2

# Allow for more PIDs (to reduce rollover problems)
kernel.pid_max = 65536
EOF

3.11. Reload all sysctl variables without rebooting the server.



sysctl -p /etc/sysctl.d/00-sysctl.conf

3.12. Configure firewall for Nginx and Keepalived.



# Enable ans start firewalld.service
systemctl enable --now firewalld

# You must allow VRRP traffic to pass between the keepalived nodes
firewall-cmd --permanent --add-rich-rule='rule protocol value="vrrp" accept'

# Enable Kubernetes API
firewall-cmd --permanent --add-port=6443/tcp

# Reload firewall rules
firewall-cmd --reload

3.13. Create Local DNS records.



cat <<'EOF' | sudo tee /etc/hosts > /dev/null
# localhost
127.0.0.1     localhost        localhost.localdomain

# When DNS records are updated in the DNS server, remove these entries.
192.168.16.80  kube-api.example.local
192.168.16.102 kubemaster01  kubemaster01.example.local
192.168.16.103 kubemaster02  kubemaster02.example.local
192.168.16.104 kubemaster03  kubemaster03.example.local
192.168.16.105 kubeworker01  kubeworker01.example.local
192.168.16.106 kubeworker02  kubeworker02.example.local
192.168.16.107 kubeworker03  kubeworker03.example.local
EOF

3.14. Configure keepalived failover on kubelb01 and kubelb02.

🔵 Important

Don't forget to change auth_pass to something more secure.

Change interface ens192 to match your interface name.

Change virtual_ipaddress from 192.168.16.80 to a valid IP.

The priority specifies the order in which the assigned interface takes over in a failover; the higher the number, the higher the priority.

3.14.1. Please execute the following command on kubelb01 Server.



cat <<'EOF' | sudo tee /etc/keepalived/keepalived.conf > /dev/null
# Global definitions configuration block
global_defs {

    router_id LVS_LB

}

vrrp_instance VI_1 {

    # The state MASTER designates the active server, the state BACKUP designates the backup server.
    state MASTER

    virtual_router_id 100

    # The interface parameter assigns the physical interface name 
    # to this particular virtual IP instance.
    interface ens192

    # The priority specifies the order in which the assigned interface
    # takes over in a failover; the higher the number, the higher the priority.
    # This priority value must be within the range of 0 to 255, and the Load Balancing 
    # server configured as state MASTER should have a priority value set to a higher number 
    # than the priority value of the server configured as state BACKUP.
    priority 150

    advert_int 1

    authentication {

        auth_type PASS

        # Don't forget to change auth_pass to something more secure.
        # auth_pass value MUST be same in both nodes.
        auth_pass Bx3ae3Gr

    }

    virtual_ipaddress {

        192.168.16.80

    }
}
EOF

3.14.2. Please execute the following command on kubelb02 Server.



cat <<'EOF' | sudo tee /etc/keepalived/keepalived.conf > /dev/null
# Global definitions configuration block
global_defs {

    router_id LVS_LB

}

vrrp_instance VI_1 {

    # The state MASTER designates the active server, the state BACKUP designates the backup server.
    state BACKUP

    virtual_router_id 100

    # The interface parameter assigns the physical interface name 
    # to this particular virtual IP instance.
    interface ens192

    # The priority specifies the order in which the assigned interface
    # takes over in a failover; the higher the number, the higher the priority.
    # This priority value must be within the range of 0 to 255, and the Load Balancing 
    # server configured as state MASTER should have a priority value set to a higher number 
    # than the priority value of the server configured as state BACKUP.
    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        # Don't forget to change auth_pass to something more secure.
        # auth_pass value MUST be same in both nodes.
        auth_pass Bx3ae3Gr

    }

    virtual_ipaddress {

        192.168.16.80

    }
}
EOF

3.15. Start and enable keepalived service on both load balancer nodes.



# Start and enable keepalived service
systemctl enable --now keepalived

# Check if the keepalived service is running
systemctl status keepalived

3.16. To determine whether a server is acting as the master, you can use the following command to see whether the virtual address is active.



ip addr show ens192

3.17. Configure nginx on both load balancer nodes.



cat <<'EOF' | sudo tee /etc/nginx/nginx.conf > /dev/null
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {

    worker_connections 2048;

}

stream {

    upstream stream_backend {

        # Load balance algorithm
        least_conn;

        # kubemaster01
        server kubemaster01.example.local:6443;

        # kubemaster02
        server kubemaster02.example.local:6443;

        # kubemaster03
        server kubemaster03.example.local:6443;

    }

    server {

        listen                  6443;
        proxy_pass              stream_backend;

        proxy_timeout           300s;
        proxy_connect_timeout   60s;

    }

}
EOF

3.18. Start and enable nginx service on both load balancer nodes.



# Start and enable nginx service
systemctl enable --now nginx

# Check if the nginx service is running
systemctl status nginx

3.19. The servers need to be restarted before continue further.



reboot

3.20. Verify the load balancer.



curl -k https://kube-api.example.local:6443

🔵 Important

If the load balancers are working, you should get the following output.



curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to [https://kube-api.example.local:6443](https://kube-api.example.local:6443)

4. Install and Configure Kubernetes

4.1. Install prerequisites on BOTH Master and Worker nodes

🔵 Important

Verify the MAC address and product_uuid are unique for every node. You can get the MAC address of the network interfaces using the following command.
ip link | grep link/ether
The product_uuid can be checked using the below command.
cat /sys/class/dmi/id/product_uuid
Verify the Linux Kernel version is greater than 4.5.0. It can be checked using the following command.
uname -r
Docker, Rocky Linux 8 and the XFS filesystem could be a trouble giving combination if you don't meet all the specifications of the overlay/overlay2 storage driver.

The overlay storage driver relies on a technology called "directory entry type" (d_type) and is used to describe information of a directory on the filesystem. Make sure you have a d_type enabled filesystem.
xfs_info / | grep ftype
The ftype value must be set to 1. If not do not continue further.

The disk.EnableUUID parameter must be set to TRUE on all VMware VMs. In order to do this, power off the VMs and go to Edit Settings. Then, navigate to VM Options -> Advanced -> Edit Configuration. Please go through the existing configurations and verify disk.EnableUUID property exists, and its value already set to true. If the disk.EnableUUID property does not exist, add it using ADD CONFIGURATION PARAMS.

4.1.1. Set server hostname.



# Example:
# hostnamectl set-hostname kubelb01

hostnamectl set-hostname <hostname>

4.1.2. Install prerequisites.



# Clean YUM repository cache
dnf clean all

# Update packages
dnf update -y

# Install prerequisites
dnf install -y vim net-tools chrony ntpstat

4.1.3. Synchronize server time with Google NTP server.



# Add Google NTP Server
sed -i '/^pool/c\pool time.google.com iburst' /etc/chrony.conf

# Set timezone to Asia/Colombo
timedatectl set-timezone Asia/Colombo

# Enable NTP time synchronization
timedatectl set-ntp true

4.1.4. Start and enable chronyd service.



# Start and enable chronyd service
systemctl enable --now chronyd

# Check if chronyd service is running
systemctl status chronyd

4.1.5. Display time synchronization status.



# Verify synchronisation state
ntpstat

# Check Chrony Source Statistics
chronyc sourcestats -v

4.1.6. Permanently disable SELinux.



# Permanently disable SELinux
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config

4.1.7. Enable IP masquerade at the Linux firewall.



# Enable IP masquerade at the firewall
firewall-cmd --permanent --add-masquerade
firewall-cmd --reload

4.1.8. Disable IPv6 on network interface.



# Disable IPv6 on ens192 interface
nmcli connection modify ens192 ipv6.method ignore

4.1.9. Execute the following commands to turn off all swap devices and files.



# Permanently disable swapping
sed -i '/ swap / s/^/#/' /etc/fstab

#d Disable all existing swaps from /proc/swaps
swapoff -a

4.1.10. Enable auto-loading of required kernel modules.



# Enable auto-loading of required kernel modules
cat <<'EOF' | sudo tee /etc/modules-load.d/crio.conf > /dev/null
overlay
br_netfilter
EOF

# Add overlay and br_netfilter kernel modules to the Linux kernel
# The br_netfilter kernel modules will enable transparent masquerading and facilitate Virtual Extensible LAN (VxLAN) traffic for communication between Kubernetes pods across the cluster
modprobe overlay
modprobe br_netfilter

4.1.11. Disable File Access Time Logging and enable Combat Fragmentation to enhance XFS file system performance. Add noatime,nodiratime,allocsize=64m to all XFS volumes under /etc/fstab.



# Edit /etc/fstab
vim /etc/fstab

# Modify XFS volume entries as follows
# Example:
UUID="03c97344-9b3d-45e2-9140-cbbd57b6f085"  /  xfs  defaults,noatime,nodiratime,allocsize=64m  0 0

4.1.12. Tweaking the system for high concurrancy and security.



cat <<'EOF' | sudo tee /etc/sysctl.d/00-sysctl.conf > /dev/null
#############################################################################################
# Tweak virtual memory
#############################################################################################

# Default: 30
# 0 - Never swap under any circumstances.
# 1 - Do not swap unless there is an out-of-memory (OOM) condition.
vm.swappiness = 1

# vm.dirty_background_ratio is used to adjust how the kernel handles dirty pages that must be flushed to disk.
# Default value is 10.
# The value is a percentage of the total amount of system memory, and setting this value to 5 is appropriate in many situations.
# This setting should not be set to zero.
vm.dirty_background_ratio = 5

# The total number of dirty pages that are allowed before the kernel forces synchronous operations to flush them to disk
# can also be increased by changing the value of vm.dirty_ratio, increasing it to above the default of 30 (also a percentage of total system memory)
# vm.dirty_ratio value in-between 60 and 80 is a reasonable number.
vm.dirty_ratio = 60

# vm.max_map_count will calculate the current number of memory mapped files.
# The minimum value for mmap limit (vm.max_map_count) is the number of open files ulimit (cat /proc/sys/fs/file-max).
# map_count should be around 1 per 128 KB of system memory. Therefore, max_map_count will be 262144 on a 32 GB system.
# Default: 65530
vm.max_map_count = 2097152

#############################################################################################
# Tweak file handles
#############################################################################################

# Increases the size of file handles and inode cache and restricts core dumps.
fs.file-max = 2097152
fs.suid_dumpable = 0

#############################################################################################
# Tweak network settings
#############################################################################################

# Default amount of memory allocated for the send and receive buffers for each socket.
# This will significantly increase performance for large transfers.
net.core.wmem_default = 25165824
net.core.rmem_default = 25165824

# Maximum amount of memory allocated for the send and receive buffers for each socket.
# This will significantly increase performance for large transfers.
net.core.wmem_max = 25165824
net.core.rmem_max = 25165824

# In addition to the socket settings, the send and receive buffer sizes for
# TCP sockets must be set separately using the net.ipv4.tcp_wmem and net.ipv4.tcp_rmem parameters.
# These are set using three space-separated integers that specify the minimum, default, and maximum sizes, respectively.
# The maximum size cannot be larger than the values specified for all sockets using net.core.wmem_max and net.core.rmem_max.
# A reasonable setting is a 4 KiB minimum, 64 KiB default, and 2 MiB maximum buffer.
net.ipv4.tcp_wmem = 20480 12582912 25165824
net.ipv4.tcp_rmem = 20480 12582912 25165824

# Increase the maximum total buffer-space allocatable
# This is measured in units of pages (4096 bytes)
net.ipv4.tcp_mem = 65536 25165824 262144
net.ipv4.udp_mem = 65536 25165824 262144

# Minimum amount of memory allocated for the send and receive buffers for each socket.
net.ipv4.udp_wmem_min = 16384
net.ipv4.udp_rmem_min = 16384

# Enabling TCP window scaling by setting net.ipv4.tcp_window_scaling to 1 will allow
# clients to transfer data more efficiently, and allow that data to be buffered on the broker side.
net.ipv4.tcp_window_scaling = 1

# Increasing the value of net.ipv4.tcp_max_syn_backlog above the default of 1024 will allow
# a greater number of simultaneous connections to be accepted.
net.ipv4.tcp_max_syn_backlog = 10240

# Increasing the value of net.core.netdev_max_backlog to greater than the default of 1000
# can assist with bursts of network traffic, specifically when using multigigabit network connection speeds,
# by allowing more packets to be queued for the kernel to process them.
net.core.netdev_max_backlog = 65536

# Increase the maximum amount of option memory buffers
net.core.optmem_max = 25165824

# Number of times SYNACKs for passive TCP connection.
net.ipv4.tcp_synack_retries = 2

# Allowed local port range.
net.ipv4.ip_local_port_range = 2048 65535

# Protect Against TCP Time-Wait
# Default: net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rfc1337 = 1

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# The maximum number of backlogged sockets.
# Default is 128.
net.core.somaxconn = 4096

# Turn on syncookies for SYN flood attack protection.
net.ipv4.tcp_syncookies = 1

# Avoid a smurf attack
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Turn on protection for bad icmp error messages
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Enable automatic window scaling.
# This will allow the TCP buffer to grow beyond its usual maximum of 64K if the latency justifies it.
net.ipv4.tcp_window_scaling = 1

# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1

# Tells the kernel how many TCP sockets that are not attached to any
# user file handle to maintain. In case this number is exceeded,
# orphaned connections are immediately reset and a warning is printed.
# Default: net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_max_orphans = 65536

# Do not cache metrics on closing connections
net.ipv4.tcp_no_metrics_save = 1

# Enable timestamps as defined in RFC1323:
# Default: net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_timestamps = 1

# Enable select acknowledgments.
# Default: net.ipv4.tcp_sack = 1
net.ipv4.tcp_sack = 1

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks.
# net.ipv4.tcp_tw_recycle has been removed from Linux 4.12. Use net.ipv4.tcp_tw_reuse instead.
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_reuse = 1

# The accept_source_route option causes network interfaces to accept packets with the Strict Source Route (SSR) or Loose Source Routing (LSR) option set. 
# The following setting will drop packets with the SSR or LSR option set.
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Turn on reverse path filtering
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP redirect acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0

# Disables sending of all IPv4 ICMP redirected packets.
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Disable IPv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

#############################################################################################
# Kubernetes related settings
#############################################################################################

# Enable IP forwarding.
# IP forwarding is the ability for an operating system to accept incoming network packets on one interface,
# recognize that it is not meant for the system itself, but that it should be passed on to another network, and then forwards it accordingly.
net.ipv4.ip_forward = 1

# These settings control whether packets traversing a network bridge are processed by iptables rules on the host system.
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

# To prevent Linux conntrack table is out of space, increase the conntrack table size.
# This setting is for Calico networking.
net.netfilter.nf_conntrack_max = 1000000

#############################################################################################
# Tweak kernel parameters
#############################################################################################

# Address Space Layout Randomization (ASLR) is a memory-protection process for operating systems that guards against buffer-overflow attacks.
# It helps to ensure that the memory addresses associated with running processes on systems are not predictable,
# thus flaws or vulnerabilities associated with these processes will be more difficult to exploit.
# Accepted values: 0 = Disabled, 1 = Conservative Randomization, 2 = Full Randomization
kernel.randomize_va_space = 2

# Allow for more PIDs (to reduce rollover problems)
kernel.pid_max = 65536
EOF

4.1.13. Reload all sysctl variables without rebooting the server.



sysctl --system

4.1.14. Create Local DNS records.



cat <<'EOF' | sudo tee /etc/hosts > /dev/null
# localhost
127.0.0.1     localhost        localhost.localdomain

# When DNS records are updated in the DNS server, remove these entries.
192.168.16.80  kube-api.example.local
192.168.16.102 kubemaster01  kubemaster01.example.local
192.168.16.103 kubemaster02  kubemaster02.example.local
192.168.16.104 kubemaster03  kubemaster03.example.local
192.168.16.105 kubeworker01  kubeworker01.example.local
192.168.16.106 kubeworker02  kubeworker02.example.local
192.168.16.107 kubeworker03  kubeworker03.example.local
EOF

4.1.15. Configure NetworkManager before attempting to use Calico networking.



# Create the following configuration file to prevent NetworkManager from interfering with the interfaces
cat <<'EOF' | sudo tee /etc/NetworkManager/conf.d/calico.conf > /dev/null
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:tunl*
EOF

4.1.16. The servers need to be restarted before continue further.



reboot

4.1.17. Configure CRI-O Container Runtime Interface repositories.

🔵 Important

The CRI-O major and minor versions must match the Kubernetes major and minor versions. For more information, see the CRI-O compatibility matrix.



# Set environment variables according to the operating system and Kubernetes version
OS=CentOS_8
VERSION=1.19

# Configure YUM repositories
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo

4.1.18. Install CRI-O package.



# Install cri-o package
dnf install -y cri-o

4.1.19. Start and enable CRI-O service.



# Start and enable crio service
systemctl enable --now crio

# Check if the crio service is running
systemctl status crio

4.1.20. Add Kubernetes repository.



cat <<'EOF' | sudo tee /etc/yum.repos.d/kubernetes.repo > /dev/null
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

4.1.21. Install kubeadm, kubelet and kubectl packages.



dnf install -y --disableexcludes=kubernetes kubelet-1.19* kubeadm-1.19* kubectl-1.19*

4.1.22. Pull latest docker images used by kubeadm.



kubeadm config images pull

4.2. Configure MASTER nodes

4.2.1. Prepare Master Nodes

4.2.1.1. Open necessary firewall ports used by Kubernetes.



# Open necessary firewall ports
firewall-cmd --zone=public --permanent --add-port={6443,2379,2380,10250,10251,10252}/tcp

# Allow docker access from another node
firewall-cmd --zone=public --permanent --add-rich-rule 'rule family=ipv4 source address=192.168.16.0/24 accept'

# Apply firewall changes
firewall-cmd --reload

4.2.1.2. Configure runtime cgroups used by kubelet service.

🔵 Important

For Kubernetes versions 1.9.x and above your vsphere.conf file should be placed only on the Kubernetes MASTER nodes.



# Configure runtime cgroups used by kubelet on ALL master nodes
cat <<'EOF' | sudo tee /etc/sysconfig/kubelet > /dev/null
KUBELET_EXTRA_ARGS="--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --cloud-provider=vsphere --cloud-config=/etc/kubernetes/vsphere.conf"
EOF

4.2.1.3. Enable kubelet service.



systemctl enable kubelet

4.2.1.4. Create vSphere config file.

🔵 Important

Please make sure to change vSphere config values as appropriate. For details about how this file can be constructed, refer the VMware Documentation



cat <<'EOF' | sudo tee /etc/kubernetes/vsphere.conf > /dev/null
[Global]
secret-name = "vmware"
secret-namespace = "kube-system"
port = "443"
insecure-flag = "1"

[VirtualCenter "vcenter.example.local"]
datacenters = "MAIN"

[Workspace]
server = "vcenter.example.local"
datacenter = "MAIN"
default-datastore = "MAIN-DS"
resourcepool-path = "MAIN/Resources"
folder = "Kubernetes"

[Disk]
scsicontrollertype = pvscsi
EOF

4.2.2. Configure the First Master Node (kubemaster01)

4.2.2.1. Create the kubeadm config file.

🔵 Important

Please make sure to change controlPlaneEndpoint value as appropriate.



cat <<'EOF' | sudo tee /etc/kubernetes/kubeadm.conf > /dev/null
---
apiServer:
  extraArgs:
    cloud-config: /etc/kubernetes/vsphere.conf
    cloud-provider: vsphere
    endpoint-reconciler-type: lease
  extraVolumes:
  - hostPath: /etc/kubernetes/vsphere.conf
    mountPath: /etc/kubernetes/vsphere.conf
    name: cloud
  timeoutForControlPlane: 4m0s
---
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: kube-api.example.local:6443
controllerManager:
  extraArgs:
    cloud-config: /etc/kubernetes/vsphere.conf
    cloud-provider: vsphere
  extraVolumes:
  - hostPath: /etc/kubernetes/vsphere.conf
    mountPath: /etc/kubernetes/vsphere.conf
    name: cloud
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
networking:
  dnsDomain: example.local
  podSubnet: 192.168.0.0/16
  serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: "systemd"
EOF

4.2.2.2. Initialize the first control plane.



kubeadm init \
    --config /etc/kubernetes/kubeadm.conf \
    --upload-certs \
    --v=5

🟢 You will get an output like this. Please make sure to record MASTER and WORKER join commands.

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:
kubeadm join kube-api.example.local:6443 --token ti2ho7.t146llqa4sn8y229 \
--discovery-token-ca-cert-hash sha256:9e73a021b8b26c8a2fc04939729acc7670769f15469887162cdbae923df906f9 \
--control-plane --certificate-key d9d631a0aef1a5a474faa6787b54814040adf1012c6c1922e8fe096094547b65 \
--v=5
Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:
kubeadm join kube-api.example.local:6443 --token ti2ho7.t146llqa4sn8y229 \
--discovery-token-ca-cert-hash sha256:9e73a021b8b26c8a2fc04939729acc7670769f15469887162cdbae923df906f9 \
--v=5

4.2.2.3. To start using kubectl, you need to run the following command.



mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

4.2.2.4. Create a Kubernetes secret with the base64 encoded vCenter credentials.

🔵 Important

In order to create a Kubernetes secret, we must generate base64 encoded credentials. Please make sure NOT to use special characters in vCenter password.

Let's assume your vCenter username is kubernetes@vsphere.local and the password is KuB3VMwar3

Generate base64 encoded username using echo -n 'kubernetes@vsphere.local' | base64. Please make sure to record the output (Example output: a3ViZXJuZXRlc0B2c3BoZXJlLmxvY2Fs).

Generate base64 encoded password using echo -n 'KuB3VMwar3' | base64. Please make sure to record the output (Example output: S3VCM1ZNd2FyMw==).

You can refer Securing vSphere Credentials section for more details



cat <<'EOF' | kubectl create -f -
apiVersion: v1
kind: Secret
metadata:
 name: vmware
 namespace: kube-system
type: Opaque
data:
   vcenter.example.local.username: a3ViZXJuZXRlc0B2c3BoZXJlLmxvY2Fs
   vcenter.example.local.password: S3VCM1ZNd2FyMw==
EOF

4.2.2.5. Install calico CNI-plugin.



# Install calico CNI-plugin
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

4.2.2.6. Check NetworkReady status. It must be TRUE. If not, wait some time and check it again.



# Check NetworkReady status
watch crictl info

4.2.2.7. Watch the ods created in the kube-system namespace and make sure all are running.



# Watch the Pods created in the kube-system namespace
watch kubectl get pods --namespace kube-system

4.2.2.8. Check master node status.



# Check master node status
kubectl get nodes -o wide

4.2.2.9. Verify ProviderID has been added the nodes.



# You should get an output like below
# ProviderID: vsphere://4204a018-f286-cf3c-7f2d-c512d9f7d90d
kubectl describe nodes | grep "ProviderID"

4.2.3. Configure other master nodes (kubemaster02 and kubemaster03).

🔵 Important

Make sure to join other master nodes ONE BY ONE when the kubemaster01 status becomes READY.

Before execute the kubectl join command, make sure to verify all pods are up and running using the below command.
kubectl get po,svc --all-namespaces
Use --v=5 argument with kubeadm join in order to get a verbose output.

4.2.3.1. Execute the control-plane join command recorded in step 4.2.2.2.



# Control plane join command example:
kubeadm join kube-api.example.local:6443 --token ti2ho7.t146llqa4sn8y229 \
    --discovery-token-ca-cert-hash sha256:9e73a021b8b26c8a2fc04939729acc7670769f15469887162cdbae923df906f9 \
    --control-plane --certificate-key d9d631a0aef1a5a474faa6787b54814040adf1012c6c1922e8fe096094547b65 \
    --v=5

4.2.3.2. To start using kubectl, you need to run the following command.



mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

4.2.3.3. Check master node status.



# Check master node status
kubectl get nodes -o wide

4.3. Configure WORKER nodes

🔵 Important

Make sure to join worker nodes ONE BY ONE when the MASTER nodes status becomes READY.

Before execute the kubectl join command on worker nodes, make sure to verify all pods are up and running on master nodes using the below command.
kubectl get po,svc --all-namespaces
Use --v=5 argument with kubeadm join in order to get a verbose output.

4.3.1. Open necessary firewall ports used by Kubernetes.



# Open necessary firewall ports
firewall-cmd --zone=public --permanent --add-port={10250,30000-32767}/tcp

# Apply firewall changes
firewall-cmd --reload

4.3.2. Configure runtime cgroups used by kubelet service.



# Configure runtime cgroups used by kubelet
cat <<'EOF' | sudo tee /etc/sysconfig/kubelet > /dev/null
KUBELET_EXTRA_ARGS="--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --cloud-provider=vsphere"
EOF

4.3.3. Enable kubelet service.



systemctl enable kubelet

4.3.4. Execute the worker nodes join command recorded in step 4.2.2.2.



# Worker node join command example:
kubeadm join kube-api.example.local:6443 --token ti2ho7.t146llqa4sn8y229 \
    --discovery-token-ca-cert-hash sha256:9e73a021b8b26c8a2fc04939729acc7670769f15469887162cdbae923df906f9 \
    --v=5

4.4. Configure MetalLB Load Balancer

🔵 Important

You MUST execute these commands on a MASTER node.

Make sure to follow these steps only when the both MASTER and WORKER nodes status becomes READY.

Make sure to execute kubectl get po,svc --all-namespaces on a master node and verify all pods are up and running.

4.4.1. Install MetalLB Load Balancer.



# Install MetalLB Load Balancer
kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.3/manifests/metallb.yaml

4.4.2. Create MetalLB ConfigMap.



# Create MetalLB ConfigMap
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2

      # MetalLB IP Pool
      addresses:
      - 192.168.16.200-192.168.16.250
EOF

4.4.3. Watch the Pods created in the metallb-system namespace and make sure all are running.



# Watch the Pods created in the metallb-system namespace
watch kubectl get pods --namespace metallb-system

🟣 NOTE

If you want to change the MetalLB IP Pool, please follow these steps.

1. Note the old IPs allocated to services.
kubectl get svc --all-namespaces
2. Delete the old ConfigMap.
kubectl -n metallb-system delete cm config
3. Apply the new ConfigMap
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
 config: |
    address-pools:
    - name: default
      protocol: layer2

      # MetalLB IP Pool
      addresses:
      - 192.168.16.150-192.168.16.175
EOF
4. Delete the existing MetalLB pods.
kubectl -n metallb-system delete pod --all
5. New MetalLB pods will be created automatically. Please make sure the pods are running.
kubectl -n metallb-system get pods
6. Inspect new IPs of services.
kubectl get svc --all-namespaces

4.5. Configure Kubernetes Dashboard

🔵 Important

You MUST execute these commands on a MASTER node.

Make sure to follow these steps only when the both MASTER and WORKER nodes status becomes READY.

Make sure to execute kubectl get po,svc --all-namespaces on a master node and verify all pods are up and running.

4.5.1. Install Kubernetes Dashboard.



# Install Kubernetes Dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc6/aio/deploy/recommended.yaml

4.5.2. Create the Dashboard service account.



# Create the Dashboard service account
# This will create a service account named dashboard-admin in the default namespace
kubectl create serviceaccount dashboard-admin --namespace kubernetes-dashboard

4.5.3. Bind the dashboard-admin service account to the cluster-admin role.



# Bind the dashboard-admin service account to the cluster-admin role
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin \
    --serviceaccount=kubernetes-dashboard:dashboard-admin

4.5.4. When we created the dashboard-admin service account, Kubernetes also created a secret for it. List secrets using the following command.



# When we created the dashboard-admin service account Kubernetes also created a secret for it.
# List secrets using:
kubectl get secrets --namespace kubernetes-dashboard

4.5.5. Get Dashboard Access Token.



# We can see the dashboard-admin-sa service account secret in the above command output.
# Use kubectl describe to get the access token:
kubectl describe --namespace kubernetes-dashboard secret dashboard-admin-token

4.5.6. Watch Pods and Service accounts under kubernetes-dashboard namespace.



# Watch Pods and Service accounts under kubernetes-dashboard
watch kubectl get po,svc --namespace kubernetes-dashboard

4.5.7. Get logs of kubernetes-dashboard.



# Get logs of kubernetes-dashboard
kubectl logs --follow --namespace kubernetes-dashboard deployment/kubernetes-dashboard

4.5.8. Create kubernetes-dashboard load balancer.



# Create kubernetes-dashboard load balancer
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: load-balancer-dashboard
  name: dashboard-load-balancer
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      protocol: TCP
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  type: LoadBalancer
EOF

4.5.9. Get logs of kubernetes-dashboard.



# Get logs of kubernetes-dashboard
kubectl logs --follow --namespace kubernetes-dashboard deployment/kubernetes-dashboard

4.5.10. Get kubernetes-dashboard External IP.



# Get kubernetes-dashboard external IP
kubectl get po,svc --namespace kubernetes-dashboard | grep -i service/dashboard-load-balancer

4.6. Create vSphere Storage Class



# Create vSphere storage class
cat <<'EOF' | kubectl create -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: vsphere
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/vsphere-volume
parameters:
    datastore: KUBE-DS
    diskformat: thin
    fstype: xfs
EOF

4.7. Deploy a Sample WordPress Blog

🔵 Important

You MUST execute these commands on a MASTER node.

Make sure to follow these steps only when the both MASTER and WORKER nodes status becomes READY.

Make sure to execute kubectl get po,svc --all-namespaces on a master node and verify all pods are up and running.

4.7.1. Deploy a sample WordPress application using rook persistent volume claim.



# Create a MySQL container
kubectl create -f https://notebook.yasithab.com/gist/vsphere-mysql.yaml

# Create an Apache WordPress container
kubectl create -f https://notebook.yasithab.com/gist/vsphere-wordpress.yaml

4.7.2. Get detailed information about the Persistent Volumes (PV) used by the application.



# Describe PersistentVolume
kubectl describe pv

4.7.3. Describe Persistant Volume Claims (PVC) used by the application.



# Describe mysql persistance volume claim
kubectl describe pvc mysql-pv-claim

# Describe wordpress persistance volume claim
kubectl describe pvc wp-pv-claim

4.8. Clean up Kubernates

🔴 Caution

The following commands are used to RESET your nodes and WIPE OUT all components installed.

You MUST run this on ALL MASTER and WORKER nodes.

4.8.1. Remove Kubernetes Components from Nodes



# The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d
# The reset process does not reset or clean up iptables rules or IPVS tables.
# If you wish to reset iptables, you must do so manually by using the "iptables" command.
# If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) to reset your system's IPVS tables.

# Remove Kubernetes Components from Nodes
kubeadm reset --force

# The reset process does not clean your kubeconfig files and you must remove them manually
rm -rf $HOME/.kube/config

If you liked the post, then you may purchase my first cup of coffee ever, thanks in advance :)

5. References

Understanding Infrastructure as Code (IaC)

Yasitha Bogamuwa — Wed, 15 Jun 2022 13:29:44 +0000

Overview

This article discusses the benefits of implementing infrastructure as code (IaC), which is a methodology that enables organizations to manage and provision their IT infrastructure through code. IaC provides a number of advantages over traditional methods such as increased speed and accuracy, improved consistency, and easier scalability. Additionally, IaC can help reduce the amount of manual effort required to maintain infrastructure, which can lead to cost savings and increased efficiency.

What is Infrastructure as Code?

When most people think of infrastructure, they think of the physical things that make up a company’s or organization’s IT environment. Servers, storage, networking gear, and so on. However, the definition of infrastructure has been changing over the past few years to include not just the physical components but also the software that controls them. This new category of software is often referred to as Infrastructure as Code (IaC).

Before the era of infrastructure as code, changes to an organization’s IT infrastructure were made manually. This process was often time-consuming, error-prone, and difficult to track. In many cases, it also resulted in duplicate work and inconsistency across environments.

In recent years, the use of Infrastructure as Code (IaC) has gained popularity in the DevOps community. IaC is the process of managing and provisioning infrastructure using code, typically in a declarative language such as YAML or JSON. IaC enables users to version-control their infrastructure, treat it like software, and deploy it in the same way as they would their application code. This allows for more repeatable, predictable and scalable infrastructure changes, and improve the consistency and reliability of the system. Let’s explore some of the benefits of using IaC, and how you can get started using it in your own projects.

Types of IaC

There are many different types of IaC, each with its own advantages and disadvantages. IaC tools like Terraform and Pulumi allow DevOps to write code that describes the desired state of their infrastructure. This code can be checked in to source control, allowing for collaboration and versioning. The IaC tools then automatically configure the resources to match the desired state. This results in more reliable and repeatable deployments, and allows for faster iteration on infrastructure changes.

Let’s explore two of the most common open-source types of IaC: Tettaform, and Pulumi. Terraform and Pulumi are both infrastructure as code tools that allow you to declaratively define your infrastructure. However, there are some key differences.

Terraform
Terraform is HashiCorp's infrastructure as code tool. It can be used to manage both public and private cloud providers, as well as traditional data center infrastructure. However, Terraform is more mature and has a larger user base. Additionally, Terraform has better support for provider integrations.

Pulumi
Pulumi is a newer tool, and as a result, has more features. It also allows you to use familiar general-purpose languages like Python, TypeScript, JavaScript, Go, .NET, Java, and markup languages like YAML, which can be useful for certain use cases.

For more information, please refer to Pulumi docs here.

Benefits of IaC

There are many benefits to using Infrastructure as Code (IaC) for businesses. Some of the key benefits include improving efficiency and agility, reducing costs, eliminating configuration drift, and improving security.

It helps businesses improve efficiency and agility by allowing them to quickly and easily deploy new infrastructure components or make changes to existing infrastructure. This can help businesses keep up with changing business requirements and keep their systems running smoothly.
It also helps businesses reduce costs by automating the deployment process. This can save time and money by eliminating the need for manual labor to set up new servers or make changes to existing infrastructure.
It improves security by ensuring that all systems are configured in a consistent manner. This can help prevent unauthorized access to systems and data, and minimize the risk of system failures.
It eliminates configuration drift. Configuration drift is a common challenge that organizations face when it comes to managing their infrastructure. As new servers and applications are added, or as existing ones are updated, the configuration of the systems can change in ways that are not always tracked or accounted for. This can lead to an inconsistent system state, with different servers running different versions of software, or with incorrect settings that can undermine performance or security. With IaC, all aspects of the infrastructure are defined and controlled in code, rather than being configured manually. This enables automated testing and verification of the infrastructure, as well as reproducibility and predictability. By eliminating the possibility of configuration drift, IaC can help organizations maintain a stable and consistent system state.

Declarative vs Imperative

The terms imperative and declarative come up often in IaC discussions. Both terms refer to how the user provides instructions to the automation platform. The imperative approach specifies the exact commands required to achieve the desired configuration, and those commands then need to be executed in the correct order. The declarative approach defines the desired state of the system, and the IaC tool determines how to achieve that state.

The debate between declarative and imperative approaches to IaC can be contentious. Proponents of the declarative approach argue that it is more maintainable and scalable. Imperative proponents counter that the declarative approach can be verbose and less flexible.

Many IaC tools follow the declarative approach and will automatically provision the desired infrastructure for you. If you alter the desired state, a declarative IaC tool will apply any changes for you. An imperative tool will require you to figure out those changes and apply accordingly.

Ultimately, the best approach depends on your specific needs. In my experience, the best method to IaC is to use declarative definition files where possible.

So far so good. But every coin has two sides!

We spend a lot of our time underscoring the benefits of infrastructure as code. It all works. Treating infrastructure as code can add a lot of value in terms of its ability to encourage consistent, predictable, and repeatable results in the delivery of application software throughout the process of infrastructure provision and deployment. But it doesn't mean treating infrastructure as code has no downside. Because adopting a programmatic approach means you reap both the advantages and the drawbacks that come along with it.

On average, developers find and fix about 100 bugs per 1000 lines of code. Discovering one of these bugs takes 30 times longer than writing a single line. Developers typically spend about 75% of their time debugging code (about 1,500 hours per year). That’s where code review comes in.

A code review is the process of examining software code with the intent of finding defects, or potential problems with the design. The code review is often done as a pair programming exercise, where two developers work on the same code at the same time.

I won’t lie, code review can be frustrating at the start, but as soon as someone points an error that could have caused a serious security breach or an undesirable service outage, the initial pain is thoroughly compensated for. In the long run, code reviews are useful for minimizing disruptive errors.

Benefits of a Code Review

It helps to find defects in the code. By having another set of eyes look at the code, you are more likely to find problems that you may have missed.
It helps to improve the overall quality of the code. By finding and fixing defects early in the development process, you prevent them from becoming bigger problems later on.
Reviewing code also helps to ensure that the code is consistent and meets the standards of the organization.
It helps to improve communication between team members. By working together on code, team members learn how to communicate better and work more effectively together. Also, it provides an opportunity for developers to learn from each other and improve their skills.

Learn some Best Practices

As companies move to the cloud and adopt Infrastructure as Code (IaC) practices, they need to consider a number of critical success factors. The first step is understanding the IaC process and how it can be used to your advantage. Configuration management is key to managing your infrastructure in a reproducible way, while automation helps reduce the time and effort needed to manage your systems. Additionally, you’ll need to make sure your team has the proper training and tools in order to take advantage of IaC. By following these best practices, you can ensure that your company is getting the most out of its IaC investment.

Determined by IaC best practices, let me offer you a few tips for creating an effective IaC strategy.

Make sure to code everything in your infrastructure setup, and keep that file as your single source of truth. Whenever you need to confirm a setting, explore the layouts, or create any other configuration change, use the configuration files.
Always version control your configuration files. This should be self-evident, but put your config files under source control. Stick to little documentation (or none at all) for infrastructure settings. Since your configuration files should be your main source of truth, there's no need to store more documentation in external files. External documentation can easily become misaligned with the real configurations, but that is never going to happen with your config files.
Test your configuration. IaC is code, like other forms of code, it can be tested. By utilizing testing and monitoring tools for IaC such as Checkov, and TFSec, it is possible to check errors, vulnerabilities, and inconsistencies in your code before you deploy them to production.
Take your time and review the code!

In conclusion, infrastructure as code offers many benefits to businesses. It can help improve efficiency, save money, and reduce the risk of human error. By using infrastructure as code, businesses can automate the process of creating and managing their infrastructure. This can help them improve their bottom line and become more competitive in the market.

If you liked the post, then you may purchase my first cup of coffee ever, thanks in advance :)

References

1. What Is Infrastructure as Code? How It Works, Best Practices, Tutorials
2. What is Infrastructure as Code?
3. Declarative vs. Imperative in IaC
4. Four Rules of Thumb for Providing Effective Code Review Feedback
5. The Dark Side of Infrastructure as Code

Create an Amazon Machine Image (AMI) from a FreePBX Virtual Machine (VM)

Yasitha Bogamuwa — Tue, 14 Jun 2022 09:14:37 +0000

Creating an Amazon Machine Image (AMI) from a FreePBX Virtual Machine (VM) is a quick and easy way to create a ready-to-use virtual machine that, once configured, can be used to run your own PBX. This guide will show you how to create an AMI from a VM running the FreePBX open source software.

Prerequisites

Make sure to install and configure AWS Command Line Interface in your host computer. You can find the instructions here.
Please use an IAM user with administrator privileges.
This has been tested on VMware Workstation 15 Professional edition.

Instructions

1. Download the latest FreePBX Distro from here and install it on VMware Workstation.

2. SSH into the instance and install the following packages.

yum install -y cloud-init cloud-utils-growpart

3. Modify the /etc/cloud/cloud.cfg file as follows.

system_info:
  default_user:
    name: asterisk
    lock_passwd: true
    gecos: Asterisk User
    groups: [wheel, adm, systemd-journal]
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    shell: /bin/bash
  distro: rhel
  paths:
    cloud_dir: /var/lib/cloud
    templates_dir: /etc/cloud/templates
  ssh_svcname: sshd

4. Change the /etc/ssh/sshd_config file as follows.

PasswordAuthentication no
PermitRootLogin no
UseDNS no

5. Shutdown the VM and export it as an OVA file.

6. Create an S3 bucket and upload the exported OVA file either using AWS CLI or an S3 Client. I used Cyberduck S3 Client and it is freely available [here (https://cyberduck.io/download/).

7. You'll need to create the following policy documents. Make sure to change S3 bucket and OVA file name based on your configurations. In this example, S3 Bucket name and OVA file name will be ami-storage and FreePBX.ova, respectively.

7.A. Create trust-policy.json. This will be used to create the vmimport IAM role.

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": { "Service": "vmie.amazonaws.com" },
         "Action": "sts:AssumeRole",
         "Condition": {
            "StringEquals":{
               "sts:Externalid": "vmimport"
            }
         }
      }
   ]
}

# Create vmimport IAM role
aws iam create-role --role-name vmimport --assume-role-policy-document file://trust-policy.json

7.B. Create role-policy.json. This will be used to assign necessary IAM policies to the vmimport role.

{ 
   "Version": "2012-10-17", 
   "Statement": [ 
      { 
         "Effect": "Allow", 
         "Action": [ 
            "s3:ListBucket", 
            "s3:GetBucketLocation" 
         ], 
         "Resource": [ 
            "arn:aws:s3:::ami-storage" 
         ] 
      }, 
      { 
         "Effect": "Allow", 
         "Action": [ 
            "s3:GetObject" 
         ], 
         "Resource": [ 
            "arn:aws:s3:::ami-storage/*" 
         ] 
      }, 
      { 
         "Effect": "Allow", 
         "Action":[ 
            "ec2:ModifySnapshotAttribute", 
            "ec2:CopySnapshot", 
            "ec2:RegisterImage", 
            "ec2:Describe*" 
         ], 
         "Resource": "*" 
      } 
   ] 
}

# Create and assign necessary IAM policies to the vmimport role
aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document file://role-policy.json

7.C. Create containers.json. This will be used to generate an AMI from the uploaded OVA.

[ 
  { 
    "Description": "FreePBX", 
    "Format": "ova", 
    "UserBucket": { 
        "S3Bucket": "ami-storage", 
        "S3Key": "FreePBX.ova" 
    } 
  }
]

# Generate an AMI from the uploaded OVA
aws ec2 import-image --description "FreePBX" --license-type BYOL --disk-containers file://containers.json

7.D. The previous task can range in estimated completion from 15 to 60 minutes. You can check its progress with the following command by replacing the ImportTaskId shown in the above command.

aws ec2 describe-import-image-tasks --import-task-ids import-ami-0b900a870c359a58f

7.E. The task will remain active with "StatusMessage": "pending" until it reaches completion. The "Progress" attribute will indicate the percentage of work made up to that point. Once the state switches to "completed" and the previous command gives additional information about the conversion of the image to AMI format, you will be provided with a new AMI available in the same region where you creted the S3 bucket. It can be used to provision a FreePBX EC2 instance.

If you liked the post, then you may purchase my first cup of coffee ever, thanks in advance :)

References

1. Importing a VM as an Image Using VM Import/Export
2. How to create a Sentilo AWS EC2 instance from an OVA file
3. FreePBX Tango Scenes by Crisy Meschieri

Introduction to DevOps

Yasitha Bogamuwa — Tue, 14 Jun 2022 04:38:30 +0000

What does DevOps mean to You?

DevOps is a term that has been around for a few years now, and its exact definition can be a little fuzzy. But at its core, DevOps is about breaking down the barriers between development and operations teams in order to improve collaboration and productivity. The goal is to create a more agile and responsive organization that can quickly adapt to changing business needs.

The DevOps movement has gained a lot of traction in recent years, and many organizations are looking for the right tools to help them implement DevOps practices.

There are many different models for implementing DevOps in your organization. But the most common approach is to use a combination of automation tools, process improvement techniques, and culture change strategies. Automation tools like Jenkins or TeamCity can help you streamline your build and deployment process, while process improvement techniques like Kanban or Scrum can help you optimize your workflow. And culture change strategies like training and education can help you create a more collaborative and productive work environment.

How to Adopt a DevOps Model

When an organization begins to adopt a DevOps model, it must first determine what specific tools and applications will be used. Many organizations find success by using open-source DevOps tools such as Git, Jenkins, SonarQube, Ansible, and Terraform. In order to use these tools effectively, the organization must also make changes to their development and operations processes.

One of the most important changes is the introduction of automated testing into the software development process. This can help to ensure that new code is error-free and can be safely deployed into production. Automated testing can also help to speed up the development process by identifying problems early on in the cycle.

Another key change is the adoption of continuous integration (CI) and continuous delivery (CD).

CI helps to ensure that changes made to code are properly integrated into the rest of the codebase. It also allows for earlier detection of errors and issues.

CD helps to improve the quality of the software being released, to reduce the amount of time between when a change is made to the code and when that change is available to users, and to allow for more frequent feedback from users.

How can DevOps help your Business?

The goal of DevOps is to make sure that the system works as efficiently as possible while still being able to meet customer demands.

In addition to the traditional roles of developers and operations professionals, DevOps brings together people from other parts of the organization, such as Quality Assurance (QA) and Business Analysts (BA). The goal is to break down silos within the organization and to improve communication and collaboration between different groups.

Some benefits of using DevOps include shorter wait times for new features, increased reliability, shorten the feedback loop, increase the speed of software delivery and improved communication.

Shorter Release Cycles
Shorter release cycles are possible because DevOps encourages breaking down the barriers between development and operations teams. This collaboration results in developers being able to quickly get feedback from operations on how their code is actually working in production and make necessary changes. As a result, there is less waste of time and effort due to misunderstandings or integration issues.

Increased Quality
Increased quality is another key benefit of DevOps. With tight collaboration between developers and operators, issues are caught and fixed much earlier in the process which leads to higher-quality products. In addition, automated testing can be used more extensively with DevOps to further improve product quality.

Increased Security
Increased security is another advantage of DevOps. By integrating security into the development process from the outset, vulnerabilities can be identified and addressed before they become a problem. And by automating many of the manual tasks involved in deploying software, DevOps makes it easier to enforce security policies and track compliance.

DevOps Practices Explained!

DevOps practices are designed to increase the efficiency and effectiveness of software development by integrating the functions of development, operations, and reliability engineering.

There are six key tenets of DevOps: Communication and Collaboration, Continuous Integration, Continuous Delivery, Configuration Management, Infrastructure as Code, and Monitoring and Logging. These principles help to improve coordination between teams, speed up feedback loops, and reduce risk.

The benefits of DevOps practices are clear: faster time to market, improved quality and reliability, and reduced costs. Let's look at this in more detail.

Communication and Collaboration

Employees who feel heard and connected to their team are more productive. In order for businesses to thrive in the current economy, communication and collaboration are essential. By creating an open dialogue, employees feel valued and are able to contribute their best ideas. When team members collaborate, they can produce better results by sharing knowledge and pooling resources.

Businesses can benefit greatly from communication and collaboration. First, communication allows for a clear understanding of the goals and objectives of the project. With everyone on the same page, there is less chance for confusion and miscommunication. Second, collaboration enables team members to work together to come up with creative solutions to problems. This can lead to a more efficient and successful project. Finally, communication and collaboration help build trust among team members. This trust is essential for a successful DevOps initiative. When team members trust one another, they are more likely to be open and honest with each other, which leads to better results.

Continuous Integration (CI)

Continuous integration, or CI, is a software development practice that enables developers to integrate their code into a shared repository several times a day. This helps to ensure that the code is always in a working state and reduces the chances of introducing errors into the codebase.

There are several business benefits of using continuous integration:

Fewer bugs in the codebase
By integrating code multiple times a day, developers are able to catch and fix errors earlier in the development process. This leads to fewer defects in the final product.

Shorter turnaround times
With a properly implemented CI system, developers can submit their changes for testing and approval quickly and easily. This leads to faster turnaround times for new features and bug fixes.

Continuous Delivery (CD)

In today's business world, it's essential to be able to quickly and efficiently release new features and updates to your product. This is where continuous delivery comes in. CD is a process that allows you to release software updates in a more automated way, which can lead to faster turnaround times and increased efficiency.

There are many benefits of using CD for your business. One of the biggest advantages is that it enables you to push out changes more quickly and easily. This means you can get new features and updates into the hands of your customers faster, which can give you a competitive edge. Additionally, CD can help improve the quality of your product by catching errors earlier in the development cycle.

Another benefit of CD is that it helps reduce downtime and increase reliability. When releases are done in a more automated way, there is less chance for something to go wrong.

Configuration Management (CM)

Configuration management (CM) is a term for the broad category of processes and technologies that help organizations manage and maintain the state of their IT infrastructure. CM tools allow organizations to define, track, and change the configurations of their systems in a controlled manner.

There are many different CM tools on the market, but some of the most popular ones are Ansible, Puppet, and Chef. Each tool has its own strengths and weaknesses, but they all share one common goal: to help organizations manage their IT infrastructure more effectively.
There are many business benefits to using a CM tool. Perhaps the most obvious benefit is that it can help organizations save time and money by automating tasks that would otherwise have to be done manually. CM tools can also help organizations better adhere to compliance regulations, improve system reliability and uptime, and reduce the risk of human error.

Infrastructure as Code (IaC)

In the world of business, time is money. Every minute wasted on manual processes means less time to focus on the company's core competencies and goals. This is where Infrastructure as Code (IaC) comes in, offering a way to manage IT infrastructure through code-based automation.

IAC tools such as Terraform allow businesses to treat their infrastructure as a product that can be version controlled, tested, and deployed like any other application. Additionally, IaC tools can be used in conjunction with other tools such as Puppet or Chef, allowing you to automate even more of your infrastructure management tasks.

Logging and Monitoring

Businesses today realize the importance of monitoring and logging their systems. By doing so, they can detect issues early and prevent them from becoming bigger problems. Additionally, businesses can use the data collected by monitoring and logging to improve their systems.

One benefit of monitoring is that businesses can detect issues early. If a problem is detected early, it can often be fixed before it causes any damage. This saves time and money, as well as reduces the risk of data loss or system failure.

Another benefit of monitoring is that businesses can use the data collected to improve their systems. For example, if a business notices that a particular process is causing errors, they can investigate why that is happening and make changes to fix the issue. By doing this, businesses can make their systems more efficient and reliable.

What is a DevOps Pipeline?

A DevOps pipeline is a sequence of automated tasks that helps to simplify and speed up the process of software development. The pipeline begins with the collection of code from a repository and ends with the deployment of the software. In between, various tasks are performed, such as compiling code, running tests, and packaging the software.

The advantages of using a DevOps pipeline are many. First, it helps to ensure that all changes are made in a consistent manner. Second, it allows for quick and easy feedback on whether changes have caused any problems. Third, it enables automation of many tasks that would otherwise have to be done manually. And fourth, it promotes collaboration between developers and operations staff.

What are the most popular tools in the DevOps domain?

There are a variety of tools that are used in the DevOps domain. Some of the most popular tools include;

Version Control Tools such as GitLab, GitHub and Bitbucket.
Build Tools like Maven.
Continuous Integration Tools like Jenkins, and TeamCity.
Continuous Delivery Tools including Argo CD, and Go CD.
Infrastructure as Code Tools like Terraform, Pulumi, and CloudFormation.
Code Quality and Code Security Inspection Tools like SonarQube.
Configuration Management Tools such as Puppet, Chef, and Ansible.
Container Platforms like Docker and Kubernetes.
Vulnerability Scanners including Aqua Security Trivy, and Clair.
Communication and Collaboration Tools like Slack.

These tools allow for automation of tasks and help to improve collaboration between developers and IT ops professionals.

In conclusion, DevOps is a beneficial process for businesses. By implementing DevOps, businesses can improve their communication, collaboration, and productivity. Additionally, DevOps can help businesses become more agile and responsive to changes in the market. As a result, businesses that implement DevOps are more likely to be successful in the long run.

If you liked the post, then you may purchase my first cup of coffee ever, thanks in advance :)

References

Convert Your AWS EC2 Linux PV Instance into an HVM Instance

Yasitha Bogamuwa — Sun, 15 May 2022 13:01:42 +0000

Linux Amazon Machine Images use one of two types of virtualization: paravirtual (PV) or hardware virtual machine (HVM). The main differences between PV and HVM AMIs are the way in which they boot and whether they can take advantage of special hardware extensions (CPU, network, and storage) for better performance.

This post will discuss the steps for converting an EC2 Linux instance from PV to HVM.

Prerequisites

A PV based instance, HVM instance (needs to be created) and a conversion instance (needs to be created).
Snapshots of the EBS volume(s) of the instance to be converted.
You'll need to snapshot the volume of the existing PV instance. Ideally the snapshot is taken with the instance in a stopped state.

You may also have an instance store volume. During a stop action any data on the instance store will be lost.

Instructions

1. Create an HVM instance, using as close as possible your current OS, (i.e., make sure the kernel versions match).

1.A. Create the HVM instance in the same Availability Zone (AZ) as your PV instance (eu-west-1b) and with the same size EBS volume 15GB.

1.B. Once it has started and is running you can stop it. Tag the Volume with something like OS-HVM and detach it from the HVM instance.

2. Start a conversion instance wait for it to fully start. (It is important that the instance is fully started before attaching the other volumes)

2.A. SSH into the conversion instance.

2.B. Attach the HVM volume to the conversion instance as /dev/xvdf.



cd /    
sudo mkdir /hvm
sudo mount /dev/xvdf1 /hvm

The following 2.C, 2.D and 2.E steps may not be needed on all Linux versions, however it is recommended.

2.C. Ensure you keep the HVM boot directory by moving it to the tmp directory.



sudo mv /hvm/boot  /tmp/boot.hvm

2.D. Empty out the rest of the drive with:



sudo rm -Rf /hvm/*

2.E. Verify the drive is now empty with:



sudo ls -al /hvm

3. With the snapshot of the Volume from your PV instance you'll want to Create Volume in the EC2 console. Keep the current volume size and create it in the same AZ (eu-west-1b). Find the new EBS volume and tag it OS-PV.

3.A. Attach it to the conversion instance as /dev/xvdg



sudo mkdir /pv
sudo mount /dev/xvdg /pv

3.B. Verify that it is mounted and has the correct file structure.



# You should see the boot and other root directories.
sudo ls /pv

3.C. Then copy the contents of /pv to /hvm



sudo cp -p -R /pv/* /hvm

4. Change the boot directory to be HVM based.

4.A. Remove the PV boot directory and replace it with the HVM boot directory.



sudo rm -R /hvm/boot

4.B. Copy back the saved HVM boot directory.



sudo mv /tmp/boot.hvm  /hvm/boot

5. Verify there are now all the needed directories and files on the HVM volume with.



sudo ls -al /hvm

6. You can now unmount the hvm volume.



sudo umount /hvm

7. Find the OS-HVM volume and detach it from the conversion instance.

7.A. Attach it back to the HVM instance as device /dev/xvda

7.B. Start the HVM instance.

8. Clean up the conversion instance and any unwanted or left over volumes/snapshots.

If you liked the post, then you may purchase my first cup of coffee ever, thanks in advance :)

References

AWS Linux AMI Virtualization Types

DEV Community: Yasitha Bogamuwa

Analyze AWS Application Load Balancer logs using Amazon Athena

1. Introduction

2. What is Amazon Athena

3. Analyzing ALB Access Logs with Athena

4. References

Unveiling the Speed Mystery: Investigating Slow S3 Uploads from AWS EKS Pods

The Problem

Investigation

Identifying the Issue

What is IMDS and What it does here?

How does the upload work with HttpPutResponseHopLimit set to 1 despite the slow upload speed?

Resolve the issue

Conclusion

References

Multi-Master Kubernetes Cluster Setup with CRI-O and vSphere Storage on Rocky Linux 8

Overview

1. Architecture Diagram

2. System Requirements

2.1. Master Nodes

2.2. Worker Nodes

2.3. Nginx Load Balancers

2.4. IP Allocations

2.5. DNS Entries

2.6. VMware Roles and Service Accounts

3. Configure Nginx Load Balancers

🔵 Important

🔵 Important

🔵 Important

4. Install and Configure Kubernetes

4.1. Install prerequisites on BOTH Master and Worker nodes

🔵 Important

🔵 Important

4.2. Configure MASTER nodes

4.2.1. Prepare Master Nodes

🔵 Important

🔵 Important

4.2.2. Configure the First Master Node (kubemaster01)

🔵 Important

🟢 You will get an output like this. Please make sure to record MASTER and WORKER join commands.

🔵 Important

4.2.3. Configure other master nodes (kubemaster02 and kubemaster03).

🔵 Important

4.3. Configure WORKER nodes

🔵 Important

4.4. Configure MetalLB Load Balancer

🔵 Important

🟣 NOTE

4.5. Configure Kubernetes Dashboard

🔵 Important

4.6. Create vSphere Storage Class

4.7. Deploy a Sample WordPress Blog

🔵 Important

4.8. Clean up Kubernates

🔴 Caution

5. References

Understanding Infrastructure as Code (IaC)

Overview

What is Infrastructure as Code?

Types of IaC

Benefits of IaC

Declarative vs Imperative

So far so good. But every coin has two sides!

Benefits of a Code Review

Learn some Best Practices

References

Create an Amazon Machine Image (AMI) from a FreePBX Virtual Machine (VM)

Prerequisites

Instructions

References

Introduction to DevOps

What does DevOps mean to You?

How to Adopt a DevOps Model

How can DevOps help your Business?

DevOps Practices Explained!

Communication and Collaboration

Continuous Integration (CI)

Continuous Delivery (CD)

Configuration Management (CM)

Infrastructure as Code (IaC)