DEV Community: Ayaan Bordoloi

Configuring OIDC and IRSA for Amazon EKS

Ayaan Bordoloi — Thu, 15 Aug 2024 13:01:09 +0000

Learn how to configure OIDC provider with EKS and how to create IRSA. Source code is available in github.

Overview

In this blog, you will learn how to configure EKS cluster, OpenID connect (OIDC), IAM roles, and Kubernetes Service accounts using OpenTofu/Terraform. I will mainly use this configuration for granting permissions to read, write and list objects in my S3 bucket, but you can use this configuration to create your own IAM roles and policies for any Amazon service.

We will basically follow these five steps to achieve this:

Create an IAM OpenID Connect(OIDC) provider for your cluster.
Create IAM policy with permissions to write objects to your S3 bucket.
Create IAM role to access this policy.
Attach the IAM role and policy.
Use a Service Account to securely grant your pods AWS access via an IAM role.

OIDC provider configuration

OpenID Connect (OIDC) is used to securely connect your Kubernetes Service Account with an AWS IAM role. This allows your pods to assume the IAM role and access AWS services like S3 without needing AWS credentials directly inside the pods.

I created this OIDC provider using OpenTofu configuration file for my EKS cluster:

data "tls_certificate" "eks" {
  url = aws_eks_cluster.[your_cluster_name].identity[0].oidc[0].issuer
}

resource "aws_iam_openid_connect_provider" "eks" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.eks.certificates[0].sha1_fingerprint]
  url             = aws_eks_cluster.[your_cluster_name].identity[0].oidc[0].issuer
}

Replace [your_cluster_name] with your actual cluster name.

Create IAM policy

The IAM policy determines what your pods can do on AWS resources (e.g., access S3) when they assume the associated IAM role.

I created the necessary IAM policy using OpenTofu to access my S3 bucket:

resource "aws_iam_policy" "eks_s3_access_policy" {
  name        = "eks-s3-access-policy"
  policy      = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::[your_bucket_name]"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::[your_bucket_name]/*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:PutObjectAcl",
            "Resource": "arn:aws:s3:::[your_bucket_name]/*"
        }

            ]
}
EOF
}

Replace [your_bucket_name] with your actual bucket name.

Create an IAM role

The IAM role enables your Kubernetes pods to securely access AWS services by assuming the role through the ServiceAccount, inheriting the permissions specified in the attached IAM policy which we created before this step.

Creating an IAM role:

resource "aws_iam_role" "eks_s3_access" {
  name = "eks-s3-access"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "${aws_iam_openid_connect_provider.eks.arn}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub": "system:serviceaccount:default:s3-access-sa"
        }
      }
    }
  ]
}
EOF
}

AWS IAM supports federated identities using OIDC. This feature allows us to authenticate AWS API calls with supported identity providers and receive a valid OIDC JSON web token (JWT). We are passing this token to the AWS STS AssumeRoleWithWebIdentity API operation and receive IAM temporary role credentials. Such credentials can be used to communicate with services likes Amazon S3 and DynamoDB.

Attach the policy to the IAM role

I have created a OpenTofu configuration file to create this policy and role attachment for the ServiceAccount to use:

resource "aws_iam_role_policy_attachment" "eks_s3_access_policy_attachment" {
  role       = aws_iam_role.eks_s3_access.name
  policy_arn = aws_iam_policy.eks_s3_access_policy.arn
}

Finally, we will create a ServiceAccount in our cluster which serves as the Kubernetes identity that allows your pods to securely assume the AWS IAM role. This enables the pods to access AWS resources like S3 with the permissions defined in the IAM policy, without requiring AWS credentials to be manually managed or exposed within the cluster.

Create ServiceAccount

The Service Account in this setup is crucial for securely allowing Kubernetes pods to assume an AWS IAM role, enabling them to access AWS resources, such as S3, without needing to embed AWS credentials directly in the pods.

Here is YAML file I used to create the ServiceAccount in my cluster:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-access-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::[AWS account ID]:role/eks-s3-access

Replace the [AWS account ID] with your actual account ID.

The eks.amazonaws.com/role-arn: arn:aws:iam::[AWS account ID]:role/eks-s3-access is used to map your Kubernetes Service Account to the AWS IAM role.

Now, you have got the necessary permissions to give access to a Workload running inside the cluster to resources outside of the cluster such as the S3 bucket.

Conclusion

In this article, we explored how to securely grant a Kubernetes pod access to AWS resources using IAM roles. This approach is particularly useful for scenarios where workloads running inside the cluster need to interact with external AWS services.

Based on my experience, getting it up and running wasn't a quick task. However, I hope this guide helps others save time and avoid the same challenges.

Mastering Connectivity: CG-NAT Solutions with Tailscale

Ayaan Bordoloi — Mon, 29 Jan 2024 14:01:15 +0000

Recently, the CEO of Supabase tweeted about the importance of transitioning from IPv4 to IPv6. This is due to the shortage of IPv4 addresses, and major cloud providers like AWS are set to start charging for IPv4 addresses starting from February 1, 2024.

Not everyone is fully ready to adopt IPv6 because of its complexities. To lawfully and technically circumvent these issues, I will demonstrate two methods in two clouds which I learned in The Cloud Seminar recently. These solutions don't always work, but in most cases, one can definitely bypass the restrictions.

First, I will discuss what tailscale is and how we can use tailscale to translate a private IPv4 address to public IPv4 address using CGNAT CIDR.

Tailscale

Tailscale is a VPN service that enables encrypted point-to-point connections using the open source WireGuard protocol, making devices and applications accessible anywhere in the world, securely and effortlessly. It offers speed, stability, and simplicity over traditional VPNs.

We will use tailsacle to convert our private IPV4 address to public IPV4 address using CGNAT CIDR. So, what CGNAT does is it involves the translation of private IP addresses used within a local network into a smaller set of public IP addresses when the traffic passes through the ISP's network.

Cloud Machine Access via Console Connect

Now, I will be working in the Hetzner cloud to demonstrate how we can SSH into our cloud machine without a public IP using tailscale.

Steps:

Login to the console and create a project in the cloud.
Start creating your server by adding the required machine details. ( I picked an ubuntu machine)

While creating the server in my Hetzner Cloud console, I unchecked the Public IPv4 option and only checked the public IPv6 option because it was charging me a certain amount of money to enable it.

Finally, create the server after filling in all the required details for the machine you need.
After creating your server, your server dashboard will be visible. (I named my server tailscale-connect)

At the top of your server dashboard you will see a console connect option, click on that.

The instance connection will open and you will see the following screen:

Now, to log in to the server, it asks for a login username and login password, which we don't know. We will reset the login username and password to connect to the server.

Navigate to the rescue screen on your server dashboard and click "Reset Root Password." (After clicking, you will receive a new root password. Save it somewhere.)

Now, refresh the page and open the console connect terminal again to login with the new root password. (Type your new root password which you generated in the previous step.)
You will see the following screen after successfully logging into your server-

The top arrow indicates the private IPv4 address, and the arrow below indicates the IPv6 address that the cloud provided us.

Since, we didn't enable the public IPv4 address(because of extra cost) we didn't get one.

Even though we connected to the instance through the console, we still haven't connected it through our machine, and it is not possible to do so without a public IPv4 address. To circumvent this issue, we will use tailscale.

Tailscale installation

Head to the terminal and install tailscale on your linux machine with the following command:



curl -fsSL https://tailscale.com/install.sh | sh

Once your installation is complete, type tailscale up and go to the link that tailscale provided in the terminal.
You will see the following screen:

Sign in to tailscale using any of the provided methods.
You will see the following screen after singing up-

The tailscale dashboard displays all the devices connected to your network.(Currently it's only your local machine)
Now, we will connect the cloud machine to the same network and then access it through our local machine without a public IPv4 address.

Let's see how to do that in the next steps.

Tailscale: SSH connect without Public IP

Head to the cloud machine terminal that we logged into earlier.
We will install tailscale on this machine and connect it to the network that we created previously.
Type the following command to install tailscale on your cloud machine



curl -fsSL https://tailscale.com/install.sh | sh

Once your installation is complete, type tailscale up and go to the link that tailscale provided in the terminal.
On the sign-up page, sign in with the same account that you used to sign up on your local machine.( In my case, I used my github account.)
Once you have signed in, you will now see two machines on the dashboard: one being your local machine and the other being your cloud machine.

Now that we have connected the two machines on the same network, we can access the cloud machine through our local machine without requiring a public IPv4 address.
Copy the CGNAT IP of your cloud machine provided by Tailscale from your dashboard.
Open your terminal and type the following to connect to your cloud machine



ssh root@[CGNAT-IP]

Now, you have successfully connected to your cloud machine through your local machine without requiring a public IPv4 address. How cool is that?!!

We don't need the console login terminal anymore, so we can close it and directly access the machine via our machine terminal.

EC2: SSH connect without public `IPv4`

Now, to implement the above method, we need to log in to the server using the cloud provider's console connect. Unfortunately, AWS console connect doesn't support public IPv6 addresses, only IPv4. AWS will charge $0.005 per IP per hour for public IPv4, leading to high costs. To address this, create a Jump box (Bastion Host) in the public subnet using a public IPv4 address and connect other machines through it using private IPs. This reduces costs, as only one machine is billed for the public IPv4 address, with others connected via private IPs.

Steps:

Log into your AWS console.
Head over to the VPC section and create a VPC for your instances.
In the VPC section, create two subnets (Public & Private) allowing IPv4 CIDR blocks for both and a IPv6 CIDR block for private only.
Create an Internet gateway (IGW) and attach it to your VPC.
Create a route table and associate it with the public subnet. Edit the route and add a destination of 0.0.0.0, choosing the destination as the IGW you created in the previous step.
Similarly, create a Route table and associate it with the private subnet. No need to edit route here.
Head over to EC2 instances and create a key pair first.
Create an EC2 instance (JUMPBOX) in the public subnet of your VPC, allowing the option for auto-assigning a public IPv4. In the security group, allow SSH from my IP.
Try to SSH into it from your terminal and see if it's working.

Next, create one or two instances (test instances) in the private subnet of your VPC. Disable the auto-assign public IPv4 address and enable the auto-assign IPv6 address.

We cannot SSH into the Test instances because we have disabled the public IPv4 option. We only have a private IPv4 address and an IPv6 address. To SSH into the test instances, we will use the JUMPBOX, which has an IPv4 address assigned to it.
Head over to config file in the .ssh folder.



cd .ssh
code .

Paste the following in the config file



Host JUMPBOX
  HostName [Public IP of JUMPBOX]
  ForwardAgent yes
  StrictHostKeyChecking ask
  IdentityFile ~/.ssh/[private-key]
  User ubuntu

Host [Private IP of test instance]
  ProxyCommand ssh -q -W %h:%p JUMPBOX
  ForwardAgent yes
  IdentityFile ~/.ssh/[private-key]
  User ubuntu

Host [Private IP of 2nd test instance]
  ProxyCommand ssh -q -W %h:%p JUMPBOX
  ForwardAgent yes
  IdentityFile ~/.ssh/[private-key]
  User ubuntu

Please update the above file with the details of your own machine and then save and exit.
Add the following rule in the security group of your test instances and then save it.(Custom IP is the CIDR of your VPC)

Now, head over to the terminal and ssh into your test instances through your private IP.

This is how we can SSH into our instances without a public IPv4 address. This method can reduce the cost to a great extent. You can also install a NAT gateway in your public subnet to access the internet on these machines.

Conclusion

In this blog, I discussed how we can SSH into our cloud machines from our local machine without the use of public IPv4 addresses, and with the help of Tailscale using the CGNAT CIDR, thus saving the cost of using a public IPv4 address. I also discussed how we can create a JUMPBOX to do the same but with the cost of paying for only one IPv4 network rather than paying for IPv4 addresses for all the machines. These are some great alternatives to save the extra cost without transitioning into IPv6 addresses.

Kubernetes Cluster Setup Using Kubeadm on AWS

Ayaan Bordoloi — Sun, 10 Dec 2023 14:50:04 +0000

In this blog, I will explain how to setup a Kubernetes cluster with one master node and two worker nodes using Kubeadm. We will be doing it on the AWS cloud.

Prerequisites for this setup:

AWS account.
MobaXterm installed on your system.
Knowledge about Kubernetes architecture. You can checkout this video.

What is Kubeadm?

Kubeadm is a tool to setup kubernetes clusters without any complex configurations. It performs the actions necessary to get a minimum viable cluster up and running. It is developed and maintained by the official Kubernetes community. You can create a production-like cluster locally on a workstation for development and testing purposes with kubeadm.

Kubeadm prerequisites:

We need to create the EC2 instances with the following configurations:

Launch one Ubuntu instance with minimum of 2 vCPU and 2GB RAM for the master node.
Launch two Ubuntu instances with minimum of 1vCPU and 2 GB RAM for the worker nodes.

Kubeadm port requirements:

Enable the following ports in the security groups of the instances:

Control plane node: `6443/tcp` for Kubernetes API Server `2379-2380` for etcd server client API `10248-10260` for Kubelet API, Kube-scheduler, Kube-controller-manager, Read-Only Kubelet API, Kubelet health `80,8080,443` Generic Ports `30000-32767` for NodePort Services
Worker plane node: `10248-10260` for Kubelet API etc `30000-32767` for NodePort Services ## Cluster setup steps:

Here are the major steps involved in this setup:

Launch 3 EC2 t2.medium instances with the specified ports open.
Install container runtime on all nodes. We will use cri-o.
Install Kubeadm, Kubelet, and kubectl on all the nodes.
Initiate Kubeadm control plane configuration on the master node.
Install the Calico network plugin.
Join the worker node to the master node using the join command.
Validate all cluster components and nodes.
Deploy a sample app and validate the app to test our cluster.

Let's start with the setup now!

Launch EC2 instances:

Launch 3 t2.medium instances and SSH into them through MobaXterm.
Open the 3 nodes in the multi-execution mode.

Refer to this video if you're new to MobaXterm.

Run on all the nodes of the cluster as root user-

1. Disable SWAP

We need to disable Swap for kubeadm to work properly.

swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

2. Enable iptables Bridged Traffic on all the Nodes

Execute the following commands on all the nodes for IPtables to see bridged traffic. Here we are tweaking some kernel parameters and setting them using sysctl.

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

3. Install CRI-O Runtime

The basic requirement for a Kubernetes cluster is a container runtime. We can use any container runtime we want, here we will be using CRI-O.

Enable cri-o repositories for version 1.28

OS="xUbuntu_22.04"

VERSION="1.28"

cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
EOF
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
EOF

Add the GPG keys for CRI-O to the system’s list of trusted keys.

curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -

Update and install crio and crio-tools.

sudo apt-get update
sudo apt-get install cri-o cri-o-runc cri-tools -y

Reload the systemd configurations and enable cri-o.

sudo systemctl daemon-reload
sudo systemctl enable crio --now

The cri-tools include crictl, a CLI utility for interacting with containers created by the container runtime. When utilizing container runtimes other than Docker, you can employ the crictl utility for debugging containers on the nodes.

4. Install Kubeadm & Kubelet & Kubectl

Install the required dependencies

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

Download the GPG key

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://dl.k8s.io/apt/doc/apt-key.gpg

Add the Kubernetes APT repository to your system.

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Update apt repo

sudo apt-get update -y

Install the latest version

sudo apt-get install -y kubelet kubeadm kubectl

Add hold to the packages to prevent upgrades.

sudo apt-mark hold kubelet kubeadm kubectl

Add the node IP to KUBELET_EXTRA_ARGS.

sudo apt-get install -y jq
local_ip="$(ip --json a s | jq -r '.[] | if .ifname == "eth1" then .addr_info[] | if .family == "inet" then .local else empty end else empty end')"
cat > /etc/default/kubelet << EOF
KUBELET_EXTRA_ARGS=--node-ip=$local_ip
EOF

We now, have all the required utilities and tools for configuring Kubernetes components using kubeadm.

Now, let's initialize Kubeadm On Master Node To Setup Control Plane.

5. Initialize Kubeadm On Master Node

Execute the commands in this section only on the master node.
Set the following environment variables.

IPADDR=$(curl ifconfig.me && echo "")
NODENAME=$(hostname -s)
POD_CIDR="192.168.0.0/16"

Initialize the master node control plane configurations using the kubeadm command.

sudo kubeadm init --control-plane-endpoint=$IPADDR  --apiserver-cert-extra-sans=$IPADDR  --pod-network-cidr=$POD_CIDR --node-name $NODENAME --ignore-preflight-errors Swap

You should get the following output:

Copy the join command save it somewhere, we will need it for joining the worker node to the master.

Use the following commands from the output to create the kubeconfig in master so that you can use kubectl to interact with cluster API.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Now, verify the kubeconfig by executing the following kubectl command to list all the pods in the kube-system namespace.

kubectl get po -n kube-system

You should get the following output:

You verify all the cluster component health statuses using the following command.

kubectl get --raw='/readyz?verbose'

You should get the following output:

By default, apps won’t get scheduled on the master node. If you want to use the master node for scheduling apps, taint the master node.

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

Next, we will be installing CNI tool for pod networking.

6. Install Calico Network Plugin

Kubeadm does not configure any network plugin. You need to install a network plugin of your choice for kubernetes pod networking and enable network policy.

We will use the Calico network plugin for this setup.

Run the following commands to install the Calcio network plugin

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml

curl https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/custom-resources.yaml -O

kubectl create -f custom-resources.yaml

7. Joining of Worker nodes to master node

Now, let’s join the worker node to the master node using the Kubeadm join command you have got in the output while setting up the master node.

Execute the commands in this section only on the worker nodes.
Here is what the command looks like in my case

kubeadm join 3.86.197.238:6443 --token 3nf11u.w40mnymbrlkr8f88 \
        --discovery-token-ca-cert-hash sha256:d904af14e6bee5af2020e2b1a9572345403b1a1c7a095bb5e8584e04f1db3667

You should get the following output:

Execute the kubectl command from the master node to check if the node is added to the master.

kubectl get nodes

You will get the following output:

You can further add more nodes with the same join command.

8. Deploy a sample Nginx App

Create an Nginx deployment. Execute the following directly on the command line.

kubectl run nginx --image=nginx --port=80 
kubectl expose pod nginx --port=80 --type=NodePort

A random port between 30000-32767 and assigned to nginx app.
To access that port we run the following command:

kubectl get svc

You get an output like this:

Here, we can see that the nginx app has been deployed on the port 30501.

We can now access that port with our master node ip or worker nodes ip and get the following result:

This shows that we have setup our cluster succesfully!

Conclusion

In this blog, I have explained how to deploy our own Kubernetes cluster using Kubeadm on AWS. This setup is good for learning and playing around with kubernetes.

Thanks for reading!

DEV Community: Ayaan Bordoloi

Configuring OIDC and IRSA for Amazon EKS

Overview

OIDC provider configuration

Create IAM policy

Create an IAM role

Attach the policy to the IAM role

Create ServiceAccount

Conclusion

Mastering Connectivity: CG-NAT Solutions with Tailscale

Tailscale

Cloud Machine Access via Console Connect

Steps:

Tailscale installation

Tailscale: SSH connect without Public IP

EC2: SSH connect without public IPv4

Steps:

Conclusion

Kubernetes Cluster Setup Using Kubeadm on AWS

Prerequisites for this setup:

What is Kubeadm?

Kubeadm prerequisites:

Kubeadm port requirements:

Launch EC2 instances:

Run on all the nodes of the cluster as root user-

1. Disable SWAP

2. Enable iptables Bridged Traffic on all the Nodes

3. Install CRI-O Runtime

4. Install Kubeadm & Kubelet & Kubectl

5. Initialize Kubeadm On Master Node

6. Install Calico Network Plugin

7. Joining of Worker nodes to master node

8. Deploy a sample Nginx App

Conclusion

EC2: SSH connect without public `IPv4`