DEV Community: Irtiza Ali

Kubernetes Cluster Authentication using AWS IAM

Irtiza Ali — Wed, 09 Feb 2022 09:14:16 +0000

Overview

This article is about the usage of AWS IAM for authenticating to a Kubernetes cluster.

Most of the things are available in the KOps and AWS IAM authenticator documentation but in this story I will discuss how to manage access to the cluster using AWS IAM Groups. It means that the administrator just need to do one time configuration of configuring the IAM role(s) in kubernetes cluster. After that they can add or remove an IAM user from IAM Groups to grant or revoke access respectively.

Pre-requisites

I am assuming that you must have some kind of knowledge about these topics:

Kubernetes.
AWS IAM.
AWS IAM Authenticator.

I am assuming that you have deployed the Kubernetes cluster using KOps on AWS. Now, you want to allow your existing IAM users to access the cluster. Further details can be found here.

If the cluster is not deployed using KOps instead the cluster is deployed using either eksctl or terraform, then you need to go through AWS IAM authenticator documentation to make some changes in its configurations accordingly.

Details/Steps
Follow the steps given below:

Deploy AWS IAM Authenticator on the cluster.
Create an IAM Role using these steps:

when creating a new role choose Another AWS Account then enter your AWS account id in the text box. Don’t attach any policy. Add tags if you want to then give an appropriate name and description to this role. Once all the previous steps are done then create the role.

Map this IAM role to a Kubernetes RBAC group so that IAM users can access the cluster assuming this role. We can do this by updating the AWS IAM Authenticator configuration and adding a mapping in mapRoles key of the config file. After the change your config map should look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-system
  name: aws-iam-authenticator
  labels:
    k8s-app: aws-iam-authenticator
data:
  config.yaml: |
    # a unique-per-cluster identifier to prevent replay attacks
    # (good choices are a random token or a domain name that will be unique to
    # your cluster)
    clusterID: <your-cluster-id>
    server:
      # groups
      mapRoles:
      - roleARN: arn:aws:iam::<account-id>:role/<name of the role created in previous step>
        username: <name of the role created in previous step>
        groups:
        - system:masters

Now create a policy the allows the IAM users to assume this role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::<account-id>:role/<name of the role created in previous step>"
        }
    ]
}

Create an IAM Group and attach the policy created in the previous step to this group. This group will be used to manage IAM users' access to the Kubernetes cluster.
After making these changes, update your kubeconfig by adding the role that the users need to assume while accessing the cluster. Your kubeconfig should look like this:

apiVersion: v1
clusters:
  - cluster:
      certificate-authority-data: <>
      server: <server-api>
    name: <cluster-id>
contexts:
  - context:
      cluster: <cluster-id>
      user: <cluster-id>
    name: <cluster-id>
current-context: <cluster-id>
kind: Config
preferences: {}
users:
  - name: <cluster-id>
    user:
      exec:
        apiVersion: client.authentication.k8s.io/v1alpha1
        command: aws-iam-authenticator
        args:
          - "token"
          - "-i"
          - "<cluster-id>"
          - "-r"
          - "arn:aws:iam::<account-id>:role/<role-name-that-was-created-earlier>"

Now you can grant and revoke IAM user access by adding and removing them from the group that was created in the previous step.
This whole authentication mechanism is created for Administrator users. You can similarly create separate roles and groups for users who need a different level of permission on the Kubernetes cluster.

Final Thoughts

I hope this story will be helpful. Please let me know if I have missed anything or anything that can be improved.

Add Git branch name to each commit

Irtiza Ali — Mon, 08 Nov 2021 06:12:42 +0000

Overview

This story is about how to utilize git hooks in order to add branch name to each commit. I have used the script from this article and modified it to my needs to minimize the restrictions.

Details

Follow the instruction given below to enable it:

Move inside your repository's home directory.
Rename the .git/hooks/prepare-commit-msg.example to .git/hooks/prepare-commit-msg
Change file permissions:

sudo chmod 755 .git/hooks/prepare-commit-msg

Paste the script given below in that file

#!/bin/bash
#
# Automatically adds branch name and branch description to every commit message.
# Modified from the gist here https://gist.github.com/bartoszmajsak/1396344
#

# This way you can customize which branches should be skipped when
# prepending commit message.
RED="\033[1;31m"
GREEN="\033[1;32m"
ORANGE="\033[0;33m"
NOCOLOR="\033[0m"

if [ -z "$BRANCHES_TO_SKIP" ]; then
  BRANCHES_TO_SKIP=(master production staging main)
fi

AUTHORINFO=$(git var GIT_AUTHOR_IDENT) || exit 1
NAME=$(printf '%s\n\n' "${AUTHORINFO}" | sed -n 's/^\(.*\) <.*$/\1/p')
# Regex to check the valid branch name
VALID_BRANCH_REGEX="(^(feature|hotfix|bugfix|release|dev|improvement))|^([A-Z]+\-[0-9]+)$"

# Get branch name and description
BRANCH_NAME=$(git branch | grep '*' | sed 's/* //')

# Branch name should be excluded from the prepend
BRANCH_EXCLUDED=$(printf "%s\n" "${BRANCHES_TO_SKIP[@]}" | grep -c "^$BRANCH_NAME$")

# A developer has already prepended the commit in the format BRANCH_NAME
BRANCH_IN_COMMIT=$(grep -c "$BRANCH_NAME" $1)

# check the branch name is valid or not
if [[ "$BRANCH_NAME" =~ $VALID_BRANCH_REGEX ]]; then
  # if face any error in mac then run chmod u+x .git/hooks/prepare-commit-msg and restart your terminal
  if [ -n "$BRANCH_NAME" ] && ! [[ $BRANCH_EXCLUDED -eq 1 ]] && ! [[ $BRANCH_IN_COMMIT -ge 1 ]]; then 
    sed -i.bak -e "1s/^/[$BRANCH_NAME] /" $1
  fi
else
  echo -e "\n${RED}Please correct the branch name${NOCOLOR}"
  printf "\nBranch Name not as per the defined rules like:  "
  echo -e "${GREEN}dev, hotfix, bugfix, release, dev, improvement, hotfix-102\n"
  echo -e "${ORANGE}You cannot push in these branches directly: ${BRANCHES_TO_SKIP[@]}\n"
  echo -e "${ORANGE}Current BRANCH_NAME: ${BRANCH_NAME}"
  echo -e "\n${RED}Oh, please stop. I cannot allow you to commit with your current branch: ${BRANCH_NAME}${NOCOLOR}"
  exit 1;
fi

After these changes, the branch name will be added to your every commit and it will look like this

It will not allow you to commit directly to the following branch

master
production
staging
main

You can change these restrictions based on your requirements.

Final Thoughts

I hope you would love this story and let me know if anything can be improved.

Uniform Data Distribution Among Kinesis Data Stream Shards

Irtiza Ali — Thu, 12 Nov 2020 13:55:16 +0000

Overview

This post is about how to distribute data among kinesis shards uniformly but before moving forward go through the kinesis documentation to grasp the basic understanding of kinesis data stream. I used Lambda as a consumer for Kinesis data stream, so I will discuss things in Lambda’s context.

Read this article about Kinesis stream and AWS Lambda.

Problem

A Kinesis data stream is a set of shards. Each shard has a sequence of data records. Each data record has a sequence number that is assigned by Kinesis Data Streams.

To put a record in Kinesis stream Partition Key must be provided for each record. An MD5 hash function is used to map partition keys to 128-bit integer values and to map associated data records to shards using the hash key ranges of the shards.

The problem arises when the partition key is not random and MD5 hash function maps the record’s partition key in a specific shard’s hash key range more than the other.

In the above image, we can see that shard#2 has more records than other shards because of the non-random partition key assignment.

Non-uniform distribution of messages causes Lambda’s Iterator age to increase and there is a chance to lose data. If we increase the retention period it will increase the operational cost of Kinesis.

Iterator age is the time between when the last record in a batch was recorded and when Lambda reads the record. 

Iterator age depends on other parameters, such as Lambda function execution duration, shard count, and batch size. 

For more information, see AWS Lambda CloudWatch Metrics.

Kinesis can retain data up to 7 days and the default retention period is 1 day.

Solution

There are two ways to distribute data among shards uniformly:

Assign each record a unique partition key, in this way, there is a better chance of uniform distribution.
Assign ExplicitHashKey to each record. In this way, the partition key is not required and we can make sure that the message is dumped in the shard of our choice. In python, we can use list shards method to get shards information. It will return information in this format:

{  "Shards": [
            {
                "ShardId": "shardId-000000000000",
                "HashKeyRange": {
                    "StartingHashKey": "0",
                    "EndingHashKey": "170141183460469231371588410571"
                },
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "496103516369698317099491240188243335841720557371394"
                }
            },
            {
                "ShardId": "shardId-000000000001",
                "HashKeyRange": {
                    "StartingHashKey": "170141183460469231371588410571",
                    "EndingHashKey": "340282asdasd6337460743176821144"
                },
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49610351636992132529573630114381723961608490082063351826"
                }
            },
            {
                "ShardId": "shardId-000000000002",
                "HashKeyRange": {
                    "StartingHashKey": "34028236692093846346337460743176821145",
                    "EndingHashKey": "51042355038140769519506191114765231717"
                },
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49610351637014433274772160737523259679881138443569332258"
                }
            }
        ]
}

As we can see that each shard has its own hash key range, so by using that information we can generate a key that lies between shard’s StartingHashKey and EndingHashKey. So by using this approach we can implement a round-robin algorithm that will put records to each shard one by one for achieving uniform data distribution among Kinesis data stream shards.

Final Thoughts

The approach discussed in this story is one way to achieve uniform data distribution among Kinesis shards. Please share your feedback about anything that can be improved or I missed. Thank you

Resources

https://docs.aws.amazon.com/streams/latest/dev/introduction.html

AWS Elasticsearch Service Data Rotation

Irtiza Ali — Wed, 01 Jul 2020 18:51:04 +0000

Overview

Problem

Elasticsearch is normally used for application logs management and monitoring. Logs should be retained for a specific interval of time, based on the needs and later must be discarded to clean up the disk space.

Elasticsearch provides a feature that can be used to delete the old data. But it is not recommended due to this problem.

Solution

The recommended way to clean up data is by using Elasticsearch Curator.

So in this story, we will create a lambda for curator and trigger it by using the CloudWatch event after a defined interval of time. Once lambda is triggered it will clean up the data using multiple filters.

Each of the above steps will be discussed in details later in this story.

Pre-Requisites

It is better to have knowledge about these services:

Assumption

I am assuming that you have a running AWS Elasticsearch cluster and application logs are being dumped in it.

Details

In this section, I will explain each step of the solution in detail:

1. Curator’s Lambda

Create an IAM role and attach this inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "es:ESHttpGet",
                "es:ESHttpDelete"
            ],
            "Resource": "<es-cluster-arn>"
        }
    ]
}

This policy will allow lambda to perform Get and Delete operations on the Elasticsearch cluster.

Create a lambda and assign the above role to it.
Once lambda is created we need to package and publish its code. In this Github repository, you will find:

lambda’s code.
guidelines on how to use filters (it will be used to filter elasticsearch indices that need to be deleted).
how to use different environment variables.
guideline on packaging and publishing lambda’s code.

2. Curator’s Lambda

Once lambda is published, we need to implement a cron functionality to trigger the lambda on regular intervals, to do it we will use AWS CloudWatch Events. Follow the guidelines given below:

Create an Event Rule.
Choose the Schedule event source.
Create and assign a Cron Expression based on your needs. AWS cron expression is a little bit different from the one we normally use. Details can be found on this link.
In the target select the lambda created above.

Final Thoughts

I hope that you like this story and please give feedback about anything that can be improved or I have missed. Thank you :)

Data Migration to New AWS Elasticsearch Service Domain

Irtiza Ali — Wed, 03 Jun 2020 05:45:40 +0000

This story provides guidelines to migration data to the new AWS Elasticsearch Service Domain.

Overview

Data migration to the new AWS Elasticsearch Service domain consists of two steps:

Creating a manual snapshot of Elasticsearch Service domain data on the S3 bucket.
Restore the snapshot from S3 in the Elasticsearch domain.

Assumption

I am assuming that you already know how to create an AWS Elasticsearch Service domain.

Manual Snapshot/Backup

Create a bucket in the same region where the Elasticsearch domain exists.
Copy the bucket arn.
Create an IAM role, this role will allow Elasticsearch to use S3. Initially create a role of ec2 use case (it will be changed later) without any policy. The policy will be added later.
Add an inline JSON policy and use the bucket arn copied in step 2:

{
   "Version": "2012-10-17",
   "Statement": [{
       "Action": [
         "s3:ListBucket"
       ],
       "Effect": "Allow",
       "Resource": [
         "arn:aws:s3:::<bucket-name>"
       ]
     },
     {
       "Action": [
         "s3:GetObject",
         "s3:PutObject",
         "s3:DeleteObject"
       ],
       "Effect": "Allow",
       "Resource": [
         "arn:aws:s3:::<bucket-name>/*"
       ]
     }
   ]
 }

Add a trust relationship so that Elasticsearch can assume this role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create an IAM user with AWS CLI utility usage enabled. This user will be used to register the manual snapshot repository. Attach the inline JSON policy given below:

{
   "Version": "2012-10-17",
   "Statement": [
     {
       "Effect": "Allow",
       "Action": "iam:PassRole",
       "Resource": "<role-arn-created-in-step-3>"
     },
     {
       "Effect": "Allow",
       "Action": "es:ESHttpPut",
       "Resource": "<elasitcsaerch-arn>"
     }
   ]
 }

Configure the user created in step-6 using its access-id and access-secret:

aws configure

Enter data for each prompt.

Install pip and some packages

sudp install python-pip
sudo pip install requests-aws4auth

Create a python file and paste the script given below:

import boto3
import requests
from requests_aws4auth import AWS4Auth

host = '<existing elasticsearch service domain url>'
region = '<elasticsearch service domain region>'
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Register repository
path = '_snapshot/<snapshot-repository-name>' # the Elasticsearch API endpoint
url = host + path

payload = {
  "type": "s3",
  "settings": {
    "bucket": "<enter bucket name created in step-1>",
    "region": "<bucket region>",
    "role_arn": "<arn of role created in step-3>"
  }
}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers)

print(r.status_code)

Run the script, it will print the data given below:

Note
Make sure if configure (command:aws configure) aws user command is executed using sudo run the python script using sudo otherwise, there is no need to use sudo.

Take the manual snapshot either by using elasticsearch api or kibana dev tool console:

PUT _snapshot/<snapshot-repository-name>/<date/snapshot-name>

To check snapshot has been created successfully and the indices that are part of this snapshot:

GET _snapshot/<snapshot-repository-name>/_all?pretty

Check the s3 bucket to check whether data has been created successfully.

Restore Snapshot

Create a new Elasticsearch Service Domain.
A role is required that will allow the new Elasticsearch Service Domain to access the S3 that was used to store the snapshots. but we don't need to create a new role because the role created in Step-3 of Manual Snapshot can be used here.
Create an IAM user with AWS CLI utility usage enabled. This user will be used to register the manual snapshot repository with a new Elasticsearch Service Domain. Attach the inline JSON policy:

{
   "Version": "2012-10-17",
   "Statement": [
     {
       "Effect": "Allow",
       "Action": "iam:PassRole",
       "Resource": "<role-arn-refered-in-step-2>"
     },
     {
       "Effect": "Allow",
       "Action": "es:ESHttp*",
       "Resource": "<new-elasitcsaerch-arn>"
     }
   ]
 }

Configure the user on a system:

aws configure

Install pip and packages but if already exists then no need for this step:

sudo install python-pip
sudo pip install requests-aws4auth

Create a file and paste the python script given below:

import boto3
import requests
from requests_aws4auth import AWS4Auth

host = '<new existing elasticsearch service domain url>'
region = '<new elasticsearch service domain region>'
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Register repository
path = '_snapshot/<snapshot-repository-name-used-in-manual-snapshot>' # the Elasticsearch API endpoint
url = host + path

payload = {
  "type": "s3",
  "settings": {
    "bucket": "<enter bucket name created manual snapshot process>",
    "region": "<bucket region>",
    "role_arn": "<arn of role refered in step-2>"
  }
}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers)

print(r.status_code)

Run the python script

Note
Make sure if configure (command:aws configure) aws user command is executed using sudo run the python script using sudo otherwise, there is no need to use sudo.

To check snapshot repository is configured, check the existing snapshots by either by using elasticsearch api or kibana dev tool console:

GET _snapshot/<snapshot-repository-name>/_all?pretty

It must show the snapshot that was created in the manual snapshot process.
To check existing indices:

GET _aliases?pretty=true

Restore the snapshot either by using elasticsearch api or kibana dev tool console:

POST _snapshot/<snapshot-repository-name>/<date/snapshot-name>/_restore -d
{
  "indices": "<index-name>",
  "ignore_unavailable": false,
  "include_global_state": false
}

Verify that the index has been restored:

GET _aliases?pretty=true

Verify the data of the index:

GET /<index-name>/_search/

Final Thoughts

I hope you have liked this tutorial. Do give me feedback about anything that can be improved. Thank you.

Validate a URL/Domain using Node.js HTTPS Module

Irtiza Ali — Sat, 16 May 2020 23:31:42 +0000

Overview

This post provides guidelines on how to validate a xyz.com domain. By validation, I mean that the domain has a valid certificate signed by Certificate Authority.

Scenarios

The list given below contains scenarios in which you want to validate the domain/URL:

You want to upload data to a server with a URL like this (xyz.com) and you are not sure whether this server is secure or not.
You have developed a B2B service and you want to only serve requests from valid domains.

How to do it?

In node.js there are two ways to do it:

https module
ssl-validate module

1. HTTPS Module

Nodejs https module’s request method validates the domain provided against the chain of Certificate Authorities root certificate. A code example is given below:

var https = require('https');

var options = {
  hostname: 'github.com/',
  port: 443,
  path: '/',
  method: 'GET',
  rejectUnauthorized: true
};


var req = https.request(options, function(res) {
  console.log("statusCode: ", res.statusCode);
  console.log("headers: ", res.headers);

});
req.end();

req.on('error', function(e) {
  console.error(e);
});

Keypoints

rejectUnauthorized: This means that it will validate the server/domain certificate against the chain of CA's root certificate.
The only problem with this approach is that this chain should be updated regularly otherwise a new domain that is signed by a certificate authority root certificate which is not part of the chain, marked as an invalid certificate(a common example is a self-signed certificate).

2. ssl-validate Module

It can also be used but it requires another module to get the domain information.

Logging using Telegraf and Influxdb for Mongo Restore and Backup Process

Irtiza Ali — Sat, 16 May 2020 10:33:21 +0000

This story provides a way to backup and restore MongoDB data using bash scripts, data encryption, and decryption using OpenSSL, logs collection using telegraf and influxDB for data storage.

Overview

In the above diagram, MongoDB is running on VM-1, while data backup and restore with telegraf on VM-2, telegraf collects the logs generated by backup and restore processes and finally dump the logs in a measurement of influxDB running on VM-3.
The architecture given above is one of the many ways to skin this cat.

Guideline

Detailed guidelines for the above processes can be found in the README.md of this repository.

Final Thoughts

I hope you have liked this tutorial. Do give me feedback about anything that can be improved. Thank you.

Image credits

Load Credentials using IAM Role Programmatically

Irtiza Ali — Sat, 16 May 2020 10:27:40 +0000

This repository contains guidelines to call AWS API Gateway’s endpoint from EC2 instance using a node.js and python scripts. The scripts need to send sign requests to the endpoint, so therefore it needs AWS credentials. For security and automation purposes, AWS credentials are not provided as environment variables instead a role has been assigned to the EC2 instance, which allows EC2 to access the API Gateway Endpoint. In this story, I will explain how to load the credentials programmatically using node.js and python scripts.

Details

Follow the step by step guidelines provided in the README.md of this repository for implementation details.

Final Thoughts

I hope that you liked this story, please give feedback regarding anything that can be improved.

Flask Application CI/CD using Travis CI, Github Actions, and Heroku

Irtiza Ali — Sat, 16 May 2020 10:24:41 +0000

This story provides guidelines on how to configure CI/CD pipeline for a Flask Application using Travis CI, Github Actions, and Heroku.

The detailed guidelines are provided in the README.md of this repository.