DEV Community: Nuwan Arambage

Github or Gitlab | Setup personal token to use automatically in git commands

Nuwan Arambage — Mon, 13 Jan 2025 09:33:59 +0000

I came across this trick which works for me pretty well. Our internal soure code repository migrated to gitlab from bitbucket. It was bit annoying to put username and personal token eveytime.

When I googled few times. I fould this trick and thought of sharing with the community.May be someone could get a help from this. There is another method as well. I have tried and tested only the first method. You could try out other method if you would like more secure way to keep your github personal token.

Method 01: Use ~/.netrc file

Step1: Open this ~/.netrc file 
    vim ~/.netrc

Step2: Addn following entry 
cat ~/.netrc
machine github.com login <login-id> password <token-password>   <-- sample entry 

cat  ~/.netrc
machine gitlab.wyoc.com login nuwana1 password sdp-A  <-- actual entry

Method-02: Use credentials caching

Run below command in order 

    git config credential.helper store <-- This will ask your username and passwords and then remember it for future use. 

    git pull <--Please note running above commands will create a file at ~/.git-credentials and store the credentials in plain text which can be a security risk.

Note: An alternative is to store the credentials in memory rather than the disk. To do this you can run the below command.

git config credential.helper 'cache --timeout=3600'

SSH Keys | Change the label of the public key

Nuwan Arambage — Mon, 13 Jan 2025 09:05:30 +0000

Hope everyone is in the journey of year 2025. Today, let's talk something which is trivial and usually comes across when you work in DevOps or SRE space. It is about ssh public key.

I have seen many cases that ssh public key label is rarely used to specifically identify perticular public key. Most of the time the keys are generated keeping the default label whcih has the user@hostname format. General usecase is engieer generates the public and private key for automation or authentication and turns the blind eye for -C option which creates lable for public key.

Essentially, it is pretty simple. We can generate the ssh key using label or we can edit manually after ssh keys are generated. Let's learn by doing . Simply generate a ssh key by following command.

ssh-keygen -t rsa -b 4096 -f ./id_rsa

Following is the execution output when above command is run. I would not go to explain more in -t, -b and -f options. man pages could give you details.

[root@jenkin-slave-node .wyoc]# ssh-keygen  -t rsa -b 4096 -f ./id_rsa
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ./id_rsa.
Your public key has been saved in ./id_rsa.pub.
The key fingerprint is:
SHA256:cDEkDXCL9/I79yVUQ11cm0uuhiwKZLrrWLiPwbedUTY root@jenkin-slave-node.localdomain
The key's randomart image is:
+---[RSA 4096]----+
|    ..+++     .o=|
|     o o.o   .  =|
|    . + .     o+ |
|     . +     .o..|
|     o.ES   .  o |
|. . + oo. ... .  |
| + + o  .. o.o.  |
|  B + + o.o .o   |
| +o*.o ..o ..    |
+----[SHA256]-----+

The result of this command is following files. One is a private key other one is a public key.

[root@jenkin-slave-node .wyoc]# ls -l id*
-rw------- 1 root root 3243 Jan 13 08:32 id_rsa  <-- private key
-rw-r----- 1 root root  760 Jan 13 08:32 id_rsa.pub <-- public key
[root@jenkin-slave-node .wyoc]#

Let's open the public key using cat command and see the output. Scroll to the right and find string --> root@jenkin-slave-node.localdomain .
Well, that is the one we called the comment or label which we found after the ssh public key.

[root@jenkin-slave-node .wyoc]# cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCv+QOF1jiZ8C5//btEQkhsDbaG00Ftce9tbsFj2WxnEZPEnWmkG+jMDKAxp3IuXzn1H+NP0uT0whcumClCwrpwH7NFcWigxPbcQhMaIWcYovqmDqY1liCDNoCAjmgQBbypIJXGNkHzeuVOoYhfKah3ZuXNnMv3y6m+KXuLdxko1lUdYmZHYfDTNi0Ho6WeW3tVZ48pvsUsOGrrc7fn5z923CH750oOem/i4Of6cL2qIoSd4Jrfs9T/aUNHGN5G5qw08I/Zkm3J/LiWi+0C7zJFdnVWueHkBIshpqD+7vwWF7/zMzGwidr0YVATc8LEHuKfmwOjmkQFT5VkzKRCxlE711NvtXmqPqF0E603OXTs5QfAznMuFb6y6N71E6b9PM8wZeG5SXgAIMv8kr1xnsBQ2NU5EZu1TfFaO+A3lHBw4YjO0hhqZeDz9QwC02iVBS4lJJjKvYlSlmwPfuR9NZ70f9tefsSG0zEo1K2UiGoHCOacyrR3Bo+rKdiYre6Gam9rZyC7Uksl1b64KtNqkesYCUMG9CtGsTMySbljVv3AuJI/6LG7dAIjnbPTDIaW+CbWyW2yqVAAvHlx19j/Hjc6HYv5ikHhYZm6EZ7bN3C4vnLs+LOdUAi03sc2ETr5ECLN0YMduGfh7IHYcxkxEsUvXZy96OibNdiQef/7md/Lkw== _root@jenkin-slave-node.localdomain_

I would like to add my perspective on labeling ssh public key. Because it gives more clarity and maintainability. So I have mentioned below important notes.

Important Notes

You can edit the label manually after ssh public key is generated. This can be done via vim command.
Or You can use the ssh key gen command with -C option. For example following command is helpful.

[root@jenkin-slave-node .wyoc]# ssh-keygen -t rsa -b 4096 -f ./id_rsa-a -C "JenkinMasterNodePublicKey"

In conclusion, I hope this is a helpful tip and happy reading who spend time to come here. If you have a different perspective, let's make a comment.

Take care till we meet again with a new post. Cheers!

Terraform in AWS | Provision EC2 with AWS Systems Manager SSM access

Nuwan Arambage — Fri, 10 Jan 2025 14:52:25 +0000

You all come across this task which I'm going to discuss today. It is a trivial and most common one if you start to work on DevOps stuff. This is all about provisioning EC2 instance using terraform. It is somewhat extenstive version of EC2 instance provisioning because of below reasons.

All TF configurations which are environment specific inside the infra-configs.tfvars file.
All configurations are in human readable format which means no subnet_id, no vpc_id. It has only vpc_name, security_group_names etc. How we do it is , id are retrived via terraform data blocks where ever possible.
It uses the tagging which is most important in Enterprise cloud application environment
Multiple ingress rules can be configured via infra-configs.tfvars config file rather .tf file itself.
It supports configuring multuple subnets and it chooses one subnet randomly to spin up EC2 instance.
TF code written in a modularized approach where iam role, security group, ec2 provisioned via seperate modules.

Code is published into github public repository. Git repo URL is mentioned here.
https://github.com/arambage0/cloud-infra-wyoc/tree/main/terraform/provision_ec2_ssm

Well, there are prerequisites in order to setup SSM access to EC2. There is a nice blog https://dev.to/aws-builders/securely-access-your-ec2-instances-with-aws-systems-manager-ssm-and-vpc-endpoints-1bli#how-to-use-ssm-for-ssh-less-login written by https://dev.to/sunnynazar so I would like to extact details from his blog because it is nicely written and I do not want to repeat myself here. I would extact few details from his blog to show up prerequisites.

Prerequisites for SSM to work with EC2

Security Group for EC2 Instance: The minimum traffic you need to allow for SSM access to work is to add an Outbound HTTPS (port 443) in the security group for EC2 instance.

Create an IAM Role: To use SSM to log in to EC2 instances, you must first create an IAM role with the required permissions. The role must have the AmazonSSMManagedInstanceCore policy attached to it, which allows SSM to access the EC2 instances.

Install SSM Agent: After creating the IAM role, you need to install the SSM agent on each EC2 instance you want to access using SSM. The SSM agent is pre-installed on Amazon Linux 2 and Amazon Linux AMIs, but you must install it manually on other instances.

Configure EC2 Instances: Once the SSM agent is installed, you need to configure your EC2 instances to allow SSM access. You can do this by creating a VPC endpoint for SSM. VPC endpoints which are required when using Private Subnets are below:

com.amazonaws.region.ec2messages
com.amazonaws.region.ssmmessages
com.amazonaws.region.ssm
com.amazonaws.region.kms (This is needed if you want to use AWS KMS encryption for Session Manager.)

Note:The security group for VPC Endpoints must allow inbound HTTPS (port 443) traffic from the resources in your VPC that communicate with the service.

In the terraform code published into the repo works in a already provisioned cloud infrastructure where subnets, vpc endpoints are available and ssm agent is baked into ami that we used to spin up the EC2 instance.

As part of the terraform code, it provisions IAM role, EC2 isntance profile, security group and EC2 with best practices which are used in the Industry.

If you are eager to execute the code. Following steps are for you.

Step 1: Clone the git repository https://github.com/arambage0/cloud-infra-wyoc.git

  `git clone https://github.com/arambage0/cloud-infra-wyoc.git`

Step 2: Configure the tfvar file infra-configs.tfvars. The most mandatory ones are region, vpc_name, instance_type, key_name, private_subnets, pre_existing_sg_names depends on your environment.

cd terraform/provision_ec2_ssm/configs/nonprod/ vim infra-configs.tfvars

region          = "eu-west-1"
vpc_name        = "LSA-NONPROD-LTSQA-vpc"
instance_type   = "t2.large"
key_name        = "lsa-nonprod-ire-private-key"
private_subnets = ["LSA-NONPROD-LTSQA-app_internal_subnets-eu-west-1a", "LSA-NONPROD-LTSQA-app_internal_subnets-eu-west-1b", "LSA-NONPROD-LTSQA-app_internal_subnets-eu-west-1c"]
instance_name   = "app-node01"

sg_name        = "AppNodeSecurityGroup"
sg_description = "Allow multiple ports"

# Existing Security Groups
pre_existing_sg_names = ["App-Tier-LTSQA-SG", "App2Endpoint-Tier-LTSQA-SG"]


sg_ingress_rules = [
  {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["20.101.58.0/23"]
  },
  {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["20.101.58.0/23"]
  },
  {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["20.101.58.0/23"]
  }
]

role_name                       = "AppNodeSSMEC2Role"
ssm_policy_name                 = "AmazonSSMManagedInstanceCore"
permission_boundary_policy_name = "ServiceRoleBoundary"
instance_profile_name           = "AppNodeSSMInstanceProfile"


environment       = "non-prod"
environment_level = "nonprod-e2e"
project_prefix    = "saga-lts"


mandatory_tags = {
  ApplicationName    = "SAGASurveillanceAnalytics"
  ApplicationID      = "APP-80897"
  CostCentre         = "L96532-IMO"
  ProjectCode        = "P007723"
  Environment        = "LTS-NONPROD"
  DataClassification = "Restricted"
  ManagedBy          = "XYO"
  Automation         = "YES"
}



recommended_tags = {
  BusinessUnit        = "LSEGSurveillance"
  Owner               = "AWS-SAGA-LTS-NONPROD@SAGA.COM"
  BusinessCriticality = "1"
  AssignmentGroup     = "CMUKAppSupportSurv"
}


optional_tags = {
  Account  = "AWS-SAGA-LTS-NONPROD"
  Workload = "NONPROD"
}

Step 3: Configure the configs.tf file as well for mentioning terraform s3 configuration. For instance bucket, key, region, dynamodb_table

vim configs.tf

terraform {
  backend "s3" {
    bucket         = "my-wyoc-s3-bucket"
    key            = "terraform/state/my-project.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "my-wyoc-state-lock" 
    encrypt        = true
  }
}

provider "aws" {
  region = var.region
}

Step 4: After that, we are good to execute terraform commands.

cd /<YOUR_OWN_DIRECTORY_PATH>/cloud-infra-wyoc/terraform/provision_ec2_ssm terraform init terraform plan -var-file="./configs/nonprod/infra-configs.tfvars" terraform apply -var-file="./configs/nonprod/infra-configs.tfvars"

Ok cool. If you come to this stage successfully, we are really good to login to EC2 machine via SSM.

First of all check the EC2 status. You should see EC2 status is running and both ec2 status should be completed OK. Once it is okay, you try login via SSM.

Happy learning for all of you and thanks for reading. I would like to add a nice quote. It grabs my attention.

"We learn by doing things. We learn by experimenting thigs. We learn by breaking and fixing things."

Terraform in AWS | Provision TF backend using AWS S3 and DynamoDB

Nuwan Arambage — Tue, 07 Jan 2025 23:52:45 +0000

Terraform is a popular infrastructure provisioning tool. It works with AWS well. DevOps engineer may across the case where he/she needs to set up remote state for terraform. It can be done via AWS S3 bucket and Amazon DynamoDB. In order to make the life easy, anyone can use the below script to create remote state setup easily. What you need to do is, follow these steps once you log into AWS cloudshell.

mkdir 07012024 cd 07012024/ pwd git clone https://github.com/arambage0/cloud-infra-wyoc.git ls -ltr cd cloud-infra-wyoc/ cd scripts/tf-backend/ chmod u+x create_tf_backend.sh ./create_tf_backend.sh --region eu-west-1 --bucket-name my-wyoc-s3-bucket-a --dynamodb-table-name my-wyoc-state-lock-a

The command options values that you can select as you prefer.

./create_tf_backend.sh --region <REGION_CODE> --bucket-name <BUCKET_NAME> --dynamodb-table-name <DYNAMODB_TABLE_NAME>

I have mentioned below the real execution that I have done in my AWS account cloudshell.

Essentially, script creates the s3 bucket with versioning enabled and dynamodb table in pay per request mode.

Okay. It looks all good. Let's run the simple terraform code provisioning s3 bucket to see that terraform remote backend setup works.

Let's follow these steps

mkdir demo-07012024 cd demo-07012024 touch main.tf outputs.tf variables.tf config.tf vim config.tf

provider "aws" {
  region = "eu-west-1"
}



terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.32"
    }
  }

  required_version = ">= 1.4.5"

}

terraform {

  backend "s3" {
    bucket         = "my-wyoc-s3-bucket-a"
    key            = "terraform/saga-infra.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "my-wyoc-state-lock-a"
  }
}

To explain about corelation between and config.tf file, the highlighted one are the AWS resources that was created by create_tf_backend.sh script.

backend "s3" {
bucket = "my-wyoc-s3-bucket-a"
key = "terraform/saga-infra.tfstate"
region = "eu-west-1"
dynamodb_table = "my-wyoc-state-lock-a"
}

Let's run the terraform commands to reconfigure the backend. Because I already run the terraform init command. Here I wanted to show the output how it looks like when you configure remote state pointing to s3 backend.

Below is the output once the terraform init -reconfigure command ran.

Output shown that --> "Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes."

[root@jenkin-slave-node provision_s3_bucket]# terraform init -reconfigure

Initializing the backend...

**Successfully configured the backend "s3"! **Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v5.82.2

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[root@jenkin-slave-node provision_s3_bucket]#

Let's run simple S3 bucket provisioning. Follow these steps.

Add the below code block to main.tf file

resource "aws_s3_bucket" "my_bucket" {
  bucket  = "my-unique-bucket-name-wyoc-08012025"
  tags    = {
    Name          = "MyS3Bucket"
    Environment    = "Production"
  }
}

Save it and run below commands in order.

terraform init terraform plan terraform apply -auto-approve
What we can see was that execution was successful.The key observations are

terraform state file created in the s3 bucket configured (my-wyoc-s3-bucket-a) .

the bucket we configure terraform to provision is created successfully which means terraform is working correctly with s3 and dynamodb remote backend.

Moreover, we can list and describe resources from terraform commands which retrive details from remote s3 backend. Run following commands to get the output.

terraform state list terraform state show aws_s3_bucket.my_bucket

terraform state show command gave the following output.

# aws_s3_bucket.my_bucket:
resource "aws_s3_bucket" "my_bucket" {
    arn                         = "arn:aws:s3:::my-unique-bucket-name-wyoc-08012025"
    bucket                      = "my-unique-bucket-name-wyoc-08012025"
    bucket_domain_name          = "my-unique-bucket-name-wyoc-08012025.s3.amazonaws.com"
    bucket_regional_domain_name = "my-unique-bucket-name-wyoc-08012025.s3.eu-west-1.amazonaws.com"
    force_destroy               = false
    hosted_zone_id              = "Z1BKCSDXD74EZPE"
    id                          = "my-unique-bucket-name-wyoc-08012025"
    object_lock_enabled         = false
    region                      = "eu-west-1"
    request_payer               = "BucketOwner"
    tags                        = {
        "Environment" = "Production"
        "Name"        = "MyS3Bucket"
    }
    tags_all                    = {
        "Environment" = "Production"
        "Name"        = "MyS3Bucket"
    }

    grant {
        id          = "b178b9a2113d356e6a8885a371b72aa6834022341f5abc23619128125eda218ad"
        permissions = [
            "FULL_CONTROL",
        ]
        type        = "CanonicalUser"
    }

    server_side_encryption_configuration {
        rule {
            bucket_key_enabled = false

            apply_server_side_encryption_by_default {
                sse_algorithm = "AES256"
            }
        }
    }

    versioning {
        enabled    = false
        mfa_delete = false
    }
}

In this blog, we have learned

create terrafrom remote backend using AWS S3 and AWS DynamoDB.
Configure it in terraform config.tf file.
Initialize terraform backend and execute pretty simple s3 bucket provisioning.

One bonus point is, there is a way to configure backend configurations taken out of .tf file which is a good practice. There are two ways that we can do.

Via TV_VAR bash variables and terraform init -backend-config option
Via config file and terraform init -backend-config option

Option One Steps

Add following code to config.tf file. We called them terraform backend partial configuration.

provider "aws" {
  region = "eu-west-1"
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.32"
    }
  }

  required_version = ">= 1.4.5"

}

terraform {

  backend "s3" {
  }

 }

Most possibly, you may set below code snippet in CI/CD pipeline which runs terraform code. For simplicity, I have added into simple bash script, set the execution permissions and execute the script.

Script in github

https://github.com/arambage0/cloud-infra-wyoc/blob/main/scripts/tf-backend/tf_init_config.sh

chmo u+x tf_init_config.sh
./tf_init_config.sh

#!/bin/bash

# TF_VAR variables that exposed your real configurations
export TF_VAR_s3_bucket="my-wyoc-s3-bucket-a"
export TF_VAR_s3_key="terraform/saga-infra.tfstate"
export TF_VAR_s3_region="eu-west-1"
export TF_VAR_dynamodb_table="my-wyoc-state-lock-a"

# Terraform init 
terraform init   -backend-config="bucket=${TF_VAR_s3_bucket}" -backend-config="key=${TF_VAR_s3_key}" -backend-config="region=${TF_VAR_s3_region}" -backend-config="dynamodb_table=${TF_VAR_dynamodb_table}"

This steps set the TF backend.

Next step to run terraform plan command.
Run the terraform apply command.

Option Two Steps

Add the following code to the configuration file. Please note add your own specific configs that you have created.

vim tfstate_config.properties

# state.config
bucket = "my-wyoc-s3-bucket-a"
key    = "terraform/saga-infra.tfstate"
region = "eu-west-1"
dynamodb_table= "my-wyoc-state-lock-a"

Run the terraform command below.

terraform init -backend-config="./tfstate_config.properties"

Next step to run terraform plan command.
Run the terraform apply command.

Hope this will be helpful for you. Happy learning. Have a nice day.