DEV Community: Vinh Le

Multiple SSH keys for different Git accounts

Vinh Le — Sun, 11 Sep 2022 09:54:35 +0000

🗒️ Content

Before SSH Keys Setup
Connecting to GitLab with SSH
Connecting to GitHub with SSH
A Better Keys Organization

1. Before SSH Keys Setup

Checking for existing SSH keys file

$ ls ~/.ssh
id_rsa  id_rsa.pub  known_hosts

Confirm active public SSH keys (*.pub) in ssh-agent

$ ssh-add -L

Remove all old keys

$ rm -r ~/.ssh

2. Connecting to GitLab with SSH

Step 1: Generate an SSH key pair

$ ssh-keygen -t ed25519 -C "<your_email_01>@gmail.com"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/<user>/.ssh/id_ed25519): /home/<user>/.ssh/id_ed25519_gitlab
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

Step 2: Add an SSH key to your GitLab account

$ cat ~/.ssh/id_ed25519_gitlab.pub
# or
$ xclip -sel clip < ~/.ssh/id_ed25519_gitlab.pub

Copy the contents of your public key file by manually or a script. Then coming to GitLab > Preferences > SSH Keys

Step 3: Verify that you can connect

$ ssh -T git@gitlab.com
...
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Welcome to GitLab, <user_name>!

3. Connecting to GitHub with SSH

Step 1: Generate an SSH key pair

$ ssh-keygen -t ed25519 -C "<your_email_02>@gmail.com"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/<user>/.ssh/id_ed25519): /home/<user>/.ssh/id_ed25519_github
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

Step 2: Add an SSH key to your GitLab account

$ cat ~/.ssh/id_ed25519_github.pub
# or
$ xclip -sel clip < ~/.ssh/id_ed25519_github.pub

Copy the contents of your public key file by manually or a script. Then coming to GitHub > Settings > SSH and GPG keys

Step 3: Verify that you can connect

$ ssh -T git@github.com
...
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Hi <user_name>! You've successfully authenticated, but GitHub does not provide shell access.

4. A Better Keys Organization

For more information on these settings, see the man ssh_config

$ cd ~/.ssh/
$ touch config
$ vi config
# GitLab
Host gitlab.com
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/id_ed25519_gitlab

# GitHub
Host github.com
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/id_ed25519_github

AWS CloudFormation - Separately Manage Network and Server Stacks by Cross Stacks

Vinh Le — Sun, 21 Aug 2022 16:33:41 +0000

🗒️ Content

AWS CloudFormation 101
One key pair f
Hands-on Lab: Separately Manage Network and Server Stacks by Cross Stacks

1. AWS CloudFormation 101

1.1. Infrastructure as Code (IaC)

Treating infrastructure in the same way as developers treat code is one of the most fundamental principles of DevOps.

Infrastructure as code (IaC) means provisioning, configuring and managing your infrastructure resources using code and templates.

1.2. AWS CloudFormation

AWS CloudFormation is a service speeding up cloud provisioning with infrastructure as code

AWS resources and infrastructure architecture are modeled and described in CloudFormation templates written in JSON or YAML (recommended) formatted text file. After that CloudFormation takes care of provisioning and configuring those resources for you.

The CloudFormation workflow for creating stacks as the below diagram

CloudFormation Workflow (source: https://aws.amazon.com/cloudformation/)

2. CloudFormation Templates Structure

Resource (mandatory): declare the AWS resources
Parameters (optional): input custom values to template each time creating/updating a stack, using with !Ref <parameter>
Mapping (optional): map a key to a specific value, using with !FindInMap [ <map_name>, <top_key>, <second_key> ]
Condition (optional): control the creation of resources or outputs based on a condition
Output (optional): return output values of stack
Cross-stack: separate templates and refer values from different stacks, using with Export from one stack and !ImportValue <export_name> from other stacks

3. Hands-on Lab: Separately Manage Network and Server Stacks by Cross Stacks

Assume that we are developing a simple web application with the following architecture

One VPC, one public subnet, one EC2 instance
One security group opening in port 80 (for HTTP from anywhere)

Separating our infrastructure into 2 templates may help us manage them easier.

3.1. LabNetworkStack

The content of lab_network_cfn.yml:

Description: This template manages network components

Parameters:
  ApplicationName:
    Description: An application name that is prefixed to resource names
    Type: String
    Default: lab
  VpcCidr:
    Description: The IP range (CIDR notation) for VPC in form x.x.x.x/16-28
    Type: String
    AllowedPattern: "^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\\/(1[6-9]|2[0-8]))$"
    Default: 10.192.0.0/16
  PublicSubnetCidr:
    Description: The IP range (CIDR notation) for public subnet in form x.x.x.x/16-28
    Type: String
    AllowedPattern: "^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\\/(1[6-9]|2[0-8]))$"
    Default: 10.192.10.0/24

Resources:
  Vpc:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCidr
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: !Sub '${ApplicationName}-vpc'
  PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref Vpc
      CidrBlock: !Ref PublicSubnetCidr
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Sub '${ApplicationName}-public-subnet'
  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Sub '${ApplicationName}-igw'
  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref Vpc
  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref Vpc
      Tags:
        - Key: Name
          Value: !Sub '${ApplicationName}-rtb'
  DefaultPublicRoute:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttachment
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway
  PublicSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnet

Outputs:
  VpcId:
    Description: The VPC ID of lab application
    Value: !Ref Vpc
    Export:
      Name: !Sub '${AWS::StackName}-VpcId'
  SubnetId:
    Description: The Subnet ID of lab application
    Value: !Ref PublicSubnet
    Export:
      Name: !Sub '${AWS::StackName}-PublicSubnetId'

The template can be deployed with AWS CLI

aws s3 cp lab_network_cfn.yml s3://<your_bucket>
aws cloudformation create-stack --stack-name LabNetworkStack --template-url https://<your_bucket>.s3.<your_region>.amazonaws.com/lab_network_cfn.yml

3.2. LabServerStack

The content of lab_server_cfn.yml:

Description: This template manages servers components

Parameters:
  ApplicationName:
    Description: An application name that is prefixed to resource names
    Type: String
    Default: lab
  Landscape:
    Type: String
    Default: develop
    AllowedValues:
    - develop
    - production
  NetworkStackName:
    Description: The name of Network CloudFormation stack
    Type: String
    Default: LabNetworkStack

Mappings:
  RegionMap:
    ap-southeast-1:
      AMI: ami-02ee763250491e04a
    ap-southeast-2:
      AMI: ami-0e040c48614ad1327

Conditions:
  IsProduction: !Equals [ !Ref Landscape, production]

Resources:
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Sub 'Security group for ${ApplicationName} instances'
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0
      VpcId: !ImportValue
        'Fn::Sub': '${NetworkStackName}-VpcId'
      Tags:
        - Key: Name
          Value: !Sub '${ApplicationName}-srg'
  Ec2Instance: 
    Type: AWS::EC2::Instance
    Properties: 
      ImageId: !FindInMap [ RegionMap, !Ref 'AWS::Region', AMI ]
      InstanceType: !If [ IsProduction, t2.small, t2.micro ] 
      KeyName: your_key_name
      SubnetId: !ImportValue
        'Fn::Sub': '${NetworkStackName}-PublicSubnetId'
      SecurityGroupIds:
      - !Ref SecurityGroup
      BlockDeviceMappings: 
      - DeviceName: "/dev/sda1"
        Ebs:
          VolumeSize: 12
          VolumeType: gp2
          Encrypted: true
      UserData:
        "Fn::Base64":
          !Sub |
            #!/bin/bash
            apt update
            apt install apache2 -y
      Tags:
        - Key: Name
          Value: !Sub '${ApplicationName}-ec2'

Outputs:
  InstanceIp:
    Description: The IP address of instance
    Value: !GetAtt Ec2Instance.PublicIp

The template can be deployed with AWS CLI

aws s3 cp lab_server_cfn.yml s3://<your_bucket>
aws cloudformation create-stack --stack-name LabServerStack --template-url https://<your_bucket>.s3.<your_region>.amazonaws.com/lab_server_cfn.yml

Confirmation

Confirm the status of stacks are all CREATE_COMPLETE
Confirm the default Ubuntu Apache web page http://your_server_ip

3.3. Clean resources

The stacks can be deleted by consoles or AWS CLI in the following order

aws cloudformation delete-stack --stack-name LabServerStack
aws cloudformation delete-stack --stack-name LabNetworkStack

AWS CDK in Python - Schedule Lambda Functions

Vinh Le — Wed, 29 Jun 2022 05:06:32 +0000

🗒️ Content

AWS CDK 101
CDK Installation
Hands-on Lab: Schedule Lambda Functions with AWS CDK in Python

1. AWS CDK 101

1.1. Infrastructure as Code (IaC)

Treating infrastructure in the same way as developers treat code is one of the most fundamental principles of DevOps.

Infrastructure as code (IaC) means provisioning, configuring and managing your infrastructure resources using code and templates.

1.2. AWS CDK

CloudFormation templates are used to define our application infrastructure. However, thanks to AWS CDK, the full power of a programming language can be leveraged to describe our infrastructure.

AWS CDK is a framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation.

1.3. AWS CDK Basic Concepts

Supported languages: TypeScript, JavaScript, Python, Java, C#/.Net and Go
AWS CDK App: an application written in a supported language
AWS CDK Stack: the unit of deployment in the AWS CDK, equivalent to AWS CloudFormation
AWS CDK Constructs: the basic building blocks, represents a "cloud component"

CDK App Structure (source: https://docs.aws.amazon.com/cdk/v2/guide/images/AppStacks.png)

2. CDK Installation

Install Node.js

AWS CDK uses Node.js as its back-end regardless of the programming language you use, so nodejs is required.

> sudo apt install nodejs
> npm install -g aws-cdk
> cdk --version

Install virtualenv package

If your programming language is Python, using a virtual environment is recommended.

> pip install virtualenv

Configure your AWS credentials

AWS credentials also need to be configured in your local development environment, so that CDK CLI can communicate with AWS.

> aws configure

3. Hands-on Lab: Schedule Lambda Functions with AWS CDK in Python

This tutorial shows how to use AWS CDK to build our first application, which is a AWS Lambda function on a schedule ⏰.

Step 1: Initialize the app using the cdk init command

> mkdir ScheduleLambda
> cd ScheduleLambda
> cdk init app --language python
...
Executing Creating virtualenv...
✅ All done!

Note that cdk init cannot be run in a non-empty directory!

Step 2: Activate the app's Python virtual environment and install dependencies

> source .venv/bin/activate
> python -m pip install -r requirements.txt

Only in the first time, Bootstrapping is required to provision resources that AWS CDK needs to perform the deployment, such as S3 bucket and IAM roles.

> cdk bootstrap aws://<your_account_id>/<your_region>

Step 3: Add file lambda/lambda-handler.py with the below content

def handler(event, context):
    print("This is the Schedule Lambda Function's log")
    print(event['time'])

We can see that cdk prepares a virtual environment for us.
The structure of our application is as below.

├── app.py
├── cdk.json
├── lambda
│   └── lambda-handler.py
├── README.md
├── requirements-dev.txt
├── requirements.txt
├── schedule_lambda
│   ├── __init__.py
│   └── schedule_lambda_stack.py
├── source.bat
└── tests

Step 4: Update the content of schedule_lambda_stack.py as the following

from aws_cdk import (
    Duration, Stack,
    aws_lambda,
    aws_events,
    aws_events_targets
)
from constructs import Construct

class ScheduleLambdaStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # The code that defines your stack goes here
        schedule_lambda = aws_lambda.Function(self,
            'ScheduleLambda',
            handler='lambda-handler.handler',
            runtime=aws_lambda.Runtime.PYTHON_3_9,
            code=aws_lambda.Code.from_asset('lambda'),
            timeout=Duration.seconds(300)
        )
        schedule_lambda_rule = aws_events.Rule(self,
            "ScheduleLambdaRule",
            schedule=aws_events.Schedule.rate(Duration.minutes(1))
        )
        schedule_lambda_rule.add_target(aws_events_targets.LambdaFunction(schedule_lambda))

Step 5: Synthesize an AWS CloudFormation template

> cdk synth

Step 6: Deploying the stack

> cdk deploy
...
Do you wish to deploy these changes (y/n)? y
...
 ✅  ScheduleLambdaStack

Step 7: Confirm deployment in CloudFormation console
Final step: Clean your resource

> cdk destroy ScheduleLambdaStack
Are you sure you want to delete: ScheduleLambdaStack (y/n)? y
...
 ✅  ScheduleLambdaStack: destroyed

Two Approaches to AWS Solution Design

Vinh Le — Wed, 29 Jun 2022 03:59:20 +0000

Source: https://aws.amazon.com/well-architected-tool/

After several years working with AWS services in multiple projects with different roles from developer, operator to architect, when I design or evaluate a solution, I am using two approaches: AWS in DevOps model and Well-Architected Framework. They are very useful for both real work and all AWS certified examinations (specially for AWS Solutions Architect/DevOps Professional).

1. AWS in DevOps model

Development and operation processes in each organization are different, but they often contain plan, design, development, deployment and operation stages. Leveraging AWS services in our processes should be considered seriously to maximize benefits.

Plan stage: In this stage, we often collect and evaluate requirements, then select appropriate AWS services. It requires basic knowledge and experience on many AWS services across domains from network, computation, storage, management, security to developer tools. If the target is a production environment, PaaS should be preferred over IaaS, such as Aurora over MySQL installed on EC2. However, do not limit yourself and avoid vendor lock-in, sometimes combination with other SaaS (e.g., Slack or open source solution (e.g., Kubernetes can create great solutions.

Design stage: In this stage, we often configure services and decide how they communicate together. Multi-tier, serverless and micro-services are very common architectures which an architect should be familiar with. There are three common types of design task: new solutions, existing solutions improvement and migration. The first one, design for new solutions, we start from zero, but we may enjoy flexibility in decisions. We should determine requirements from multiple views including security & control, scalability & availability & reliability, and performance & cost. The second one, continuous improvement for existing solutions, we often troubleshoot and determine improvement strategy based on Well-Architected Framework. The last one, migration plan, we may start with a legacy application on-premises, and such tasks are sometime not easy due to a lot of unmaintained resources. We need to measure current workloads, select migration tools & services, and implement 6R’s strategies (rehosting, replatforming, refactoring / re-architecting, retire and retain)

Development stage: In this stage, functional requirements and designs are translated into application code and configuration. To develop and debug cloud-based applications, we need to have knowledge at application level and follow best practices of each service to leverage the power of cloud. Besides, we will find that AWS CLI and SDK (e.g. boto3) are very useful for our work.

Deployment stage: Deployment/release stage should be performed quickly and has minimum downtime and human interaction. AWS provides us a set of flexible services to realize DevOps best practices such as Infrastructure as Code (IaC), Continuous Integration (CI) and Continuous Deployment - Delivery (CD). Depending on tasks, we can use other tools (e.g. Jenkins to support our complex scenarios. To minimize downtime, we also should to be familiar with different deployment strategies, such as rolling updates, blue-green and canary release.

Operation stage: In this stage, we try to reduce operational complexity, optimize cost, and increase monitoring insights. Specially in a large organization, we should determine scalable operations with cross-account authentication & authorization & networking & monitoring & encryption and logging. Suppose that we have thousands resources for each business project, and hundreds projects across hundreds accounts, we may be unaware of unnecessary resources and waste a lot of money. We should define a cost-effective pricing model (e.g. on-demand, reservation, spot) based on requirements, cost reduction suggestion, budget notification and bill analysis to ensure cost optimization. Besides, in large application a monitoring system with a remediate and health recovery strategy should be designed carefully.

2. Well-Architected Framework

AWS published Well-Architected Framework, that describes key concepts, design principles, and architectural best practices for designing and running workloads in the cloud. It is a valuable resource, specially for an AWS solution architect, in order to design architecture, evaluate workloads and identify risk issues. Design principles are summarized as below.

Operational Excellence Pillar: the ability to support development and run workloads effectively, gain insight into their operations, and to continuously improve supporting processes and procedures to deliver business value.

There are 5 design principles for operational excellence in the cloud.

Perform operations as code: implement infrastructure | middleware | application configuration | operational procedures as code, and automatic executions by event triggers.

Make frequent, small, reversible changes: frequently release in small (CI/CD) and reversible changes (blue-green, canary).

Refine operations procedures frequently: regularly review, validate and evolve procedures appropriately as workload evaluation.

Anticipate failure: simulate - test failure scenarios (e.g. Chaos Monkey), validate their impacts, and prepare responses (notification, diagnosis and auto-recovery).

Learn from all operational failures: learn from all operational events and share knowledge across teams.

Security Pillar: the ability to protect data, systems and assets while taking advantage of cloud technologies to improve your security.

There are 7 design principles for security in the cloud.

Implement a strong identity foundation: implement least privilege, duty separation (authorization) and centralized authentication management; eliminate long-term static credentials.

Enable traceability: monitor, alert, and audit actions and changes in real time. Integrate log and metric to automatically analyze and take actions.

Apply security at all layers: deep security control from network, computing, storage, OS, code to application.

Automate security best practices

Protect data in transit and at rest: classify data into sensitivity levels and use mechanisms, such as encryption, tokenization, and access control.

Keep people away from data: reduce or eliminate the need for direct access or manual processing of data.

Prepare for security events: prepare incident management, investigation processes, and mitigation-recovery actions.

Reliability Pillar: the ability of a workload to perform its intended function correctly and consistently when it’s expected to. Furthermore, the ability to operate and test the workload through its total lifecycle.

There are 5 design principles for reliability in the cloud.

Automatically recover from failure: perform automatic notification, tracking of failures, and recovery processes by monitor KPIs measuring business value (not technical aspects) of services.

Test recovery procedures: simulate different failures, then test and validate recovery strategies.

Scale horizontally to increase aggregate workload availability: distribute requests across multiple and smaller resources to reduce the impact of a single failure.

Stop guessing capacity: automatically add or remove resources to maintain the optimal level to satisfy demand without over- or under-provisioning.

Manage change in automation: manage changes with version control, and apply changes by using automation mechanism.

Performance Efficiency Pillar: the ability to use resources efficiently to meet system requirements, and to maintain that efficiency as demand changes and technologies evolve.

There are 5 design principles for performance efficiency in the cloud.

Democratize advanced technologies: delegate complex tasks to your cloud vendor, and consider consuming the technology as a service.

Go global in minutes

Use serverless architectures: removes the operational burden of managing physical servers, and can lower transactional costs

Experiment more often

Consider mechanical sympathy: use the technology approach that aligns best with your workload goals

Cost Optimization Pillar: the ability to run systems to deliver business value at the lowest price point.

There are ﬁve design principles for cost optimization in the cloud.

Implement cloud financial management

Adopt a consumption model: pay only for the computing resources that you require and increase or decrease usage depending on business requirements, not by using elaborate forecasting

Measure overall efficiency: measure the business output of the workload and the costs associated with delivering it

Stop spending money on undifferentiated heavy lifting

Analyze and attribute expenditure: identify the usage and cost of systems, which allows transparent attribution of IT costs to individual workload owner.

DEV Community: Vinh Le

Multiple SSH keys for different Git accounts

🗒️ Content

1. Before SSH Keys Setup

2. Connecting to GitLab with SSH

3. Connecting to GitHub with SSH

4. A Better Keys Organization

AWS CloudFormation - Separately Manage Network and Server Stacks by Cross Stacks

🗒️ Content

1. AWS CloudFormation 101

1.1. Infrastructure as Code (IaC)

1.2. AWS CloudFormation

2. CloudFormation Templates Structure

3. Hands-on Lab: Separately Manage Network and Server Stacks by Cross Stacks

3.1. LabNetworkStack

3.2. LabServerStack

3.3. Clean resources

AWS CDK in Python - Schedule Lambda Functions

🗒️ Content

1. AWS CDK 101

1.1. Infrastructure as Code (IaC)

1.2. AWS CDK

1.3. AWS CDK Basic Concepts

2. CDK Installation

3. Hands-on Lab: Schedule Lambda Functions with AWS CDK in Python

Two Approaches to AWS Solution Design

1. AWS in DevOps model

2. Well-Architected Framework

Further Reading