DEV Community: Shrihari Haridass

Mastering Infrastructure with CloudPosse Atmos and Terraform

Shrihari Haridass — Fri, 09 Aug 2024 09:05:52 +0000

Introduction

In today's blog, we’ll dive deep into learning and hands-on practice with CloudPosse Atmos. I'm not sure how familiar people are with this technology, but I’m confident that after reading this and visiting Atmos' official documentation, you’ll be intrigued by it. In this blog, I've included some content from the official sources, but I've also added my own insights and projects. I hope you enjoy this technical blog, which I’m presenting in a slightly different format today. It might be a bit longer compared to my previous posts, but I only publish blogs about topics I have hands-on experience with. I hope you find it valuable—let’s get started!

-1-. What is SweetOps..?

SweetOps is a methodology for building modern, secure infrastructure on top of Amazon Web Services (AWS). It provides a toolset, library of reusable Infrastructure as Code (IaC), and opinionated patterns to help you bootstrap robust cloud native architectures. Built in an Open Source first fashion by Cloud Posse, it is utilized by many high performing startups to ensure their cloud infrastructure is an advantage instead of a liability. In short, SweetOps makes working in the DevOps world Sweet!

Who is this for?

SweetOps is for DevOps or platform engineering teams that want an opinionated way to build software platforms in the cloud. If the below sounds like you, then SweetOps is what you're looking for:

-1-. You're on AWS
-2-. You're using Terraform as your IaC tool
-3-. Your platform needs to be secure and potentially requires passing compliance audits (PCI, SOC2, HIPAA, HITRUST, FedRAMP, etc.)
-4-. You don't want to reinvent the wheel

With SweetOps you can implement the following complex architectural patterns with ease:

-1-. An AWS multi-account Landing Zone built on strong, well-established principles including Separation of Concerns and Principle of Least Privilege (POLP).
-2-. Multi-region, globally available application environments with disaster recovery capabilities.
-3-. Foundational AWS-focused security practices that make complex compliance audits a breeze.
-4-. Microservice architectures that are ready for massive scale running on Docker and Kubernetes.
-5-. Reusable service catalogs and components to promote reuse across an organization and accelerate adoption

-2-. What is Atmos and its uses

Cloudposse Atmos is an open-source tool for managing and orchestrating infrastructure and applications on cloud providers like AWS, GCP, and Azure. It's designed to simplify and automate the process of provisioning, deploying, and managing infrastructure and applications in a cloud-agnostic way.

Atmos provides a declarative configuration language that allows you to define your infrastructure and applications in a human-readable format, and then automates the provisioning and deployment process using Terraform and other tools.

Some key features of Cloudposse Atmos include:

Declarative configuration language
Cloud-agnostic architecture
Support for multiple cloud providers (AWS, GCP, Azure)
Integration with Terraform and other tools
Automated provisioning and deployment
Support for microservices and containerized applications

Cloudposse Atmos is often used by DevOps teams and cloud engineers to streamline their infrastructure and application management workflows, and to promote infrastructure-as-code (IaC) practices.

-3-. Launch a t2.micro EC2 instance, update the machine, and follow the commands below to install Atmos

# Update and install utilities

sudo apt-get update && sudo apt-get install -y apt-utils curl

# Add the Cloud Posse repository

curl -1sLf 'https://dl.cloudsmith.io/public/cloudposse/packages/cfg/setup/bash.deb.sh' | sudo bash

# Update package lists

sudo apt-get update

# Install atmos version 1.84.0

sudo apt-get install -y atmos=1.84.0-1

# Verify installation

atmos version

Also, make sure you run the command to clone the CloudPosse repo in a single line. We have successfully installed Atmos on our system.

-4-. After that, we need to create components, stacks, and some files. Make sure you follow the folder structure below, as it must be organized this way. Before creating the structure, create a main folder, and then set up the following structure within it for better understanding.

-5-. Now, we will start writing the Terraform files within that folder structure. Let’s get started!

-6-. First, add the following code to the atmos.yaml file. We need to configure atmos.yaml for our project.

base_path: "./"

components:
  terraform:
    base_path: "components/terraform"
    apply_auto_approve: false
    deploy_run_init: true
    init_run_reconfigure: true
    auto_generate_backend_file: false

stacks:
  base_path: "stacks"
  included_paths:
    - "deploy/**/*"
  excluded_paths:
    - "**/_defaults.yaml"
  name_pattern: "{stage}"

logs:
  file: "/dev/stderr"
  level: Info

To configure Atmos for your project, we’ll create a file called atmos.yaml to specify where Atmos can find the Terraform components and Atmos stacks. This file allows you to configure almost everything in Atmos.

-7-. After that, navigate to the myappcomponent to write the Terraform code/module. First, create a variables.tf file and add the following code to it.

you’ll see that everything is just plain Terraform (HCL) with nothing specific to Atmos. That’s intentional: we want to demonstrate that Atmos works seamlessly with plain Terraform. Atmos introduces conventions around how you use Terraform with its framework, which will become more evident in the subsequent lessons.

variable.tf

variable "stage" {
  description = "Stage where it will be deployed"
  type        = string
}

variable "location" {
  description = "Location for which the weather."
  type        = string
  default     = "Los Angeles"
}

variable "options" {
  description = "Options to customize the output."
  type        = string
  default     = "0T"
}

variable "format" {
  description = "Format of the output."
  type        = string
  default     = "v2"
}

variable "lang" {
  description = "Language in which the weather is displayed."
  type        = string
  default     = "en"
}

variable "units" {
  description = "Units in which the weather is displayed."
  type        = string
  default     = "m"
}

To make the best use of Atmos, ensure your root modules are highly reusable by accepting parameters, allowing them to be deployed multiple times without conflicts. This also usually means provisioning resources with unique names.

main.tf

The main.tf file is where the main implementation of your component resides. This is where you define all the business logic for what you're trying to achieve-the core functionality of your root module. If this file becomes too large or complex, you can break it into multiple files in a way that makes sense. However, sometimes that is also a red flag, indicating that the component is trying to do too much and should be broken down into smaller components.
In this example, we define a local variable that creates a URL using the variable inputs we receive. We also set up a data source to perform an HTTP request to that endpoint and retrieve the current weather. Additionally, we write this output to a file to demonstrate a stateful resource

locals {
  url = format("https://wttr.in/%v?%v&format=%v&lang=%v&u=%v",
    urlencode(var.location),
    urlencode(var.options),
    urlencode(var.format),
    urlencode(var.lang),
    urlencode(var.units),
  )
}

data "http" "weather" {
  url = local.url
  request_headers = {
    User-Agent = "curl"
  }
}

# Now write this to a file (as an example of a resource)
resource "local_file" "cache" {
  filename = "cache.${var.stage}.txt"
  content  = data.http.weather.response_body
}

version.tf

The versions.tf file is where provider pinning is typically defined. Provider pinning increases the stability of your components and ensures consistency between deployments in multiple environments.

terraform {
  required_version = ">= 1.0.0"    #you can use latest version as well

  required_providers {}
}

output.tf

The outputs.tf file is where, by convention in Terraform, you define any outputs you want to expose from your root module. Outputs are crucial for passing state between root modules and can be used with remote state or the Atmos function to retrieve the state of other components. In object-oriented terms, think of outputs as the 'public' attributes of the module, intended to be accessed by other modules. This convention helps maintain clarity and organization within your Terraform configurations.

output "weather" {
  value = data.http.weather.response_body
}

output "url" {
  value = local.url
}

output "stage" {
  value       = var.stage
  description = "Stage where it was deployed"
}

output "location" {
  value       = var.location
  description = "Location of the weather report."
}

output "lang" {
  value       = var.lang
  description = "Language which the weather is displayed."
}

output "units" {
  value       = var.units
  description = "Units the weather is displayed."
}

The sequence of defining these files does not matter

-8-. Now we are configuring our stack for deployment

components:
  terraform:
    <name-of-component>:

To specify which component to use, set the metadata.component property to the path of the component's directory, relative to the components.base_path defined in the atmos.yaml. In our case, the components.base_path is components/terraform, so you can simply specify weatheras the path

components:
  terraform:
    station:
      metadata:
        component: weather

-9-. Next, go to /stacks/catalog and create a file named station.yaml.

components:
  terraform:
    station:
      metadata:
        component: weather
      vars:
        location: Los Angeles
        lang: en
        format: ''
        options: '0'
        units: m

-10-. Next, we’ll define the environment-specific configurations for our Terraform root module. We’ll create a separate file for each environment and stage. In our case, we have three environments: dev, staging, and prod

When Atmos processes this stack configuration, it will first import and deep-merge all the variables from the imported files, then apply the inline configuration. While the order of keys in a YAML map doesn’t affect behavior, lists are strictly ordered, so the sequence of importsis important

-11-. Define the configuration for the devenvironment.

In the devstack configuration, Atmos first processes the importsin the order defined. It then applies the global varsspecified in the top-level section. Include only those varsin the globals that are applicable to every single component in the stack. For variables that aren't universally applicable, define them on a per-component basis.

For example, by setting var.stage to devat a global level, we assume that every component in this stack will have a stage variable.

Finally, in the component-specific configuration for the station, set the fine-tuned parameters for this environment. Everything else will be inherited from its baseline configuration. There are no strict rules about where to place configurations; organize them in a way that makes logical sense for your infrastructure’s data model.

To accomplish this, go to /stacks/deploy and create a file named dev.yaml.

vars:
  stage: dev

import:
  - catalog/station

components:
  terraform:
    myapp:
      vars:
        location: India
        lang: en

-12-. In this demo, we will focus on the devenvironment only. If you want to create configurations for stageand prod, you can refer to the official Atmos documentation for guidance.

-13-. After completing all the steps, the final file and folder structure should look like this:

-14-. Now that we have written all the modules in Terraform, we still need to install Terraform itself. Let’s proceed with the installation.

-1-. sudo apt-get update && sudo apt-get install -y gnupg software-properties-common

-2-. wget -O- https://apt.releases.hashicorp.com/gpg | \
gpg --dearmor | \
sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg > /dev/null

-3-. gpg --no-default-keyring \
--keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg \
--fingerprint

-4-. echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
sudo tee /etc/apt/sources.list.d/hashicorp.list

-5-. sudo apt update

-6-. sudo apt-get install terraform

-7-. terraform version

-15-. Now, let’s deploy the module using the command below. Make sure to run this command from the root folder where the atmos.yaml file is located

atmos terraform apply myapp -s dev

-16-. Here, you can view either a graph or a report for the country India. Additionally, if you need to run the initor plancommands, you can execute them as well.

-17-. Atmos can change how you think about the Terraform code you write to build your infrastructure.

When you design cloud architectures with Atmos, you will first break them apart into pieces called components. Then, you will implement Terraform "root modules" for each of those components. Finally, compose your components in any way you like using stacks, without the need to write any code or messy templates for code generation.

Conclusion

In this blog, we’ve walked through the process of setting up and configuring Atmos with Terraform. We started by creating the necessary directory structure and defining the atmos.yaml configuration file. We then wrote the Terraform code for our components and environment-specific configurations.

We covered the steps to install Terraform and deploy the module, ensuring that everything was executed from the correct directory. By the end of this guide, you should have a solid understanding of how Atmos integrates with Terraform and how to manage infrastructure using this framework.

Feel free to explore Atmos further by consulting the official documentation for additional environments like stagingand production. With this setup, you’re well-equipped to manage and scale your infrastructure effectively.

Thank you for following along, and I hope you found this tutorial helpful!

Reference:

How to Monitor your AWS EC2/Workspace with Datadog

Shrihari Haridass — Mon, 08 Jul 2024 10:30:19 +0000

-1- Log in to your AWS and Datadog accounts. In this example, I will configure AWS Workspace. If you have an EC2 instance, you can follow these steps too. My OS is Ubuntu.

-2-. After deploying an EC2 instance or logging into an AWS Workspace, proceed to update the machine.

-3-. Next, navigate to Datadog → Integrations → Agent → Ubuntu and install the Datadog agent on your host machine. This agent will send metrics to Datadog and does not integrate directly with any AWS services.

-4-. Then click on 'Select API Key,' create a new key, and give it a name. Below that, you will see a command to install the Datadog agent on your system. Copy that command and run it on your server.



DD_API_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX DD_SITE="datadoghq.com"  bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

-5-. The command will configure and install the Datadog agent on your machine.



sudo usermod -a -G docker dd-agent
systemctl status datadog-agent
datadog-agent version
hostname

You can run the above command. Since I have Docker installed on my system, I've added the Datadog agent to the Docker group to monitor Docker. After running the command, check the service status, Datadog version, and finally, the hostname. If you're using an EC2 instance, it will display the instance ID; for a Workspace, it will show the Workspace ID.

-6-. After installing the agent, navigate to your Datadog dashboard, go to 'Infrastructure,' and search for your instance ID or workspace ID. You will now see your host machine on the default dashboard.

-7-. If you notice, initially, you'll see only a few metrics. However, you can explore other options such as 'Host Info,' 'Containers,' 'Processes,' 'Network,' 'Logs,' and more. By default, you can view host info and metrics on the dashboard. If you want to see 'Processes' and 'Logs,' you'll need to enable them in the Datadog configuration file.

-8-. To enable viewing 'Processes' in the Datadog dashboard, you'll need to edit the configuration file.



vi /etc/datadog-agent/datadog.yaml

Add the following line to your configuration file to enable 'Processes' monitoring.



process_config:
  process_collection:
    enabled: true

-9-. After adding the line to the configuration file, restart the Datadog service. Then, navigate to Datadog, click on 'Processes,' and you'll now see the dashboard displaying Processes, PID, total CPU, and RSS memory.



sudo systemctl restart datadog-agent

-10-. Now that Processes are visible, you can expand the dashboard by clicking 'Open in Live Process' on the right-hand side for a larger view. Next, let's configure 'Logs.'

-11-. To configure 'Logs,' uncomment the line 'Logs: true' in the Datadog configuration file.



vi /etc/datadog-agent/datadog.yaml

-12-. Create a new directory for system logs:



sudo mkdir /etc/datadog-agent/conf.d/system_logs.d

-13-. Create the conf.yaml file in the new directory:



sudo vi /etc/datadog-agent/conf.d/system_logs.d/conf.yaml

-14-. Add the log collection configuration to the conf.yaml file:



logs:
  - type: file
    path: /var/log/syslog
    service: syslog
    source: syslog
    sourcecategory: system

  - type: file
    path: /var/log/auth.log
    service: auth
    source: auth
    sourcecategory: system

  - type: file
    path: /var/log/kern.log
    service: kernel
    source: kernel
    sourcecategory: system

  - type: file
    path: /var/log/messages
    service: messages
    source: messages
    sourcecategory: system

  # Add other log files as needed

-15-. After making the changes, save and exit the file. Typically, you can use 'wq!' to save and exit, but 'x' also works.

-16-. Ensure that the Datadog Agent user has the appropriate permissions to read the log files. Afterward, restart the agent.



sudo usermod -a -G adm dd-agent
sudo systemctl restart datadog-agent

-17-. Now go to Datadog again and go to "Logs" tab

-18-. Finally, with all these different metrics, create a new dashboard and consolidate all windows into one.

-19-. Of course, if you want to monitor additional services or metrics, feel free to explore the official Datadog website and refer to the documentation.

-20. In conclusion, this guide has demonstrated how to effectively monitor your AWS EC2 or Workspace using Datadog. We covered installing the Datadog agent, configuring metrics like Processes and Logs, and creating a unified dashboard. It's important to note that Datadog is a paid tool, so if you're practicing as a single user or student, be mindful of potential costs. However, Datadog offers a 14-day trial period, allowing you to explore its features for free during this time. Remember, depending on your specific use case, you can further explore Datadog to unlock additional metrics and services tailored to your needs. For more detailed options, visit the official Datadog website and consult their documentation.

Importance of Security Groups (SGs) and Network Access Control Lists (NACLs) in AWS

Shrihari Haridass — Thu, 25 Apr 2024 09:41:33 +0000

Note: If I'm mistaken or if you disagree with any point, please feel free to share your thoughts. My aim is to highlight the importance of these concepts for newcomers and individuals starting their journey in the cloud.

-1-. So let's assume you're going to the office. When you enter the office premises, firstly, the security guard identifies your identity and allows you to proceed. Then, as you enter the building, another guard verifies your identity before you swipe your card to gain entry. Of course, swiping your card marks your attendance, but it also verifies that you are indeed the right person to enter the office.

-2-. Now, you are part of the cybersecurity analyst team, and you have a lot of responsibilities. For this, you have a separate team to monitor security. When you enter the office, everyone can access it, but the security room is accessible only to those who have the authority. This adds another level of security. Your card includes these additional steps, so when you enter that room, it will check if you are an authorized person to access the room, ensuring that not everyone can access this area.

-3-. Now, these last two security steps are most important and play a very crucial role in the organization, right?

-4-. Now, the point I'm making is, consider this: You have an application inside your VPC, which consists of various components like IGW, subnets, ELB, route tables, SG, NACL, NAT gateway, etc. Now, let's relate all of these to the story I mentioned earlier. What I'm trying to convey is that, just like in the story where multiple security checkpoints were in place before reaching a secure area, in your VPC environment, there are similar security measures. The last two security points I mentioned - subnet-level security and instance-level security - act as crucial checkpoints. These ensure that user requests are securely directed to your application or server after passing through these layers of protection.

-5-. Of course, before reaching the subnet and instance level security, there are several other security checks you can implement. However, in today's blog, I'm focusing more on Security Groups, which operate on the instance level, and Network Access Control Lists (NACLs), which work on subnet-level security. I'll delve into how you can set up these security measures in your organization or application layer.

-6-. In AWS, security is a mutual responsibility shared between AWS and the user. AWS follows the principle of denying all inbound traffic by default when launching instances. If you want to allow inbound traffic, you must specifically allow access to particular ports. However, opening ports to all traffic can pose security risks. It's essential to restrict access to specific IP addresses or employ other secure methods to ensure a safer environment.

-7-. As a DevOps engineer or AWS admin, managing Security Groups (SGs) for all instances individually can be time-consuming. However, Network Access Control Lists (NACLs), which operate at the subnet level, offer a solution. By defining port and traffic rules at the subnet level, changes automatically apply to all instances within that subnet. This not only reduces administrative tasks but also enhances security by providing consistent rules across multiple instances.

-8-. In this blog, we'll walk through the process of creating a Virtual Private Cloud (VPC) on AWS. Within this VPC, we'll set up an EC2 instance and deploy a Python application. Throughout the tutorial, we'll explore the configuration of Security Groups (SGs) and Network Access Control Lists (NACLs) to better understand their roles in securing our infrastructure.

-9-. Let's begin by logging into your AWS account and navigating to the 'VPC' service. "click on create VPC"

-10- Next, select the 'VPC and more' option, as we're currently focusing on creating a VPC. Provide a name for the VPC, specify the CIDR block, choose the Availability Zone (AZ), select subnets including private subnets, and set up a VPC endpoint as an S3 gateway. Finally, click on 'Create VPC' to complete the process.

-11-. You'll also have a visual representation of your VPC, subnets, route tables, and network connections, which helps you understand how traffic flows within the VPC.

-12-. After creating the VPC, navigate to the EC2 service and launch a t2.micro instance with default settings. Simply choose the VPC you created and any public subnet, then proceed to launch the instance.

-13-. After launching the instance, wait for it to start. Once it's running, log in to your server. Since I am using an Ubuntu image, update your machine to ensure you have the latest version of Python installed.

sudo apt-get update

-14-. Run the following command to start a simple Python server:
python3 -m http.server 8080

-15-. If you copy the public IP and try to access the application using port 8080, you may find it inaccessible. As I mentioned earlier, AWS implements mutual security measures, so by default, all traffic is denied when you launch a server. To address this, navigate to the "Security" tab and click on your "SG-Name". You'll notice that by default, only port 22 is allowed for server login.

-16-. To access the application, click on the Security Group (SG) associated with your instance. Then, select "Edit Inbound Rules" and allow port 8080 for all traffic. Keep in mind that opening all traffic is not recommended for production environments; instead, you can choose to allow traffic only from specific IP addresses. However, for study purposes, we're opening all traffic.

-17-. After saving the rule, copy the public IP address and access it on port 8080. You should now be able to see your application.

-18-. Up until now, you've witnessed the importance of Security Groups at the instance level in providing security. Of course, we opened all traffic for study purposes, but I must reiterate that opening all traffic is not recommended. You should allow access only from specific IP addresses to enhance security. Keep this in mind as you proceed.

-19-. In an organization with numerous instances running across different environments like dev, prod, and uat, managing each instance's security group policies individually becomes impractical. Developers may also launch instances for their purposes and define their own rules, potentially conflicting with organizational security policies.

To address this challenge, defining rules at the subnet level becomes crucial. By setting up Network Access Control Lists (NACLs) at the subnet level, you can control the traffic entering and leaving that subnet. This way, regardless of how many instances are launched within the subnet, they all adhere to the defined rules, ensuring consistent and manageable security across your infrastructure.

-20-. let's navigate back to the VPC console and select "Network ACLs". Then, open the VPC that you created earlier.

-21-. In the "Inbound Rules" section, you'll notice that rules start from Rule 100 and continue with an asterisk (*), indicating that they follow the order from 100 until N. To modify the inbound rules, let's first remove Rule 100. Then, click on "Add New Rule" and set it to deny port 8080.

-22-. Now, if you refresh the application page, you'll notice that the application is no longer accessible on port 8080. Although we allowed port 8080 at the instance level, modifying the rule at the subnet level denied traffic for all instances within that subnet. Even if someone launches an instance and allows traffic on port 8080, it won't work due to the NACL policy we set up. This demonstrates the importance of NACLs in VPC from a security standpoint, as they provide an additional layer of control over traffic flow within subnets, ensuring consistent security policies across instances.

-23-. Exactly, by modifying the NACL rules to allow traffic, the application will be accessible again. This flexibility in NACL rules allows you to adjust security measures as needed while maintaining control over traffic flow within your VPC.

-24-. n the NACL group, if I set Rule 100 to deny port 8080 and Rule 200 to allow traffic for port 8080, will our application be accessible or not? The answer is within this blog. Feel free to try it out and let me know.

Terraform & HashiCorp Vault Integration: Seamless Secrets Management

Shrihari Haridass — Sat, 23 Mar 2024 04:00:00 +0000

what is Hashicorp vault.?

Imagine HashiCorp Vault as a secure digital vault for all your sensitive information like passwords, API keys, and encryption certificates. It acts like a central location to store, access, and manage these secrets. Here's why it's useful:

Strong Security: Vault encrypts your secrets and controls access with different permission levels. This minimizes the risk of unauthorized access and keeps your sensitive information safe.
Centralized Management: No more scattered secrets! Vault keeps everything in one place, making it easier to control and audit who can access what.
It can store various secrets, from database credentials to cloud API keys. Vault also integrates with different tools and platforms you might already be using.

Overall, using Vault helps organizations improve security, simplify secret management, and streamline access control for critical information. While Vault has a free open-source version with limited features, most businesses opt for the paid edition with additional functionalities like advanced auditing and disaster recovery

-1-. In today's demo, we will explore Terraform Vault integration with a real example and troubleshooting. Let's get started!

-2-. Launch one EC2 instance with basic configuration and access it.

-3-. Then update your machine and install GPG as well.

sudo apt update && sudo apt install gpg

-4-. Then download the signing key to a new keyring.

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

-5-. Then verify the key's fingerprint.

gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint

-6-. Clone the Vault repository from GitHub.

mkdir -p $GOPATH/src/github.com/hashicorp && cd $_
git clone https://github.com/hashicorp/vault.git
cd vault

-7-. Then install Vault.

snap install vault

-8-. To start Vault, you can use the following command:

vault server -dev -dev-listen-address="0.0.0.0:8200"

-9-. You may need to set the following environment variables. To do so, create a duplicate session and copy the command, then paste it into the new session.

export VAULT_ADDR='http://0.0.0.0:8200'

-10- Then go to your EC2 instance's security group and open port 8200 to access the Vault UI.

-11-. And then copy the public IP and paste it into your browser's address bar followed by port 8200. You will see the login page appear.

-12-. For logging in as root, select 'Token' as the default method. Copy the token from your server when you start Vault; it will be displayed to you. After that, you will see the dashboard.

-13-. Then, in this demo, we will store the secret as 'Key & value'. To do so, click on 'Secret Engines', then select 'KV' which stands for 'Key/value', and click on 'Next'.

-14-. Next, on the following window, provide a path and click 'Next' to proceed, where you will notice that no credentials are created or stored.

-15-. Now, click on 'Create Secret' and input your secret. I will use 'Username & password' as an example.

-16-. Then, to access these credentials in Terraform, you need to assign policies or roles similar to IAM. Click on 'Policies', then click on 'Enable New Method', and select 'App Role' as the authentication method. This is similar to IAM. Enable the role. Using this App Role, I will authenticate Terraform or whichever tool you are using.

-17-. Then, create a duplicate session of your machine and run the following command on it.

-18-. Because HashiCorp Vault does not support the creation of users in the UI, go to the console and create a user. It's very easy. Before creating the role, we need to create a policy for that role. This policy enables Terraform to access the 'kv' and 'secret' folders in Vault.

vault policy write terraform - <<EOF
path "*" {
  capabilities = ["list", "read"]
}

path "secrets/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "kv/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "secret/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "auth/token/create" {
capabilities = ["create", "read", "update", "list"]
}
EOF

-19-. Now, create a role.

vault write auth/approle/role/terraform \
    secret_id_ttl=10m \
    token_num_uses=10 \
    token_ttl=20m \
    token_max_ttl=30m \
    secret_id_num_uses=40 \
    token_policies=terraform

-20-. Similar to AWS, in Vault, we have Role ID and Secret ID. This is sensitive information, so do not share it with anyone.

vault read auth/approle/role/terraform/role-id

vault write -f auth/approle/role/terraform/secret-id

-21-. Now that we are done with Vault, the next step is to write down the Terraform project and check whether Terraform is able to read the secret from Vault.

#main.tf

provider "aws" {
  region = "us-east-1"
}

provider "vault" {
  address = "http://<your-ip_address>:8200"
  skip_child_token = true

  auth_login {
    path = "auth/approle/login"

    parameters = {
      role_id = "<your-role_id>"
      secret_id = "<your-secret_id>"
    }
  }
}

data "vault_kv_secret_v2" "example" {
  mount = "kv" // change it according to your mount
  name  = "test-secret" // change it according to your secret
}

-22-. Then, to check if Terraform is able to retrieve the credentials from Vault, run the 'terraform init' and 'terraform apply' commands.

-23-. Now, let's add the block to create an EC2 instance and apply it again. You will see that in the EC2 tag, our password will be taken.

#main.tf

provider "aws" {
  region = "us-east-1"
}

provider "vault" {
  address = "http://<your-ip_address>:8200"
  skip_child_token = true

  auth_login {
    path = "auth/approle/login"

    parameters = {
      role_id = "<your-role_id>"
      secret_id = "<your-secret_id>"
    }
  }
}

data "vault_kv_secret_v2" "example" {
  mount = "kv" // change it according to your mount
  name  = "test-secret" // change it according to your secret
}

resource "aws_instance" "example" {
  ami = "ami-0c7217cdde317cfec"
  instance_type = "t2.micro"

  tags = {
    secret = data.vault_kv_secret_v2.example.data["username"]
  }
}

-24-. If you encounter the following error while running 'terraform apply' command, then run it again to recreate 'role-id' and 'secret-id' with the same command as mentioned in step no. 20. After that, paste the new credentials into the code. This will solve your problem.

Automating ECR Image Notifications in Slack with EventBridge and Lambda.

Shrihari Haridass — Sat, 17 Feb 2024 03:51:00 +0000

This blog covers how you can enable notifications in Slack whenever new Docker images are pushed into an AWS ECR repo. This is achieved with the assistance of AWS services such as EventBridge and Lambda. Additionally, to send notifications on Slack automatically, a Slack Webhook is also required.

-1-. As a first step, create an account on Slack, then navigate to the Slack API: Applications | Slack. Click on 'Create an App,' and choose the option to create it from scratch.

-2-. Provide an 'App Name,' choose your workspace created to receive notifications, and then click on 'Create App.'

-3-. On the left side menu bar, locate the 'Incoming Webhook' option and click on it.

-4-. Activate the incoming webhook by clicking on the 'On' toggle.

-5-. To add the webhook to the workspace, click on 'Add new webhook to workspace.' A pop-up will appear with Slack channel permission window; click 'Allow' to enable the reception of incoming webhooks.

-6-. Once done, you will see a generated 'Webhook URL.' Copy this URL and save it on Notepad or any text editor.

-7-. Next, log in to your AWS account and navigate to 'AWS Lambda.' Create a new function, choose the runtime as 'Python 3.12,' and then create the function. Since Lambda doesn't support the 'request' package, you'll need to write the Python code on your local machine where Python is already installed. Alternatively, you can use the provided code below:

import json
import requests

def lambda_handler(event, context):
    # Print the entire event to CloudWatch Logs for debugging
    print(json.dumps(event))

    # Access imageDigest and repositoryName dynamically
    image_digest = event.get('imageDigest') or event.get('detail', {}).get('imageDigest')
    repo_name = event.get('repositoryName') or event.get('detail', {}).get('repositoryName')

    if not image_digest or not repo_name:
        return {
            'statusCode': 400,
            'body': json.dumps('Invalid event structure. Missing required keys.')
        }

    slack_webhook_url = 'Your Slack URL'

    message = f"New image pushed to ECR: {repo_name}:{image_digest}"
    requests.post(slack_webhook_url, json={'text': message})

    return {
        'statusCode': 200,
        'body': json.dumps('Notification sent to Slack!')
    }

-8-. After running the command 'pip install requests,' multiple files will be generated. Zip all these files, excluding the Slack webhook file.

-9-. Navigate back to the Lambda function, and on the right-hand side, click on 'Upload from' and choose '.zip.' Select your zipped file and upload your code. In the test event, set the key-value pairs accordingly.

-10-. Finally, deploy the function.

-11-. Now, go to 'EventBridge,' and click on 'Create Rule.'

-12-. Enter a name and description for the rule, then click on 'Next.'

-13-. Now, an important step is to configure the 'Event Pattern.' Select the value as 'my image' and click on 'Next.'

-14-. Choose the target as 'Lambda,' select your function, click 'Next,' and create the rule.

-15-. Now, go to Amazon ECR, create a public or private repository, push an image into that repository, and you will see a notification received in your Slack workspace as configured.

-16-. So, from now on, whenever you push a new image to ECR using EventBridge and Lambda, it will automatically send a notification to your Slack group.

Conclusion:

So, this is how you can configure notifications for ECR images. By leveraging EventBridge and Lambda along with Slack webhooks, you can receive automatic notifications in your Slack workspace whenever a new image is pushed to your ECR repository.

AWS CodePipeline

Shrihari Haridass — Wed, 07 Feb 2024 05:20:39 +0000

-1-. Now, all stages have been properly configured, and it is running smoothly. However, we are currently executing it manually. Therefore, as the final step in this series, we will configure AWS CodePipeline and explore its functionalities.

-2-. Navigate to 'CodePipeline' and then click on 'Create Pipeline'.

-3-. Next, provide a name for your pipeline, select the pipeline type, choose 'New Role', and then click on 'Next'.

-4-. Next, in the following window, select 'CodeCommit' as the source provider since our code is stored there. Choose the 'Repository Name' where your code resides, and then select the branch from which you want to build and deploy your application. For the detection option, choose 'AWS CodePipeline', as it is essential for tracking changes in the repository. This ensures that any new changes trigger our pipeline. Finally, click on 'Next'.

-5-. In the next stage, the 'Build Stage,' select the 'Build Provider,' indicating where you want to build your code. Choose your region, specify your project name, and then click on 'Next'.

-6-. In the deploy stage, select the 'Deploy Provider' and specify the application name and deployment group where we have configured the deployment. Finally, click on 'Next'.

-7-. Review your pipeline settings and configurations, ensuring they are accurate. Then, click on 'Create Pipeline' to initiate the process, which will fetch your code from CodeCommit, build it, and deploy it on EC2.

-8-. Here, you can observe that our pipeline has run successfully, with all stages completing without errors. This indicates that our application has been deployed. You can verify this by checking the latest commit that we made, which triggered the pipeline automatically. To confirm the deployment, you can make changes to the 'index.html' file and see if the changes reflect correctly.

-9-. Exactly! AWS CI/CD, including services like AWS CodePipeline, is indeed a critical component of DevOps practices. Building pipelines in AWS allows for automated and efficient deployment of applications, ensuring faster delivery and higher reliability. I'm glad you found the process of configuring the code pipeline informative. It's an essential skill for modern software development and deployment. If you have any more questions or need further clarification on any aspect, feel free to ask!

This is the last part of our 'AWS DevOps' series. We covered topics such as 'Introduction to AWS DevOps,' 'AWS CodeCommit,' 'AWS CodeBuild,' 'AWS CodeDeploy,' and 'AWS CodePipeline.' We delved into each step in detail, exploring their importance. Make sure to try this out, and after you're finished, remember to terminate or delete the pipeline to avoid unnecessary billing. Please share, like, and let me know in the comment box your thoughts for the next series or any questions you may have. Happy learning!

Automating Dockerfile Vulnerability Scanning in GitHub Actions Using Snyk and CodeQL

Shrihari Haridass — Fri, 02 Feb 2024 18:30:00 +0000

What will be covered in this blog?

-> Building secure software is like building a sturdy house - you wouldn't wait until it's finished to check for termites, right? That's where Dockerfile scanning comes in. It's like checking your construction plans for weak spots before hammering any nails.

-> Think of Snyk and CodeQL as your security inspectors. They scan your Dockerfile, a blueprint for your container image, and point out any hidden vulnerabilities, like rickety doors or leaky windows.

-> By integrating this scanning into your GitHub Actions, like an automated construction manager, you catch these issues early, even before building the image. This saves you time, money, and headaches later on, because fixing a broken door after the house is built is much harder than preventing it in the first place!

-> This is what DevSecOps is all about - baking security into every step of building software, not just the final inspection. By automating Dockerfile scanning with Snyk and CodeQL, you build stronger, more secure software with confidence and peace of mind, just like a well-built house that can withstand any storm.

-> So in this setup what we do we are using snyk to scan Dockerfile and then outputting the result in SARIF format to upload to GitHub code scanning

-> SARIF (Static Analysis Results Interchange Format) is an OASIS Standard that defines an output file format. The SARIF standard is used to streamline how static analysis tools share their results.

(1). Go to 'Snyk,' create your account, then navigate to the Dashboard. In the left-side menu bar, click on your account name, and select 'Account Settings' from the dropdown menu.

(2). Once inside, proceed to the 'General' tab. You will find the 'Generate Auth Token' option; click on it, then copy the token. Save it in Notepad. In my case, I have already created that token.

(3). Next, navigate to your GitHub account and create a repository with your desired name. Create a 'Dockerfile'; for demonstration, you can use the following example Dockerfile.

# Use an official Node.js runtime as a parent image
FROM node:14

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json ./

# Install app dependencies
RUN npm install

# Copy the rest of the application code to the working directory
COPY . .

# Expose the port the app runs on
EXPOSE 3000

# Define the command to run the application
CMD ["npm", "start"]

save this file

(4). Proceed to 'Settings' → 'Settings and variables' → 'Actions.' Click on 'New Repository Secret,' provide the name as 'SNYK_TOKEN,' paste the token, and save it.

(5). Navigate to the 'Actions' tab, search for 'Snyk,' and select 'Snyk Container.'

(6). It will generate a '.yaml' file. You can customize this script based on your requirements, but for this demo, it's okay. Now, you can 'Commit' the changes.

(7). Navigate to the 'Actions' tab and click on the pipeline; your pipeline will be triggered automatically.

(8). Now you can see that my job has run successfully.

(9). Now, go to the 'Snyk Dashboard.' Note that when creating an account on Snyk, you can connect it with your 'GitHub' account for access to your repository. Alternatively, you can go to the 'Projects' option and click on 'Add Project.'

(10). Select the 'GitHub' option, then choose your repository and click on 'Add selected repos.'

(11). Here, you will find your project report and the scan report for the 'Dockerfile.'

(12). Now, return to our 'GitHub Account,' navigate to 'Security Tab,' and click on 'Code Scanning.'

(13). Click on 'Code Scanning,' and you will find a detailed report.

(14). you can clone My repo also

shri2904 / snyk-blog

Description

snyk-blog

Description

View on GitHub

AWS CodeDeploy

Shrihari Haridass — Wed, 31 Jan 2024 08:55:34 +0000

Hi, guys! I hope you are practicing AWS DevOps and enhancing your skills. Last week, we covered the AWS CodeBuild service, and today we are moving on to the next part, which is CodeDeploy. After building your code, we need to deploy it, right? With the help of the CodeDeploy service, we can automate our deployment process as well.

AWS CodeDeploy is a service that automates code deployments to any instance, including Amazon EC2 instances and instances running on-premises.

-1-. Let's deploy our code with the AWS CodeDeploy service. To deploy an application using CodeDeploy, follow these steps:

Navigate to CodeDeploy.
Click on "Create Application".

-2-. Then, provide an "application name" and select the compute platform as "EC2." Finally, click on "Create."

-3-. So, since we are using EC2 for our study, once your application is built, it will run on that system or be deployed on the server. This concept is referred to as a "deployment group." Click on "Create Deployment Group." A deployment group typically consists of more than one server. If you want to deploy the application on multiple servers simultaneously, you'll need to create a deployment group for that purpose.

-4-. Then, provide a deployment group name, create a service role, and select the deployment type. In the service role, attach the necessary access policies. If needed, you can attach those policies to your role to ensure the deployment is done successfully.

-5-. Then, create a basic EC2 instance and go back to our CodeDeploy group. We need the EC2 instance because our application is deployed on that server. In the Environment Configuration, select your EC2 instance.

-6-. Select the "never" value for installing the AWS CodeDeploy agent, disable the load balancer, and click on "Create Deployment Group." For the time being, I've chosen the "never" option, but we may need to change this setting later.

-7-. The reason I selected the "never" option for installing the agent is because your application runs on an EC2 instance, and the EC2 instance needs specific requirements for the application to function. Installing the "agent" ensures compatibility between the EC2 instance and CodeDeploy. To address any potential version mismatches between the EC2 instance and CodeDeploy, I will provide you with a script. You'll need to connect to the instance and run the script to resolve the version mismatch.

#!/bin/bash 
# This installs the CodeDeploy agent and its prerequisites on Ubuntu 22.04.  
sudo apt-get update 
sudo apt-get install ruby-full ruby-webrick wget -y
cd /tmp 
wget https://aws-codedeploy-us-east-1.s3.us-east-1.amazonaws.com/releases/codedeploy-agent_1.3.2-1902_all.deb 
# wget cmd end here
mkdir codedeploy-agent_1.3.2-1902_ubuntu22 
dpkg-deb -R codedeploy-agent_1.3.2-1902_all.deb codedeploy-agent_1.3.2-1902_ubuntu22
# dpkg end here
sed 's/Depends:.*/Depends:ruby3.0/' -i ./codedeploy-agent_1.3.2-1902_ubuntu22/DEBIAN/control
# sed end here
dpkg-deb -b codedeploy-agent_1.3.2-1902_ubuntu22/
sudo dpkg -i codedeploy-agent_1.3.2-1902_ubuntu22.deb
systemctl list-units --type=service | grep codedeploy
sudo service codedeploy-agent status

-8-. Now, if you recall, just as we used the buildspec.yaml for building the code, we need an appspec.yaml file for deploying the code.

version: 0.0
os: linux
files:
  - source: /
    destination: /var/www/html
hooks:
  AfterInstall:
    - location: scripts/install_nginx.sh
      timeout: 300
      runas: root
  ApplicationStart:
    - location: scripts/start_nginx.sh
      timeout: 300
      runas: root

-9-. After that, create a "scripts" folder, and within it, create two new files named "install_nginx.sh" and "start_nginx.sh." Push these files to our AWS code repository. The file contents are as follows:

---------------------install_nginx.sh---------------------
#!/bin/bash

sudo apt-get update
sudo apt-get install -y nginx

----------------------start_nginx.sh-------------------
#!/bin/bash

sudo service nginx start

-10-. Afterward, we need to build this code so that our latest code is stored in "S3." Navigate to "Build" and initiate the build process.

-11-. Now, go to "Deployment" again and click on "Create Deployment." Choose the "Revision type," select the file type, and then click on "Create Deployment." You will see the deployment process starting. Also, choose the S3 location where our "Zip" file is stored; this is crucial. Click on that zip file, and it will show the details of the zip. Copy the zip URL and paste it in the revision location.

-12-. But if you click on "Events," you will notice that all are pending because the EC2 instance doesn't have the permission to retrieve artifacts from S3. Additionally, the EC2 instance lacks permission to access the CodeDeploy service. Let's create one more role in IAM to address this issue.

-13-. Now, we need to grant these permissions to our instance. Go to the EC2 dashboard and select your instance.

-14-. Now, connect to the instance and restart the CodeDeploy agent service. After that, check the CodeDeploy dashboard, and you should see that all steps have run successfully.

sudo service codedeploy-agent restart

-15-. Now, go to EC2, copy your public IP, and paste it into the browser. You should see the Nginx page, indicating that your deployment was successful.

so this is how code-pipeline works

AWS CodeBuild

Shrihari Haridass — Wed, 24 Jan 2024 05:39:32 +0000

Today, we will explore the AWS CodeBuild service, which is the next phase after CodeCommit. AWS CodeBuild is a fully managed continuous integration service that compiles source code, runs tests, and produces software packages ready for deployment. You don't need to provision, manage, and scale your own build servers. I hope you have read the previous blog about the CodeCommit service.

So, let's get started

-1-. Now, let's build our code. To do that, go to CodeBuild and click on 'Create build project.'

-2-. In the project configuration settings, provide the 'Project Name,' then select the source provider as 'AWS CodeCommit.' After that, choose the 'Branch.' Next, select the OS as 'Ubuntu,' runtime as 'Standard,' and use the latest image.

-3-. Now, we need to create a 'Service Role.' Essentially, it requires access to other services. For instance, for building, it needs access to the code repository to fetch code, and the same applies to other services. If you observe, it creates a role automatically for you. After that, you will see the 'Buildspec' file option, which is important as it is the 'Configuration' file that we need to create.

-4-. To create the file, let's go to VS Code, create a file named 'buildspec.yml,' save it, and push it to the master branch.

version: 0.2

phases:
  install:
    commands:
      - echo Installing NGINX
      - sudo apt-get update
      - sudo apt-get install nginx -y
  build:
    commands:
      - echo Build started on `date`
      - cp index.html /var/www/html/
  post_build:
    commands:
      - echo configuring NGINX

artifacts:
  files:
    - '**/*'

-5-. If you save the file as 'buildspec.yml,' you don't need to provide the file name; it will be fetched automatically from that branch. After that, click on 'Create build project.'

-6-. Next, click on the 'Start Build' button to initiate your CodeBuild. You can observe that our build is successful.

-7-. If you want to store the 'Artifacts,' you can edit the configuration and set the location to 'S3'.

-8-. Choose 'S3' as the option. If you already have a bucket, you can provide its name; otherwise, create one in S3. Within that bucket, create another folder where we will store our artifacts in '.zip' format. also choose Artifacts packaging as "ZIP" Copy that location and paste it, then click on 'Update artifacts.' This way, when we build next time, our artifacts will be stored in S3.

-9-. You can run the build again to check if your build is stored in S3. Before starting the build, go to IAM → roles, find our build role, and add the 'Access S3' permission. This step is necessary to upload the artifacts; otherwise, you will encounter an 'access denied' error. Here, I've attached that role, and our build is successful.

-10-. Now, go to the S3 location to check if the artifacts are uploaded. By mistake, I pasted the wrong location path, so that's why the path is long.

So, this is how we can build our code using the CodeBuild service. This series is designed for those who are starting with AWS DevOps or are new to AWS DevOps. That's why I am covering the basic pipeline steps. Once this series concludes, we will create a real scenario pipeline. At that time, we can skip the basic steps. I hope you enjoyed today's part.

AWS CodeCommit

Shrihari Haridass — Wed, 17 Jan 2024 04:40:00 +0000

As you can see in the title, today we are covering the first service of AWS DevOps, which is 'CodeCommit.' I hope you've read the introduction part of this series; if not, please check it out. AWS CodeCommit is similar to other centralized code repositories like GitHub, GitLab, GitBucket, and so on; the only difference is that it's an AWS private repository that you can use. For more information, you can read about it on AWS CodeCommit

So today, we'll learn how to create a repository in CodeCommit, create a user for it, provide necessary permissions to that user using the IAM service, clone that repository into our local machine, make some changes, and push it back to the repository. Additionally, we'll explore how to create branches, merge them in CodeCommit, and cover some other important options. Let's get started!

-1-. Begin by searching for the 'CodeCommit' service. As you know, the first step involves pushing code into the repository.

-2-. Next, navigate to 'Repository,' where we push our code. To do this, click on 'Create Repository.'

Then, provide a name and description for the repository. You'll notice 'CodeGuru,' which is similar to 'SonarQube Scanner' for code scanning. However, at this point, we don't want to proceed directly to 'create repository.'

-3-. You'll encounter a warning indicating that using the root user is not recommended. To address this, let's create another user in 'IAM' to enable the use of services in AWS DevOps. Proceed to 'IAM' now.

-4-. In 'IAM,' navigate to 'Users,' click on 'Add Users,' and provide a name for the user.

-5-. In the 'Set Permissions' window, click 'Next.' On the 'Review' page, you'll notice that we currently have only one permission, which is the ability to change the password. Proceed to click 'Create User.'

-6-. You can save this password, as it is displayed only once, or download the '.csv file.'

-7-. Now that our user is created, click on it. We want to grant permissions for 'CodeCommit,' so first, navigate inside it and then click on 'Security Credentials.'

-8-. Scroll down, and you'll find 'HTTPS Git credentials for AWS CodeCommit.' Click on 'Generate Credentials.' Here, you will see your credentials generated. You can also download them, and with these credentials, you can access.

-9-. Return to the 'CodeCommit' window, click on 'Clone URL,' and choose the 'HTTPS' option. Create a folder, open your preferred code editor (like VS Code), and clone the repository to your local computer.

-10-. While cloning, a pop-up will appear asking you to enter the Git credentials created for 'CodeCommit.' Simply copy and paste them in this window.

-11-. If the cloning process is not working, we might have missed setting up IAM permissions for cloning and committing. Go back to the 'Users' section, select the user, navigate to the 'Permissions' tab. As there is only one permission currently, click on 'Add permission' → 'Attach Policy Directly' → 'AWSCodeCommitPowerUsers.'

-12-. Now, try the process again in 'VS Code.' This time, you should see a 'Clone successful, empty repository' message.

-13-. Next, add an 'index.html' file with some sample text. After making changes, commit and push this code into our repository.

-14-. Navigate to 'CodeCommit' → 'Repository,' and refresh. You should now see our files in the repository.

-15-. This is how you can connect your local machine to AWS CodeCommit using IAM credentials, similar to connecting to GitHub. Now, let's create another branch, push the code, and check the 'Branches' section in CodeCommit to see the newly created 'dev' branch.

-16-. Now, let's merge the changes into 'Master.' Click on the branch, create a pull request, provide a name, create the pull request, merge it using fast-forward, and delete the source branch after merging. Refresh the 'Master' branch to see your changes successfully merged.

-17-. So, this is how you can use the 'CodeCommit' service.

How to Connect AWS Application Composer in VS Code

Shrihari Haridass — Tue, 16 Jan 2024 09:39:18 +0000

As you know, last month, AWS announced AWS Application Composer in VS Code, allowing you to use it within VS Code and work seamlessly. However, when I started working on it, I found it challenging to integrate because it was recently announced, and there were no video documentation available. Many steps were missing at a high level. After troubleshooting and spending hours on it, I finally managed to make it work.

So, I'm sharing a step-by-step document on how you can connect AWS Application Composer to VS Code locally and sync up. I hope you find it helpful. Let's get started!

-1- So, let's begin by logging into the AWS account.

-2- Next, search for the 'Application Composer' service.

-3- After opening, you'll find information about the 'Application Composer.' You can then click on 'Create project.'

-4- Once you click on Create project, a blank canvas dashboard will open. On the right-hand side, you will see the Menu option. Click on it, and you will see multiple options. Select 'Active local sync.'

-5- Next, select the folder from your local machine.

-6- I created a new folder named 'application-compose' and selected it.

-7- After selecting, you will see the 'Active' button. Click on it.

-8- Open VS Code, go to the Extensions, and search for 'AWS Toolkit.' Install it and configure it.

-9- Click on 'Sign In to get Started,' and another window will appear. Sign in using your 'Builder ID.' If you haven't created one, you can do so at the time of signing in, and you'll be ready to use AWS services.

-10- Now, navigate to the folder you created.

-11- Now, you will see the 'template.yaml' file there, which is currently empty since we haven't created anything yet. Additionally, on the right side, you'll notice a small 'Application Composer' symbol. Clicking on this symbol will open the same AWS GUI.

-12- From there, you can start working on your project. This is how you can use AWS Application Composer in VS Code.

Introduction to AWS DevOps

Shrihari Haridass — Thu, 11 Jan 2024 10:41:05 +0000

Introduction

AWS provides a set of flexible services designed to enable companies to more rapidly and reliably build and deliver products using AWS and DevOps practices. These services simplify provisioning and managing infrastructure, deploying application code, automating software release processes, and monitoring your application and infrastructure performance.

DevOps is a methodology. When we talk about AWS DevOps, the main components are:

A. CodeCommit
B. CodeBuild
C. CodeDeploy
D. CodePipeline

But, as we delve deeper, the journey begins with code. How do you deploy this code on servers in an automated fashion or in a continuous cycle? This is the essence of the journey.

If you are a DevOps engineer, especially focused on AWS DevOps, you should be familiar with the following services. I'll discuss each service in-depth, so don't worry about that.

IAM

Suppose you want to grant access to a specific DevOps service for a particular user. In AWS, where there are over 400 services, you can provide access to specific services, and these services also require access to certain resources. All these rules are defined in IAM.

KMS

KMS, or Key Management Service, is utilized when working with the need to encrypt or decrypt files. The code for encryption and decryption belongs to KMS.

Artifacts

When you build your code, it's essential to save it somewhere. For this purpose, we use the 'Artifact' and 'S3' services.

CodeCommit

CodeCommit is similar to GitHub or GitBucket; the only difference is that it is owned by AWS. It functions as a private service of AWS, serving as a central repository where developers can push their code to 'AWS CodeCommit'.

CodeBuild

After pushing the code to the repository, the next steps involve 'building' and 'testing' the code, forming a crucial part of the Continuous Integration (CI) process. Once the developer pushes the code into the repository, it undergoes a build process, and the resulting code is saved to S3 as artifacts. These artifacts can then be deployed whenever and at any time.

CodeDeploy

In CodeDeploy, its role involves fetching the build from S3 and deploying that code onto compute or serverless services. This marks the next phase in the AWS DevOps service. I'll elaborate on the following aspects in the next part, as they fall under the Continuous Deployment (CD) process.

CodePipeline

After completing both CI and CD processes, the next step is to automate the workflow by building a pipeline. CodePipeline comes into play for this purpose. It essentially creates stages for you, following the sequence of source → build → deploy.

EC2

Amazon EC2 empowers you to run secure and scalable compute capacity in the cloud. Additionally, if you need to manage services within your application, EC2 is suitable. It is especially recommended for small-scale applications.

ECS

Amazon ECS is a fully managed orchestration service that is ideal for scaling applications, especially those dealing with high traffic. In AWS DevOps, it is commonly utilized for efficient management and scaling of applications.

Lambda

AWS Lambda allows you to run code without the need for provisioning or managing servers. In the realm of AWS DevOps, Lambda is frequently employed for various purposes such as automating tasks, handling serverless functions, and orchestrating workflows seamlessly.

This concludes the first part of AWS DevOps, covering essential services commonly used and taught. In the upcoming segments of the DevOps series, we will delve into more services integral to AWS DevOps. Stay tuned for the new series, and thank you for the positive response to my Azure DevOps series. Your support is appreciated, and I look forward to the next part.