DEV Community: Peter Eskandar

[Boost]

Peter Eskandar — Thu, 28 Nov 2024 10:29:47 +0000

Installing AWS CloudWatch Agent on On-Premises Servers Using SSM

Peter Eskandar ・ Aug 31

Mastering the AWS Phone Interview: Key Technical Concepts and Preparation Guide

Peter Eskandar — Mon, 14 Oct 2024 11:32:35 +0000

The AWS Phone Interview is a critical step in landing a role at Amazon Web Services (AWS). Known for its rigorous process, the interview assesses not only your technical acumen but also how well you align with AWS' Leadership Principles.Preparing for this stage can seem daunting, but with the right approach, you can navigate the process with confidence and set yourself up for success.

This article is actually based on my own study notes that I prepared while getting ready for the AWS phone interview. These notes helped me pass the interview, and I hope they will be helpful for you as well. While I focused on the technical aspects of the interview, you should also expect to be asked behavioral questions based on Amazon’s Leadership Principles. In this guide, however, I will walk you through the main technical topics you’ll likely encounter.

It’s important to note that for most of the content and screenshots in this article, credits go to Adrian Cantril’s Tech Fundamentals Course and the BeSA Program, which have been instrumental in structuring this guide.

Check out Adrian Cantril's Tech Fundamentals Course here.
Learn more about the BeSA Program here.

Your feedback is invaluable! If you find these study notes helpful or if you have any suggestions on how to improve them, please feel free to share.

Main Items

OSI Model Introduction
Network Address Translation
Distributed Denial of Service (DDos)
Encryption
Encryption In Transit "SSL & TLS"
Hashing
DNS – Domain Name System
Multi-Tenant Applications
Databases

OSI Model Introduction :

The OSI (Open Systems Interconnection) model is a conceptual framework that describes the communication functions of a telecommunication or computer system. The model is divided into seven layers, each of which performs a specific function in the transmission of data between networked devices. These layers are designed to provide a standardized approach to network communication, allowing different types of devices to communicate with each other regardless of their underlying hardware or software.

OSI Model

1. Layer 1 – Physical :
Layer 1, also known as the physical layer, is the first and lowest layer in the OSI model. It is responsible for the physical transmission of data over a communication channel, such as a wire, cable, or wireless signal. This layer defines the physical characteristics of the communication medium, including voltage levels, cable types, and signaling methods, to ensure that data is transmitted accurately and reliably between devices.

Main characteristics :
- Physical Shared Medium
- Define Standards for transmitting onto the medium
- Define Standards for receiving from the medium
- No Access Control
- No uniquely identified devices
- No Device to Device Communication “Everything is broadcast using transmission”
- Layer2 – Data Link adds a lot of intelligence on top of layer 1

OSI Model Layer1

2. Layer 2 – Data Link :

Layer 2, also known as the data link layer, is the second layer in the OSI model. It is responsible for the reliable transfer of data between two nodes on the same network segment. This layer defines protocols for the access and use of the physical network, such as addressing, flow control, and error detection and correction.

OSI Model Layer2

Main characteristics :
- Adds a controlled access to the physical medium
- Use MAC Addresses for both Source and Destination
- In case of broadcast the Destination MAC address is all FFs “ff:ff:ff:ff:ff:ff”
- The EtherType field in the Frame defines which L3 Protocol is used “For Example : Internet Protocol IP or Address Resolution Protocol ARP”
- FCS Attribute is for Frame Check Consequence : which is an error-detecting code added to a frame in a communication protocol. Frames are used to send payload data from a source to a destination.

Advantages :
- Identifiable devices
- Media Access Control
- Collision Detection
- Unicast 1:1
- Broadcast 1:All
- Switched – Like Hubs with Super Powers (Layer 2)

3. Layer 3 – Network Layer :

Layer 3, also known as the network layer, is the third layer in the OSI model. It is responsible for the end-to-end delivery of data between devices on different network segments. This layer defines protocols for logical addressing, routing, and fragmentation of data packets to ensure that they are delivered to the correct destination across different networks.

Layer 3 connects multiple local Layer2 LANs widely together using IP Addresses by handling logical addressing, routing, and forwarding of packets across different networks.

OSI Model Layer3

How a Router Defines it’s Next HOP ?
- Communications between different routers are made on Layer2, Source Router wraps the packet in a frame and adds the source and destinations MAC Addresses, the packet itself doesn’t change the frame though.
- The MAC Address of the Destination router can be obtained using Address Resolution Protocol ARP

OSI Model Layer3 Routing

Layer 3 Summary :
- IP Addresses (IPv4/v6) – Cross Network Addressing
- ARP – Find the MAC Address for this IP
- Route – Where to forward this packet
- Router Tables – Multiple Routes
- Router – Moves packer from SRC to DST – Encapsulating in L2 on the way
- Device to Device communication over the internet
- No method for channels of communications, just SRC IP to DST IP
- Can be delivered Out of Order “for packets ordering Layer4 should be used”

4. Layer 4 & 5 – Transport & Session :

Layers 4 and 5, also known as the transport and session layers respectively, are responsible for managing the end-to-end communication between applications running on different devices.

Layer 4, the transport layer, is responsible for ensuring that data is delivered reliably and efficiently from one device to another. It does this by establishing connections between applications on different devices, and managing flow control and error recovery. This layer defines two main protocols: the Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP).

Layer 5, the session layer, is responsible for managing the communication sessions between applications running on different devices. It defines protocols for session establishment, maintenance, and termination, as well as for managing session security and synchronization.

Why use Layer 4 "Layer3 Problems" ?
- With Layer3 – There are no communication channels – packets have a source and destination IP but no method of splitting by APP or Channel
- No Flow Control – if the source transmits faster than the destination can receive it can saturate the destination causing packet loss
- Each Packet is routed independently – Per Packet routing can introduce delays to packets en route. Different packets can experience different delays
- Routing Decisions are per packet – Different routes can result in Out of Order packet at the destination. L3 provides no ordering mechanism

OSI Model Layer3 Problems

Transmission Control Protocol (TCP) vs User Datagram Protocol (UDP) :

Factor	TCP	UDP
Best For	Web Browsing HTTP/HTTPS File Transfer SSH Email or Texting SMTP	Video Chat Live Streaming Online Gaming
Connection type		Connection-less No connection is needed to start and end a data transfer
Data sequence	Can sequence data (send in a specific order)	Cannot sequence or arrange data
Data retransmission	Can retransmit data if packets fail to arrive	No data retransmitting. Lost data can’t be retrieved
Delivery	Delivery is guaranteed	Delivery is not guaranteed
Check for errors	Thorough error-checking guarantees data arrives in its intended state	Minimal error-checking covers the basics but may not prevent all errors
Broadcasting	Not supported	Supported
Speed	Slow, but complete data delivery	Fast, but at risk of incomplete data delivery

Network Address Translation :

NAT is a fundamental technology used in computer networking to allow multiple devices to share a single public IP address. This is essential for businesses and individuals who have more devices than available public IP addresses.

NAT is designed to overcome IPv4 shortages
Also provides some Security Benefits
Translates private IPv4 Addresses to Public
Types :
- Static NAT : 1 private IP to 1 fixed public IP (this is how AWS IGW works)
- Dynamic NAT : 1 private to 1st available Public, The Router maintains a NAT Table, it maps private IP : Public IP. Public IP allocations are temporary allocations from a a Public IP Pool
- Port Address Translation (PAT) : many private to 1 public (AWS NAT Gateway & Home Router)

Port Address Translation (PAT)

Distributed Denial of Service (DDos) :

A Distributed Denial of Service (DDoS) attack is a type of cyber attack where multiple compromised devices, often from different locations, are used to flood a targeted website or network with traffic, causing the website or network to become unavailable to its users. In a DDoS attack, the attackers usually exploit vulnerabilities in the devices or applications to gain control over them and use them to generate a massive amount of traffic. DDoS attacks can be financially motivated or politically motivated, and they can cause significant damage to the targeted organization or individual, such as loss of revenue, loss of reputation, or even loss of sensitive data.

Attacks designed to overload websites
Compete against legitimate connections
Distributed – hard to block individual Ips/Ranges
Examples :
- Application Layer – HTTP Flood
- Protocol Attack – SYN Flood
- Volumetric – DNS Amplification

Application Layer Attach

Protocol Attack

Amplification Attack

Encryption :

Encryption is the process of converting data into a form that is unreadable known as a ciphertext to unauthorized users. It is an essential technique for protecting the confidentiality and integrity of data both in transit and at rest.
Encryption is a two-way process. Data can be encrypted (to protect it) and later decrypted (to access it).

Encryption Concepts

- Encryption at Rest :

Definition: Encryption at rest refers to the protection of data stored on a disk, database, or any storage medium when it is not actively being used or accessed.
Purpose: The goal is to protect data from unauthorized access in case the storage medium is compromised (e.g., if a hard drive is stolen or a server is hacked).
How It Works: Data is encrypted before it is written to storage and decrypted when it is read back into memory. Common methods include full-disk encryption, database encryption, and file-level encryption.
Examples: Encrypting files on a hard drive, encrypting a database, or using a cloud provider's storage encryption features.
Tools/Technologies: BitLocker (Windows), FileVault (macOS), Transparent Data Encryption (TDE) for databases, Amazon S3 Server-Side Encryption.

- Encryption in Transit :

Definition: Encryption in transit refers to the protection of data as it moves across a network, from one system to another, such as from a client to a server.
Purpose: The goal is to protect data from being intercepted or tampered with while it is being transmitted across potentially insecure networks (e.g., the Internet).
How It Works: Data is encrypted before it is sent over the network and decrypted upon arrival. Secure protocols like TLS/SSL are commonly used to encrypt data in transit.
Examples: HTTPS for secure web browsing, VPNs for secure remote access, SSH for secure command-line access, encrypted emails.
Tools/Technologies: TLS/SSL (used in HTTPS), Secure Shell (SSH), IPsec for VPNs, SMTP over TLS for email.

Encryption Approaches

Different Types of Encryption :

- Asymmetric Encryption :

Definition: Two keys are used—a public key for encryption and a private key for decryption.
Examples: RSA (Rivest–Shamir–Adleman), ECC (Elliptic Curve Cryptography).
Use Cases: Often used for secure key exchange, digital signatures, and in scenarios where secure key distribution is challenging.

- Symmetric Encryption :

Definition: The same key is used for both encryption and decryption.
Examples: AES (Advanced Encryption Standard), DES (Data Encryption Standard).
Use Cases: Faster and efficient for large amounts of data, but key management is critical.

- Hybrid Encryption :

Definition: Combines both symmetric and asymmetric encryption to leverage the benefits of both.
Examples: TLS (Transport Layer Security) in HTTPS uses asymmetric encryption to exchange a symmetric key, which is then used for the session.

- Envelop Encryption :

Envelope encryption is a technique used to secure data by encrypting it with multiple layers of keys. In envelope encryption, a symmetric data encryption key is generated for each piece of data or user, and it is then encrypted with a separate key known as the key encryption key (KEK). This KEK is typically an asymmetric key, allowing for greater security and flexibility in key management. The encrypted data and encrypted data encryption key (DEK) are then stored separately, providing an additional layer of security.

Envelop Encryption - Encryption

Envelop Encryption - Decryption

Encryption In Transit "SSL & TLS" :

Secure Socket Layer (SSL) and Transport Layer Security (TLS) are two cryptographic protocols used to provide secure communication over the internet. SSL was developed by Netscape in the mid-1990s and TLS is its successor. These protocols are used to secure web traffic, email, instant messaging, and other types of internet traffic. SSL and TLS use a combination of symmetric and asymmetric encryption to encrypt data, ensuring that information transmitted over the internet is secure and cannot be intercepted by unauthorized parties.

Benefits :
- Privacy & Data Integrity Between Client & Server
- Privacy – Communications are Encrypted
- Asymmetric and then Symmetric encryption
- Identity (Server or Client/Server) Verification
- Reliable Connection – Protects against alteration

- How it works ?

Client Hello :
- The client initiates the connection by sending a "Client Hello" message to the server.
- This message includes the client's supported TLS versions, cipher suites (encryption algorithms), and a randomly generated number (client random).
Server Hello :
- The server responds by sending a “Hello Server” message to the client
- It includes the TLS version, cipher suite and a random generated number (server random)
- At this point, the Client & the Server have agreed how to communicate and the client has The Server Public Certificate, the certificate contains The Server Public Key
- The server can also request a client certificate for mutual authentication.
Server Certificate Verification :
- Trust Chain: Verifies the certificate chain up to a trusted CA. “The Client verifies that the Server Certificate is issued by a Public Certificate Authority trusted by the Operation system or the Browser”
- Expiration: Ensures the certificate is within its valid date range.
- Domain Match: Confirms the certificate is valid for the domain being accessed.
- Revocation: Checks if the certificate has been revoked.
- Signature: Verifies the certificate’s integrity with the CA’s public key
Client Key Exchange :
- The client generates a "pre-master secret" and encrypts it using the server's public key (from the server's certificate).
- The encrypted pre-master secret is sent to the server.
Session Keys Generation :
- Based on the Encryption Algorithm, both the Client & the Server use the pre-master secret along with the random values for the Client & the Server to generate the same session keys for encryption and decryption “symmetric encryption keys”
Finished Messages :
- The client sends a "Finished" message, encrypted with the session key, to confirm that the handshake was successful.
- The server responds with its own "Finished" message, also encrypted.

SSL & TLS

Hashing :

Hash functions are mathematical algorithms that transform input data into a fixed-length string of characters, called a hash or message digest. Hashing is the process of applying a hash function to data to produce a unique and irreversible representation of the original data.
Hash functions are widely used in computer security and cryptography for data integrity and authentication, digital signatures, password storage, and more.

The main characteristics of a hash function are its one-way property, where it is easy to compute the hash value of the input data but computationally infeasible to reconstruct the original data from the hash value, and its collision resistance, where it is highly unlikely for two different inputs to produce the same hash value.

Examples: SHA-256, MD5 (though MD5 is not recommended for security purposes anymore “collision – where two different pieces of information can generate the same hash/digest message”).

Hashing

Hashing Examples

DNS – Domain Name System :

Turns domain names into IP addresses, which allow browsers to get to websites and other internet resources.

DNS

- What happens when you enter a domain name like Netflix.com in your web browser ?

Browser Cache Check “Modern browsers have their own DNS Cache”:
- The browser first checks its internal cache to see if it has recently resolved IP address for that domain name
- If found, it uses this cached IP address to establish the connection. If not, it proceeds to the next step
Operating System (OS) Cache Check :
- If the browser doesn’t contain the IP address, the browser sends a query to the operating system’ DNS resolver (usually part of the networking stack)
- The OS checks it’s own cache for a recently resolved IP address and if found it returns the IP to the web browser.
Host File Check :
- If the OS cache doesn’t have the IP address, the DNS resolver checks the local “hosts” file.
- If “Netflix.com” is listed in the hosts file, the corresponding IP is returned to the browser.
DNS Query to DNS Server :
- If the IP address isn’t found in the hosts file, the DNS resolver queries the configured DNS Server (often provided by the ISP or a public DNS like Google’s 8.8.8.8)
- The query first goes to DNS resolver server (often called a recursive resolver)
Recursive DNS Resolution :
- If the DNS resolver server doesn’t have the IP address in it’s cache, it begins a recursive query :
  - Root Name Servers : The DNS resolver contacts one of the Root Name Servers to find out which Name Server is authoritative for the top-level domain “TLD” .com
  - TLD Name Servers : The Root Name Server responds with the address of the of the .com TLD Name Servers
  - Authoritative DNS Server : The DNS resolver then queries the .com TLD server, which responds with the IP Address of the authoritative Name Server for Netflix.com
  - Final DNS Query : The DNS Resolver queries the authoritative Name Server for Netflix.com which returns the IP address of the that domain.
Connection Establishment :
- The browser uses the IP address to establish a connection to the web server at www.netflix.com, typically over HTTPS.
Caching the results :
- The resolved IP address is cached at each level (browser cache, OS cache, DNS server cache) to speed up future requests.

DNS - Walking The Tree

Multi-Tenant Applications :

Designing a multi-tenant web application involves creating an architecture where multiple customers (tenants) share the same application instance, while their data remains isolated and secure.

Or in other words, talking about multi-tenancy in a SaaS environment is when to offer a service to multiple Group of Users “The so called Tenants” sharing similar experience while ensuring that their data remains isolated and secure, so no tenant can access data of the other tenants.

There are multiple Architectural Models :

The Silo Model “Isolated Tenancy” :
- Description : Each tenant has its own instance of the application, including its own database, application logic and underlying application infrastructure.
- Characteristics :
  - Data Isolation : Complete Physical Isolation
  - Customization : High as it’s tenant has it’s own environment
  - Scalability : Scale by adding more resources
  - Cost : Higher due to separate and non shared resources between different tenants
  - Management Complexity : High, as every tenant infrastructure should be managed individually.
Pool Model :
- Description : All Tenants share the same instance of the application and the underlying database, and data separation is done logically.
- Characteristics :
  - Data Isolation : Logically, data is stored in the same database and partitioned using the tenant_id or data can be stored in different schemas inside the same database
  - Customization : Limited, as all tenants share the same application instance
  - Scalability : High, since resource are shared
  - Cost : Optimized, as resources are shared among multiple tenants with a minimum risk that resources become idle
- Management Complexity : Lower with a single instance to manage
Bridge Model “Hybrid Model” :
- Description : A Hybrid approach where the application layer is shared between different tenants but each tenant has it’s own database.
- Characteristics :
  - Data Isolation : Physical Data Isolation but shared application logic
  - Customization : Moderate with shared logic
  - Scalability : Balanced, combinig shared and isolated resources
  - Cost : Moderate as application layer is still shared among different tenants
  - Management Complexity : Moderate with shared application layer

Databases :

When to Use NoSQL Databases:
- Flexibility: NoSQL databases are more flexible as the are schema-less or have flexible schemas, making them ideal for unstructured or semi-structured data. This is useful for scenarios where data formats may evolve over time, like user profiles or content management systems.
- Scalability: NoSQL databases are designed for horizontal scaling “adds more servers”, making them better suited for handling large volumes of data and high traffic loads, particularly in distributed environments.
- Consistency: Loss of Consistency due to it’s distributed nature and peer to peer replication
- Performance: NoSQL databases can offer better performance for certain types of workloads, such as real-time data processing, by allowing for denormalized data storage and optimized queries for specific use cases.
- Examples: MongoDB, Cassandra, DynamoDB.
When to Use SQL Databases:
- Structured Data: SQL databases are ideal when working with structured data that fits well into tables with a predefined schema. They are well-suited for applications requiring complex queries, joins, and transactions.
- Easy Querying: Allow Easy Querying between multiple tables through tables relationships
- Scalability : Can Scale only vertically by increasing the size of the used instances “Ram & CPU”
- ACID Compliance: SQL databases guarantee ACID (Atomicity, Consistency, Isolation, Durability) properties, making them ideal for applications where data integrity and consistency are critical, such as financial systems.
- ACID Transactions: SQL Transactions are a group of statements that are executed atomically this means that they are either all executed or not executed
- Relational Data: When your data is highly relational, and you need to maintain complex relationships between entities (e.g., foreign keys), SQL databases are more appropriate.
- Examples: MySQL, PostgreSQL, Microsoft SQL Server.

Thank you for reading through the whole article! If you found it helpful, please consider adding a like or sharing it with others who might benefit from it.

Installing AWS CloudWatch Agent on On-Premises Servers Using SSM

Peter Eskandar — Sat, 31 Aug 2024 22:56:22 +0000

Introduction

In this guide, we'll walk you through the process of installing the AWS CloudWatch Agent on on-premises servers using AWS Systems Manager (SSM). This is particularly useful for those managing hybrid environments where both on-premises servers and cloud-based resources are monitored using AWS CloudWatch.

To make this guide practical, we'll simulate an on-premises server using an EC2 instance created in another AWS account. We'll cover everything from registering the on-premises server with SSM, installing the CloudWatch Agent, configuring it, and then using it to collect and send logs to CloudWatch.

What You'll Learn

How to register an on-premises Debian server with AWS Systems Manager using a Hybrid Activation.
How to install and configure the CloudWatch Agent on the server.
How to send logs from your server to AWS CloudWatch

Prerequisites

An AWS Account with necessary permissions.
A Debian-based Server (simulated using an EC2 instance created in a different AWS Account for this guide).

Step 1: Create an SSM Hybrid Activation

Before registering your on-premises server with AWS Systems Manager, you need to create a Hybrid Activation. This step will provide you with an Activation Code and Activation ID, which are required to register your server.

1. Navigate to the Systems Manager Console

2. Create a New Hybrid Activation

In the Systems Manager navigation pane, choose Hybrid Activations under Node Management.
Click on Create Activation.
Fill in the following details:
- Activation Description: Provide a meaningful description, like "On-Premises Server Registration".
- Instance Limit: Set the number of on-premises servers you want to register.
- IAM Role: Choose or create an IAM role that has the necessary permissions for Systems Manager.
- Registration Expiration Date: Set the expiration date for this activation, after which it can no longer be used.
Click Create Activation.

3. Save the Activation Code and Activation ID

After creating the activation, you'll receive an Activation Code and Activation ID. Make sure to note these down, as you'll need them later to register your on-premises server.

Step 2: Onboard a Debian Server to AWS Systems Manager (SSM)

With your Hybrid Activation in hand, you can now register your Debian server with AWS Systems Manager.

1. Update Your Package List

Start by updating your server's package list:

sudo apt-get update

2. Install the SSM Agent

Next, download and install the SSM Agent:

mkdir /tmp/ssm
cd /tmp/ssm
wget https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/debian_amd64/amazon-ssm-agent.deb
sudo dpkg -i amazon-ssm-agent.deb

3. Register the Server with SSM

sudo amazon-ssm-agent -register -code "<your-activation-code>" -id "<your-activation-id>" -region "<your-region>"

For example:

sudo amazon-ssm-agent -register -code "h7FfWBbOrDCeXexxxxxx" -id "914e2266-e1c1-4c3a-b638-2azzzzzzzzzz" -region "eu-central-1"

4. Start the SSM Agent

Once registered, start the SSM Agent:

sudo systemctl start amazon-ssm-agent

5. Enable the SSM Agent to Start on Boot

Ensure the agent starts automatically on boot:

sudo systemctl enable amazon-ssm-agent

6. Verify the SSM Agent Status

Finally, confirm that the agent is running:

sudo systemctl status amazon-ssm-agent

Your Debian server should now be successfully registered with AWS Systems Manager, making it manageable through the AWS Management Console.

Step 3: Install and Configure Nginx (For Log Collection)

To generate some logs for the CloudWatch Agent, let’s install Nginx on the Debian server.

Install Nginx

sudo apt update
sudo apt install nginx
sudo systemctl status nginx

The Nginx log files that we’ll be sending to Cloudwatch are:

/var/log/nginx/error.log /var/log/nginx/access.log

Step 4: Install CloudWatch Agent Using SSM

Now, let’s use the SSM Agent to install the CloudWatch Agent on our server.

1. Access the Systems Manager Console

Open the Systems Manager console at AWS Systems Manager Console.

2. Run the Command to Install CloudWatch Agent

Navigate to Run Command.
Select AWS-ConfigureAWSPackage from the list of Command documents.
Choose the on-premises server as the target.
Set Action to Install.
Enter AmazonCloudWatchAgent in the Name box.
Leave the Version field blank to install the latest version.
Choose Run.

The CloudWatch Agent will now be installed on your server.

Step 5: Configure CloudWatch Agent

To enable the CloudWatch Agent to send logs from your on-premises server to AWS CloudWatch, you need to set up an IAM user with the necessary permissions, configure your server to use this IAM user's credentials, and ensure that the CloudWatch Agent is properly configured to use these credentials.

1. Create an IAM User with the Necessary Permissions

First, you'll need to create an IAM user that has permissions to send logs to CloudWatch.

Steps to Create the IAM User :

1. Log in to the AWS Management Console and open the IAM console.

2. Create a New User:

Navigate to Users and click on Add user.
Enter a user name (e.g., CloudWatchAgentUser).
Under Access type, select Programmatic access to generate an access key ID and secret access key for this user.

3. Assign Permissions:

Click on Attach policies directly.
Attach the following managed policies to the user:

CloudWatchAgentServerPolicy CloudWatchAgentAdminPolicy AmazonSSMManagedInstanceCore
These policies grant the necessary permissions to send logs to CloudWatch, access SSM, and interact with the CloudWatch Agent.

4. Complete the User Creation:

Proceed to review and create the user.
On the final page, make sure to download the .csv file containing the Access Key ID and Secret Access Key, or copy them to a secure location. You’ll need these credentials in the next step.

2. Configure the Server with IAM User Credentials

Now that you have the Access Key ID and Secret Access Key, you need to configure your server to use these credentials by creating an AWS CLI profile named AmazonCloudWatchAgent.

Configure AWS CLI with the IAM User Credentials:

On your on-premises server, run the following command to configure the AWS CLI with the IAM user credentials:

sudo aws configure --profile AmazonCloudWatchAgent

When prompted, enter the following details:

AWS Access Key ID: Enter the Access Key ID you obtained earlier.
AWS Secret Access Key: Enter the Secret Access Key.
Default region name: Enter the region where you want the logs to be sent (e.g., eu-central-1).
Default output format: Leave this field blank or enter json.

This creates a profile named AmazonCloudWatchAgent on your server that the CloudWatch Agent will use to send logs to AWS CloudWatch.

3. Update the CloudWatch Agent Configuration

If you're simulating an on-premises environment using an EC2 instance, you might need to update the CloudWatch Agent configuration file (common-config.toml) to use the newly created profile.

Update the Configuration File:

Open the common-config.toml file:

sudo nano /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

Uncomment and update the following section to include the profile name and credentials file:

[credentials]
shared_credential_profile = "AmazonCloudWatchAgent"
shared_credential_file = "/root/.aws/credentials"

Save and exit the file.

Step 6: Start CloudWatch Agent with a Pre-Created Config File Using SSM

In this step, we’ll use a configuration file created in advance and saved in the AWS Systems Manager Parameter Store to start the CloudWatch Agent.

1. Create and Save the Configuration File in SSM Parameter Store

First, create a CloudWatch Agent configuration file on your local machine. Here’s an example configuration:

{
  "agent": {
    "metrics_collection_interval": 60,
    "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/nginx/access.log",
        "log_group_class": "INFREQUENT_ACCESS",
            "log_group_name": "{instance_id}-nginx-access.log",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 7
          },
          {
            "file_path": "/var/log/nginx/error.log",
            "log_group_class": "INFREQUENT_ACCESS",
            "log_group_name": "{instance_id}-nginx-error.log",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 7
          }
        ]
      }
    }
  }
}

Once your configuration file is ready, save it to the AWS Systems Manager Parameter Store:

aws ssm put-parameter --name "CloudWatchAgentConfig" --type "String" --value file://configuration_file_pathname

Replace configuration_file_pathname with the actual path to your configuration file.

For more information about how to create cloudwatch agent configuration file, please visit Link

2. Access the Systems Manager Console

Return to the Systems Manager console.

3. Run the Command to Start CloudWatch Agent

Navigate to Run Command.
Select AmazonCloudWatch-ManageAgent from the Command documents.
Select the on-premises server as the target.
Set Action to configure.
Set Mode to onPremise.
In the Optional Configuration Location box, enter the name of the configuration file stored in the Parameter Store (e.g., CloudWatchAgentConfig).
Choose Run.

The CloudWatch Agent will now start with the specified configuration.

Verify CloudWatch Agent Logs
You can monitor the CloudWatch Agent's logs by running:

sudo tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Note on KMS Encryption for Log Groups

At the moment, the CloudWatch Agent does not support KMS encryption for log groups during their creation. The workaround is to allow the agent to create the log groups first and then manually associate them with a KMS key using the following command:

aws logs associate-kms-key --log-group-name LOG_GROUP_NAME --kms-key-id KEY_ARN

There is an ongoing feature request regarding this issue, which you can track here.

Conclusion

I created this blog post as a practical reference for anyone who needs to register on-premises servers with AWS Systems Manager (SSM) and install the CloudWatch Agent using SSM. Instead of having to sift through a whole bunch of AWS documentation each time you need to perform these tasks, you can use this guide to streamline the process. By following the steps outlined here, you can quickly and efficiently onboard your servers to SSM and configure the CloudWatch Agent to monitor and log your system's activity, ensuring you maintain visibility and control over your infrastructure, whether it's on-premises or in the cloud.

Additional References

SSM Agent PreInstalled AMI: Link
SSM Supported OS and Machine Types: Link
Install SSM Agent on Debian: Link
Check SSM Agent Status : Link

Cross-Account log data sharing using Kinesis Data Firehose

Peter Eskandar — Sun, 05 Feb 2023 00:35:53 +0000

Introduction :

Nowadays, One of the most fundamental security measures when working on a multi-account AWS environment is the ability to consolidate, manage, and analyze logs coming from various AWS Services in multiple accounts and multiple AWS Regions in one single place.

Solution Overview :

Based on the necessity to create a single Dashboard from where our Security Team can analyze all the logs coming from multiple AWS Application Accounts. We ended up with the following solution :

Using the above solution we were able to stream CloudWatch logs for a specific set of AWS Services (for example: AWS WAF) from multiple Application Accounts to a Centralized Log Archive Account.

Solution Setup :

let's start by setting up the required resources in our Centralized Log Account, then we will move to the Application Account

Centralized Log Account :

We are going to create the following resources :

1- A Centralized Kinesis Firehose Stream & S3 Bucket :

Using the AWS Console, create your Kinesis Firehose Stream by setting the Source as Direct Put and the Destination as Amazon S3.

You can choose an already existing Bucket as a destination otherwise you can create a new one.

For the rest you can keep the default values, otherwise you can follow the AWS Documentation for any further customizations : Link

2- IAM Role for CloudWatch Logs Destination :

To enable the CloudWatch Logs Destination which We are going to create in the next step to send data to the Kinesis Firehose Stream We already setup in the previous step, We need to create a IAM Role "CWLtoKinesisFirehoseRole" with the following permissions:

{
    "Statement":[
      {
        "Effect":"Allow",
        "Action":["firehose:*"],
        "Resource":["arn:aws:firehose:region:LogAccountID:*"]
      }
    ]
}

With the following Trust Relationships to allow the CloudWatch service to assume it :

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

3- CloudWatch Logs Destination :

The CloudWatch Logs Destination will work as the Access Point for your remote AWS Accounts to stream their logs to your centralized Kinesis Firehose in the Log Account.

The CloudWatch Logs Destination is a regional resource but can stream data to a Kinesis Firehose Stream in a different region, So you can create multiple CloudWatch Logs Destinations in different regions targeting your Centralized Kinesis Firehose Stream "This is what we've done in our case".

The necessity to create Multiple CloudWatch Logs Destinations in different regions is based on the regions from where you want to stream logs in your Application Accounts.

The Application Accounts should stream CloudWatch logs to the Centralized Log Account using the CloudWatch Logs Destination in the same region.

We are going to create it using the CLI as there is no way to do so using the AWS Console.

CLI Command :

aws logs put-destination --destination-name "demoFirehoseCrossAccount" --target-arn "arn:aws:firehose:region:LogAccountID:deliverystream/DEMO-FIREHOSE-CROSSACCOUNT" --role-arn "arn:aws:iam::LogAccountID:role/CWLtoKinesisFirehoseRole"

target-arn and role-arn are refering to the Kinesis Firehose stream and the IAM Role we've created in the previous steps.

Command Result :

        {
            "destination": {
                "destinationName": "demoFirehoseCrossAccount",
                "targetArn": "arn:aws:firehose:region:LogAccountID:deliverystream/DEMO-FIREHOSE-CROSSACCOUNT",
                "roleArn": "arn:aws:iam::LogAccountID:role/CWLtoKinesisFirehoseRole",
                "arn": "arn:aws:logs:region:LogAccountID:destination:demoFirehoseCrossAccount",
                "creationTime": 1675210769461
            }
        }

Save the command result somewhere as we are going to need the Destination ARN during resources creation in the Application Account.

4- CloudWatch Logs Destination Policy :

Each CloudWatch Logs Destination should have a policy attached to it.

Using the Destination Policy, you can decide which Remote Accounts can stream their logs through this CloudWatch Logs Destination.

Create a policy.json file with the following content, to allow remote Account/s to create CloudWatch Subscription Filters targeting the CloudWatch Logs Destination indicated in the Resource section of the Policy :

{ 
        "Version" : "2012-10-17", 
        "Statement" : [ 
            { 
                "Sid" : "", 
                "Effect" : "Allow",
                "Principal" : {
                   "AWS" : "ApplicationAccountID"
                },
                "Action" : "logs:PutSubscriptionFilter", 
                "Resource" : "arn:aws:logs:region:logAccountID:destination:demoFirehoseCrossAccount"
            } 
        ] 
    }

CloudWatch Subscription Filters are what we are going to create in the Application Accounts

The AWS Principal should be the Account ID/IDs of the Application Accounts and Resource is the ARN **of the **CloudWatch Logs Destination created in the previous step.

CLI Command :

aws logs put-destination-policy --destination-name "DEMO-FIREHOSE-CROSSACCOUNT" --access-policy "file://policy.json"

Application Account :

In your Application Accounts all you need to create is a CloudWatch Kinesis Firehose Subscription Filter for the CloudWatch Log Group you want to stream to your Centralized Log Account :

as a Destination choose Cross-Account and then insert the CloudWatch Logs Destination ARN for the one created in the Centralized Log Account :

Finally :

Now, We've all the required resources to stream CloudWatch Logs from multiple Application Accounts to a Centralized Log Account.

If you've the necessity to elaborate your data before storing it in S3, Kinesis Data Firehose can invoke a Lambda function to transform incoming source data and deliver the transformed data to destination LINK.

Helpful Links :

Delete Default VPC in multiple AWS Accounts using SNS & Lambda

Peter Eskandar — Wed, 18 Jan 2023 17:56:45 +0000

Problem Introduction :

When working on AWS to setup a large-scale Architecture based on your own Network "VPCs, Transit Gateways, Firewalls etc...", Usually you don't use the Default VPC created by AWS in every enabled region.

So, as a best practice the Default VPCs in the regions that you're using to deploy your application workload should be deleted.

Solution :

In our case, We've created an SNS Topic and a Lambda function in our Automation account.

We've added a subscription to trigger the Lambda function each time we publish something on the SNS Topic.

One of our Automation Pipelines will publish a message to the SNS Topic each time a new AWS Account is being created.

The Lambda function based on the information recieved from SNS "Usually the AccountID from where we want to delete the default VPCs" will assume a role in the destination account and then delete the Default VPCs.

Something similar to what is shown in the diagram below :

Lambda Permissions :

In the Automation Account, the Lambda function should have the permission to assume a role in the destination account :

        {
            "Sid": "AllowAssumeRole",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::*:role/DeleteDefaultVPCRole"
        }

Where DeleteDefaultVPCRole is the name of the IAM Role we are using in our Destination Account/s.

In the Destination Account , the DeleteDefaultVPCRole should have the following permissions, so that the Lambda can delete the Default VPC and all of its resources (IGW, Route Tables, Subnets and Security Groups) :

        {
            "Sid": "AllowDeleteDefaultVPCResources",
            "Effect": "Allow",
            "Action": [
                "ec2:DeleteVpc",
                "ec2:DetachInternetGateway",
                "ec2:DeleteInternetGateway",
                "ec2:DeleteSubnet",
                "ec2:DeleteRouteTable"
            ],
            "Resource": "*"
        }

Lambda Code :

The Lambda based on the following environment variable will operate in multiple regions and switch from one region to another in the destination account:

The Lambda will search for all the VPCs in different regions with the flag isDefault equals true.

Using the function getCrossAccountCredentials the Lambda will assume the DeleteDefaultVPCRole in the Destination account.

Below you can find the code in node.js "index.js & utils.js files"

index.js :


const aws = require('aws-sdk');
const utils = require('./utils')(aws);

exports.handler = async (event, context) => {

   try {
      // get accountId from the sns published message
      // the message should contain only the crossAccount accountId
      const snsEvent  = event['Records'][0]['Sns'];
      const accountId = snsEvent ? snsEvent['Message'] : null;
      // get list of all regions
      const regions = process.env.REGIONS.split(',');
      // get only the regions with default vpc <RegionName, VpcId>
      let regionVpcMap = accountId ? await deleteDefaultVPCByRegion(regions, accountId) : null;
      return regionVpcMap
   } catch (e) {
      console.log(e);
   }

};

// get Current default VPC and start deleting all dependencies
async function deleteDefaultVPCByRegion(regions, accountId) {
   let regionVpcMap = new Map();
   let vpcParams = { Filters: [{'Name' : 'isDefault', 'Values' : ['true']}]}
   for(const region of regions)
   {
      try {
         const token = await utils.getCrossAccountCredentials(`arn:aws:iam::${accountId}:role/DeleteDefaultVPCRole`)
         const regionalEC2 = new aws.EC2({ region: region, credentials: token });
         const result = await regionalEC2.describeVpcs(vpcParams).promise();
         console.log(result, region, regions);
         if(result['Vpcs'].length > 0) {
            regionVpcMap.set(region, result['Vpcs'][0]['VpcId'])
         /**
              - Action to follow : 
              1.) Detach & Delete the internet-gateway
              2.) Delete subnets
              3.) Delete route-tables if not main
              4.) Delete security-groups if not default
              5.) Delete the VPC 
         **/  

            await utils.deleteInternetGateway(regionalEC2, result['Vpcs'][0]['VpcId']);
            await utils.deleteSubnetsDefaultVPC(regionalEC2, result['Vpcs'][0]['VpcId']);
            await utils.deleteSGDefaultVPC(regionalEC2, result['Vpcs'][0]['VpcId']);
            await utils.deleteRouteTablesDefaultVPC(regionalEC2, result['Vpcs'][0]['VpcId']);
            await utils.deleteDefaultVPC(regionalEC2, result['Vpcs'][0]['VpcId']);
         } else {
            console.log('No Default VPCs found !');
         }
      } catch (e) {
         throw new Error(e);
      }
   }

   return regionVpcMap.size > 0 ? regionVpcMap : null;
}

utils.js :

module.exports = (AWS) => {
   // get crossAccount temp Credentials
   async function getCrossAccountCredentials (roleArn)  {
      const sts = new AWS.STS();
      const timestamp = (new Date()).getTime();
      const params = {
        RoleArn: roleArn,
        RoleSessionName: `VPC-Deleter-${timestamp}`,
      };
      const assumeRoleResult = await sts.assumeRole(params).promise();
      return {
        accessKeyId: assumeRoleResult.Credentials.AccessKeyId,
        secretAccessKey: assumeRoleResult.Credentials.SecretAccessKey,
        sessionToken: assumeRoleResult.Credentials.SessionToken,
      }
    }

   // delete  Internet Gateway for attached to the default VPC
   async function deleteInternetGateway(ec2, vpcId) {
       const params = {Filters: [ { Name: "attachment.vpc-id", Values: [vpcId]}]};
       try {
         const result =  await ec2.describeInternetGateways(params).promise();
         for(const igw of result['InternetGateways']){
             // detach IGW from VPC
             await ec2.detachInternetGateway({ InternetGatewayId: igw['InternetGatewayId'],  VpcId: vpcId}).promise();
             // delete IGW
             await ec2.deleteInternetGateway({ InternetGatewayId: igw['InternetGatewayId']}).promise();
         }
       } catch (e) {
          throw new Error(`Unable to detach IGW for VPC - ${vpcId} - reason :  ${e}`)
       }
      }


      // delete all subnets related to the default VPC
      async function deleteSubnetsDefaultVPC(ec2, vpcId) {
      const params = {Filters: [ { Name: "vpc-id", Values: [vpcId]}]};
       try {
         const subnetsResult =  await ec2.describeSubnets(params).promise();
         for(const subnet of subnetsResult['Subnets']){
           // delete each subnet attached to the current default VPC   
           await ec2.deleteSubnet({SubnetId: subnet['SubnetId']}).promise();
         }
       } catch (e) {
          throw new Error(`Unable to delete subnets for VPC - ${vpcId} - reason :  ${e}`);
       }
      }


      // delete all security groups related to the default VPC
      async function deleteSGDefaultVPC(ec2, vpcId) {
      const params = {Filters: [ { Name: "vpc-id", Values: [vpcId]}]};
       try {
          const sgResult = await ec2.describeSecurityGroups(params).promise();
          for(const sg of sgResult['SecurityGroups']){
             if(sg['GroupName'] !== 'default')
               await ec2.deleteSecurityGroup({GroupId: sg['GroupId']}).promise();
         }
       } catch (e) {
          throw new Error(`Unable to delete Security Groups for VPC - ${vpcId} - reason :  ${e}`);
       }
      }


      // delete all route tables related to the default VPC
      async function deleteRouteTablesDefaultVPC(ec2, vpcId) {
      const params = {Filters: [ { Name: "vpc-id", Values: [vpcId]}]};
       try {
          const rtbResult = await ec2.describeRouteTables(params).promise();
          for(const rtb of rtbResult['RouteTables']){
             if(!rtb['Associations'].length ) {
                 await ec2.deleteRouteTable({RouteTableId: rtb['RouteTableId'] }).promise();
             }
         }
       } catch (e) {
          throw new Error(`Unable to delete Route Tables for VPC - ${vpcId} - reason :  ${e}`);
       }
      }

      // delete default VPC
      async function deleteDefaultVPC(ec2, vpcId) {
       try {
          await ec2.deleteVpc({VpcId: vpcId}).promise();
          console.log(`Default VPC ${vpcId} has been deleted`);
       } catch (e) {
          throw new Error(`Unable to delete Default VPC - ${vpcId} - reason :  ${e}`);
       }
      }

      return {
        getCrossAccountCredentials, 
        deleteInternetGateway,
        deleteSubnetsDefaultVPC,
        deleteSGDefaultVPC,
        deleteRouteTablesDefaultVPC,
        deleteDefaultVPC
      }

   }

The End

Hope this article hepls you to organize your AWS Accounts in a better way

If you end up here, I want to say, thank you for reading this article! 🙌

Peter

VPC Interface endpoints sharing in a Multi-Account Organization

Peter Eskandar — Mon, 05 Dec 2022 13:56:24 +0000

When you start working on a Multi-Account Organization in AWS, there are two things that you will think about most of the time :

Security
Cost Optimization

So, to secure connections between your VPC and AWS Services you can do so by using the VPC interface endpoints, they will allow your resources to connect to AWS Services without the need of an Internet Gateway, NAT device, VPN or a AWS Direct Connect connection.

Instances in your VPC won’t use public IP addresses to communicate with AWS services, instead it uses VPC Endpoint for that.

But when you start thinking about using them, the first thing that comes to your mind is :

VPC Interface Endpoints Pricing :
Below you can find and example of how much 5 VPC Interface endpoints will cost you for 10 GB of total data processed by all VPCE Interface endpoints in the AWS region.

Actually too much money, specially if your going to use them in each account of your AWS Organization

So, there is a way to share them from one Account with other Accounts ?

Yes, there are multiple ways to do so and today I’m going to share with you one of the ways we’ve already used in our AWS Organization.

Let’s get to work
We can say that you have an Architecture similar to the one shown below, one Network/Shared Services Account, multiple Application Accounts where you’ve most of your workload, multiple VPCs, multiple VPC Interface endpoints in each VPC per Account and everything is connected together using a Transit Gateway.

What we are going to do ?

We are going to keep the VPC Interface endpoints in our Shared Services/Network Account, working on sharing them with the other accounts in our Organization and then remove the interface endpoints from the other accounts.

Which AWS Services will help us to do so ?

Route53
Resource Access Manager

So, let’s move to our account from where we want to share the interface endpoints.

When you create an Interface endpoint (for example for : EC2 Service), AWS automtically enables the Private DNS Name for this endpoint “ec2.eu-west-3.amazonaws.com” as shown below :

to be able to share this endpoint with the other accounts we need to disable the Private DNS Name so we can create a private hosted zone on Route53 for it.

to do so, just select the interface endpoint and go to Actions and then select Modify private DNS Name and disable it

Now, let’s go to Route53 to create a private hosted zone for our interface endpoint with it’s Private DNS Name, associate it to the VPC where we’ve already created the interface endpoint and then create a DNS record which will target our VPC endpoint as show below :

What are going to do next ?

In our centralized account we need setup some other few things :

Inbound Resolver : The inbound resolver will receive queries forwarded from other VPCs’ DNS servers and from workloads running in participating AWS accounts. The Inbound resolver should be created on the same VPC where we’ve created the interface endpoint and associated to the Route53 Private hosted zone.

DNS queries reaches the default DNS server of our VPC and because the VPC is associated with the private hosted zone ec2.eu-west-3.amazonaws.com, the default DNS server will be able to resolve this domain.

Outbound Resolver : The Outbound Resolver will forward the DNS queries coming from the other VPCs to the IP addresses of the Inbound Resolver.

Resolver Rules : indicates that queries for ec2.eu-west-3.amazonaws.com should be forwarded through the Outbound Resolver.

I’ve created a rule called “ec2 rule” for the VPC interface endpoint Private DNS Name “ec2.eu-west-3.amazonaws.com” and targeting the IP Addresses of our Inbound Resolver

Resolver rules is what we are going to share with the other accounts in our organization.

Using Resource Access Manager, you can create a resource share and then share the resolver rule/rules with all the accounts in your organization

Final Step :

Now move to the other accounts, delete the EC2 VPC Interface endpoint and then switch to Route53.

Under Rules section, you can find the resolver rule already shared with you from the centeralized account, what you need to do now is just to associate your VPCs in this account with the resolver rule so that they will be able to resolve the ec2.eu-west-3.amazonaws.com using the DNS Server of the centeralized VPC in the Shared Services/Network Account.

IMPORTANT:

VPC interface endpoints, resolver rules, inbound & outbuond resolvers are all regional services, so if you want to share VPC interface endpoints in multiple regions, then you’ve to replicate everything.

In this case you will need to create multiple interface endpoints in different regions in the centeralized account, multiple private hosted zones (for example: ec2.eu-west-3.amazonaws.com & ec2.eu-central-1.amazonaws.com) targeting the endpoint in that region and multiple resolver rules one for each domain name.

This blog Post helped us during the setup of this solution : https://aws.amazon.com/blogs/security/simplify-dns-management-in-a-multiaccount-environment-with-route-53-resolver/

Hope this will help you to save some money on AWS 💰

Thanks

Peter