DEV Community: kranthi

[Boost]

kranthi — Thu, 05 Jun 2025 07:05:04 +0000

kranthi

Jun 5 '25

Mastering AWS Storage Services: S3, EBS, and EFS

Comments

6 min read

Mastering AWS Storage Services: S3, EBS, and EFS

kranthi — Thu, 05 Jun 2025 06:56:11 +0000

1. Amazon S3 (Simple Storage Service)

• Object storage designed to store and retrieve any volume of data from anywhere on the web.
• Provides 99.999999999% durability across multiple AZs.
• Ideal for backups, data lakes, static websites, machine learning datasets, and log archives.

Amazon S3 Storage Classes

1. S3 Standard
This is the default and most commonly used storage class in Amazon S3. It's ideal for data that is frequently accessed, such as active application files, dynamic websites, and hot data analytics. It offers high availability, millisecond retrieval, and 11 nines (99.999999999%) of durability.
Security features include SSE-S3, SSE-KMS encryption, IAM policies, ACLs, and bucket policies.

2. S3 Intelligent-Tiering
Designed for data with unpredictable access patterns, this class automatically moves objects between frequent and infrequent tiers based on usage. It offers the same high availability and millisecond latency as S3 Standard and maintains 11 nines of durability.
It supports SSE-KMS encryption and features like Object Lock for compliance and immutability. Ideal when you want to optimize cost without affecting performance.

3. S3 Standard-IA (Infrequent Access)
This class is perfect for data that is not accessed frequently but must be quickly retrievable when needed—like backups or older logs. It provides high availability, millisecond retrieval times, and 11 nines of durability.
Security options include MFA Delete, SSE-KMS encryption, and fine-grained access controls. It is more cost-effective than S3 Standard, but retrieval costs apply.

4. S3 One Zone-IA
This is a cheaper version of Standard-IA but stores data in a single Availability Zone instead of multiple ones. It’s suitable for non-critical, infrequently accessed data like secondary backups or easily reproducible data.
While it maintains millisecond access latency and 11 nines durability, it has lower availability and high risk if that one AZ fails. Encryption and access features include SSE-KMS, Object Lock, and IAM controls.

5. S3 Glacier Instant Retrieval
Used for archival data that still requires instant access (e.g., medical images, financial records). It combines low-cost storage with millisecond retrieval, while still offering 11 nines durability.
You get the benefits of Object Lock, SSE-KMS, and legal hold capabilities. Ideal when long-term storage is needed, but latency cannot be compromised.

6. S3 Glacier Flexible Retrieval
Formerly known as "S3 Glacier", this class is great for long-term archives that are rarely accessed but occasionally needed. Retrieval times range from minutes to hours, and it’s cheaper than Instant Retrieval.
It provides 11 nines durability, high availability, and supports event-based restore, encryption via SSE-KMS, and audit control through logging and monitoring tools.

7. S3 Glacier Deep Archive
The lowest-cost storage option for rarely accessed data—like compliance archives or historical records. Retrieval times take hours, but it still guarantees 11 nines durability and high availability.
Used with audit logging, SSE-KMS, and lifecycle policies, it’s perfect for long-term cold storage where retrieval time is not urgent.

Advanced Features:
• Lifecycle Policies: Automate transitions between classes to reduce costs.
• Versioning & MFA Delete: Protects against accidental overwrites/deletions.
• S3 Object Lock: WORM (Write Once Read Many) for regulatory compliance.
• Cross-Region Replication (CRR): For DR and latency optimization.
• Event Notifications: Trigger Lambda, SNS, or SQS for automated workflows.

Security Best Practices:
• Enforce least-privilege access using IAM policies.
• Enable default encryption (SSE-KMS) at the bucket level.
• Block all public access unless explicitly required.
• Use S3 Access Analyzer and AWS Config rules for compliance checks.

When to Use S3:
• Hosting static content like HTML/CSS for websites.
• Storing backups, logs, or data for ML pipelines.
• Archiving compliance data (e.g., logs, financial records).

2. Amazon EBS (Elastic Block Store)

• High-performance block storage used in conjunction with EC2.
• Provides persistent storage with low latency.
• Supports dynamic scaling and high IOPS workloads.

Amazon EBS Volume Types

1. gp3 (General Purpose SSD)
The gp3 volume is the default choice for most general-purpose workloads on AWS. It is designed for use cases like boot volumes, small databases, and development/test environments. gp3 offers up to 16,000 IOPS and 1,000 MB/s throughput, regardless of volume size, making it more performant and cost-effective than its predecessor, gp2. It’s an SSD-backed volume, supporting encryption, snapshots, and provisioned IOPS, allowing you to tailor performance to workload needs without increasing capacity.

2. io2 and io2 Block Express (Provisioned IOPS SSD)
These high-performance SSD volumes are built for mission-critical applications such as large-scale relational and NoSQL databases, SAP HANA, and latency-sensitive transactional workloads. io2 supports up to 256,000 IOPS and 4,000 MB/s throughput, especially with Block Express architecture, which delivers consistent sub-millisecond latency. They offer higher durability (99.999%), multi-attach capability, and enhanced resiliency, making them ideal for enterprise-grade deployments. Encryption and snapshot support are fully integrated.

3. st1 (Throughput Optimized HDD)
st1 volumes are HDD-based and designed for high-throughput workloads such as big data, data warehouses, and log processing systems. With up to 500 MB/s throughput and 500 IOPS, they are a cost-effective option for workloads that require sequential access over random I/O. While not ideal for boot volumes or transactional databases, st1 is a good fit for large-scale data lakes or analytical platforms.

4. sc1 (Cold HDD)
The sc1 volume is optimized for infrequently accessed data, offering the lowest-cost magnetic storage on EBS. It is suitable for archival workloads, large-volume cold storage, and backups that are rarely retrieved. Performance is lower than st1, with up to 250 IOPS and 250 MB/s throughput, making it unsuitable for active use cases but valuable for minimizing costs in long-term storage scenarios.

5. Magnetic (Standard – Deprecated)
This legacy magnetic volume type is still available for older EC2 instances or legacy applications that were designed with it. It offers low performance, both in terms of throughput and IOPS, and is not recommended for new workloads unless absolutely required for compatibility reasons. AWS recommends moving to gp3 or st1 for modern applications.

Advanced Features:
• Snapshots: Point-in-time backups that can be copied across regions.
• Encryption: Fully managed using AWS KMS.
• Elastic Volumes: Modify size, IOPS, or type without downtime.
• Multi-Attach: Attach io2 volumes to multiple instances (Linux only).

Security Best Practices:
• Always encrypt volumes using KMS.
• Implement automated snapshots using AWS Backup.
• Enable CloudTrail for auditing volume operations.
• Use tags for cost allocation and access control.

When to Use EBS:
• Boot volume for EC2 instances.
• Databases like MySQL, PostgreSQL, Oracle.
• Applications requiring high-throughput and low-latency.

3. Amazon EFS (Elastic File System)

• Managed NFS-based file system that auto-scales to petabytes.
• Accessible from multiple EC2s, ideal for shared workloads.
• Available in multiple AZs for HA.

EFS Performance & Throughput Modes

General Purpose Mode
This is the default performance mode for Amazon EFS. It's optimized for low-latency operations and is ideal for workloads like CMS platforms, developer home directories, web serving, and shared development environments. This mode supports a burst model, allowing short-term high throughput and IOPS for small to medium-sized workloads, providing a great balance between cost and performance.
Max I/O Mode
The Max I/O performance mode is designed for highly parallel and large-scale workloads, such as big data analytics, media processing pipelines, and machine learning datasets. While it provides higher aggregate throughput and IOPS, it may introduce slightly higher latencies compared to General Purpose mode. It's best suited for environments where scalability is more important than single-instance latency.

Storage Classes:
• Standard: Primary tier for hot data.
• IA (Infrequent Access): Lower-cost tier for cold data.

Lifecycle Management:
• Automatically moves files between Standard and IA based on access
patterns.

Security and Compliance:
• Supports encryption in transit and at rest (using KMS).
• IAM policies + VPC security groups for access control.
• POSIX file permissions for Linux workloads.

When to Use EFS:
• Shared storage for containers, Lambda, EC2.
• Hosting WordPress, MediaWiki, or similar.
• Real-time data processing or analytics.

Best Practices:
• Use lifecycle management to control costs.
• Enable logging with CloudTrail.
• Choose the right performance mode based on workload.

Conclusion
Choosing the right AWS storage service depends on workload type, access
pattern, and cost-performance tradeoffs.

• S3 is unmatched for object storage, backups, and analytics.
• EBS is critical for applications needing block-level performance and
persistence.
• EFS simplifies scalable file sharing and NFS-based workloads.
Always benchmark performance, enable encryption, and apply access control
policies across all services for a secure and optimized architecture.

Mastering Amazon EC2

kranthi — Thu, 05 Jun 2025 06:34:44 +0000

1. What is Amazon EC2?
Amazon EC2 (Elastic Compute Cloud) is a web service that provides resizable compute capacity in the cloud. It’s designed to make web-scale cloud computing easier for developers. You can launch virtual servers (instances), configure networking and security, and manage storage.

Core Capabilities:

Launch and terminate instances on demand.
Choose AMIs to define OS and software.
Select instance types based on workload.
Automate scaling and high availability.
Integrate with Elastic Load Balancer and Auto Scaling.

2. EC2 Instance Lifecycle

Pending – Instance is being launched.
Running – Instance is active.
Stopping – Instance is shutting down.
Stopped – Instance is off but data is intact.
Terminated – Instance is permanently deleted.
EC2 Instance Types (With Use Cases)

General Purpose
• t4g, t3, t2 – Low-cost, burstable performance.
• Use Case: Development, testing, web servers.
Compute Optimized
• c7g, c6g, c5 – High-performance CPU.
• Use Case: Batch processing, game servers, ML inference.
Memory Optimized
• r6g, r5, x2idn – Large memory capacity.
• Use Case: In-memory DBs, real-time analytics.
Storage Optimized
• i4i, d3en, h1 – High IOPS and throughput.
• Use Case: NoSQL DBs, big data workloads.
Accelerated Computing
• p4, inf2, g5 – GPU-based.
• Use Case: AI/ML, video processing, 3D rendering.

4. Purchasing Options

On-Demand Instances
• No upfront cost. Pay per second.
• Ideal for unpredictable workloads.
Reserved Instances (RIs)
• Commitment (1 or 3 years).
• Up to 75% cost savings.
• Convertible or Standard RIs.
Savings Plans
• Commitment on usage (EC2, Fargate, Lambda).
• More flexibility than RIs.
Spot Instances
• Use spare capacity.
• Up to 90% discount.
• Suitable for fault-tolerant workloads.
Dedicated Hosts
• Physical server for your use.
• Bring Your Own License (BYOL).

5. Elastic Load Balancing (ELB)
Distributes traffic to multiple EC2s.
Types:
• Application Load Balancer (ALB): HTTP, Web apps.
• Network Load Balancer (NLB): TCP, low latency.
• Gateway Load Balancer (GWLB): Third-party appliances.

Features:
• Health checks.
• Sticky sessions.
• SSL termination.

6. EC2 Auto Scaling
Auto Scaling ensures availability and cost efficiency.
Components:
• Launch Template/Config
• Auto Scaling Group (ASG)
• Scaling Policies: Target, step, scheduled

Advanced Strategies:
• Predictive scaling
• Lifecycle hooks
• Warm pools

7. EC2 Storage and Volumes
EBS (Elastic Block Store):
• gp3: General purpose
• io1/io2: High IOPS
• st1/sc1: Throughput and archival

Instance Store:
• High-speed, ephemeral
• Data lost on stop/terminate

EFS (Elastic File System):
• NFS file system
• Scalable across multiple EC2

Amazon FSx:
• Windows File Server, Lustre, NetApp ONTAP

8. AMIs and Snapshots
Amazon Machine Image (AMI):
• Template to launch instances
• Includes OS, configuration, apps

Snapshots:
• Point-in-time backups of EBS volumes
• Used to create AMIs

Golden AMI Strategy:
• Hardened base image
• Pre-installed apps and security settings

9. Backup Strategies

EBS Snapshots:
• Manual or scheduled
• Incremental backups

AMI-Based Backups:
• Save complete OS and data state

AWS Backup:
• Centralized backup across services

Cross-region Backup:
• Enable DR and compliance

10. EC2 Security Best Practices
Key Concepts:
• Security Groups: Instance-level firewall
• NACLs: Subnet-level control
• Key Pairs: SSH authentication
• IAM Roles: Secure access to AWS services
• SSM: Secure shell-free management

Hardening Tips:
• Disable root login
• Regular patching
• Use least privilege IAM policies

11. Monitoring and Logging

CloudWatch:
• EC2 metrics, custom alarms
• Log agent for file-level monitoring

CloudTrail:
• Record API activity

EC2 Detailed Monitoring:
• 1-minute interval metrics

AWS Config:
• Audit and compliance checks

12. Server Connectivity & Session Management
Linux:
• SSH with PEM key
• SSM Session Manager
Windows:
• RDP (Remote Desktop Protocol)
• EC2 Connect for browser-based access
Connection Tools:
• Putty, Mobaxterm, VS Code SSH plugin

13. High Availability and Cost Optimization
High Availability:
• Deploy in multiple AZs
• Use Load Balancer + Auto Scaling
• Elastic IP for static access

Cost Optimization:
• Choose right instance type
• Use Spot and RIs where suitable
• Use Auto Scaling to scale down
• Schedule non-production shutdown

14. Real-Time Use Cases

Web Hosting: Scalable app hosting with Auto Scaling + ALB
Batch Jobs: Use Spot Instances for cost-effective processing
Gaming: Low-latency game server on EC2
CI/CD Runners: Host Jenkins or GitHub runners
Dev/Test Environments: Spin up/down quickly
AI/ML Training: Use GPU-based instances
EC2 Interview Questions
Basic:
• What is EC2?
• How do you connect to EC2?
• What are the differences between AMI and Snapshot?
Intermediate:
• How does Auto Scaling work?
• Difference between Security Group and NACL?
• What is the difference between instance store and EBS?
Advanced:
• Design a fault-tolerant EC2 architecture.
• How do you implement patch management in EC2?
• How do you maintain golden AMIs?
• How do you monitor 100+ EC2 instances efficiently?
```
                            © 2025 Kranthi – AWS Community Builder
```

Automating detection and alerts of VPC endpoint changes using AWS CloudTrail, EventBridge, Lambda, SNS, to Email.

kranthi — Wed, 21 May 2025 12:43:14 +0000

kranthi

May 21 '25

VPC Endpoint Monitoring & Alerting

Comments

8 min read

VPC Endpoint Monitoring & Alerting

kranthi — Wed, 21 May 2025 12:42:26 +0000

Automating detection and alerts of VPC endpoint changes using AWS
CloudTrail, EventBridge, Lambda, SNS, to Email.

This demo demonstrates the automated monitoring of VPC
endpoint events. The goal is to quickly detect when VPC endpoints are created, modified, or deleted and to alert the operations team in real time.

Business Rationale:
In production environments, immediate detection of changes can help prevent configuration drift, reduce security risks, and ensure compliance.

Scope:
The PoC covers event capture using CloudTrail, event filtering with EventBridge, message formatting with Lambda, and alert distribution via SNS to email (with future integration to Microsoft Teams).

AWS CloudTrail:
Captures API calls made in your AWS account. All VPC endpoint events (CreateVpcEndpoint, ModifyVpcEndpoint, DeleteVpcEndpoints) are logged.

Amazon EventBridge:
Filters CloudTrail events using a specific event pattern. Only relevant VPC endpoint events trigger the next step.

AWS Lambda:
A Lambda function is triggered by EventBridge. This function formats the raw CloudTrail event data into a clear alert message.

Amazon SNS:
The formatted alert is published to an SNS topic, which then distributes notifications to subscribed email addresses.

             **_Step-by-Step Implementation_**

Step 1: Create the Sample VPC and Network Components

Create a VPC: • Navigate to VPC Console → Create VPC • Enter details (e.g., CIDR 12.0.0.0/16) and name it “endpoint-vpc.”
Create Subnets: • Create at least two subnets in different Availability Zones.
Create a Security Group: • Define necessary inbound/outbound rules.
Create a Route Table: • Associate with the VPC for gateway-type endpoints.

Added an inbound rule (TCP 443) to the security group and attached it to the VPC endpoint for secure HTTPS communication.

                     **_CloudTrail Setup_**

Step 1: Create a New Trail
• Navigated to AWS CloudTrail service.
• Clicked on “Create trail”. • Provided a Trail name (e.g., VpcEndpointMonitoringTrail).
• Selected “Create new S3 bucket” as the storage location.
• Entered an appropriate bucket name (e.g., vpc-endpoint-logs-kranthi).
• Enabled Log file validation for added integrity checks.

Step 2: Enable CloudWatch Logs Integration
• Under CloudWatch Logs, enabled “Send to CloudWatch Logs”. • Selected “Create new IAM role”.
• Assigned a recognizable role name (CloudTrailDeliveryRoleVpcEndpoint).
• This allows CloudTrail to send logs to CloudWatch for real-time
monitoring and alerting.
• Reviewed all settings, “Create trail”.

• Confirmation message appeared stating the trail was created
successfully.

Step 4: Verify S3 Bucket and Log Delivery
• Navigated to the S3 console.
• Confirmed that the bucket (vpc-endpoint-logs-kranthi) was successfully created.
• Verified that log files were being delivered to the bucket under the
specified prefix structure ( AWSLogs//CloudTrail/).

Amazon SNS Topic for Alerts setup
Create Amazon SNS Topic for Alerts
• Navigated to the Amazon SNS console.
• Chose to create a new topic.

1.Selected Standard type.
• Standard topics offer high throughput and best-effort ordering.
• In contrast, FIFO topics provide strict ordering and exactly-once message delivery, but with lower throughput.
2.Provided a name: vpc-endpoint-alert
3.Create SNS topic.

4.Create SNS Subscription
• Clicked on the newly created topic to create a subscription.

• Set the protocol as Email.
• Entered the email address as the endpoint (alerts@yourdomain.com).
• Used the Topic ARN of the previously created SNS topic.
• Subscription created successfully.
• Verified that a confirmation email was sent to the specified email address.

• Opened the subscription email and clicked the confirmation link to approve the SNS subscription.
• The email endpoint was successfully confirmed to receive notifications.

Steps to Create Amazon EventBridge Rule
1.Navigate to EventBridge Console
• Open the AWS Management Console and go to Amazon EventBridge.

2.Create Rule
• Click on "Create rule" and proceed to define the rule details.
3.Define Rule Details
• Enter rule name: vpcendpointchangerule
• Select Event Source as "Event Source with an event pattern"

3.Configure Event Pattern
• Choose "Build event pattern" > Other
• Select Custom pattern (JSON editor)

{
 "source": ["aws.ec2"],
 "detail-type": ["AWS API Call via CloudTrail"],
 "detail": {
 "eventSource": ["ec2.amazonaws.com"],
 "eventName": [
 "CreateVpcEndpoint",
 "ModifyVpcEndpoint",
 "DeleteVpcEndpoints"
 ]
 }}

• Write custom event pattern that listens to VpcEndpoints actions.

4.Add Target
• Select Target 1 as AWS Service, Choose SNS topic.
• Select the previously created topic: vpc-endpoint-alerts.
• Choose to Create a new execution role.
• Define a recognizable role name for easy identification.

5.Review and Create

• The final image confirms that the EventBridge rule vpcendpointchangerule was successfully created and is now active.

                  **_About VPC Endpoints_**

What is a VPC Endpoint?
A VPC endpoint enables private connectivity between your VPC and supported AWS services or VPC endpoint services, powered by PrivateLink, without requiring internet access, NAT gateway, or VPN.
Use Cases
• Access Amazon S3 or DynamoDB from private subnets.
• Enhance security by keeping traffic within the AWS network.
• Reduce latency and data transfer costs. • Meet compliance and audit requirements for isolated networks.

   **_VPC Endpoint Creation & Email Notification Testing_**

Navigate to VPC Console • Open the AWS Management Console and go to the VPC service. • In the Endpoints section, observe that no endpoints currently exist.

Create a New VPC Endpoint • Click “Create endpoint”. • For Service category, select AWS Services. • In Service name, choose any supported AWS service.

3. Configure VPC Settings
• Choose the VPC you created in the initial step of the POC.
• Select appropriate subnets and route tables associated with your VPC.
• Leave the default options or customize Policy as needed.
• Click “Create endpoint”.

4. Trigger and Validate Email Notification
• After creating the endpoint, the EventBridge rule triggers an SNS notification.
• Check the configured email inbox and observe that anotification was received.
• The notification, however, appears unstructured and raw, making it difficult to read or parse.

Lambda Integration for Structured Notifications

Why Use AWS Lambda?
AWS Lambda lets you run backend code without provisioning or managing servers. It's event-driven and integrates well with other AWS services.
Common Use Cases for Lambda:
• Format and forward notifications.
• Filter or enrich incoming events.
• Automatically remediate certain changes.
• Connect multiple AWS services efficiently.
Purpose in This POC
In this POC, Lambda will:
• Process the raw EventBridge message.
• Convert it into a structured and readable format.
• Send the refined output to the SNS topic, which emails it to recipients.

 **_Lambda Setup for Structuring Notifications_**

1. Create Lambda Function
• Navigate to the AWS Lambda Console and click “Create function.”
• Select Author from scratch.
• Enter the function name: formatvpcendpoint.
• Choose Python 3.13 as the runtime.
• Under Permissions, select Create a new role with basic Lambda
permissions.

2. Add Notification Formatting Code
• Paste the Python code that:
Parses the incoming EventBridge JSON -->Formats key fields (action, time, VPC ID) -->Publishes the structured message to the SNS topic --> Save your changes.

import os
import boto3
sns = boto3.client('sns')
def lambda_handler(event, context):
 topic_arn = os.environ.get('SNS_TOPIC_ARN')

 if not topic_arn:
 print(" SNS_TOPIC_ARN environment variable is not set.")
 return
 # build your message as you did
 message = (
 " *VPC Endpoint Alert*\n"
 f" Time: {event['time']}\n"
 f" Region: {event['region']}\n"
 f" User: {event['detail']['userIdentity'].get('userName', 'N/A')}\n"
 f" Action: {event['detail']['eventName']}\n"
 f" VPC ID: {event['detail']['requestParameters'].get('VpcId', 'N/A')}\n"
 f" Service: {event['detail']['requestParameters'].get('ServiceName', 'N/A')}\n"
 f"Check AWS Console for more details."
 )
 print("Formatted message:")
 print(message)
 sns.publish(
 TopicArn=topic_arn,
 Message=message,
 Subject=" VPC Endpoint Action Alert"
 )

3. Deploy and Test the Function
• Click “Deploy” to publish your function changes.
• Use the “Test” feature to run the function with sample EventBridge event data and verify the output is formatted as expected.

Configure Environment Variables • Go to the Configuration tab → Environment variables. • Add a key-value pair: o Key: SNS_TOPIC_ARN o Value: (Paste the ARN of the SNS topic created earlier, arn:aws:sns:ap-southeast-1:xxxxxxx:vpc-endpoint-alert)

Update EventBridge Rule Target
• Go to Amazon EventBridge → Rules → Select your rule vpcendpointchangerule.
• Edit the rule’s target:
o Change the target from SNS to Lambda function.
o Select the Lambda function formatvpcendpoint.
• Save the updated rule configuration.

Important Troubleshooting Step – Resolving Lambda Permission Issue
I successfully created a VPC endpoint, but did not receive any email notification.
This happened because the newly created Lambda function role lacked authorization to publish to the SNS topic.

Handling Lambda Permission Error for SNS
Step 1: Identifying the Issue
• After deploying the Lambda function and triggering the EventBridge rule, CloudWatch Logs showed an error indicating sns:Publish permission was denied.
• This was an expected error, as the Lambda execution role was created with only basic Lambda permissions.

     **_2: Resolving the Permission Issue: _**

• Created the following inline policy to allow the Lambda function to
publish messages to the SNS topic:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "sns:Publish"
 ],
 "Resource": [
 "arn:aws:sns:ap-southeast-1:<AWS account number>:vpc-endpoint-alerts"
 ]
 }
 ]}

• Attached this policy to the Lambda execution role.

Step 3: Verification
• Triggered a new VPC Endpoint creation to test the fix.
• Checked CloudWatch Logs – the authorization error was resolved.

• A structured email notification was received as expected, confirming end-to-end functionality.

Build a Multi-Tenant Monitoring System with RDS, Athena & Glue – Step-by-Step!

kranthi — Fri, 25 Apr 2025 08:35:02 +0000

kranthi

Apr 25 '25

Building a Multi-Tenant Cost Tracking & Performance Monitoring Solution with Amazon RDS, Athena, and Glue

#devops #aws #cloud #database

Comments

15 min read

Building a Multi-Tenant Cost Tracking & Performance Monitoring Solution with Amazon RDS, Athena, and Glue

kranthi — Fri, 25 Apr 2025 08:32:08 +0000

Welcome! In this blog, we'll guide you through creating a multi-tenant performance and cost tracking solution using Amazon RDS (PostgreSQL), Athena, AWS Glue, and supporting AWS services. This solution is perfect for SaaS products requiring centralized monitoring, log analysis, and schema-level tenant isolation.

Architecture Diagram - Multi-Tenant RDS Cost & Performance Analytics.

Step 1: Networking Setup

Create a secure environment for hosting the RDS instance:

VPC CIDR: 10.0.0.0/16
Subnets: Create at least three across different Availability Zones
Internet Gateway: Attach and update routing
Route Table: Add custom routes for outbound traffic
DNS Hostnames: Enable to facilitate RDS endpoint resolution

Step 2: Deploy Amazon RDS (PostgreSQL)

Engine: PostgreSQL
Instance Class: db.t3.micro (Free Tier eligible)
Database Name: Champrds
Storage: gp2 with autoscaling
Network: Place the RDS in the custom VPC and subnet group
Security: Allow port 5432 internally; enable deletion protection
Monitoring: Enable Enhanced Monitoring and Performance Insights
Logs: Export PostgreSQL logs to Amazon CloudWatch

Subnet Group Configuration:
To define a group of subnets in multiple Availability Zones for Amazon RDS, enabling high availability and failover support.

Steps:

Created a DB Subnet Group using the previously created custom VPC named champ.
Selected multiple Availability Zones to ensure redundancy.

Added the three private subnets to the subnet group

The image shows the final confirmation screen of the DB Subnet Group named champsubnet-group, associated with three subnets across different AZs.

Amazon RDS (PostgreSQL) Configuration:
To provision a managed PostgreSQL database instance with support for multi-tenant architecture and performance monitoring.
Why PostgreSQL?

Open-source, highly reliable, and feature-rich.
Strong support for schema-based multi-tenancy.
Compatible with AWS services like Performance Insights, CloudWatch, and extensions like pg_stat_statements.

*Steps: *
• Selected Amazon RDS from the AWS Console.

• Chose PostgreSQL as the database engine.

• In Templates, selected Free Tier.

Set DB instance identifier as champDB-1.
Under Credential Settings, chose Self-managed credentials.
Provided custom master username.
Set a strong password.

• Selected Burstable instance class (db.t3.micro) for cost-effective performance.

• Chose General Purpose (SSD) – gp2
• Allocated required initial storage
• Enabled Storage Auto Scaling.
• Set Maximum storage threshold

• Chose Not to connect to EC2 as the DB will be accessed using pgAdmin.
• Selected the custom VPC created earlier (champ)

• Selected the DB Subnet Group previous
• Created a new Security Group: - Opened port 5432 (PostgreSQL default) for inbound access.

• Selected Password Authentication for database login.

• Enabled Enhanced Monitoring for real-time metrics on the database instance.
• Enabled Performance Insights for in-depth query and load analysis.
_Purpose and Benefits of Enhanced Monitoring_

- Provides real-time OS-level metrics such as CPU, memory, disk I/O, and network usage.
- Helps in identifying performance bottlenecks at the instance level.

Performance Insights

- Visualizes database load and query performance over time.
- Helps in identifying slow or expensive queries.
- Supports deeper optimization by showing wait events and resource usage patterns.

Enabled PostgreSQL log under Log exports to Amazon CloudWatch Logs.
To capture and centralize PostgreSQL database logs (including general activity, connections, and errors) in CloudWatch Logs for further analysis, alerting, and integration with downstream services like S3, Athena, and Glue.

• Set the Initial database name as Champrds.

• Enabled Deletion Protection to prevent accidental deletion of the RDS instance.

• Created the PostgreSQL database instance with the name Champrds.

Once the RDS instance reached Available status: The Current Activity tab became visible in the RDS console

This tab provides real-time query and session metrics using Performance Insights.
Helps in monitoring active connections, SQL queries, waits, and database load trends.

• Navigated to Amazon CloudWatch Logs: Verified that a new Log Group was created for the RDS instance.

• Installed the latest version of pgAdmin from the official pgAdmin website.

Collected the following connection details from the RDS console:

Endpoint (hostname)
Port (5432)
Master username and password
Database name: Champrds
In pgAdmin:
Created a new server connection.

Entered the RDS endpoint and credentials.

Successfully established the connection to the PostgreSQL RDS instance.

Multi-Tenancy in PostgreSQL – Concept and Use Case

What is Multi-Tenancy?

Multi-tenancy is a database architecture pattern where a single database instance serves multiple users or organizations (called tenants). Each tenant's data is isolated logically, even though they share the same infrastructure.

Why Use Multi-Tenancy?
This approach is widely adopted for:
• Cost efficiency – one DB instance for many users
• Simplified management – easier to maintain and monitor
• Scalability – add new users without deploying separate databases

Step 1: Login as Admin

Connect to RDS database (Champrds) using the master user credentials via pgAdmin.

Step 2: Create Tenant Users

-- Create tenant users
CREATE USER advent WITH PASSWORD 'AdventStrongPass1';
CREATE USER "6thgen" WITH PASSWORD 'SixthGenStrongPass1';
CREATE USER isalam WITH PASSWORD 'IsalamStrongPass1';

Create Schemas For AUTHORIZATION

-- Create schemas WITHOUT the AUTHORIZATION clause
CREATE SCHEMA advent;
CREATE SCHEMA "6thgen";
CREATE SCHEMA isalam;
-- Grant access to respective users
GRANT USAGE ON SCHEMA advent TO advent;
GRANT USAGE ON SCHEMA "6thgen" TO "6thgen";
GRANT USAGE ON SCHEMA isalam TO isalam;
-- Allow tenants to create objects in their schemas
GRANT CREATE ON SCHEMA advent TO advent;
GRANT CREATE ON SCHEMA "6thgen" TO "6thgen";
GRANT CREATE ON SCHEMA isalam TO isalam;

Set Default Schema for Each User (Search Path) You can set the default schema for each user, so they don’t need to set it manually:

ALTER ROLE advent SET search_path = advent;
ALTER ROLE "6thgen" SET search_path = "6thgen";
ALTER ROLE isalam SET search_path = isalam;

Now each user:

Has their own schema
Can create and use their own tables
Won’t get into privilege issues

Verifying Multi-Tenant Access via pgAdmin:
Used pgAdmin to connect individually as each tenant user: 6thgen

Verified successful login and access to the shared database for all tenant users via separate sessions.

Monitoring and Analyzing Tenant Activity

Running Sample Queries for Each Tenant
Simulating High CPU Load & Slow Queries
Checking Tenant-wise Usage Patterns
Analyzing Performance with Performance Insights Dashboard
Identifying Top Queries and High-Load Users
Mimicking Real-Time Multi-Tenant Application Behavior

Simulate Activity for Each Tenant User
You'll now open a query tool in each tenant session and run different types of queries to simulate: Normal usage, Heavy joins, Slow queries, High CPU usage.

Query To Simulate moderate insert/update activity:

CREATE TABLE IF NOT EXISTS test_data_advent (id SERIAL, name TEXT, created_at 
TIMESTAMP DEFAULT now());
INSERT INTO test_data_advent (name) 
SELECT 'User ' || generate_series(1, 10000);

Query To Simulate a CPU-heavy query:

WITH RECURSIVE nums AS (
 SELECT 1 AS n
 UNION ALL
 SELECT n + 1 FROM nums WHERE n < 100000
)
SELECT COUNT(*) FROM nums;

Query To Simulate a slow query using a sleep function:

SELECT pg_sleep(10);

Monitoring with RDS Performance Insights:

Amazon RDS Performance Insights is a powerful tool that helps monitor and analyze database performance in real time. It provides insights into SQL queries, wait events, top resource-consuming users, and helps identify performance bottlenecks. In a multi-tenant setup, this is especially useful to observe how different tenants are impacting the database
load and optimize resource usage accordingly

*Check Performance Insights: *
Once the queries are running:

Go to RDS > Performance Insights
Choose your PostgreSQL RDS instance
Check for:

Top SQL queries
Load by user (advent, 6thgen, isalam)
CPU, I/O graphs You’ll see these users now show up in the “Top Load” section.

Top Users Overview:

Performance Insights dashboard displays the top users accessing the database.
Verified all three tenant users (advent, 6thgen, isalam) are actively visible.

Individual User CPU Usage:
• Shows detailed CPU consumption for each user.
• Helps identify resource-heavy users or inefficient queries per tenant.

Verified current host IP online and matched it with dashboard.

Connected from a different network (host) by sharing user credentials

• Connected from a different network (host) by sharing user credentials. Dashboard

Using AWS security groups or AWS Network ACLs, we can restrict access to the RDS instance by allowing connections only from trusted IP addresses (hosts). This ensures only known networks can access the database, enhancing security in a multi-tenant setup.
ex: (optional)

Exporting RDS Logs from CloudWatch to S3 and Analyzing with Athena:
The main goal is to make RDS logs easily accessible and queryable for operational insights, debugging, and audit trails. By exporting logs from CloudWatch to Amazon S3 and integrating with Athena, we can run SQL queries on log data without maintaining infrastructure—providing a serverless, cost-effective, and scalable analytics solution

Approach:
To achieve this, two methods were used:

1. AWS Glue Integration:

Set up a Glue Crawler to scan and catalog log data in S3.
Created a Glue ETL Job to transform logs for structured querying.
Integrated the cataloged data with Athena for optimized analysis.

2. Direct Athena Table Setup:

Used Athena to directly point to the S3 bucket containing exported logs.
Created an external table by specifying the log file format and S3 path.

Pre-requisites:

Create an Amazon S3 bucket to store exported RDS logs from CloudWatch.

Attach the appropriate bucket policy to allow CloudWatch Logs to write data to the S3 bucket.

Steps to Export RDS Logs from CloudWatch to Amazon S3:

Go to CloudWatch > Logs > Log groups /aws/rds/instance/champdb-1/postgresql

Create an Export Task from CloudWatch:

Go to the slowquery log group
Click “Actions” → Export Data to Amazon S3

Choose:

Time range (based on requirement)
Target S3 bucket and prefix (folder)
Start export

It will create a folder in your S3 like:

Transforming RDS Logs with AWS Glue ETL for Structured Analysis

*AWS Glue *
Amazon Glue is a fully managed ETL (Extract, Transform, Load)
service that helps prepare and transform data for analytics. It
integrates with S3, Athena, and the Glue Data Catalog, making it easy to automate data pipelines and query structured data.

** Problem:**

After exporting RDS logs from CloudWatch Logs to Amazon S3, the files are in .gz compressed text format.
AWS Glue Crawlers do not support parsing .gz plain text files directly for schema inference.
AWS Glue supports formats like JSON, CSV, Parquet, Avro, and ORC etc.
This limits our ability to use Glue Data Catalog or Athena for querying logs efficiently.

Solution:

We created an AWS Glue ETL Job to transform .gz log files into structured JSON format.
JSON format is fully supported by Glue, allowing schema detection and query execution in Athena.

Benefits of Using Glue ETL:

Automates the transformation of raw logs into readable formats.
Enables seamless querying in Amazon Athena for log analysis.
Mimics real-world operational use cases like monitoring slow queries, high CPUusage, or user access patterns.
Supports building dashboards in tools like Amazon QuickSight using structured log data.

ETL Job Creation for Log Format Transformation
To convert exported .gz log files from CloudWatch into a JSON-compatible format that AWS Glue can process for querying in Athena.

Steps to Create ETL Job:
1.Go to AWS Console, search and open AWS Glue, then select "Jobs" under the ETL section.
2.Click “Add job” and choose “Script editor” to manually write the ETL script.

Other available options:

Visual ETL – Drag-and-drop interface, ideal for quick workflows.
Notebook – Use Jupyter-style interactive environment for advanced development.

In the script editor:

Wrote and added a transformation script to parse .gz text logs to structured JSON.
Gave a job title for identification.

Selected Spark as the engine.

Wrote and added a transformation script to parse .gz text logs to structured JSON.

import sys
import boto3
import gzip
import json
import re
from io import BytesIO
# Initialize the Glue context
from awsglue.context import GlueContext
from pyspark.context import SparkContext
sc = SparkContext()
glueContext = GlueContext(sc)
# S3 Bucket and Paths for Input and Output
input_s3_bucket = 'new-rds-champ'
input_s3_key = 'logs/6f8dd69a-ff1a-4fb7-a5b9-cf029a6380de/champdb-1.0/000000.gz'
output_s3_bucket = 'new-rds-champ'
output_s3_key = 'parsed_logs/output/'
# Read the .gz file from S3
def read_gz_file_from_s3(bucket, key):
 s3 = boto3.client('s3')
 obj = s3.get_object(Bucket=bucket, Key=key)
 gzipped_content = obj['Body'].read()
 with gzip.GzipFile(fileobj=BytesIO(gzipped_content), mode='rb') as f:
 return f.read().decode('utf-8')
# Parse the log line to extract timestamp, IP, log_level, and message using regex
def parse_log_line(log_line):
 # Example regex: Modify based on your log format
 log_pattern = 
r'(?P<timestamp>\S+)\s+(?P<ip>\S+)\s+(?P<log_level>\S+)\s+(?P<message>.+)'
 match = re.match(log_pattern, log_line)

 if match:
 return match.groupdict()
 return None
# Read the gzipped file from S3
log_data = read_gz_file_from_s3(input_s3_bucket, input_s3_key)
# Split the data into lines and parse each line
parsed_logs = []
for log_line in log_data.splitlines():
 parsed_log = parse_log_line(log_line)
 if parsed_log:
 parsed_logs.append(parsed_log)
# Write the parsed logs to S3 as JSON
def write_parsed_logs_to_s3(bucket, key, parsed_logs):
 s3 = boto3.client('s3')
 json_data = json.dumps(parsed_logs)
 s3.put_object(Bucket=bucket, Key=key, Body=json_data)
# Output path for parsed logs
output_s3_path = f'{output_s3_key}parsed_logs.json'
write_parsed_logs_to_s3(output_s3_bucket, output_s3_path, parsed_logs)
print(f"Parsed logs successfully written to: s3://{output_s3_bucket}/{output_s3_path}")

Gave a job title for identification.
Provided an appropriate IAM Role with access to S3 and Glue.

• Updated and ran the job successfully.

After completion:
A new folder was created in S3 (parsed_log/).
Inside that folder, parsed and structured log files were saved (parsed_logs.json).

Creating Glue Database for Athena Integration
Purpose:
To catalog and manage structured metadata from parsed S3 data, enabling Athena to run SQL queries on it.

Go to AWS Console, search and open AWS Glue.
In the Databases section, click “Add database”.

Provide a name for the Glue database (e.g., rdslogdb) and click Create.
This database will be used as the metadata catalog for Athena to reference and query the structured log data stored in S3.

Creating Glue Crawler to Catalog Parsed RDS Logs
Purpose:
To automatically scan the structured JSON files in S3 (created via ETL job), detect schema, and populate the Glue Data Catalog for Athena querying.
• Go to AWS Glue, select Crawlers, and click “Create crawler”.

Crawler Properties: Provide a name for the crawler ( rds-log-crawler).

Choose Data Source:
• Select S3 as the data source and give the path to the parsed logs folder generated by the ETL job.

• Select the IAM role with necessary permissions.

Set Output and Scheduling:
• Choose the target database (e.g., rdslogdb) created earlier.

Review and Create:
• Confirm the details and click Create crawler

• After creating the crawler, select it from the list and click "Run crawler".

• Once the crawler finishes, it creates a table inside the selected Glue database by detecting the schema from the JSON files.

• This table can now be queried using Amazon Athena for insights into your RDS logs.

Direct Integration from S3 to Athena (Without Glue)
Purpose: Quickly query log files stored in S3 without creating a Glue ETL or Crawler
setup.

Use Case:
Best suited for ad-hoc analysis, lightweight querying, or temporary exploration when
you have structured data like JSON, CSV, or Parquet directly stored in S3.
Simply point Athena to the S3 path and define the table schema manually using DDL
statements.

Created an external table pgrds_logs in Athena, specifying the S3 bucket path where
the logs are stored and defining table properties including compression type as gzip.

Specified the S3 bucket URI pointing to the .gz log files exported from CloudWatch, which serves as the data source for the Athena external table.

Extracting Valuable Insights with Athena Queries on RDS Logs

To analyze and monitor the PostgreSQL RDS usage across multiple tenants, we executed a series of Athena queries on the structured log data stored in Amazon S3. These queries returned critical details such as timestamps, tenant (user) activities, query patterns, and potential performance issues.

This approach enables:
• Granular visibility into individual tenant behavior
• Identification of slow or long-running queries
• Enhanced observability for resource optimization and cost management
• Support for proactive database performance tuning
These insights mimic real-world application usage and help drive informed decisions for multi-tenant architectures.

Sample Queries :

Conclusion & Key Takeaways
This PoC illustrates a comprehensive approach to managing and analyzing performance in a multi-tenant PostgreSQL database on Amazon RDS, with a focus on data-driven cost and performance insights.

Key Outcomes:

Achieved efficient tenant isolation using PostgreSQL schemas.
Enabled real-time performance monitoring with RDS Performance Insights and CloudWatch.
Implemented log export automation to Amazon S3 for advanced analytics.
Leveraged Athena & AWS Glue for scalable, serverless log analysis.

The architecture is designed with security, observability, and scalability in mind, offering a solid framework for SaaS platforms or internal multi-tenant systems that require
detailed visibility into tenant behavior and system performance.
Presented by KRANTHI PUTTI https://www.linkedin.com/in/kranthi-putti/

DEV Community: kranthi

[Boost]

Mastering AWS Storage Services: S3, EBS, and EFS

Mastering AWS Storage Services: S3, EBS, and EFS

Mastering Amazon EC2

Automating detection and alerts of VPC endpoint changes using AWS CloudTrail, EventBridge, Lambda, SNS, to Email.

VPC Endpoint Monitoring & Alerting

VPC Endpoint Monitoring & Alerting

Build a Multi-Tenant Monitoring System with RDS, Athena & Glue – Step-by-Step!

Building a Multi-Tenant Cost Tracking & Performance Monitoring Solution with Amazon RDS, Athena, and Glue

Building a Multi-Tenant Cost Tracking & Performance Monitoring Solution with Amazon RDS, Athena, and Glue

Mastering Amazon EC2