DEV Community: Samuel Ajisafe

Beyond the Switchover: Using RDS Snapshots to Safely Test MySQL 8.4 Compatibility Before Your Blue/Green Cutover

Samuel Ajisafe — Thu, 13 Aug 2026 20:16:53 +0000

AWS RDS Blue/Green deployment is a powerful zero-downtime tool, but its Green environment is read-only by design. Here is how to layer a snapshot-based test instance on top of it to validate application compatibility before committing to the full switchover.

There is a moment every DevOps engineer, DBA, and SRE dreads: a database engine reaches end-of-life, AWS begins billing you for Extended Support, and the upgrade clock is ticking. For MySQL 8.0, that moment arrived in 2026. AWS began charging Extended Support fees for any RDS instance still running 8.0, and teams across the industry started planning migrations to MySQL 8.4, the current Long Term Support (LTS) release.

AWS provides an excellent managed feature for this exact scenario: RDS Blue/Green Deployments. It is powerful, well-documented, and purpose-built for zero-downtime upgrades. But there is a constraint baked into how it works that trips up a lot of engineers the first time they reach for it, and it becomes a real blocker when you are managing a shared RDS instance across multiple projects with different readiness timelines.

This post documents exactly what that constraint is, why it exists, what error you will see in your application logs, and how to layer a snapshot-based testing pattern on top of Blue/Green to get the best of both worlds: safe compatibility validation per project, followed by a single zero-downtime cutover for everything.

The Context: A Shared RDS Instance Serving Multiple Projects

Picture a common real-world setup. You have one RDS MySQL 8.0 instance - call it proddb - running in us-east-1. It hosts several logical databases across seven different projects/environments: a SaaS platform in Node.js, several Laravel applications, a portfolio site. Some of these applications have been running for years. The code was written before MySQL 8.4 existed. You have no guarantee that everything will work after the engine upgrade.

You cannot afford to do a big bang upgrade. If one application breaks on MySQL 8.4, you need the others to continue running on the working engine while you fix it. You need a way to validate each project individually before committing everything.

This is the scenario where Blue/Green deployments look like the perfect answer - and they are, but not in the way you might initially expect.

What AWS RDS Blue/Green Deployment Actually Does

Before getting into the limitation, it is worth being precise about what Blue/Green deployment is and why it exists.

When you create a Blue/Green deployment in RDS, AWS provisions a Green environment that is an exact replica of your Blue environment (your current production instance). RDS manages the replication between them automatically, using physical replication for MySQL. The Green instance stays continuously synchronised with Blue. Your schema, your data, your parameter groups - everything is mirrored.

The workflow is designed like this:

Create the Blue/Green deployment
Upgrade the Green instance to MySQL 8.4 (you do this manually after creation, or configure it during setup)
Test the Green environment
Trigger the switchover

During switchover, AWS stops writes to Blue, waits for replication lag to reach zero, renames the instances (Green takes the original Blue endpoint, Blue gets an -old1 suffix and is set to read-only), and brings write traffic live on the new MySQL 8.4 instance. Typical downtime for single-region configurations is under five seconds. The old Blue instance is retained so you can roll back if needed.

This is genuinely excellent engineering. For a planned, coordinated upgrade where all your applications are ready to move at the same time, it is the right tool.

The Constraint You Will Hit: Green Is Read-Only by Design

Here is where the trouble starts if your goal is gradual, per-project migration.

The AWS documentation is explicit about this:

"During testing, we recommend that you keep your databases in the green environment read only. Enable write operations on the green environment with caution because they can result in replication conflicts. They can also result in unintended data in the production databases after switchover."

The Green environment is read-only by default. RDS sets the read_only parameter to 1 on the Green instance to protect replication integrity. If writes reach the Green during the testing phase, they could cause replication conflicts and potentially corrupt data that flows back after switchover.

There is a technical reason for this. MySQL physical replication works by replaying binary log events from the Blue primary on the Green replica. If you write directly to the Green instance, those writes are not in the Blue binlog. When the switchover happens and the old Blue becomes the replica, those writes have no origin on the Blue side. The replication relationship breaks, or worse, data that was never in production gets silently promoted into what is now your production database.

AWS therefore makes the protection the default. You can override it with a parameter group change, but the official guidance is to leave it alone.

What This Looks Like in Your Application Logs

If you follow the logical path - "I have a Green instance running MySQL 8.4, let me point my Dev application at it and test" - this is the error you will see immediately:

Error: The MySQL server is running with the --read-only option
       so it cannot execute this statement
    at PromisePool.query (/app/node_modules/express-mysql-session/
       node_modules/mysql2/promise.js:356:22)
    at /app/node_modules/express-mysql-session/index.js:384:30 {
  code: 'ER_OPTION_PREVENTS_STATEMENT',
  errno: 1290,
  sql: 'INSERT INTO `sessions` (`session_id`, `expires`, `data`)
        VALUES (\'KYzPpAzZpm9vHHp7WI_6W7H0UHkBHnaq\', 1786734753,
        \'{"cookie":...}\')
        ON DUPLICATE KEY UPDATE `expires` = VALUES(`expires`),
        `data` = VALUES(`data`)',
  sqlState: 'HY000',
  sqlMessage: 'The MySQL server is running with the --read-only
               option so it cannot execute this statement'
}

This surfaces immediately on the first write operation - in this case, the session store trying to persist a session record during an ELB health check. The application cannot even initialise properly. The cascade is:

ELB health checker hits /admin/login
Application attempts to create or update a session record in MySQL
Green instance rejects the write with ER_OPTION_PREVENTS_STATEMENT (errno 1290)
Application returns HTTP 500
A secondary error ERR_HTTP_HEADERS_SENT follows because the error handler tries to write a response after the response has already been partially sent
Health check fails, ELB marks the target unhealthy

[API][/admin/login][500][user_id:-] - {
  "request_id":"msrwaseu-3r7f2m",
  "method":"GET",
  "duration_ms":29,
  "headers":{
    "host":"172.31.3.155:4000",
    "user-agent":"ELB-HealthChecker/2.0"
  }
}
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at ServerResponse.setHeader (node:_http_outgoing:655:11)
    at ServerResponse.header (/app/node_modules/express/lib/response.js:684:10)

The ERR_HTTP_HEADERS_SENT error is a cascade failure, not the root cause. Fix the MySQL read-only problem and this disappears with it.

The important thing to understand is that this is not a misconfiguration on your part. The Green instance is behaving exactly as designed. You pointed an application with write operations at a managed replica. The replica correctly refused the writes.

Why the Naive Fixes Do Not Work

"Can I just set read_only=0 on the Green instance via a parameter group?"

Technically yes. The read_only parameter is dynamic in MySQL, meaning you can change it without a reboot. Some community members on AWS rePost have suggested this as a workaround. However, AWS's own guidance explicitly warns against it because it breaks the replication integrity guarantee. If your application writes data to the Green during testing and that data does not exist on Blue, the switchover will either fail guardrail checks or promote inconsistent data. This is not a theoretical risk - it is the exact failure mode the read-only default exists to prevent.

"Can I just do the switchover now and test on the new production instance?"

This defeats the purpose of testing before the switchover. If your application breaks on MySQL 8.4 and you have already cut over, you are now dealing with a production incident. The rollback path (replicating from the new MySQL 8.4 back to the old -old1 MySQL 8.0 instance using binlog position) is complex and time-sensitive.

"Can I selectively switchover just one project's database?"

No. Blue/Green deployment is an instance-level operation, not a database-level operation. When you trigger the switchover, every logical database on the instance moves simultaneously. You cannot switch one logical database while leaving others on Blue.

The Solution: Snapshot the Green, Test in Isolation, Switch Everything at Once

This is the pattern that gives you everything:

Per-project compatibility testing against a writable MySQL 8.4 instance
Replication on the Blue/Green deployment continues uninterrupted throughout
A single managed switchover when all projects are validated
No manual dump and restore for any project at cutover time

Here is how it works.

Step 1: Keep your Blue/Green deployment intact and revert your application immediately

If you pointed an application at the Green endpoint and it is throwing 500s, revert DB_HOST in your secrets manager or SSM Parameter Store back to the Blue endpoint and redeploy. Get the application healthy again first.

aws ssm put-parameter \
  --name "/your-project/dev/DB_HOST" \
  --value "proddb.xxxxxxxx.us-east-1.rds.amazonaws.com" \
  --type "String" \
  --overwrite \
  --region us-east-1

Do not delete the Blue/Green deployment. It is doing its job. Replication is running, the Green is staying synchronised with Blue, and you will use it for the final switchover.

Step 2: Take a snapshot of the Green instance

In the AWS Console, navigate to Databases, select your Green instance (proddb-green-grigwa in this example), and choose Actions - Take snapshot. Give it a descriptive name like green-84-compat-test-20260813.

Via CLI:

aws rds create-db-snapshot \
  --db-instance-identifier proddb-green-grigwa \
  --db-snapshot-identifier green-84-compat-test-20260813 \
  --region us-east-1

Wait for the snapshot status to reach available. This typically takes a few minutes.

Step 3: Restore the snapshot as a standalone MySQL 8.4 instance

Restore the snapshot as a new, independent DB instance. This instance is not part of any Blue/Green deployment, not a replica of anything, and is fully writable from the moment it comes online.

In the console: RDS - Snapshots - select your snapshot - Actions - Restore snapshot. Set the DB instance identifier to something clear like proddb-84-test.

Via CLI:

aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier proddb-84-test \
  --db-snapshot-identifier green-84-compat-test-20260813 \
  --db-instance-class db.t3.medium \
  --vpc-security-group-ids sg-xxxxxxxxxx \
  --db-subnet-group-name your-subnet-group \
  --no-publicly-accessible \
  --region us-east-1

This instance contains every logical database from your Green snapshot, running on MySQL 8.4. It accepts full read/write operations. It costs the same as any other RDS instance of that class and size, prorated to the hours it runs - typically a few days at most for a dev validation exercise.

Step 4: Point your first project at the test instance and validate

Update the DB_HOST for your first project's dev environment to the test instance endpoint and redeploy.

aws ssm put-parameter \
  --name "/project/dev/DB_HOST" \
  --value "proddb-84-test.xxxxxxxx.us-east-1.rds.amazonaws.com" \
  --type "String" \
  --overwrite \
  --region us-east-1

Now run your application against MySQL 8.4. Exercise every critical path: authentication flows, session management, file uploads, report generation, background jobs - whatever your application does. Check application logs for any MySQL 8.4 compatibility errors. Common ones to watch for include deprecated functions, changes to default values for SQL mode, and authentication plugin changes.

All other projects remain on the Blue instance throughout this process. Your production workloads are completely unaffected.

Step 5: Validate additional projects as they are ready

For each subsequent project, either point it at the same test instance (all logical databases are present since the snapshot captured everything) or repeat the process per environment. The test instance holds a point-in-time copy of every database. For projects that have been running production writes since the snapshot was taken, you may want to do a fresh mysqldump of just that logical database from Blue and restore it into the test instance to get current data for realistic testing. But for application compatibility validation, the snapshot data is usually sufficient.

Step 6: Delete the test instance and trigger the Blue/Green switchover

Once every project that shares the RDS instance is validated on MySQL 8.4, delete the test instance. It has served its purpose and you no longer need to pay for it.

aws rds delete-db-instance \
  --db-instance-identifier proddb-84-test \
  --skip-final-snapshot \
  --region us-east-1

Then trigger the switchover on your Blue/Green deployment. Because replication has been running continuously throughout the entire testing phase, the Green instance is fully synchronised with Blue. The switchover is clean, the downtime is under five seconds, and every project moves to MySQL 8.4 simultaneously with no manual dump or restore required.

aws rds switchover-blue-green-deployment \
  --blue-green-deployment-identifier bgd-xxxxxxxxxx \
  --switchover-timeout 300 \
  --region us-east-1

After switchover, the original Blue endpoint now points to the MySQL 8.4 instance. All your applications that were pointing at the Blue endpoint continue working without any DB_HOST changes. The old MySQL 8.0 instance is retained with an -old1 suffix. Keep it for at least 24-48 hours as a rollback option, then delete it once you are confident everything is stable.

Why This Pattern Works: The SRE Reasoning

This approach is grounded in a core SRE principle: validate before you commit, not while you commit.

Blue/Green deployment is optimised for the commitment phase - the actual cutover. It gives you replication, endpoint swapping, automatic rollback, and near-zero downtime. What it does not give you is a writable staging environment for compatibility testing, because enabling writes on the Green would compromise the very replication integrity that makes the switchover safe.

The snapshot test instance fills exactly that gap. It is an isolated sandbox that costs almost nothing for a short-lived validation exercise. It carries no risk to the Blue/Green replication. If the application breaks badly on MySQL 8.4 on the test instance, you simply delete the test instance, fix the code, create a new snapshot from the Green (which has continued replicating and is still current), and restore again. The Blue/Green deployment is completely untouched.

This is defence in depth for database migrations: one layer for safety (Blue/Green with continuous replication and managed switchover), one layer for confidence (snapshot-based isolated testing with full read/write access).

Summary: The Full Migration Sequence

Phase	Action	Blue/Green Replication
1	Create Blue/Green deployment, upgrade Green to MySQL 8.4	Running
2	Take snapshot of Green instance	Running
3	Restore snapshot as standalone writable test instance	Running
4	Point project Dev environment at test instance, validate	Running
5	Fix any MySQL 8.4 compatibility issues in application code	Running
6	Repeat for each project and environment as ready	Running
7	Delete test instance	Running
8	Trigger Blue/Green switchover - all projects move at once	Completes
9	Validate production, delete old Blue instance	Done

Things to Keep in Mind

Snapshot data is a point in time. The test instance holds data as of the moment you took the snapshot. For high-volume databases, the data on the test instance may be hours or days old by the time you finish testing. This is fine for compatibility validation. If you need realistic load testing with current data volumes, consider scripting a fresh snapshot restore closer to your planned switchover date.

The test instance incurs cost. Even a db.t3.micro or db.t3.medium running for three days is a small number, but track it. Delete the test instance as soon as validation is complete. It has no reason to linger.

MySQL 8.4 changes the default authentication plugin. If your application uses mysql_native_password, check whether your database users are configured correctly on the Green instance. MySQL 8.4 makes caching_sha2_password the default. Most modern MySQL client libraries handle this transparently, but older applications or direct connection strings may surface authentication errors during testing.

Run the MySQL Shell upgrade checker before any of this. The AWS blog post on MySQL 8.0 to 8.4 upgrades recommends restoring a snapshot of your 8.0 Blue instance and running mysqlsh -- util check-for-server-upgrade against it before you even create the Blue/Green deployment. This catches error-level incompatibilities (corrupted tables, deprecated syntax, partitioning issues) before you spend time on the migration pipeline.

Pre-switchover, monitor ReplicaLag. Before triggering the switchover, confirm the ReplicaLag CloudWatch metric on the Green instance is at or near zero. A switchover with significant lag means a longer write blackout window while RDS waits for the Green to catch up.

Closing Thoughts

RDS Blue/Green deployment is not a limitation to work around - it is the right tool for the final mile of a database upgrade. The read-only Green environment is not a bug; it is an intentional protection that keeps your data consistent through the switchover.

The pattern described here does not replace Blue/Green. It layers a temporary, low-cost, isolated testing environment on top of it so that by the time you press the switchover button, you have already confirmed that every application works correctly on the new engine. The switchover becomes a formality rather than a risk event.

For any engineering team managing a shared RDS instance across multiple projects with different codebase ages and readiness timelines, this two-layer approach - snapshot test instance for validation, Blue/Green for the cutover - is the pattern that gives you both site reliability and migration confidence.

References

Fixing the AWS Amplify “Make SSL Changes and Try Again” Error (Root Cause & Solution)

Samuel Ajisafe — Fri, 28 Nov 2025 18:36:42 +0000

Introduction

Connecting a custom domain to AWS Amplify should be simple but sometimes you may run into this frustrating error:

Make SSL changes and try again.
Consult the troubleshooting guide, make any necessary changes, and retry activation.

If you’ve spent hours checking DNS records, regenerating SSL certificates, or digging through AWS documentation with no luck—you’re not alone. I recently encountered this issue while supporting a customer, and after extensive troubleshooting, I discovered the real root cause that AWS doesn't clearly point out.

This post walks you through the exact problem and the simple fix that finally resolved it.

❗️The Error in Amplify

When activating a custom domain in AWS Amplify, you may see:

Make SSL changes and try again.

Amplify advises you to check the SSL configuration, but the real issue is usually not related to your certificate at all.

⚡ My Troubleshooting Journey

Like most engineers, I started with the standard checks:

Regenerating the SSL certificate
Ensuring the certificate was issued in us-east-1
Validating domain and subdomain spelling
Reviewing Route 53 DNS records
Searching AWS re:Post, StackOverflow, and several AI agents

Nothing worked.

At that point, I decided to isolate every possible cause manually. That’s when I noticed something that Amplify does not highlight clearly.

🧠 Understanding the Real Root Cause

AWS Amplify, although regionally displayed in the console, behaves like a global service when it comes to domain management similar to S3, CloudFront.

This means:

👉 You cannot map the same domain or subdomain to more than one Amplify app, even if the other app is old, inactive, or already deleted from your current workflows.

If a domain or subdomain is still attached to ANY Amplify or CloudFront project anywhere in AWS Accounts, the new SSL validation will fail silently and repeatedly.

Also another scenario is the customer already tried using cloudfront and has created the same domain as “Alternate name” for the distribution

Amplify does not explicitly warn you that the domain is already mapped somewhere else—making the error appear unrelated.

✔️ The Solution That Worked

When I asked the client whether they had an older Amplify app before migrating, they confirmed yes.

That old app still had the domain attached and was blocking SSL activation.

Fix Steps:

Go to the previous Amplify app (if deleted, check the Amplify console history).
Remove any custom domain mappings. If Cloudfront, ensure the Distribution is disabled and deleted
Return to the new Amplify app.
Re-add the domain.

After deleting the old domain mapping, the SSL validation completed instantly.

🛠 Code Snippet — Domain Cleanup (If Using CLI)

If you manage Amplify via CLI, you can list and remove domain mappings like this:

# List domain associations for an Amplify app
aws amplify list-domain-associations \
  --app-id YOUR_APP_ID

# Delete an old domain mapping
aws amplify delete-domain-association \
  --app-id YOUR_APP_ID \
  --domain-name yourdomain.com

✅ Final Thoughts

This issue is surprisingly common, and the fix is much simpler than the error message implies.
If you’re stuck on the SSL activation step in AWS Amplify:

Always check whether your domain or subdomain is mapped to another Amplify app or CloudFront Distribution.

Removing that mapping is often the key to solving the problem instantly.

How to Setup Pritunl on Amazon Linux 2023 | Centos | RedHAT

Samuel Ajisafe — Thu, 21 Aug 2025 22:02:35 +0000

Unlock Your Secure Network: A Step-by-Step Guide to Setting up Pritunl VPN on Amazon Linux 2023 🚀

Welcome back! Following the great feedback on my last guide for setting up Pritunl VPN on Ubuntu, I'm excited to dive into a new environment. This time, we're tackling Amazon Linux 2023, a fantastic choice given its long-term support (EOL is still years away). This guide will get your secure VPN server up and running, and I'll be sure to update it as new versions are released.

So, let's get started and secure your network! 🔒

Prerequisites: Setting the Stage 🛠️
Before we dive into the Pritunl setup, we need to ensure our system is ready. This involves installing Docker and Docker Compose, which we'll use to run Pritunl in a containerized environment. This approach is clean, efficient, and avoids conflicts with other software on your server.

Bash

# Install Docker
sudo dnf install -y docker

# Start and enable the Docker service
sudo systemctl start docker
sudo systemctl enable docker

# Add your user to the docker group to run Docker commands without sudo
sudo usermod -aG docker $USER

# Install Docker Compose as a plugin for the Docker CLI
sudo curl -SL https://github.com/docker/compose/releases/latest/download/docker-compose-linux-$(uname -m) -o /usr/libexec/docker/cli-plugins/docker-compose
sudo chmod +x /usr/libexec/docker/cli-plugins/docker-compose

Note: After running the usermod command, it's best to log out and log back in (or run newgrp docker) for the changes to take effect.

Step 1: Configuring the System for VPN Traffic ⚙️
Pritunl requires specific system settings to handle VPN traffic correctly. We'll set up iptables rules and enable IP forwarding, which are crucial for routing traffic through the VPN tunnel.

Bash

Create a directory for our Pritunl configuration

mkdir pritunl && cd pritunl

Allow traffic to and from the VPN tunnel interface (tun+)

sudo iptables -A FORWARD -i tun+ -j ACCEPT
sudo iptables -A FORWARD -o tun+ -j ACCEPT

Install iptables-services to persist the rules

sudo dnf install -y iptables-services
sudo service iptables save

Enable IP forwarding, which allows packets to be forwarded between interfaces

sudo sysctl -w net.ipv4.ip_forward=1

Make this change permanent

echo 'net.ipv4.ip_forward=1' | sudo tee -a /etc/sysctl.conf

Load the TUN module, which is essential for the VPN tunnel

sudo modprobe tun

Make the TUN module persistent on reboots

echo 'tun' | sudo tee -a /etc/modules-load.d/tun.conf

Step 2: Deploying Pritunl with Docker Compose 🐳
Now, let's define our Pritunl container using a docker-compose.yml file. This file simplifies the deployment process, ensuring all dependencies are handled correctly.

Bash

Open a text editor to create the docker-compose.yml file

sudo vi docker-compose.yml

Paste the following configuration into the file:

services:
  mongodb:
    image: mongo:latest
    container_name: mongodb
    restart: always
    network_mode: host
    volumes:
      - mongodb_data:/data/db

  pritunl:
    image: jippi/pritunl
    container_name: pritunl
    privileged: true
    restart: always
    network_mode: host
    volumes:
      - pritunl_data:/var/lib/pritunl
    devices:
      - /dev/net/tun:/dev/net/tun
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
      - SYS_ADMIN
    environment:
      - PRITUNL_MONGODB_URI=mongodb://localhost:27017/pritunl
    depends_on:
      - mongodb

Save the file and exit the editor. Now, launch the container with a single command:

Bash

docker compose up -d

This command will download the Pritunl image and start the container in the background.

Step 3: Initial Setup & Configuration ✨
With the container running, we need to retrieve the initial setup key and default credentials to access the web interface.

Bash

Retrieve the Pritunl setup key

docker exec pritunl pritunl setup-key

Retrieve the default username and password

docker exec pritunl pritunl default-password

Note down these credentials carefully. Now, navigate to your server's public IP address <https://ip-address> in a web browser. You'll be prompted to enter the setup key and then the default credentials to log in. The Port 443 must be Open on security Group

Once inside the Pritunl admin console, you can create a new Server, add a User, and start the server. The final step is to create a crucial NAT rule to allow traffic to exit your server.

While creating the Server, note down 2 things, UDP port and Virtual Network, UDP Port must be open on the EC2 Security Group, and the Virtual Network must be open must be used in the next command

This rule ensures VPN traffic can be routed out to the internet
Replace 192.168.248.0/24 with your Pritunl server's network range if you change it
Replace eth0 with your server's primary network interface

sudo iptables -t nat -A POSTROUTING -s 192.168.248.0/24 -o eth0 -j MASQUERADE

Step 4: Final Touches on AWS ☁️
If you're hosting this on AWS, you must open the necessary ports in your security group to allow clients to connect.

HTTPS (TCP 443): Required for the Pritunl web interface and VPN connection.

OpenVPN Ports (UDP): Pritunl uses a range of ports for its VPN tunnels. Check your server's configuration and open those specific ports. The default is often UDP 1194, but Pritunl can use others.

By completing these steps, you've successfully deployed a robust and secure Pritunl VPN server. You can now download the client configuration file from the web interface and connect to your new private network.

If you have any questions or run into issues, drop a comment below. Happy securing!

pritunl #vpn #vpnserver #amazonlinux2023 #centos #redhat #docker #dockercompose #openvpn #devops #linux #cloud #aws #tutorial #security #technology #howto

Access an EC2 Instance Using Session Manager (Even Without SSH Keys

Samuel Ajisafe — Sat, 07 Jun 2025 15:00:14 +0000

Sometimes you're dropped into an AWS environment where EC2 instances already exist — maybe from a team handover or an inherited project. But here's the issue:
You don't have the SSH key pair to access the instances. It might have been lost, never shared, or never created in the first place.

Thankfully, there are several ways to regain access:

Common Solutions When SSH Access is Lost

Use EC2 Instance Connect
Use AWS Systems Manager Session Manager
Create an AMI of the instance and launch a new one

While each option has its use case, the most secure, scalable, and production-friendly solution is Option 2 — Session Manager.

Why Session Manager is the Best Option

Over time, I've seen many clients face this exact issue. In most cases, EC2 Instance Connect either doesn't work due to misconfigurations, or it's blocked by security groups and firewalls.

Session Manager, on the other hand, works consistently and securely. Plus, it doesn't require any open ports — a huge win for environments with strict compliance requirements.

Benefits of Session Manager

No SSH keys or bastion hosts required
No open inbound ports needed
Works over AWS Systems Manager Agent (SSM Agent)
Supports centralized logging (CloudWatch, S3)
Secured via IAM and AWS KMS

First: Check if the Instance is Already Registered

Before doing any configuration, verify whether your instance is already registered with Session Manager.

Go to this link:
Start a Session in AWS Session Manager
Click "Start session".
If your instance is listed, you can connect directly — no further setup needed.

If no instances appear, follow the steps below to enable access.

Step-by-Step: Enable Session Manager on EC2

Step 1: Create an IAM Policy for Session Manager

Go to the IAM Console.
In the sidebar, click "Policies" > "Create Policy".
Select the JSON tab and replace the contents with the following policy (adjust values as needed):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssmmessages:CreateControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:OpenDataChannel",
        "ssm:UpdateInstanceInformation"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::your-bucket-name/s3-prefix/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetEncryptionConfiguration"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
      ],
      "Resource": "arn:aws:kms:your-region:your-account-id:key/your-key-id"
    }
  ]
}

Click "Next: Tags" (optional).
Click "Next: Review".
Name the policy something like SessionManagerPermissions.
Click "Create Policy".

Step 2: Create an IAM Role for EC2 Instances

Go to "Roles" > "Create Role" in the IAM console.
Select AWS service as the trusted entity, and choose EC2.
Click "Next".
Attach the SessionManagerPermissions policy you just created.
Name the role something like EC2SessionManagerRole.
Click "Create Role".

Step 3: Attach the IAM Role to the EC2 Instance

Open the EC2 Console.
Select the instance you want to access.
Choose "Actions > Security > Modify IAM Role".
Select the role you just created (EC2SessionManagerRole).
Click "Update IAM Role".

Step 4: Connect via Session Manager

Go to Systems Manager > Session Manager.
Wait a few minutes — the instance should now appear.
Click "Start session" and select your instance.
Click "Connect".

You now have shell access to the EC2 instance — without SSH or open ports.

Final Notes

Ensure the SSM Agent is installed and running. Most Amazon Linux, Ubuntu, and Windows AMIs include it by default.
The instance must have internet access (via a NAT Gateway or Internet Gateway) unless you're using VPC endpoints.
Session Manager access is auditable, secure, and great for managing access at scale.

Summary

Session Manager is the go-to tool for securely accessing EC2 instances when SSH is not an option. Whether you're dealing with lost keys or just want to manage access more securely and efficiently — this is the approach I recommend and use in production environments.

Let me know in the comments if you’ve used Session Manager or if you ran into any issues while setting it up.

AWSCommunityBuilder #Cloud #AWS #Server #Compute #SystemsManager #EC2 #Passwordless #DevOps

Send AWS CloudWatch Alerts to Google Chat Using SNS and Lambda

Samuel Ajisafe — Wed, 09 Apr 2025 10:00:41 +0000

Monitoring and alerting are key pillars of any production-grade AWS environment. In this guide, I’ll walk you through integrating AWS CloudWatch Alarms with Google Chat using Amazon SNS and a Lambda function ideal for teams using Google Workspace instead of Slack or Microsoft Teams.

🎯 Use Case

My ECS service scales up when CPU or memory exceeds 75%. I want real-time alerts in Google Chat—not just email.

🚧 The Challenge

While CloudWatch can trigger alarms and send notifications via SNS, Google Chat isn't a native target. SNS supports protocols like email, Lambda, SQS, and HTTP/S—but not Google Chat directly.

✅ The Solution

We can bridge the gap with this flow:

CloudWatch Alarm → SNS Topic → Lambda Function → Google Chat Webhook

🧰 Prerequisites

To complete this integration, you'll need:

An AWS account with permission to create SNS topics and Lambda functions
A Google Chat space with an active webhook URL
Basic Python and AWS Lambda familiarity

🔧 Step 1: Set Up a Google Chat Webhook

In Google Chat, open or create a Space.
Click the arrow next to the space name → Manage Webhooks.
Click Add Webhook, name it (e.g., AWS Alerts), and copy the Webhook URL.

🐍 Step 2: Write Your Lambda Function (Python)

Create a simple Lambda function to forward SNS messages to Google Chat using httplib2.

lambda_function.py

from httplib2 import Http
from json import dumps

def lambda_handler(event, context):
    url = "<WEBHOOK-URL>"  # Replace with your actual webhook URL
    message = {'text': event['Records'][0]['Sns']['Message']}
    headers = {'Content-Type': 'application/json; charset=UTF-8'}

    http_obj = Http()
    response = http_obj.request(
        uri=url,
        method='POST',
        headers=headers,
        body=dumps(message)
    )
    return response

📦 Step 3: Package the Lambda Code

On any Linux machine:

mkdir -p python/lambda
cd python/lambda

# Add your lambda_function.py here
vi lambda_function.py

# Install dependencies
pip3 install httplib2 -t .
pip3 install requests -t .

# Zip the code
zip -r python_code.zip .

⚠️ Install pip3 if missing:
Ubuntu/Debian: sudo apt install python3-pip
RHEL/CentOS: sudo dnf install python3-pip

🚀 Step 4: Deploy the Lambda Function

Go to AWS Lambda → Create function
Choose Author from scratch
Set Runtime to Python 3.x
Upload python_code.zip under Function Code
Set the Handler to lambda_function.lambda_handler
Click Deploy

🧪 Step 5: Test the Integration

Manual Test (Lambda)

In your Lambda function, click Test
Select the SNS Topic Notification template
Use the default sample event
Confirm the alert is posted in your Google Chat room

📢 Step 6: Set Up SNS Topic & Subscription

Create SNS Topic

Go to Amazon SNS → Topics → Create topic
Choose Standard
Name it something like alert-notifications
Click Create topic

Subscribe Lambda to SNS

Go to Subscriptions → Create subscription
Set protocol to AWS Lambda
Select your deployed Lambda function
Confirm the subscription

Manual Test (SNS)

In the SNS topic, click Publish message
Add a subject and message body
Publish and check Google Chat for the alert

✅ Final Thoughts

With this integration, you can pipe AWS alerts directly into Google Chat—keeping your team informed in real time, even if you're not using Slack or Microsoft Teams.

You can attach this SNS topic to any CloudWatch alarm or event, including:

ECS Service scaling
EC2 high CPU alerts
ALB 5XX errors

🔗 Resources

If you found this useful, follow me for more practical AWS DevOps content.

Let’s keep building resilient cloud-native systems—one alert at a time!

Tags:
#DevOps #CloudEngineer #AWS #GoogleChat #Lambda #SNS #Alerting #Monitoring #CloudWatch #Automation #AWSCommunityBuilders

My one stop for solution open source setup of software on Linux

Samuel Ajisafe — Sun, 06 Apr 2025 01:22:19 +0000

Still Building this post.

Reference this docs: https://devtutorial.io/how-to-install-rabbitmq-on-centos-stream-10-p3704.html

Unlocking a Hidden Power of CloudFront: Integrating On-Premise Servers with AWS CloudFront

Samuel Ajisafe — Wed, 02 Apr 2025 23:13:12 +0000

Introduction

AWS CloudFront is widely known for improving web performance, reducing latency for global users, and enhancing security with tight integrations AWS WAF and Shield Advanced. Many AWS Cloud Engineers and DevOps professionals associate it primarily with delivering static content from Amazon S3 websites, but its potential goes far beyond that.

In this guide, I’ll walk you through an innovative use case—leveraging AWS CloudFront to optimize and secure applications running on-premises. If you manage a hybrid environment or entirely on-premise infrastructure and need to enhance performance for global users or further tighten the security of your application by protecting it against DDOS and Vulnerabilities this solution is for you.

The Challenge

If your applications are hosted within your private corporate network, accessing them via HTTPS can be a headache. You might encounter errors like ERR_SSL_VERSION_OR_CIPHER_MISMATCH due to DNS issues and SSL/TLS misconfigurations.

For example, I deployed multiple applications within my private network (192.168.10.0/24) using Nginx as a load balancer. While external users could access the application seamlessly on the secure channel (https), internal users faced SSL-related issues, despite setting up a private DNS.

After extensive research with many read on AWS Whitepapers, I found an efficient solution:

CloudFront with On-Premises Web Servers

By integrating AWS CloudFront with an on-premises web server, we can achieve:

Enhanced security with CloudFront’s native DDoS protection
Global performance optimization
Seamless HTTPS support
Reduced operational overhead

Prerequisites

Before you proceed with the setup, ensure the following prerequisites are met:

SSL Certificate in AWS ACM: You must generate an SSL certificate in AWS Certificate Manager (ACM) for your domain in us-east-1.
Public DNS Mapping: The origin domain name (analytics-origin.example.com) should already be mapped and pointing to the public IP of analytics.example.com in a DNS manager such as AWS Route 53, GoDaddy, Namecheap etc.
Network Configuration: Ensure that the firewall and security group rules allow inbound connections from CloudFront IP ranges to your on-premise server.

Step-by-Step Implementation

In my case I have already configured NGINX as Load balancer

1. Configure Nginx as a Load Balancer

If you are using Nginx to route traffic to an internal application, set up your configuration as follows:

server {
    listen       443 ssl http2;
    listen       [::]:443 ssl http2;
    server_name  analytics.example.com;

    ssl_certificate "/etc/nginx/ssl/public.example.crt";
    ssl_certificate_key "/etc/nginx/ssl/private.example.com.key";
    ssl_session_cache shared:SSL:5m;
    ssl_session_timeout  10m;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
    ssl_prefer_server_ciphers on;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    ignore_invalid_headers off;
    client_max_body_size 100m;
    proxy_buffering off;
    proxy_request_buffering off;

    location / {
        proxy_buffer_size          128k;
        proxy_buffers              4 256k;
        proxy_busy_buffers_size    256k;
        client_max_body_size       50M;
        proxy_pass  http://192.168.10.68:3000;
    }
}

2. Configure AWS CloudFront

Origin Settings

Origin Domain Name: analytics-origin.example.com (must already be mapped and pointing to the public IP of analytics.example.com in a DNS manager like AWS Route 53, GoDaddy, etc.)

Origin Protocol Policy: HTTPS Only
Origin Port: 443
Origin Path: Leave blank

Viewer Settings

Viewer Protocol Policy: Redirect HTTP to HTTPS
Allowed HTTP Methods: GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE
Cache Policy: Use CachingDisabled (recommended for dynamic content)

Additional Configuration but Compulsory

Alternate Domain Names (CNAMEs): analytics.example.com
SSL Certificate: Use AWS-managed certificate for your domain

3. Configure DNS Records

Create a CNAME record in your DNS provider or Alias record in Route53:

Name: analytics.example.com
Value: Your CloudFront distribution domain (*.cloudfront.net)

4. Network Considerations

Allow traffic from CloudFront IP ranges in your firewall/security group.
Restrict direct access to Your Public IP (your on-prem server's public IP).
Ensure CloudFront can reach Your Public IP.

5. Testing and Validation

✅ Verify DNS resolution for analytics-origin.example.com
✅ Check CloudFront distribution status
✅ Test application access at https://analytics.example.com
✅ Validate functionality
✅ Review Nginx and CloudFront logs

Conclusion

CloudFront isn’t just for static content—it’s a powerful tool for accelerating and securing on-prem applications. By following this setup, you can:

Improve latency for global users
Secure on-premise applications using AWS-managed SSL/TLS
Leverage AWS’s DDoS protection and WAF

If you found this guide helpful, drop a comment below or share your experience integrating CloudFront with on-premise servers! 🚀

AWS #Cloud #DevOps #SRE #AWSCommunityBuilder #s3 #Infrastructure #CloudEngineer #CDN #Security #Nginx #Network

Redirect a Top-Level Domain Hosted on AWS to an External URL

Samuel Ajisafe — Thu, 30 Jan 2025 00:28:45 +0000

In today’s interconnected digital landscape, businesses often need to redirect traffic from their primary domain to external URLs hosted outside their cloud infrastructure. Whether it’s for rebranding, marketing campaigns, mergers, or compliance, redirecting a top-level domain (TLD) is a common yet critical task.

In this article, we’ll explore real-world use cases for domain redirection and walk you through a step-by-step guide to achieve this using AWS services like Route 53, S3, ACM, and CloudFront.

*Scenarios Requiring Domain Redirection *
Here are some common situations where mapping a TLD to an external URL is necessary:

Rebranding or Domain Consolidation
- Example: A company rebrands from example.com to tester.com/agents/ and wants legacy traffic to seamlessly redirect to the new site.
Marketing Campaigns
- Example: A short-term campaign uses a memorable domain (e.g., promo.example.com) to redirect users to a third-party landing page (e.g., tester.com/summer-sale).
Mergers and Acquisitions
- Example: After acquiring a company, redirect its domain (e.g., acquired-company.com) to a section of the parent company’s site (e.g., tester.com/partners).
Compliance or Legal Requirements
- Example: Redirecting users from a non-compliant legacy domain to an updated, compliant URL hosted externally.

If any of these scenarios sound familiar, this walkthrough is for you!

Solution Overview: AWS Architecture for Domain Redirection

For the use case example.com → www.tester.com/agents/dns, the following AWS services are leveraged:

AWS Route 53: Managed DNS routing for the TLD.
Amazon S3: Hosts a simple redirect rule.
AWS Certificate Manager (ACM): Provides SSL/TLS certificates for secure redirection.
Amazon CloudFront: Ensures HTTPS support and improves performance.

Step-by-Step Walkthrough

Here’s how to redirect example.com and www.example.com to www.tester.com/agents/dns/ using AWS services:

Step 1: Create S3 Buckets for Redirection

Create the root domain bucket:
- Go to AWS S3 Console → Create bucket.
- Bucket name: example.com (must match your domain).
- Region: Choose a region (e.g., US East (N. Virginia)).
- Uncheck Block all public access
- Click Create bucket.
Configure static website hosting for the root domain bucket:
- Open the bucket → Properties tab → Static website hosting.
- Select Redirect all requests to another host name.
- Target domain: www.tester.com/agents/dns/.
- Protocol: https (if the target supports HTTPS).

Step 2: Request an SSL Certificate (AWS Certificate Manager)

Request a certificate:
- Go to AWS Certificate Manager (ACM) → Request a certificate.
- Domain names:
  - example.com
  - *.example.com (covers subdomains like www).
- Validation method: DNS validation.
- Click Request.
Validate the certificate:
- ACM will ask you to add CNAME records to your Route 53 hosted zone.
- Click Create records in Route 53
- Wait for validation (status changes to Issued).

Step 3: Create CloudFront Distributions

Create a CloudFront distribution for the root domain:
- Go to CloudFront Console → Create Distribution.
- Origin Domain: Select the S3 bucket example.com (use the S3 website endpoint URL).
- Origin Path: Leave blank.
- Viewer Protocol Policy: Redirect HTTP to HTTPS.
- Alternate Domain Names (CNAMEs): example.com and www.example.com.
- SSL Certificate: Select the ACM certificate you created.
- Default Root Object: Leave blank.
- Click Create Distribution.

Step 4: Update Route 53 DNS Records

Point the root domain to CloudFront:
- Go to Route 53 Console → Hosted Zones → example.com.
- Edit the A record for the root domain:
  - Record name: Leave blank.
  - Alias: Yes.
  - Route traffic to: Alias to CloudFront distribution → Select the root domain CloudFront distribution.
- Save.
Point the www subdomain to CloudFront:
- Create/edit the A record for www:
  - Record name: www.
  - Alias: Yes.
  - Route traffic to: Alias to CloudFront distribution → Select the root domain CloudFront distribution.
- Save.

Step 5: Test the Redirection

Wait for DNS propagation (up to 48 hours, but usually faster).
Test using:
- Browser: Visit https://example.com → Should redirect to https://www.tester.com/agents/dns/.

Troubleshooting Tips

SSL/TLS Errors:
- Ensure the ACM certificate is created in us-east-1 (required for CloudFront).
- Verify the certificate is attached to the CloudFront distribution.
Infinite Redirect Loops:
- Ensure S3 bucket redirects point directly to the external URL (not to another S3 bucket).
DNS Issues:
- Use dig example.com to confirm DNS resolution.

Conclusion

Redirecting a top-level domain hosted on AWS to an external URL is a powerful strategy for maintaining brand continuity, complying with legal requirements, or executing marketing campaigns. By leveraging AWS services like Route 53, S3, ACM, and CloudFront, you can achieve seamless, secure, and cost-effective redirection without infrastructure overhead.

Call to Action

Have you implemented domain redirection on AWS? Share your experience in the comments below! If you have questions, feel free to ask—we’re here to help.

Tags: #AWS #DomainManagement #Route53 #CloudComputing #WebHosting #HTTPS #DevOps

How to Create an IAM User

Samuel Ajisafe — Mon, 27 Jan 2025 04:30:55 +0000

How to Create an IAM User

Log in to the AWS Management Console
- Go to AWS Console.
- Use your root account credentials (the email and password you used to sign up for AWS).

Access the IAM Dashboard
- In the search bar at the top of the page, type IAM and click on it.
- This will take you to the IAM (Identity and Access Management) dashboard.

Create a New User
- On the left-hand menu, click on Users.
- Click the Add users button.

Enter the User Details
- In the User name field, type the name for the user (e.g., "New-User").
- Under AWS access type, select Management Console access.
- Create a custom password or select the Autogenerated password option.
- (Optional) Enable the "User must create a new password at next sign-in" option.

Assign Permissions
- On the next page, under Set permissions, choose Attach policies directly.
- Search for AdministratorAccess in the list.
- Check the box next to AdministratorAccess to assign full admin rights.

Review and Create
- Review the details on the final page to ensure everything is correct.
- Click Create user.

Share Login Information
- Once the user is created, AWS will provide a link to the console login page.
- Share the following details with me securely:
  - The login link.
  - The username.
  - The temporary password (if autogenerated).

How to Setup Pritunl on Ubuntu Server

Samuel Ajisafe — Fri, 24 Jan 2025 07:40:09 +0000

In this tutorial, we'll walk through the complete setup of the Pritunl VPN server on Ubuntu 22.04.

Pritunl is an open-source enterprise VPN server and management platform. With its intuitive web interface and strong security features, it offers a powerful alternative to commercial VPN solutions. Pritunl supports both OpenVPN and WireGuard, and is designed to scale easily to thousands of users—perfect for modern cloud environments.

🌟 Key Features of Pritunl

Easy installation and configuration
Multi-cloud VPN peering support for AWS, Google Cloud, Azure, and Oracle Cloud
Up to five layers of authentication
Supports both OpenVPN and WireGuard
Scales seamlessly to thousands of users with high availability
Official clients available for all major platforms (compatible with any OpenVPN client)
Built on MongoDB, with fast and reliable replication support

Pritunl Architecture Overview
Pritunl uses a server-client architecture, where VPN servers and users are managed centrally via the web interface. Clients connect using downloaded profiles.

It is built on top of MongoDB, a scalable NoSQL database. This makes it easy to deploy a Pritunl cluster with built-in database replication and high availability, all without the need for expensive hardware.

✅ This setup has been fully tested on Ubuntu 22.04. I’ll continue updating this guide if there are any future changes.

Step 1: Install Required Dependencies
Start by updating your system and installing essential packages:

sudo apt update && sudo apt install -y \
  wget vim curl gnupg2 software-properties-common \
  apt-transport-https ca-certificates lsb-release

Step 2: Install Legacy OpenSSL (Required for MongoDB 6)

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 7AE645C0CF8E292A
wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2_amd64.deb
sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2_amd64.deb

Step 3: Add MongoDB and Pritunl Repositories

wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add -

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" \
  | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list

Step 4: Install MongoDB and Pritunl

sudo apt update
sudo apt install -y pritunl mongodb-org

Step 5: Start and Enable Services

sudo systemctl start pritunl mongod
sudo systemctl enable pritunl mongod
sudo systemctl status pritunl mongod

You should see both services running without issues.

Step 6: Access the Pritunl Web Interface
Now that Pritunl is running, you can access the web interface:

Open your browser and go to:
http://

Note: Make sure ports 80 and 443 are open on your server.

Generate setup-key by running the command below:

sudo pritunl setup-key

Once you enter the setup-key and MongoDB URL, it will prompt you for a username and password.

The default username and password are obtained with the below command:

sudo pritunl default-password

When you log in with the provided credentials. Set your new password and save and you should be taken to a page to configure organizations, users, and servers.

Step 8. Create a User

To add users, click on ‘Users’. This takes you to a window to first add organization.

Click on ‘Add organization’ then provide it a name then click ‘Add’.

Your organization should now be added.

Click on ‘Add user’ to create a user. Provide the required details and click ‘Add’.

If you want to add many users at once, click on ‘Bulk Add user’.

Let’s now create a Vpn server. Click on ‘servers’ then ‘Add server’.

Provide server particulars and click ‘Add’. You should see that the server has successfully been added.

Your Firewall should mimic the image below:

🧩 Step 9: Link Server to Organization
Once logged in, create an Organization, User, and Server. Then link the server to the organization to begin assigning user profiles.

Click on Server, then click on Attach Organisation

Once Attached Click on Start Server.

The setup is now completed.

Here is the guide on how to setup the client side

Follow this link 'https://client.pritunl.com/' to download the Pritunl client, mobile device download OpenVPN from Playstore or Appstore.

Install client and upload downloaded profile to client and click connect.

📚 Reference
This guide was inspired by Vinayak's Pritunl Setup Guide, with updates and improvements based on the latest best practices.

🚧 Note: The original reference has been updated. I will continue to maintain this documentation as new changes emerge.

DevOps #SRE #HybridCloud #Cloud #Security #VPN #PrivateNetwork #Network #OpenVPN #Pritunl

How to disable AWS Route53 Resolver Network Interface

Samuel Ajisafe — Thu, 09 Jan 2025 11:23:20 +0000

How I Tackled Unexpected AWS Route 53 Resolver Costs

Are you getting unexpected AWS charges for Route 53 Resolver? The costs can quickly become worrisome, especially if you're not actively using the service. As someone managing an AWS account, cost optimization was my primary goal for this project, and I leaned on several AWS tools to help with this analysis.

Tools I Used:

AWS Compute Optimizer
Billing Dashboard
Cost Explorer
Savings Plan
Reserved Instances

Identifying the Problem

While analyzing costs using Cost Explorer, I noticed a line item labeled EU-ResolverNetworkInterface. At first, I couldn't figure out where this cost was coming from. After thoroughly exploring AWS Route 53 via the console, I found no visible resolver endpoints, and searching the web didn’t offer any direct solutions. Ultimately, I realized that resolving this issue would require the AWS CLI.

What is AWS Route 53 Resolver?

Amazon Route 53 Resolver is a DNS service that enables you to manage and route DNS queries between your VPCs, on-premises networks, and the internet. If you have workloads spanning both AWS VPCs and on-premises resources, you'll likely need to resolve DNS records hosted in both environments.

Route 53 Resolver supports this by providing:

Inbound Resolver Endpoints: Allow DNS queries to your VPC from on-premises or another VPC.
Outbound Resolver Endpoints: Allow DNS queries from your VPC to on-premises resources or another VPC.

Additionally, Resolver rules enable you to create forwarding rules for specific domain names, specifying where to route DNS queries. These rules can be applied to individual VPCs and shared across multiple AWS accounts.

How I Resolved the Issue

Once I identified the cost was related to a Resolver in the EU region, I followed these steps to resolve it:

List the Resolver Endpoints: First, I ran the following CLI command to check for existing resolver endpoints in the EU West (eu-west-1) region:

   aws route53resolver list-resolver-endpoints --region eu-west-1

This showed that a resolver endpoint existed.

List Resolver Rule Associations: Next, I checked the resolver rule associations to see where the rules were applied:

   aws route53resolver list-resolver-rule-associations --region eu-west-1

This revealed the associated rule and the linked VPC.

Disassociate the Resolver Rule: To unlink the resolver rule from the VPC, I ran the following command:

   aws route53resolver disassociate-resolver-rule --vpc-id <vpc-id> --resolver-rule-id rslvr-rr-33c7383d12cb43919 --region eu-west-1

Delete the Resolver Rule: Once the rule was disassociated, I proceeded to delete it:

   aws route53resolver delete-resolver-rule --resolver-rule-id rslvr-rr-33c7383d12cb43919 --region eu-west-1

Delete the Resolver Endpoint: Finally, I deleted the resolver endpoint using the following command:

   aws route53resolver delete-resolver-endpoint --resolver-endpoint-id <resolver-endpoint-id> --region eu-west-1

Double Check: I ran the list-resolver-endpoints command again to ensure no resolver endpoints remained. If any were still there, I repeated the process.

Conclusion

In my case, it was an Outbound Resolver Endpoint causing the issue. Following the steps above helped me clear the associated costs. If you’re seeing similar charges, don’t hesitate to follow these steps and clean up unused resolver endpoints or rules.

Good luck with your AWS cost optimization!

AWS #DevOps #Cloud #Cost_Optimization #Route53 #DNS

AWS CodeDeploy: How to Fix the "Cannot Reach Instance Service" Error

Samuel Ajisafe — Sat, 16 Nov 2024 11:19:23 +0000

If you're a DevOps engineer, System engineer, or Cloud engineer using AWS CodePipeline, CodeBuild, and CodeDeploy to deploy applications to EC2 instances, you may encounter a deployment failure after a successful build. If the logs from CodeDeploy show the following error:

CodeDeploy agent was not able to receive the lifecycle event. Check the CodeDeploy agent logs on your host and make sure the agent is running and can connect to the CodeDeploy server.

Don’t panic! This guide will help you troubleshoot and resolve the issue.

Step 1: Check the Status of the CodeDeploy Agent on the EC2 Instance

The first step is to verify if the CodeDeploy agent is running on your EC2 instance. To check the status, run the following command:

sudo service codedeploy-agent status

If the agent is stopped, start it by running:

sudo service codedeploy-agent start

Step 2: Confirm IAM Role Permissions

Ensure that the EC2 instance has an IAM role attached to it, and this role must have the necessary permissions to interact with AWS CodeDeploy. The policy should include actions for CodeDeploy, S3, and CloudWatch Logs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "codedeploy:*",
        "s3:GetObject",
        "s3:ListBucket",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

Step 3: Validate IAM Role Trust Relationship

Next, ensure that the IAM role trust relationship is set up correctly. It should allow EC2 instances to assume the role. The trust relationship policy should look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Step 4: Restart the CodeDeploy Agent

Once you've confirmed that the IAM role is correctly configured, restart the CodeDeploy agent:

sudo service codedeploy-agent restart

Step 5: Check the CodeDeploy Agent Logs

If the problem persists, inspect the CodeDeploy agent logs for additional error messages that might provide insight into why the lifecycle event failed. To tail the log file:

sudo tail -f /var/log/aws/codedeploy-agent/codedeploy-agent.log

Look for any errors similar to this one:

ERROR [codedeploy-agent(3313518)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Cannot reach InstanceService: Aws::CodeDeployCommand::Errors::AccessDeniedException - Aws::CodeDeployCommand::Errors::AccessDeniedException

Step 6: Remove AWS Credentials from the Instance (if applicable)

If the error mentions AccessDeniedException, it's possible that an AWS credentials file exists on the instance (e.g., /root/.aws/credentials or /home/{user}/.aws/credentials). If such a file exists, it might be interfering with the CodeDeploy agent’s ability to connect.

To fix this:

Delete the credentials file:

sudo rm -rf /root/.aws/credentials
# or for a specific user:
sudo rm -rf /home/{user}/.aws/credentials

Restart the CodeDeploy agent:

sudo systemctl restart codedeploy-agent

Conclusion

By following these steps, you should be able to resolve the "CodeDeploy cannot reach instance service" error and get your deployments back on track. If the issue persists, revisit the IAM role permissions and the CodeDeploy agent logs to gather more information.

References:

Cloud #AWS #DevOps #Automation #CI/CD #System #Engineer #CodeDeploy #CodePipeline #EC2