DEV Community: gokcedemirdurkut

Upgrading RDS MySQL from 8.0 to 8.4 behind RDS Proxy

gokcedemirdurkut — Thu, 09 Jul 2026 10:09:34 +0000

RDS for MySQL 8.0 reaches end of standard support on July 31, 2026 after that AWS moves you onto paid Extended Support until you upgrade, so moving our production databases to 8.4 wasn't really optional. The engine upgrade itself is a non-event: pick the target version, RDS runs its prechecks, takes a snapshot, and a while later you're on 8.4. The part that actually needed thinking was authentication, because all of our services connect through RDS Proxy, and 8.4 changes how the old password plugin works.

If you run the same setup, here's what's worth knowing before you click the button.

The moving parts

If RDS Proxy and MySQL auth plugins are already familiar, skip ahead.

Amazon RDS is a managed MySQL database AWS runs the server so you don't have to.
RDS Proxy sits in front of it like a hotel front desk. Your applications don't walk up to the room (the database) holding a key; they show ID at the desk, and the desk opens the room for them. It also pools connections so the database isn't swamped.
An authentication plugin is just how MySQL checks a password think of it as the type of lock on a door. The old one, mysql_native_password, uses SHA-1 (an older, weaker recipe). The newer caching_sha2_password uses SHA-256 (stronger). MySQL 8.4 stops shipping the old lock by default and expects the new one.

So the whole story below is really: we changed the locks, and the only door that turned out to matter was the one between the front desk and the room.

There are two auth legs, not one

The thing to internalize first: with RDS Proxy in the path, a connection authenticates twice.

Service to proxy. Our services don't hold a database password at all. A sidecar mints an IAM auth token and the service uses that to reach the proxy over TLS. This leg is pure IAM the MySQL auth plugin has nothing to do with it.
Proxy to database. The proxy then opens the real backend connection using a username/password it pulls from Secrets Manager (one secret per DB user: the app users, a read-only user, the master).

That split matters because the 8.4 authentication change only touches the second leg. Whatever you read about mysql_native_password being disabled applies to how the database authenticates the proxy's connection not to how your services reach the proxy.

The mysql_native_password scare that wasn't

Community MySQL 8.4 doesn't load mysql_native_password by default anymore. Our users including the RDS master user were all created with that plugin, so the obvious worry was: the moment we're on 8.4, the proxy can't authenticate to the backend and everything falls over.

We were ready to flip a parameter to keep the plugin on. Then we actually looked at the 8.4 parameter group:

On RDS for MySQL 8.4, AWS keeps the plugin loaded for you, and you can't turn it off through the parameter group. So existing mysql_native_password users master included keep working after the upgrade with zero changes. The thing we'd braced for simply never happened. The flip side: you can't disable native password on RDS 8.4 either, so a hard "everyone must be caching_sha2" cutover isn't a thing you do here. That only becomes real whenever RDS eventually ships a 9.x line, where the plugin is gone for good.

So the honest answer for an RDS 8.0 → 8.4 upgrade: on the auth front, you don't have to do anything. It keeps working.

Moving to caching_sha2_password anyway

We still moved every user we control to caching_sha2_password not because 8.4 forced us, but because 9.x will. Whenever RDS ships a 9.x line, the native password plugin is gone for good, and a database full of native-password users turns a routine engine bump into an auth scramble at the worst possible time. Doing the migration now, while it's optional and the old plugin is still around as a safety net, means that future upgrade is just a version change. It's also the better plugin on its own merits (SHA-256 instead of SHA-1).

Two preconditions that usually make this annoying were already true for us:

RDS Proxy has supported caching_sha2_password since December 2024, and
it requires TLS which we already enforce on both the proxy (require_tls) and the instance (require_secure_transport).

So there was no new TLS work to do. All of it lives in Terraform pointing the proxy's client auth at caching_sha2, keeping require_tls on the proxy and require_secure_transport on the instance so the change was a small reviewable diff rather than console clicking. The users themselves we migrate in the small Lambda that initializes the database, using a create-or-migrate pattern:

CREATE USER IF NOT EXISTS `app`@`%` IDENTIFIED WITH caching_sha2_password BY '...';
ALTER USER `app`@`%` IDENTIFIED WITH caching_sha2_password BY '...';

The CREATE ... IF NOT EXISTS handles fresh environments; the ALTER flips the plugin for users that already exist. The order matters: we applied the Terraform first, then triggered that Lambda so the ALTERs actually ran against the live users. It normally only fires once, when the proxy is first created so migrating an already-running environment meant re-running it on purpose. Our services needed no changes for this no driver upgrades, no config, no redeploys. Modern MySQL drivers already speak caching_sha2_password, so the switch was invisible to the applications.

The master user we never touched directly. It's RDS-managed (manage_master_user_password), so RDS rotates its password on a schedule and on 8.4 that managed rotation moved the master onto caching_sha2_password on its own, with no action from us. (Worth knowing: this is RDS's own managed rotation doing it. A plain rotation that runs SET PASSWORD only changes the password and keeps the existing plugin it wouldn't migrate anything.) The only accounts left on native password are the internal ones AWS runs (the RDS admin user, the mysql.* system schemas), and those keep working untouched.

That's why you never have to set a parameter here: RDS keeps mysql_native_password loaded by default for its own accounts, so anything not yet on caching_sha2_password just keeps authenticating.

One catch: Aurora read replicas

If you have an Aurora MySQL read replica hanging off your RDS instance often a leftover from a half-finished move to Aurora it can block the upgrade outright. A replica can't run an older major version than its source, and Aurora MySQL only got an 8.4-compatible version in May 2026, about a year and a half after community 8.4. Until then, moving the RDS source to 8.4 would have broken replication, so the replica quietly vetoes the whole upgrade a big reason some teams stayed on 8.0. The version gap is closed now, but if you run this setup, check the replica side before you touch the source.

In short

The hard part of moving to 8.4 behind RDS Proxy isn't the engine bump it's the auth model, and three things carry most of it. Connections authenticate in two separate legs, and the 8.4 plugin change only touches the proxy-to-database one. RDS keeps mysql_native_password loaded for you, so nothing breaks on its own. And switching to caching_sha2_password costs almost nothing once TLS is already enforced. Get those right and there's very little left to worry about.

Why Your Nginx Security Headers Disappear (add_header Inheritance Explained)

gokcedemirdurkut — Thu, 12 Mar 2026 09:53:58 +0000

Security headers can disappear in Nginx even when they are correctly configured.

Some endpoints may silently drop important headers like:

Content-Security-Policy
Strict-Transport-Security
X-Frame-Options

This behavior is subtle and has existed in Nginx for a long time.

How the Problem Appears

Everything looks correct at first.

curl -I https://example.com/api/health

Response headers include security policies as expected.

But then another endpoint is checked:

curl -I https://example.com/

And suddenly:

❌ Content-Security-Policy missing
❌ Strict-Transport-Security missing
❌ X-Frame-Options missing

Tools like securityheaders.com will immediately detect this.

The confusing part is that the headers are clearly defined in the configuration.

A Common Configuration

A typical setup may look like this:

server {
    add_header Content-Security-Policy "...";
    add_header Strict-Transport-Security "max-age=31536000";

    location ~* \.html$ {
        add_header Cache-Control "no-store";
    }
}

The intention is simple:

disable caching for HTML responses
apply security headers globally

However, this configuration does not behave as expected.

What Actually Happens

Responses served from:

location ~* \.html$

will only include:

Cache-Control: no-store

All security headers defined in the server block disappear.

There are no warnings and no errors in the logs.
The headers are simply missing.

The Reason

This happens because of how Nginx handles header inheritance.

If a location block defines any add_header directive, it does not inherit add_header directives from parent blocks.

Even adding a single header like:

Cache-Control

causes Nginx to drop previously defined headers such as:

Content-Security-Policy
Strict-Transport-Security
X-Frame-Options
Referrer-Policy
Permissions-Policy

This behavior has existed for many years and has caused confusion in many configurations.

The Previous Workarounds

Before recently, the usual approaches were:

1. Duplicating headers

Define all security headers again in every location block.

2. Using includes

Create a file like:

include security_headers.conf;

and include it everywhere.

Both approaches work but are difficult to maintain and easy to forget.

The Fix in Nginx 1.29+

Recent Nginx versions introduced a better solution:

add_header_inherit merge;

This directive changes how headers are inherited.

Instead of replacing headers defined in parent blocks, child blocks merge them.

This feature was introduced in newer Nginx releases and is documented in the official release notes:

https://blog.nginx.org/blog/nginx-open-source-1-29-3-and-1-29-4

A Safer Configuration

With the new directive, the configuration can be written like this:

server {

    add_header_inherit merge;

    add_header Content-Security-Policy "...security policy..." always;
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;

    location ~* \.html$ {
        add_header Cache-Control "no-store, no-cache, private, max-age=0" always;
    }

}

Now the behavior is predictable:

HTML responses are not cached
Security headers are applied consistently
Headers do not need to be duplicated

Final Takeaway

When using add_header inside location blocks in Nginx 1.29+, enabling the following directive is recommended:

add_header_inherit merge;

This prevents accidental loss of security headers and makes configurations easier to maintain.

Preventing Silent ECS Deployment Failures with Circuit Breaker

gokcedemirdurkut — Thu, 26 Feb 2026 10:57:55 +0000

AWS Elastic Container Service (ECS) provides a built-in feature called the deployment circuit breaker, designed to make service deployments safer and more resilient.

This feature continuously monitors the health of tasks during a deployment and automatically rolls back changes if newly launched tasks fail to become healthy. When enabled, it prevents failed deployments from leaving services in a degraded or non-functional state.

Without this safeguard, deployment failures can easily go unnoticed. For example, if new tasks fail to start or never pass health checks, the service may still appear to be running while it is effectively broken. These silent failures can result in data loss, financial impact, or operational issues depending on the workload.

In this post, I’ll walk through how to enable the ECS deployment circuit breaker using Terraform, how to observe deployment failures via EventBridge, and how to send real-time alerts to Slack.

Why the ECS Deployment Circuit Breaker Matters

Enabling the deployment circuit breaker provides several important benefits:

Automatic rollback – Failed deployments are reverted to the last known healthy service revision
Improved visibility – ECS emits structured events whenever a deployment fails or rolls back
Reduced operational overhead – Failures are mitigated automatically without immediate manual intervention

Together, these significantly reduce the risk of production incidents caused by faulty deployments.

Enabling the Circuit Breaker with Terraform

The deployment circuit breaker can be enabled directly in your ECS service definition. In Terraform, this is done using the deployment_circuit_breaker block:

resource "aws_ecs_service" "default" {
  name            = "tuve"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.default.arn
  desired_count   = 2

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }

  ...
}

With this configuration in place, ECS will automatically stop and roll back a deployment if the new tasks fail to reach a healthy state.

Once enabled, the AWS Management Console clearly indicates that the Deployment circuit breaker is turned on.

Observing Deployment Failures

Automatic rollback is useful, but visibility is just as important.

When the ECS deployment circuit breaker triggers, ECS emits events to Amazon EventBridge with the following detail type:

ECS Deployment State Change

Here is an example event payload:

{
  "version": "0",
  "id": "ddca6449-b258-46c0-8653-e0e3aEXAMPLE",
  "detail-type": "ECS Deployment State Change",
  "source": "aws.ecs",
  "account": "111122223333",
  "time": "2020-05-23T12:31:14Z",
  "region": "eu-central-1",
  "resources": [
    "arn:aws:ecs:eu-central-1:111122223333:service/default/servicetest"
  ],
  "detail": {
    "eventType": "ERROR",
    "eventName": "SERVICE_DEPLOYMENT_FAILED",
    "deploymentId": "ecs-svc/123",
    "updatedAt": "2020-05-23T11:11:11Z",
    "reason": "ECS deployment circuit breaker: task failed to start."
  }
}

Key Fields to Monitor

Some fields in this event are particularly useful for monitoring and alerting:

eventName
- SERVICE_DEPLOYMENT_FAILED
- SERVICE_DEPLOYMENT_ROLLBACK_COMPLETED
reason – Explains why the deployment failed
resources – Identifies the affected ECS service
updatedAt – Indicates when the failure occurred

Tracking these fields ensures that deployment issues are visible immediately instead of being discovered hours later.

Deployment Rollback in the AWS Console

The AWS Management Console also provides clear visibility into rollback activity. After a failed deployment, the Deployments tab shows the rollback status along with the target service revision.

This view is particularly useful for confirming that the circuit breaker worked as expected.

Sending Deployment Alerts to Slack

To ensure deployment failures are noticed immediately, ECS deployment events can be routed to Slack using EventBridge and Lambda.

The overall flow looks like this:

ECS → EventBridge → Lambda → Slack

Lambda Handler Example

The Lambda function listens for ECS deployment state changes and sends notifications when a deployment fails or rolls back:

def lambda_handler(event, context):
    detail_type = event.get("detail-type", "")

    if detail_type == "ECS Deployment State Change":
        event_name = event.get("detail", {}).get("eventName")

        if event_name in [
            "SERVICE_DEPLOYMENT_FAILED",
            "SERVICE_DEPLOYMENT_ROLLBACK_COMPLETED"
        ]:
            detail = event.get("detail", {})
            resources = event.get("resources", [])
            service_name = resources[0].split("/")[-1] if resources else "unknown"
            reason = detail.get("reason", "Unknown")
            updated_at = detail.get("updatedAt", "Unknown")

            send_slack_notification(
                service=service_name,
                reason=reason,
                event_type=event_name,
                timestamp=updated_at
            )

EventBridge Rule (Terraform)

The following EventBridge rule filters ECS deployment events and forwards them to the Lambda function:

resource "aws_cloudwatch_event_rule" "ecs_deployment" {
  name = "ecs-deployment-events"

  event_pattern = jsonencode({
    "source": ["aws.ecs"],
    "detail-type": ["ECS Deployment State Change"],
    "detail": {
      "eventName": [
        "SERVICE_DEPLOYMENT_FAILED",
        "SERVICE_DEPLOYMENT_ROLLBACK_COMPLETED"
      ]
    }
  })
}

resource "aws_cloudwatch_event_target" "lambda" {
  rule = aws_cloudwatch_event_rule.ecs_deployment.name
  arn  = aws_lambda_function.notification.arn
}

Final Outcome

After enabling the ECS deployment circuit breaker and adding Slack notifications:

Failed deployments automatically roll back
Silent service failures are eliminated
Deployment issues become visible in real time
ECS services are safer by default

By combining automated rollback with real-time alerts, you can significantly reduce operational risk and increase confidence in your ECS deployments.

Deploying and Customizing AWS ParallelCluster Service (PCS) for HPC Workloads

gokcedemirdurkut — Sun, 26 Oct 2025 09:15:02 +0000

I recently worked on a project involving AWS ParallelCluster Service (PCS).

The main goal was to build an HPC cluster that meets our specific requirements such as using an image with Python 3.10, installing the necessary dependencies and deploying a PCS cluster.
To achieve this, I built a custom AMI(Amazon Machine Image) using Packer, then launched a PCS cluster based on that image.
Throughout the process, I automated the workflow using Terraform, GitHub Actions and shell scripts.
After deploying the cluster, I verified that everything was working correctly by running several SLURM job commands, which confirmed that our setup was operational.

In this article, I’ll walk you through the entire process from building the custom AMI to running jobs on PCS.

What is AWS PCS?

AWS ParallelCluster Service (PCS) is a managed service that enables high-performance computing (HPC) on AWS. It’s designed for running parallel workloads such as simulations, ML training, or large-scale data analysis.
Using PCS, you can deploy and manage HPC clusters without manually configuring compute nodes, networking or schedulers. It integrates seamlessly with services like Amazon S3 and AWS Batch, supporting complex workloads efficiently.

Why Use AWS PCS?

Traditionally, deploying and managing HPC clusters required deep expertise in cluster configuration, job scheduling, and infrastructure management.
AWS PCS abstracts much of this complexity by offering:

Fully managed cluster orchestration
Integration with SLURM scheduler
Elastic scaling based on job demand
Infrastructure as Code (IaC) support with Terraform and CloudFormation

This makes AWS PCS an excellent choice for researchers, data scientists, and DevOps engineers who want to focus on workloads rather than infrastructure.

How Was the PCS Cluster Created?

The PCS cluster was provisioned using Terraform with the awscc
provider. The infrastructure is defined as code and includes the cluster, login/compute node groups and job queue.

Note: The awscc provider was required because PCS resources are not yet supported by the standard AWS provider. It uses the AWS Cloud Control API to manage newer services like PCS.

📢 Official Announcement — Terraform Support for AWS ParallelCluster Service (March 2025)

PCS Cluster Components

A PCS setup typically includes three main components:

Cluster: Defines general settings, scheduler configuration, and networking.
Node Groups: Specifies login and compute nodes.
Queue: Manages job scheduling and execution PCS Setup with Terraform.

Below is a simplified example of how a PCS cluster can be created using Terraform and the awscc provider:

# Create PCS Cluster using awscc provider
resource "awscc_pcs_cluster" "this" {
  count = var.enable_cluster ? 1 : 0
  name  = var.cluster_name
  size  = var.cluster_size

  scheduler = {
    type    = "SLURM"
    version = var.cluster_slurm_version
  }

  networking = {
    security_group_ids = [aws_security_group.this.id]
    subnet_ids         = var.subnet_ids
  }
}

Variables like cluster_name, subnet_ids, and cluster_slurm_version can be parameterized to adapt across environments.

Cost Awareness & Usage Control

Running a PCS cluster can become expensive depending on your configuration.
Each compute node in the cluster uses Amazon EC2 instances, and costs increase while those instances are running.

To manage this, we added a Terraform variable that controls whether the PCS cluster should be deployed:

# Control cluster deployment
enable_pcs_cluster = true   # create the cluster
# enable_pcs_cluster = false  # skip deployment

This allows you to enable or disable the PCS cluster as needed.
For example, during development or testing, you can set enable_pcs_cluster = false to avoid unnecessary charges.

Creating Custom AMIs with Packer

Custom AMIs let you pre-install software and dependencies on compute nodes.

Steps:

Create a Packer template based on a base AMI (e.x. Ubuntu or Amazon Linux 2).
Install the required software packages.
Build the AMI and use it in your PCS cluster.

Example Packer Template:

{
  "builders": [{
      "type": "amazon-ebs",
      "region": "{{user `aws_region`}}",
      "source_ami": "{{user `source_ami`}}",
      "instance_type": "{{user `instance_type`}}",
      "ssh_username": "{{user `ssh_username`}}",
      "ami_name": "{{user `ami_name_prefix`}}-{{user `distribution`}}-{{user `architecture`}}-{{isotime `2006.01.02-15.04`}}",
      "ami_description": "{{user `ami_description`}}"
  }],
  "provisioners": [{
    "type": "shell",
    "inline": [
      "sudo yum update -y",
      "sudo yum install -y gcc make python3"
      ...
    ]
  }]
}

This is just an example. You can add other dependencies as needed.

We used SLURM as the job scheduler within PCS to manage HPC workloads efficiently.
It handles queueing, job submission, and node allocation allowing multiple users or jobs to share compute resources dynamically.

During our setup, we required Python 3.10, but the default Amazon Linux 2 AMI provided only Python 3.7.
To solve this, we created a custom AMI using Packer, based on the following Ubuntu Marketplace image:

Ubuntu Server 22.04 LTS (arm64)
AMI ID: ami-0f45e2f16611e3139
Source: AWS Marketplace

This custom AMI included:

Python 3.10 preinstalled
Common HPC dependencies (gcc, make, python3-pip)
Additional Python libraries required for our workloads

By using a custom AMI, we ensured compatibility with our codebase and reduced setup time for new compute nodes.

Note: Custom AMIs are especially helpful when your HPC workloads require specific compiler versions, Python environments, or third-party libraries.

Testing the SLURM Cluster

Once the PCS cluster was up and running, we validated the environment by executing a few basic SLURM commands:

# Check the cluster and nodes
sinfo

# Submit a test job
echo "echo Hello from SLURM" > test.sh
sbatch test.sh

# View the job queue
squeue

These simple tests confirmed that the SLURM scheduler was active, compute nodes were responding, and jobs were successfully executed across the PCS cluster.

Conclusion

AWS PCS simplifies HPC cluster management with a managed, scalable, and elastic environment.
Terraform (awscc) enables modern Infrastructure as Code deployment.
SLURM provides flexible and familiar scheduling for HPC users.
Custom AMIs and bootstrap scripts enable deep customization and reproducibility.
Ideal for research, AI/ML training, and data-intensive simulations.

Thanks for reading!

#terraform #aws #devops #cloud #hpc #slurm #packer #ami #python