<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: gokcedemirdurkut</title>
    <description>The latest articles on DEV Community by gokcedemirdurkut (@gokcedemirdurkut).</description>
    <link>https://dev.to/gokcedemirdurkut</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3529539%2Fcb6545d7-dba4-4a74-8480-2fe490dcac93.jpeg</url>
      <title>DEV Community: gokcedemirdurkut</title>
      <link>https://dev.to/gokcedemirdurkut</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/gokcedemirdurkut"/>
    <language>en</language>
    <item>
      <title>Trivy vs AWS Security Hub: Terraform Misconfiguration Scanning on AWS, Explained</title>
      <dc:creator>gokcedemirdurkut</dc:creator>
      <pubDate>Mon, 16 Nov 2026 13:12:13 +0000</pubDate>
      <link>https://dev.to/gokcedemirdurkut/trivy-vs-aws-security-hub-terraform-misconfiguration-scanning-on-aws-explained-12en</link>
      <guid>https://dev.to/gokcedemirdurkut/trivy-vs-aws-security-hub-terraform-misconfiguration-scanning-on-aws-explained-12en</guid>
      <description>&lt;p&gt;In the &lt;a href="https://dev.to/gokcedemirdurkut/trivy-vs-amazon-inspector-container-image-scanning-on-aws-explained-35bo"&gt;previous post&lt;/a&gt; I used Trivy to scan container images. But an image is only part of what we ship. The rest is the infrastructure around it: security groups, S3 buckets, databases. If you write that in Terraform, the same tool can check it too.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does an IaC scan look for?
&lt;/h2&gt;

&lt;p&gt;An image scan looks for &lt;strong&gt;known vulnerabilities&lt;/strong&gt; in software, like an old OpenSSL version.&lt;/p&gt;

&lt;p&gt;An IaC scan looks for &lt;strong&gt;risky settings&lt;/strong&gt; in your infrastructure code. Trivy calls these &lt;em&gt;misconfigurations&lt;/em&gt;. The code works fine. It just does something you probably don't want, for example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a security group that allows SSH from anywhere,&lt;/li&gt;
&lt;li&gt;an S3 bucket with no public access block,&lt;/li&gt;
&lt;li&gt;an RDS instance with unencrypted storage.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Trivy reads your &lt;code&gt;.tf&lt;/code&gt; files and checks them against its built-in rules. It doesn't need AWS credentials and doesn't talk to AWS, so you can run it on your laptop or in CI before anything is applied.&lt;/p&gt;

&lt;h2&gt;
  
  
  A first scan
&lt;/h2&gt;

&lt;p&gt;Here's a small &lt;code&gt;main.tf&lt;/code&gt; with a few problems on purpose:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="n"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_security_group"&lt;/span&gt; &lt;span class="s2"&gt;"web"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;name&lt;/span&gt;        &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"web"&lt;/span&gt;
  &lt;span class="n"&gt;description&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"Web server"&lt;/span&gt;

  &lt;span class="n"&gt;ingress&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;description&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"SSH"&lt;/span&gt;
    &lt;span class="n"&gt;from_port&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;22&lt;/span&gt;
    &lt;span class="n"&gt;to_port&lt;/span&gt;     &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;22&lt;/span&gt;
    &lt;span class="n"&gt;protocol&lt;/span&gt;    &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"tcp"&lt;/span&gt;
    &lt;span class="n"&gt;cidr_blocks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"0.0.0.0/0"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_s3_bucket"&lt;/span&gt; &lt;span class="s2"&gt;"uploads"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;bucket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"my-app-uploads"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_db_instance"&lt;/span&gt; &lt;span class="s2"&gt;"main"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;identifier&lt;/span&gt;          &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"main"&lt;/span&gt;
  &lt;span class="n"&gt;engine&lt;/span&gt;              &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"postgres"&lt;/span&gt;
  &lt;span class="n"&gt;instance_class&lt;/span&gt;      &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"db.t4g.micro"&lt;/span&gt;
  &lt;span class="n"&gt;allocated_storage&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;
  &lt;span class="n"&gt;username&lt;/span&gt;            &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"app"&lt;/span&gt;
  &lt;span class="n"&gt;password&lt;/span&gt;            &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kd"&gt;var&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;db_password&lt;/span&gt;
  &lt;span class="n"&gt;skip_final_snapshot&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;variable&lt;/span&gt; &lt;span class="s2"&gt;"db_password"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;type&lt;/span&gt;      &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;
  &lt;span class="n"&gt;sensitive&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run Trivy on the folder:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;trivy config ./terraform
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For these 31 lines it reports &lt;strong&gt;14 findings&lt;/strong&gt;: 7 HIGH, 4 MEDIUM and 3 LOW. That's normal for code nobody has scanned before. To see only the important ones first:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;trivy config --severity HIGH,CRITICAL ./terraform
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;main.tf (terraform)
===================
Tests: 7 (SUCCESSES: 0, FAILURES: 7)
Failures: 7 (HIGH: 7, CRITICAL: 0)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each finding looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AWS-0107 (HIGH): Security group rule allows unrestricted ingress from any IP address.
════════════════════════════════════════
Security groups provide stateful filtering of ingress and egress network traffic to AWS
resources. It is recommended that no security group allows unrestricted ingress access to
remote server administration ports, such as SSH to port 22 and RDP to port 3389.


See https://avd.aquasec.com/misconfig/aws-0107
────────────────────────────────────────
 main.tf:10
   via main.tf:5-11 (ingress)
    via main.tf:1-12 (aws_security_group.web)
────────────────────────────────────────
   1   resource "aws_security_group" "web" {
   .
  10 [     cidr_blocks = ["0.0.0.0/0"]
  ..
  12   }
────────────────────────────────────────
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You get the check ID (&lt;code&gt;AWS-0107&lt;/code&gt;), a severity, a short explanation, a link with the fix, and the exact file and line. Older Trivy versions print the ID as &lt;code&gt;AVD-AWS-0107&lt;/code&gt;. It's the same check.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fixing them
&lt;/h2&gt;

&lt;p&gt;Most fixes are small:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="n"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_security_group"&lt;/span&gt; &lt;span class="s2"&gt;"web"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;# ...&lt;/span&gt;
  &lt;span class="n"&gt;ingress&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;description&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"SSH from the office VPN only"&lt;/span&gt;
    &lt;span class="n"&gt;from_port&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;22&lt;/span&gt;
    &lt;span class="n"&gt;to_port&lt;/span&gt;     &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;22&lt;/span&gt;
    &lt;span class="n"&gt;protocol&lt;/span&gt;    &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"tcp"&lt;/span&gt;
    &lt;span class="n"&gt;cidr_blocks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kd"&gt;var&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;admin_cidr&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_s3_bucket_public_access_block"&lt;/span&gt; &lt;span class="s2"&gt;"uploads"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;bucket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;aws_s3_bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uploads&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;

  &lt;span class="n"&gt;block_public_acls&lt;/span&gt;       &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="n"&gt;block_public_policy&lt;/span&gt;     &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="n"&gt;ignore_public_acls&lt;/span&gt;      &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="n"&gt;restrict_public_buckets&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_db_instance"&lt;/span&gt; &lt;span class="s2"&gt;"main"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;# ...&lt;/span&gt;
  &lt;span class="n"&gt;storage_encrypted&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="n"&gt;deletion_protection&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The public access block alone clears five findings, because Trivy checks each of its four settings separately and also checks that the block exists at all. So don't panic at the first number: 14 findings doesn't mean 14 pieces of work.&lt;/p&gt;

&lt;p&gt;Not every finding needs a fix, either. For example, Trivy flags S3 buckets that don't use your own KMS key (&lt;code&gt;AWS-0132&lt;/code&gt;). S3 already encrypts new objects by default, so for many buckets that's a decision rather than a bug. We'll come back to how to record that decision.&lt;/p&gt;

&lt;h2&gt;
  
  
  What AWS gives you
&lt;/h2&gt;

&lt;p&gt;AWS can find the same problems, but from the other side: it looks at the &lt;strong&gt;resources in your account&lt;/strong&gt;, not at your code.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://aws.amazon.com/config/" rel="noopener noreferrer"&gt;AWS Config&lt;/a&gt;&lt;/strong&gt; records the configuration of your resources and evaluates them against rules. AWS provides managed rules for common checks, and you can write your own.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://docs.aws.amazon.com/securityhub/latest/userguide/what-is-securityhub.html" rel="noopener noreferrer"&gt;AWS Security Hub CSPM&lt;/a&gt;&lt;/strong&gt; runs security standards on top of Config. Its &lt;em&gt;AWS Foundational Security Best Practices&lt;/em&gt; standard is a ready-made set of controls, and it gathers the results from all your accounts in one place.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The checks overlap a lot. Every HIGH finding we fixed above has a matching Security Hub control:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Problem&lt;/th&gt;
&lt;th&gt;Trivy&lt;/th&gt;
&lt;th&gt;Security Hub control&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;SSH open to the internet&lt;/td&gt;
&lt;td&gt;AWS-0107&lt;/td&gt;
&lt;td&gt;EC2.19: Security groups should not allow unrestricted access to ports with high risk (port 22 is on its list)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S3 public access not blocked&lt;/td&gt;
&lt;td&gt;AWS-0086 / 0087 / 0091 / 0093&lt;/td&gt;
&lt;td&gt;S3.8: S3 general purpose buckets should block public access&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RDS storage not encrypted&lt;/td&gt;
&lt;td&gt;AWS-0080&lt;/td&gt;
&lt;td&gt;RDS.3: RDS DB instances should have encryption at-rest enabled&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RDS deletion protection off&lt;/td&gt;
&lt;td&gt;AWS-0177&lt;/td&gt;
&lt;td&gt;RDS.8: RDS DB instances should have deletion protection enabled&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;So the real difference isn't &lt;em&gt;what&lt;/em&gt; they check. It's &lt;strong&gt;when&lt;/strong&gt; and &lt;strong&gt;where&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Side by side
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Trivy &lt;code&gt;config&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;AWS Config / Security Hub CSPM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Looks at&lt;/td&gt;
&lt;td&gt;your Terraform code&lt;/td&gt;
&lt;td&gt;real resources in your account&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;When&lt;/td&gt;
&lt;td&gt;before &lt;code&gt;terraform apply&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;after the resource exists, continuously&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Changes made in the console&lt;/td&gt;
&lt;td&gt;not seen&lt;/td&gt;
&lt;td&gt;caught&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Needs AWS access&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;td&gt;runs inside your account&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Where you see results&lt;/td&gt;
&lt;td&gt;CI log, GitHub issues&lt;/td&gt;
&lt;td&gt;AWS console, across all accounts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost&lt;/td&gt;
&lt;td&gt;free&lt;/td&gt;
&lt;td&gt;Config: per configuration item recorded (USD 0.003) and per rule evaluation; Security Hub CSPM: per security check (USD 0.001 for the first 100,000), plus Config's configuration items&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The cost row is the same story as Inspector in part 1: AWS's tools are priced per resource and per evaluation, so the bill grows with your account. Trivy costs nothing but a few seconds of CI time.&lt;/p&gt;

&lt;p&gt;The "console changes" row matters too. If someone opens a port by hand at 2 a.m. during an incident, Trivy will never know, because it only reads the code. AWS Config will.&lt;/p&gt;

&lt;h2&gt;
  
  
  Running it in GitHub Actions
&lt;/h2&gt;

&lt;p&gt;In part 1, I left the choice between blocking the build and reporting the findings up to you. For Terraform, we report: every finding becomes a &lt;strong&gt;GitHub issue&lt;/strong&gt;, the build keeps going, and the issues get picked up like any other work.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Terraform scan&lt;/span&gt;

&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;push&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;branches&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;main&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;paths&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"terraform/**"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="na"&gt;permissions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
  &lt;span class="na"&gt;issues&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt;

&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;trivy-config&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v7&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Scan Terraform with Trivy&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;aquasecurity/trivy-action@v0.36.0&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;scan-type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;config&lt;/span&gt;
          &lt;span class="na"&gt;scan-ref&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;terraform&lt;/span&gt;
          &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;MEDIUM,HIGH,CRITICAL&lt;/span&gt;
          &lt;span class="na"&gt;format&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;json&lt;/span&gt;
          &lt;span class="na"&gt;output&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;trivy-config.json&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Open an issue per finding&lt;/span&gt;
        &lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;GH_TOKEN&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ github.token }}&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
          &lt;span class="c1"&gt;# Titles we already know about: open issues, and ones closed as "not planned"
&lt;/span&gt;          &lt;span class="s"&gt;known=$(gh issue list --state all --limit 1000 --json title,state,stateReason \&lt;/span&gt;
            &lt;span class="s"&gt;--jq '.[] | select(.state == "OPEN" or .stateReason == "NOT_PLANNED") | .title')&lt;/span&gt;

          &lt;span class="na"&gt;jq -c '.Results[]? | .Target as $file | .Misconfigurations[]? | . + {File&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$file}' trivy-config.json |&lt;/span&gt;
          &lt;span class="s"&gt;while read -r finding; do&lt;/span&gt;
            &lt;span class="na"&gt;title=$(jq -r '"[\(.Severity)] \(.ID)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;\(.CauseMetadata.Resource) - \(.Title)"' &amp;lt;&amp;lt;&amp;lt; "$finding")&lt;/span&gt;
            &lt;span class="s"&gt;if grep -qxF -- "$title" &amp;lt;&amp;lt;&amp;lt; "$known"; then&lt;/span&gt;
              &lt;span class="na"&gt;echo "Already tracked&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$title"&lt;/span&gt;
              &lt;span class="s"&gt;continue&lt;/span&gt;
            &lt;span class="s"&gt;fi&lt;/span&gt;
            &lt;span class="s"&gt;body=$(jq -r '"**File:** \(.File)\n**Resource:** \(.CauseMetadata.Resource)\n\n\(.Description)\n\n**How to fix:** \(.Resolution)\n\n\(.PrimaryURL)"' &amp;lt;&amp;lt;&amp;lt; "$finding")&lt;/span&gt;
            &lt;span class="s"&gt;gh issue create --title "$title" --body "$body"&lt;/span&gt;
          &lt;span class="s"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;How it works:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The scan doesn't fail the job.&lt;/strong&gt; Without &lt;code&gt;exit-code&lt;/code&gt;, Trivy exits with 0 even when it finds something. It writes the results to &lt;code&gt;trivy-config.json&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One issue per finding.&lt;/strong&gt; The title has the severity, check ID and resource, for example &lt;code&gt;[HIGH] AWS-0107: aws_security_group.web - Security groups should not allow unrestricted ingress to SSH or RDP from any IP address.&lt;/code&gt; That makes the issue list easy to sort and search.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No duplicates.&lt;/strong&gt; If an open issue with the same title exists, it's skipped. So the workflow can run on every push without flooding the repository.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"Won't fix" is a closed issue.&lt;/strong&gt; For a finding you decide to accept, like the KMS key example above, close the issue as &lt;strong&gt;Not planned&lt;/strong&gt; and leave a comment explaining why. The workflow sees that and won't open it again. The decision and the reason stay in the issue history, where anyone can find them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fixed means closed as completed.&lt;/strong&gt; If the finding comes back later, the next run opens a new issue for it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A note on cost: the GitHub Security tab could show these findings too (&lt;code&gt;format: sarif&lt;/code&gt;), but as I mentioned in part 1, for private repositories that requires the paid GitHub Code Security add-on. Issues are free and good enough for us.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which one should you pick?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Trivy is enough if:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;all (or almost all) of your infrastructure is in Terraform,&lt;/li&gt;
&lt;li&gt;you want to find problems while the code is still in a PR, not after it's live,&lt;/li&gt;
&lt;li&gt;you want the findings in GitHub, next to the code,&lt;/li&gt;
&lt;li&gt;and you'd rather not pay per resource.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Add AWS Config / Security Hub CSPM if:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;people still change things in the console,&lt;/li&gt;
&lt;li&gt;you have resources that were never imported into Terraform,&lt;/li&gt;
&lt;li&gt;you manage many accounts and want one view across all of them,&lt;/li&gt;
&lt;li&gt;or an auditor needs a continuous compliance report.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Or use both.&lt;/strong&gt; They don't compete. Trivy catches the problem before it's created, and AWS catches whatever got past it or was changed by hand. If you only pick one to start with and your infrastructure is mostly Terraform, Trivy gives you the most for free.&lt;/p&gt;

&lt;h2&gt;
  
  
  Wrap-up
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;trivy config&lt;/code&gt; checks your Terraform for risky AWS settings before anything is applied. It's free and needs no AWS access.&lt;/li&gt;
&lt;li&gt;Expect a long list on the first run. Many findings share one fix, and some are decisions, not bugs.&lt;/li&gt;
&lt;li&gt;You don't have to fail the build. Turning findings into issues, and closing the ones you accept as &lt;em&gt;not planned&lt;/em&gt; with a reason, keeps every decision on record.&lt;/li&gt;
&lt;li&gt;AWS Config and Security Hub CSPM check the same kinds of things, but on live resources. That's the only way to catch changes made outside Terraform.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Put together with part 1, the picture looks like this: Trivy checks your images and your Terraform before they ship, and AWS's own services watch what's actually running. Neither is the whole story on its own.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>terraform</category>
      <category>security</category>
      <category>githubactions</category>
    </item>
    <item>
      <title>Trivy vs Amazon Inspector: Container Image Scanning on AWS, Explained</title>
      <dc:creator>gokcedemirdurkut</dc:creator>
      <pubDate>Mon, 09 Nov 2026 15:27:19 +0000</pubDate>
      <link>https://dev.to/gokcedemirdurkut/trivy-vs-amazon-inspector-container-image-scanning-on-aws-explained-35bo</link>
      <guid>https://dev.to/gokcedemirdurkut/trivy-vs-amazon-inspector-container-image-scanning-on-aws-explained-35bo</guid>
      <description>&lt;p&gt;You push a container image to &lt;a href="https://aws.amazon.com/ecr/" rel="noopener noreferrer"&gt;Amazon ECR&lt;/a&gt;. Is it safe to run?&lt;/p&gt;

&lt;p&gt;There are two common ways to answer that on AWS. You can use &lt;strong&gt;AWS's own scanning&lt;/strong&gt; (ECR basic scanning, or Amazon Inspector), or an &lt;strong&gt;open-source scanner like Trivy&lt;/strong&gt; running in your CI. This post explains both from scratch, compares them, and shows a practical Trivy setup in GitHub Actions. That setup also covers the one thing people usually forget: images that are already running.&lt;/p&gt;

&lt;p&gt;No prior Trivy knowledge needed.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is Trivy?
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://github.com/aquasecurity/trivy" rel="noopener noreferrer"&gt;Trivy&lt;/a&gt; is an open-source security scanner from Aqua Security. It's a single binary that you point at a &lt;em&gt;target&lt;/em&gt;, and it reports problems.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Targets&lt;/strong&gt; it can scan:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;container images (&lt;code&gt;trivy image&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;a local folder or a git repository (&lt;code&gt;trivy fs&lt;/code&gt;, &lt;code&gt;trivy repo&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Infrastructure as Code: Terraform, CloudFormation, Dockerfiles, Kubernetes manifests (&lt;code&gt;trivy config&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;a Kubernetes cluster (&lt;code&gt;trivy k8s&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;an existing SBOM (&lt;code&gt;trivy sbom&lt;/code&gt;)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;What it looks for&lt;/strong&gt; (called &lt;em&gt;scanners&lt;/em&gt;):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;vuln&lt;/code&gt;: known CVEs in OS packages (apk, apt, rpm…) and language packages (pip, npm, Go modules, Maven, Cargo…)&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;misconfig&lt;/code&gt;: insecure settings, for example a security group open to &lt;code&gt;0.0.0.0/0&lt;/code&gt; or a container running as root&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;secret&lt;/code&gt;: API keys and passwords left in files or image layers&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;license&lt;/code&gt;: licenses of your dependencies&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  How it works
&lt;/h3&gt;

&lt;p&gt;Trivy keeps a &lt;strong&gt;vulnerability database&lt;/strong&gt; built from public sources (NVD, GitHub advisories, distro security trackers and others). The database is updated several times a day. On each run, Trivy:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;downloads the latest database (or uses a cached copy),&lt;/li&gt;
&lt;li&gt;inventories what's inside the target: which OS, which packages and versions,&lt;/li&gt;
&lt;li&gt;matches that inventory against the database.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Nothing is uploaded anywhere. Your image never leaves your machine or runner.&lt;/p&gt;

&lt;h3&gt;
  
  
  Your first scan
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# macOS&lt;/span&gt;
brew install trivy

&lt;span class="c"&gt;# Scan a public image&lt;/span&gt;
trivy image python:3.12-slim
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The output is a table per layer type: OS packages first, then language packages. Each row shows the package, installed version, CVE ID, severity, and &lt;strong&gt;fixed version&lt;/strong&gt;. The fixed version is the most useful column, because it tells you whether upgrading helps at all.&lt;/p&gt;

&lt;p&gt;A few flags you'll use constantly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Only what matters, and only what you can actually fix today&lt;/span&gt;
trivy image --severity HIGH,CRITICAL --ignore-unfixed python:3.12-slim

&lt;span class="c"&gt;# Fail the command (exit code 1) if anything is found: this is your CI gate&lt;/span&gt;
trivy image --severity CRITICAL --exit-code 1 myapp:latest

&lt;span class="c"&gt;# Machine-readable output&lt;/span&gt;
trivy image --format json  --output result.json  myapp:latest
trivy image --format sarif --output result.sarif myapp:latest

&lt;span class="c"&gt;# Generate an SBOM (list of everything inside the image)&lt;/span&gt;
trivy image --format cyclonedx --output sbom.json myapp:latest
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To accept a risk on purpose, list the CVE in a &lt;code&gt;.trivyignore&lt;/code&gt; file next to where you run Trivy:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Not reachable in our usage; revisit 2027-01-01
CVE-2024-12345
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Scanning a private ECR image works with normal AWS credentials. Trivy uses them to authenticate to ECR:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws sso login --profile dev
&lt;span class="nv"&gt;AWS_PROFILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;dev trivy image 123456789012.dkr.ecr.eu-central-1.amazonaws.com/myapp:1.4.0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What AWS gives you
&lt;/h2&gt;

&lt;h3&gt;
  
  
  ECR basic scanning
&lt;/h3&gt;

&lt;p&gt;This is free and built into ECR. You turn on &lt;em&gt;scan on push&lt;/em&gt; (or trigger scans manually), and findings appear in the ECR console next to the image. Its limits:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;It covers &lt;strong&gt;OS packages only&lt;/strong&gt;, not your Python or Node dependencies.&lt;/li&gt;
&lt;li&gt;An image is scanned &lt;strong&gt;when it's pushed&lt;/strong&gt;, and not again unless you trigger it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  ECR enhanced scanning (Amazon Inspector)
&lt;/h3&gt;

&lt;p&gt;When you switch ECR to &lt;em&gt;enhanced&lt;/em&gt; scanning, &lt;a href="https://aws.amazon.com/inspector/" rel="noopener noreferrer"&gt;Amazon Inspector&lt;/a&gt; takes over. It adds:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;language package scanning&lt;/strong&gt; on top of OS packages,&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;continuous scanning&lt;/strong&gt;: when a new CVE is published, Inspector re-evaluates images it already scanned. You choose how long images stay monitored after their last push or pull.&lt;/li&gt;
&lt;li&gt;findings in the Inspector console, &lt;strong&gt;Security Hub&lt;/strong&gt;, and &lt;strong&gt;EventBridge&lt;/strong&gt; (so you can route them to Slack, Jira, and so on).&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Amazon Inspector beyond containers
&lt;/h3&gt;

&lt;p&gt;Inspector also scans &lt;strong&gt;EC2 instances&lt;/strong&gt; (agent-based through SSM, or agentless through EBS snapshots) and &lt;strong&gt;Lambda functions&lt;/strong&gt;. It adds an &lt;em&gt;Inspector score&lt;/em&gt;: it adjusts the CVSS score using context such as network reachability, so a critical CVE on a private, unreachable instance ranks lower than the same CVE behind a public load balancer.&lt;/p&gt;

&lt;p&gt;Pricing is per resource. For ECR it's a fee per image on its first scan plus a smaller fee for each rescan. For EC2 it's per instance per month. Check the &lt;a href="https://aws.amazon.com/inspector/pricing/" rel="noopener noreferrer"&gt;Inspector pricing page&lt;/a&gt; for your region. A detail that's easy to miss: with Auto Scaling groups, every instance that lives for a while counts, so bills can grow faster than you expect.&lt;/p&gt;

&lt;h2&gt;
  
  
  The key difference: when the scan happens
&lt;/h2&gt;

&lt;p&gt;Trivy is &lt;strong&gt;point-in-time&lt;/strong&gt;: it scans when you run it. Inspector is &lt;strong&gt;continuous&lt;/strong&gt;: it rescans when the world changes.&lt;/p&gt;

&lt;p&gt;This matters more than any feature list. Picture this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Monday: you build &lt;code&gt;myapp:1.4.0&lt;/code&gt;. Trivy says it's clean. You deploy it.&lt;/li&gt;
&lt;li&gt;Wednesday: a critical CVE is published for a library inside that image.&lt;/li&gt;
&lt;li&gt;Nobody rebuilds &lt;code&gt;myapp:1.4.0&lt;/code&gt;, because nothing changed in your code.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Inspector would flag it on Wednesday. Trivy in CI never looks at it again.&lt;/p&gt;

&lt;p&gt;So &lt;strong&gt;if you use Trivy instead of Inspector, you have to schedule rescans yourself&lt;/strong&gt;. The good news is that it's a few lines of YAML. We'll get there in a moment.&lt;/p&gt;

&lt;h2&gt;
  
  
  Side by side
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;ECR basic&lt;/th&gt;
&lt;th&gt;Inspector (ECR enhanced)&lt;/th&gt;
&lt;th&gt;Trivy&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Cost&lt;/td&gt;
&lt;td&gt;Free&lt;/td&gt;
&lt;td&gt;Per image scan + per rescan&lt;/td&gt;
&lt;td&gt;Free (open source)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OS packages&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Language packages&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;✅ (check supported ecosystems)&lt;/td&gt;
&lt;td&gt;✅ (broad support)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rescans on new CVEs&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;✅ automatic&lt;/td&gt;
&lt;td&gt;Only if you schedule it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Where it runs&lt;/td&gt;
&lt;td&gt;ECR&lt;/td&gt;
&lt;td&gt;AWS-managed&lt;/td&gt;
&lt;td&gt;Anywhere: laptop, CI, cluster&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scan before push / fail the build&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;Possible with Inspector's CI/CD scanning API&lt;/td&gt;
&lt;td&gt;✅ native (&lt;code&gt;--exit-code&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Where results go&lt;/td&gt;
&lt;td&gt;ECR console&lt;/td&gt;
&lt;td&gt;Inspector, Security Hub, EventBridge&lt;/td&gt;
&lt;td&gt;Wherever you send them: terminal, JSON, SARIF, GitHub&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;IaC, secrets, licenses&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;Partly (Lambda code scanning)&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;EC2 / Lambda&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;❌ (not as a managed service)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Context-aware scoring&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;td&gt;❌ (plain severity)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Works outside AWS&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  A practical Trivy setup in GitHub Actions
&lt;/h2&gt;

&lt;p&gt;Here are two workflows: one that scans &lt;strong&gt;before&lt;/strong&gt; an image ships, and one that rescans &lt;strong&gt;what's running&lt;/strong&gt; every day.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Scan in CI, before pushing to ECR
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;build&lt;/span&gt;

&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;push&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;branches&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;main&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="na"&gt;permissions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
  &lt;span class="na"&gt;id-token&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt; &lt;span class="c1"&gt;# OIDC to AWS, no access keys
&lt;/span&gt;
&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v7&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Build image&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;docker build -t myapp:${{ github.sha }} .&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Scan image&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;aquasecurity/trivy-action@&amp;lt;pinned-sha&amp;gt;&lt;/span&gt; &lt;span class="c1"&gt;# pin to a commit SHA
&lt;/span&gt;        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;image-ref&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;myapp:${{ github.sha }}&lt;/span&gt;
          &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;CRITICAL,HIGH&lt;/span&gt;
          &lt;span class="na"&gt;ignore-unfixed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
          &lt;span class="na"&gt;exit-code&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"1"&lt;/span&gt; &lt;span class="c1"&gt;# fail the job, so the push below never runs
&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;aws-actions/configure-aws-credentials@v6&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;role-to-assume&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;arn:aws:iam::123456789012:role/github-actions-ecr-push&lt;/span&gt;
          &lt;span class="na"&gt;aws-region&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;eu-central-1&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ecr&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;aws-actions/amazon-ecr-login@v2&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Push&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;IMAGE=${{ steps.ecr.outputs.registry }}/myapp:${{ github.sha }}&lt;/span&gt;
          &lt;span class="s"&gt;docker tag myapp:${{ github.sha }} "$IMAGE"&lt;/span&gt;
          &lt;span class="s"&gt;docker push "$IMAGE"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A high or critical CVE that already has a fix &lt;strong&gt;fails the job&lt;/strong&gt;, so the image never reaches ECR. The findings table is printed in the job log, right where the developer is already looking.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;ignore-unfixed&lt;/code&gt; keeps the gate fair: a CVE with no fixed version yet would block every build without giving you anything to do about it.&lt;/p&gt;

&lt;p&gt;There's no issue here on purpose: the person who pushed sees the failure right away, and the vulnerable image never ships. Issues are for problems that are already running in production, which is what the next workflow handles.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Gate or report?&lt;/strong&gt; This example is the strictest option: it blocks the push. Many teams don't want that. When a new CVE lands in a common base image, every build can turn red on the same day, and people start adding everything to &lt;code&gt;.trivyignore&lt;/code&gt; just to ship. The softer alternative is to push first, scan without stopping the build, and open an issue for what's found. To do that, add &lt;code&gt;continue-on-error: true&lt;/code&gt; to the scan step and reuse the issue step from the next workflow with &lt;code&gt;if: steps.scan.outcome == 'failure'&lt;/code&gt;. A middle ground is to gate only on &lt;code&gt;CRITICAL&lt;/code&gt; and track everything else in issues. Pick what fits your team.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3&gt;
  
  
  2. Daily scan of the images actually running in ECS
&lt;/h3&gt;

&lt;p&gt;This is the part that replaces Inspector's continuous scanning. Instead of rescanning every image in ECR (most of them are old builds nobody runs), ask ECS &lt;strong&gt;which images are running right now&lt;/strong&gt; and scan only those.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;scan-running-images&lt;/span&gt;

&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;schedule&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;cron&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"0 6 * * *"&lt;/span&gt; &lt;span class="c1"&gt;# every day
&lt;/span&gt;  &lt;span class="na"&gt;workflow_dispatch&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

&lt;span class="na"&gt;permissions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
  &lt;span class="na"&gt;id-token&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt;
  &lt;span class="na"&gt;issues&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt; &lt;span class="c1"&gt;# open an issue when a running image is vulnerable
&lt;/span&gt;
&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;list&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;outputs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;images&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ steps.list.outputs.images }}&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;aws-actions/configure-aws-credentials@v6&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;role-to-assume&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;arn:aws:iam::123456789012:role/github-actions-ecs-read&lt;/span&gt;
          &lt;span class="na"&gt;aws-region&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;eu-central-1&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;list&lt;/span&gt;
        &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;List images of running ECS tasks&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;for cluster in $(aws ecs list-clusters --query 'clusterArns[]' --output text); do&lt;/span&gt;
            &lt;span class="s"&gt;tasks=$(aws ecs list-tasks --cluster "$cluster" --desired-status RUNNING \&lt;/span&gt;
              &lt;span class="s"&gt;--query 'taskArns[]' --output text)&lt;/span&gt;
            &lt;span class="p"&gt;[&lt;/span&gt; &lt;span class="s"&gt;-n&lt;/span&gt; &lt;span class="s2"&gt;"$tasks"&lt;/span&gt; &lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt; &lt;span class="s"&gt;continue&lt;/span&gt;
            &lt;span class="c1"&gt;# Unquoted on purpose: one argument per task ARN (max 100 per call)
&lt;/span&gt;            &lt;span class="c1"&gt;# shellcheck disable=SC2086
&lt;/span&gt;            &lt;span class="s"&gt;aws ecs describe-tasks --cluster "$cluster" --tasks $tasks \&lt;/span&gt;
              &lt;span class="s"&gt;--query 'tasks[].containers[].image' --output text&lt;/span&gt;
          &lt;span class="s"&gt;done | tr '\t' '\n' | { grep '\.dkr\.ecr\.' || true; } | sort -u &amp;gt; images.txt&lt;/span&gt;
          &lt;span class="s"&gt;echo "images=$(jq -Rsc 'split("\n") | map(select(length &amp;gt; 0))' images.txt)" &amp;gt;&amp;gt; "$GITHUB_OUTPUT"&lt;/span&gt;
          &lt;span class="s"&gt;cat images.txt&lt;/span&gt;

  &lt;span class="na"&gt;scan&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;needs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;list&lt;/span&gt;
    &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;needs.list.outputs.images != '[]'&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;strategy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;fail-fast&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
      &lt;span class="na"&gt;matrix&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ fromJson(needs.list.outputs.images) }}&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;aws-actions/configure-aws-credentials@v6&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;role-to-assume&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;arn:aws:iam::123456789012:role/github-actions-ecs-read&lt;/span&gt;
          &lt;span class="na"&gt;aws-region&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;eu-central-1&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;aws-actions/amazon-ecr-login@v2&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Scan&lt;/span&gt;
        &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;scan&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;aquasecurity/trivy-action@&amp;lt;pinned-sha&amp;gt;&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;image-ref&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ matrix.image }}&lt;/span&gt;
          &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;CRITICAL,HIGH&lt;/span&gt;
          &lt;span class="na"&gt;ignore-unfixed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
          &lt;span class="na"&gt;exit-code&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"1"&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Open an issue (once per image)&lt;/span&gt;
        &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;failure() &amp;amp;&amp;amp; steps.scan.outcome == 'failure'&lt;/span&gt;
        &lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;GH_TOKEN&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ github.token }}&lt;/span&gt;
          &lt;span class="na"&gt;IMAGE&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ matrix.image }}&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
          &lt;span class="na"&gt;export title="Vulnerable image running in production&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$IMAGE"&lt;/span&gt;
          &lt;span class="c1"&gt;# search narrows the list; the jq filter keeps exact title matches only
&lt;/span&gt;          &lt;span class="s"&gt;open=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open \&lt;/span&gt;
            &lt;span class="s"&gt;--search "\"$title\" in:title" --json title \&lt;/span&gt;
            &lt;span class="s"&gt;--jq 'map(select(.title == env.title)) | length')&lt;/span&gt;
          &lt;span class="s"&gt;if [ "$open" -eq 0 ]; then&lt;/span&gt;
            &lt;span class="s"&gt;gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" \&lt;/span&gt;
              &lt;span class="na"&gt;--body "The daily Trivy scan found fixable HIGH/CRITICAL CVEs. Details&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"&lt;/span&gt;
          &lt;span class="s"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Why it works:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Trivy downloads a fresh vulnerability database on every run.&lt;/strong&gt; A CVE published yesterday is matched today against an image you built three months ago. That's the continuous-scanning property, on a daily schedule.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A red run is the alert, and the issue is the to-do.&lt;/strong&gt; Each image is its own matrix job (&lt;code&gt;fail-fast: false&lt;/code&gt;), so the run shows which running image has a fixable high or critical CVE. The last step opens one GitHub issue per vulnerable image and skips it if one is already open, so the daily run doesn't pile up duplicates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scanning only running images keeps it cheap and relevant.&lt;/strong&gt; You don't spend time on the 400 old tags sitting in ECR.&lt;/li&gt;
&lt;li&gt;The IAM role only needs &lt;code&gt;ecs:ListClusters&lt;/code&gt;, &lt;code&gt;ecs:ListTasks&lt;/code&gt;, &lt;code&gt;ecs:DescribeTasks&lt;/code&gt; and ECR pull permissions.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;grep '\.dkr\.ecr\.'&lt;/code&gt; filter keeps only images from ECR. Sidecars pulled from Docker Hub, Quay or &lt;code&gt;public.ecr.aws&lt;/code&gt; are left out. Drop the filter if you want those scanned too. They're public, so Trivy can pull them without credentials.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two things to watch:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"Nothing found" should never look like success.&lt;/strong&gt; If the role loses permissions or a cluster is renamed, the list is empty, the scan job is skipped, and the workflow shows green while scanning nothing. If you always expect running images, add a step that fails when the list is empty.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Many accounts?&lt;/strong&gt; Run the &lt;code&gt;list&lt;/code&gt; step once per account by assuming a read-only role in each one (for example, accounts listed from an AWS Organizations OU), then deduplicate images before scanning.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Where should the results go?
&lt;/h3&gt;

&lt;p&gt;Three options, from simplest to richest:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Option&lt;/th&gt;
&lt;th&gt;Cost&lt;/th&gt;
&lt;th&gt;Good for&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Job log + red run&lt;/td&gt;
&lt;td&gt;Free&lt;/td&gt;
&lt;td&gt;the CI gate: the developer sees the table right where the build failed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GitHub issues (as above)&lt;/td&gt;
&lt;td&gt;Free&lt;/td&gt;
&lt;td&gt;tracking: assign, discuss, close when fixed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GitHub &lt;strong&gt;Security&lt;/strong&gt; tab (&lt;code&gt;format: sarif&lt;/code&gt; + &lt;code&gt;github/codeql-action/upload-sarif&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Free for public repos; private repos need &lt;strong&gt;GitHub Code Security&lt;/strong&gt;, a paid add-on priced per active committer&lt;/td&gt;
&lt;td&gt;a dashboard with per-finding history and dismiss reasons&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The Security tab is the nicest view, but for private repositories it's a paid product. If you don't already pay for it, issues give you most of the value for free.&lt;/p&gt;

&lt;h3&gt;
  
  
  Bonus: the same tool scans your Terraform
&lt;/h3&gt;

&lt;p&gt;Inspector looks at what's already deployed. Trivy can also check your infrastructure code &lt;strong&gt;before&lt;/strong&gt; it's applied:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;trivy config ./terraform
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It flags things like SSH open to &lt;code&gt;0.0.0.0/0&lt;/code&gt; in a security group, unencrypted S3 buckets and RDS instances, or a database without deletion protection. Like the image scan, it can fail CI with &lt;code&gt;--exit-code 1&lt;/code&gt;. I'll cover IaC scanning in detail in the next post.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which one should you pick?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Choose Inspector if:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;you need EC2 and Lambda coverage, not just containers,&lt;/li&gt;
&lt;li&gt;your security team works in Security Hub,&lt;/li&gt;
&lt;li&gt;you want zero maintenance and context-aware prioritisation,&lt;/li&gt;
&lt;li&gt;and the per-scan and per-instance pricing fits your scale.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Choose Trivy if:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;your workloads are mostly containers,&lt;/li&gt;
&lt;li&gt;you want to block vulnerable images &lt;strong&gt;before&lt;/strong&gt; they're pushed,&lt;/li&gt;
&lt;li&gt;you want findings where developers already work (GitHub, GitLab, your CI),&lt;/li&gt;
&lt;li&gt;you also want IaC, secret and license scanning from the same tool,&lt;/li&gt;
&lt;li&gt;or you run in more than one cloud.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Or combine them.&lt;/strong&gt; A common, cheap setup is: Trivy in CI as the gate and the developer-facing report, ECR basic scan-on-push as a free second opinion, and Inspector only where you really need it (for example, EC2 fleets that hold sensitive data).&lt;/p&gt;

&lt;h2&gt;
  
  
  Wrap-up
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Trivy is a free, fast scanner for images, code and IaC that runs anywhere. Inspector is AWS's managed, continuous scanner for ECR, EC2 and Lambda.&lt;/li&gt;
&lt;li&gt;The real difference is &lt;strong&gt;when&lt;/strong&gt; they scan. CI scanning alone misses CVEs published after the build.&lt;/li&gt;
&lt;li&gt;If you replace Inspector with Trivy, &lt;strong&gt;scan what's running on a schedule&lt;/strong&gt;, and make sure an empty scan fails loudly instead of passing silently.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>aws</category>
      <category>security</category>
      <category>docker</category>
      <category>trivy</category>
    </item>
    <item>
      <title>Upgrading RDS MySQL from 8.0 to 8.4 behind RDS Proxy</title>
      <dc:creator>gokcedemirdurkut</dc:creator>
      <pubDate>Thu, 09 Jul 2026 10:09:34 +0000</pubDate>
      <link>https://dev.to/gokcedemirdurkut/upgrading-rds-mysql-from-80-to-84-behind-rds-proxy-1g3m</link>
      <guid>https://dev.to/gokcedemirdurkut/upgrading-rds-mysql-from-80-to-84-behind-rds-proxy-1g3m</guid>
      <description>&lt;p&gt;RDS for MySQL 8.0 reaches end of standard support on &lt;strong&gt;July 31, 2026&lt;/strong&gt; after that AWS moves you onto paid Extended Support until you upgrade, so moving our production databases to 8.4 wasn't really optional. The engine upgrade itself is a non-event: pick the target version, RDS runs its prechecks, takes a snapshot, and a while later you're on 8.4. The part that actually needed thinking was authentication, because all of our services connect through RDS Proxy, and 8.4 changes how the old password plugin works.&lt;/p&gt;

&lt;p&gt;If you run the same setup, here's what's worth knowing before you click the button.&lt;/p&gt;

&lt;h2&gt;
  
  
  The moving parts
&lt;/h2&gt;

&lt;p&gt;If RDS Proxy and MySQL auth plugins are already familiar, skip ahead.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Amazon RDS&lt;/strong&gt; is a managed MySQL database AWS runs the server so you don't have to.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;RDS Proxy&lt;/strong&gt; sits in front of it like a &lt;strong&gt;hotel front desk&lt;/strong&gt;. Your applications don't walk up to the room (the database) holding a key; they show ID at the desk, and the desk opens the room for them. It also pools connections so the database isn't swamped.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An authentication plugin&lt;/strong&gt; is just &lt;em&gt;how MySQL checks a password&lt;/em&gt;  think of it as the type of lock on a door. The old one, &lt;code&gt;mysql_native_password&lt;/code&gt;, uses SHA-1 (an older, weaker recipe). The newer &lt;code&gt;caching_sha2_password&lt;/code&gt; uses SHA-256 (stronger). MySQL 8.4 stops shipping the old lock by default and expects the new one.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the whole story below is really: we changed the locks, and the only door that turned out to matter was the one between the front desk and the room.&lt;/p&gt;

&lt;h2&gt;
  
  
  There are two auth legs, not one
&lt;/h2&gt;

&lt;p&gt;The thing to internalize first: with RDS Proxy in the path, a connection authenticates twice.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Service to proxy.&lt;/strong&gt; Our services don't hold a database password at all. A sidecar mints an IAM auth token and the service uses that to reach the proxy over TLS. This leg is pure IAM the MySQL auth plugin has nothing to do with it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Proxy to database.&lt;/strong&gt; The proxy then opens the real backend connection using a username/password it pulls from Secrets Manager (one secret per DB user: the app users, a read-only user, the master).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F06xbefpkdzrjf184pij4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F06xbefpkdzrjf184pij4.png" width="800" height="341"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;That split matters because the &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.MySQL.Major.html" rel="noopener noreferrer"&gt;8.4 authentication change&lt;/a&gt; only touches the second leg. Whatever you read about &lt;code&gt;mysql_native_password&lt;/code&gt; being disabled applies to how the &lt;em&gt;database&lt;/em&gt; authenticates the proxy's connection not to how your services reach the proxy.&lt;/p&gt;

&lt;h2&gt;
  
  
  The mysql_native_password scare that wasn't
&lt;/h2&gt;

&lt;p&gt;Community MySQL 8.4 doesn't load &lt;code&gt;mysql_native_password&lt;/code&gt; by default anymore. Our users including the RDS master user were all created with that plugin, so the obvious worry was: the moment we're on 8.4, the proxy can't authenticate to the backend and everything falls over.&lt;/p&gt;

&lt;p&gt;We were ready to flip a parameter to keep the plugin on. Then we actually looked at the 8.4 parameter group:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5udqpe4z4lixkp4zbcf4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5udqpe4z4lixkp4zbcf4.png" alt="8.4 parameter group" width="800" height="145"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;On RDS for MySQL 8.4, AWS keeps the plugin loaded for you, and you can't turn it off through the parameter group. So existing &lt;code&gt;mysql_native_password&lt;/code&gt; users master included keep working after the upgrade with zero changes. The thing we'd braced for simply never happened. The flip side: you can't &lt;em&gt;disable&lt;/em&gt; native password on RDS 8.4 either, so a hard "everyone must be caching_sha2" cutover isn't a thing you do here. That only becomes real whenever RDS eventually ships a 9.x line, where the plugin is gone for good.&lt;/p&gt;

&lt;p&gt;So the honest answer for an RDS 8.0 → 8.4 upgrade: on the auth front, you don't have to do anything. It keeps working.&lt;/p&gt;

&lt;h2&gt;
  
  
  Moving to caching_sha2_password anyway
&lt;/h2&gt;

&lt;p&gt;We still moved every user we control to &lt;code&gt;caching_sha2_password&lt;/code&gt; not because 8.4 forced us, but because 9.x will. Whenever RDS ships a 9.x line, the native password plugin is gone for good, and a database full of native-password users turns a routine engine bump into an auth scramble at the worst possible time. Doing the migration now, while it's optional and the old plugin is still around as a safety net, means that future upgrade is just a version change. It's also the better plugin on its own merits (SHA-256 instead of SHA-1).&lt;/p&gt;

&lt;p&gt;Two preconditions that usually make this annoying were already true for us:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;RDS Proxy has supported &lt;code&gt;caching_sha2_password&lt;/code&gt; since December 2024, and&lt;/li&gt;
&lt;li&gt;it requires TLS which we already enforce on both the proxy (&lt;code&gt;require_tls&lt;/code&gt;) and the instance (&lt;code&gt;require_secure_transport&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So there was no new TLS work to do. All of it lives in Terraform pointing the proxy's client auth at caching_sha2, keeping &lt;code&gt;require_tls&lt;/code&gt; on the proxy and &lt;code&gt;require_secure_transport&lt;/code&gt; on the instance so the change was a small reviewable diff rather than console clicking. The users themselves we migrate in the small Lambda that initializes the database, using a create-or-migrate pattern:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;USER&lt;/span&gt; &lt;span class="n"&gt;IF&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;EXISTS&lt;/span&gt; &lt;span class="nv"&gt;`app`&lt;/span&gt;&lt;span class="o"&gt;@&lt;/span&gt;&lt;span class="nv"&gt;`%`&lt;/span&gt; &lt;span class="n"&gt;IDENTIFIED&lt;/span&gt; &lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="n"&gt;caching_sha2_password&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="s1"&gt;'...'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;USER&lt;/span&gt; &lt;span class="nv"&gt;`app`&lt;/span&gt;&lt;span class="o"&gt;@&lt;/span&gt;&lt;span class="nv"&gt;`%`&lt;/span&gt; &lt;span class="n"&gt;IDENTIFIED&lt;/span&gt; &lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="n"&gt;caching_sha2_password&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="s1"&gt;'...'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;CREATE ... IF NOT EXISTS&lt;/code&gt; handles fresh environments; the &lt;code&gt;ALTER&lt;/code&gt; flips the plugin for users that already exist. The order matters: we applied the Terraform first, then triggered that Lambda so the &lt;code&gt;ALTER&lt;/code&gt;s actually ran against the live users. It normally only fires once, when the proxy is first created so migrating an already-running environment meant re-running it on purpose. Our services needed no changes for this no driver upgrades, no config, no redeploys. Modern MySQL drivers already speak &lt;code&gt;caching_sha2_password&lt;/code&gt;, so the switch was invisible to the applications.&lt;/p&gt;

&lt;p&gt;The master user we never touched directly. It's RDS-managed (&lt;code&gt;manage_master_user_password&lt;/code&gt;), so RDS rotates its password on a schedule and on 8.4 that managed rotation moved the master onto &lt;code&gt;caching_sha2_password&lt;/code&gt; on its own, with no action from us. (Worth knowing: this is RDS's own managed rotation doing it. A plain rotation that runs &lt;code&gt;SET PASSWORD&lt;/code&gt; only changes the password and keeps the existing plugin it wouldn't migrate anything.) The only accounts left on native password are the internal ones AWS runs (the RDS admin user, the &lt;code&gt;mysql.*&lt;/code&gt; system schemas), and those keep working untouched.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl7xd2amybpkb6to5bqfa.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl7xd2amybpkb6to5bqfa.png" alt=" " width="800" height="155"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;That's why you never have to set a parameter here: RDS keeps &lt;code&gt;mysql_native_password&lt;/code&gt; loaded by default for its own accounts, so anything not yet on &lt;code&gt;caching_sha2_password&lt;/code&gt; just keeps authenticating.&lt;/p&gt;

&lt;h2&gt;
  
  
  One catch: Aurora read replicas
&lt;/h2&gt;

&lt;p&gt;If you have an Aurora MySQL read replica hanging off your RDS instance often a leftover from a half-finished move to Aurora it can block the upgrade outright. A replica can't run an older major version than its source, and Aurora MySQL only got an 8.4-compatible version in &lt;strong&gt;May 2026&lt;/strong&gt;, about a year and a half after community 8.4. Until then, moving the RDS source to 8.4 would have broken replication, so the replica quietly vetoes the whole upgrade a big reason some teams stayed on 8.0. The version gap is closed now, but if you run this setup, check the replica side before you touch the source.&lt;/p&gt;

&lt;h2&gt;
  
  
  In short
&lt;/h2&gt;

&lt;p&gt;The hard part of moving to 8.4 behind RDS Proxy isn't the engine bump it's the auth model, and three things carry most of it. Connections authenticate in two separate legs, and the 8.4 plugin change only touches the proxy-to-database one. RDS keeps &lt;code&gt;mysql_native_password&lt;/code&gt; loaded for you, so nothing breaks on its own. And switching to &lt;code&gt;caching_sha2_password&lt;/code&gt; costs almost nothing once TLS is already enforced. Get those right and there's very little left to worry about.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>mysql</category>
      <category>devops</category>
      <category>database</category>
    </item>
    <item>
      <title>Why Your Nginx Security Headers Disappear (add_header Inheritance Explained)</title>
      <dc:creator>gokcedemirdurkut</dc:creator>
      <pubDate>Thu, 12 Mar 2026 09:53:58 +0000</pubDate>
      <link>https://dev.to/gokcedemirdurkut/nginx-addheader-inheritance-the-silent-security-header-killer-p0f</link>
      <guid>https://dev.to/gokcedemirdurkut/nginx-addheader-inheritance-the-silent-security-header-killer-p0f</guid>
      <description>&lt;p&gt;Security headers can disappear in Nginx even when they are correctly configured.&lt;/p&gt;

&lt;p&gt;Some endpoints may silently drop important headers like:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Content-Security-Policy&lt;/li&gt;
&lt;li&gt;Strict-Transport-Security&lt;/li&gt;
&lt;li&gt;X-Frame-Options&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This behavior is subtle and has existed in &lt;a href="https://nginx.org/" rel="noopener noreferrer"&gt;Nginx&lt;/a&gt; for a long time.&lt;/p&gt;




&lt;h2&gt;
  
  
  How the Problem Appears
&lt;/h2&gt;

&lt;p&gt;Everything looks correct at first.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-I&lt;/span&gt; https://example.com/api/health
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Response headers include security policies as expected.&lt;/p&gt;

&lt;p&gt;But then another endpoint is checked:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-I&lt;/span&gt; https://example.com/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And suddenly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;❌ Content-Security-Policy missing
❌ Strict-Transport-Security missing
❌ X-Frame-Options missing
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Tools like &lt;a href="https://securityheaders.com/" rel="noopener noreferrer"&gt;&lt;strong&gt;securityheaders.com&lt;/strong&gt;&lt;/a&gt; will immediately detect this.&lt;/p&gt;

&lt;p&gt;The confusing part is that the headers are clearly defined in the configuration.&lt;/p&gt;




&lt;h2&gt;
  
  
  A Common Configuration
&lt;/h2&gt;

&lt;p&gt;A typical setup may look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;server&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kn"&gt;add_header&lt;/span&gt; &lt;span class="s"&gt;Content-Security-Policy&lt;/span&gt; &lt;span class="s"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;add_header&lt;/span&gt; &lt;span class="s"&gt;Strict-Transport-Security&lt;/span&gt; &lt;span class="s"&gt;"max-age=31536000"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="kn"&gt;location&lt;/span&gt; &lt;span class="p"&gt;~&lt;/span&gt;&lt;span class="sr"&gt;*&lt;/span&gt; &lt;span class="err"&gt;\&lt;/span&gt;&lt;span class="s"&gt;.html&lt;/span&gt;$ &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kn"&gt;add_header&lt;/span&gt; &lt;span class="s"&gt;Cache-Control&lt;/span&gt; &lt;span class="s"&gt;"no-store"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The intention is simple:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;disable caching for HTML responses&lt;/li&gt;
&lt;li&gt;apply security headers globally&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;However, this configuration does not behave as expected.&lt;/p&gt;




&lt;h2&gt;
  
  
  What Actually Happens
&lt;/h2&gt;

&lt;p&gt;Responses served from:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;location&lt;/span&gt; &lt;span class="p"&gt;~&lt;/span&gt;&lt;span class="sr"&gt;*&lt;/span&gt; &lt;span class="err"&gt;\&lt;/span&gt;&lt;span class="s"&gt;.html&lt;/span&gt;$
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;will only include:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;Cache-Control: no-store
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;All security headers defined in the &lt;code&gt;server&lt;/code&gt; block disappear.&lt;/p&gt;

&lt;p&gt;There are no warnings and no errors in the logs.&lt;br&gt;
The headers are simply missing.&lt;/p&gt;


&lt;h2&gt;
  
  
  The Reason
&lt;/h2&gt;

&lt;p&gt;This happens because of how Nginx handles header inheritance.&lt;/p&gt;

&lt;p&gt;If a &lt;code&gt;location&lt;/code&gt; block defines &lt;strong&gt;any &lt;code&gt;add_header&lt;/code&gt; directive&lt;/strong&gt;, it &lt;strong&gt;does not inherit&lt;/strong&gt; &lt;code&gt;add_header&lt;/code&gt; directives from parent blocks.&lt;/p&gt;

&lt;p&gt;Even adding a single header like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cache-Control
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;causes Nginx to drop previously defined headers such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Content-Security-Policy&lt;/li&gt;
&lt;li&gt;Strict-Transport-Security&lt;/li&gt;
&lt;li&gt;X-Frame-Options&lt;/li&gt;
&lt;li&gt;Referrer-Policy&lt;/li&gt;
&lt;li&gt;Permissions-Policy&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This behavior has existed for many years and has caused confusion in many configurations.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Previous Workarounds
&lt;/h2&gt;

&lt;p&gt;Before recently, the usual approaches were:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Duplicating headers&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Define all security headers again in every &lt;code&gt;location&lt;/code&gt; block.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Using includes&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Create a file like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;include&lt;/span&gt; &lt;span class="n"&gt;security_headers.conf&lt;/span&gt;;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and include it everywhere.&lt;/p&gt;

&lt;p&gt;Both approaches work but are difficult to maintain and easy to forget.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Fix in Nginx 1.29+
&lt;/h2&gt;

&lt;p&gt;Recent Nginx versions introduced a better solution:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;add_header_inherit&lt;/span&gt; &lt;span class="s"&gt;merge&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This directive changes how headers are inherited.&lt;/p&gt;

&lt;p&gt;Instead of replacing headers defined in parent blocks, child blocks &lt;strong&gt;merge them&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;This feature was introduced in newer Nginx releases and is documented in the official release notes:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://blog.nginx.org/blog/nginx-open-source-1-29-3-and-1-29-4" rel="noopener noreferrer"&gt;https://blog.nginx.org/blog/nginx-open-source-1-29-3-and-1-29-4&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  A Safer Configuration
&lt;/h2&gt;

&lt;p&gt;With the new directive, the configuration can be written like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;server&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;

    &lt;span class="kn"&gt;add_header_inherit&lt;/span&gt; &lt;span class="s"&gt;merge&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="kn"&gt;add_header&lt;/span&gt; &lt;span class="s"&gt;Content-Security-Policy&lt;/span&gt; &lt;span class="s"&gt;"...security&lt;/span&gt; &lt;span class="s"&gt;policy..."&lt;/span&gt; &lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;add_header&lt;/span&gt; &lt;span class="s"&gt;Strict-Transport-Security&lt;/span&gt; &lt;span class="s"&gt;"max-age=63072000&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="kn"&gt;includeSubDomains"&lt;/span&gt; &lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;add_header&lt;/span&gt; &lt;span class="s"&gt;X-Frame-Options&lt;/span&gt; &lt;span class="s"&gt;"SAMEORIGIN"&lt;/span&gt; &lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;add_header&lt;/span&gt; &lt;span class="s"&gt;X-Content-Type-Options&lt;/span&gt; &lt;span class="s"&gt;"nosniff"&lt;/span&gt; &lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="kn"&gt;location&lt;/span&gt; &lt;span class="p"&gt;~&lt;/span&gt;&lt;span class="sr"&gt;*&lt;/span&gt; &lt;span class="err"&gt;\&lt;/span&gt;&lt;span class="s"&gt;.html&lt;/span&gt;$ &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kn"&gt;add_header&lt;/span&gt; &lt;span class="s"&gt;Cache-Control&lt;/span&gt; &lt;span class="s"&gt;"no-store,&lt;/span&gt; &lt;span class="s"&gt;no-cache,&lt;/span&gt; &lt;span class="s"&gt;private,&lt;/span&gt; &lt;span class="s"&gt;max-age=0"&lt;/span&gt; &lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the behavior is predictable:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;HTML responses are not cached&lt;/li&gt;
&lt;li&gt;Security headers are applied consistently&lt;/li&gt;
&lt;li&gt;Headers do not need to be duplicated&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Final Takeaway
&lt;/h2&gt;

&lt;p&gt;When using &lt;code&gt;add_header&lt;/code&gt; inside &lt;code&gt;location&lt;/code&gt; blocks in &lt;strong&gt;Nginx 1.29+&lt;/strong&gt;, enabling the following directive is recommended:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;add_header_inherit&lt;/span&gt; &lt;span class="s"&gt;merge&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This prevents accidental loss of security headers and makes configurations easier to maintain.&lt;/p&gt;

</description>
      <category>backend</category>
      <category>devops</category>
      <category>infosec</category>
      <category>security</category>
    </item>
    <item>
      <title>Preventing Silent ECS Deployment Failures with Circuit Breaker</title>
      <dc:creator>gokcedemirdurkut</dc:creator>
      <pubDate>Thu, 26 Feb 2026 10:57:55 +0000</pubDate>
      <link>https://dev.to/gokcedemirdurkut/preventing-silent-ecs-deployment-failures-with-circuit-breaker-2k5j</link>
      <guid>https://dev.to/gokcedemirdurkut/preventing-silent-ecs-deployment-failures-with-circuit-breaker-2k5j</guid>
      <description>&lt;p&gt;AWS Elastic Container Service (ECS) provides a built-in feature called the &lt;strong&gt;deployment circuit breaker&lt;/strong&gt;, designed to make service deployments safer and more resilient.&lt;/p&gt;

&lt;p&gt;This feature continuously monitors the health of tasks during a deployment and automatically rolls back changes if newly launched tasks fail to become healthy. When enabled, it prevents failed deployments from leaving services in a degraded or non-functional state.&lt;/p&gt;

&lt;p&gt;Without this safeguard, deployment failures can easily go unnoticed. For example, if new tasks fail to start or never pass health checks, the service may still appear to be running while it is effectively broken. These silent failures can result in data loss, financial impact, or operational issues depending on the workload.&lt;/p&gt;

&lt;p&gt;In this post, I’ll walk through how to enable the ECS deployment circuit breaker using Terraform, how to observe deployment failures via EventBridge, and how to send real-time alerts to Slack.&lt;/p&gt;




&lt;h2&gt;
  
  
  Why the ECS Deployment Circuit Breaker Matters
&lt;/h2&gt;

&lt;p&gt;Enabling the deployment circuit breaker provides several important benefits:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Automatic rollback&lt;/strong&gt; – Failed deployments are reverted to the last known healthy service revision&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Improved visibility&lt;/strong&gt; – ECS emits structured events whenever a deployment fails or rolls back&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reduced operational overhead&lt;/strong&gt; – Failures are mitigated automatically without immediate manual intervention&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Together, these significantly reduce the risk of production incidents caused by faulty deployments.&lt;/p&gt;




&lt;h2&gt;
  
  
  Enabling the Circuit Breaker with Terraform
&lt;/h2&gt;

&lt;p&gt;The deployment circuit breaker can be enabled directly in your ECS service definition. In Terraform, this is done using the &lt;code&gt;deployment_circuit_breaker&lt;/code&gt; block:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="nx"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_ecs_service"&lt;/span&gt; &lt;span class="s2"&gt;"default"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;name&lt;/span&gt;            &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"tuve"&lt;/span&gt;
  &lt;span class="nx"&gt;cluster&lt;/span&gt;         &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;aws_ecs_cluster&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;main&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;
  &lt;span class="nx"&gt;task_definition&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;aws_ecs_task_definition&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;default&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;arn&lt;/span&gt;
  &lt;span class="nx"&gt;desired_count&lt;/span&gt;   &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;

  &lt;span class="nx"&gt;deployment_circuit_breaker&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;enable&lt;/span&gt;   &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="nx"&gt;rollback&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="p"&gt;...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With this configuration in place, ECS will automatically stop and roll back a deployment if the new tasks fail to reach a healthy state.&lt;/p&gt;

&lt;p&gt;Once enabled, the AWS Management Console clearly indicates that the &lt;strong&gt;Deployment circuit breaker&lt;/strong&gt; is turned on.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Flbyxeejxs9hkimyzwv9h.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Flbyxeejxs9hkimyzwv9h.png" alt=" " width="800" height="149"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Observing Deployment Failures
&lt;/h2&gt;

&lt;p&gt;Automatic rollback is useful, but visibility is just as important.&lt;/p&gt;

&lt;p&gt;When the ECS deployment circuit breaker triggers, ECS emits events to Amazon EventBridge with the following detail type:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ECS Deployment State Change
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here is an example event payload:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"0"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ddca6449-b258-46c0-8653-e0e3aEXAMPLE"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"detail-type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ECS Deployment State Change"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"aws.ecs"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"account"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"111122223333"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"time"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2020-05-23T12:31:14Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"eu-central-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"resources"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"arn:aws:ecs:eu-central-1:111122223333:service/default/servicetest"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"detail"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"eventType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ERROR"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"eventName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"SERVICE_DEPLOYMENT_FAILED"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"deploymentId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ecs-svc/123"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"updatedAt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2020-05-23T11:11:11Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"reason"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ECS deployment circuit breaker: task failed to start."&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Key Fields to Monitor
&lt;/h3&gt;

&lt;p&gt;Some fields in this event are particularly useful for monitoring and alerting:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;eventName&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;SERVICE_DEPLOYMENT_FAILED&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;SERVICE_DEPLOYMENT_ROLLBACK_COMPLETED&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;reason&lt;/strong&gt; – Explains why the deployment failed&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;resources&lt;/strong&gt; – Identifies the affected ECS service&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;updatedAt&lt;/strong&gt; – Indicates when the failure occurred&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Tracking these fields ensures that deployment issues are visible immediately instead of being discovered hours later.
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Deployment Rollback in the AWS Console
&lt;/h3&gt;

&lt;p&gt;The AWS Management Console also provides clear visibility into rollback activity. After a failed deployment, the &lt;strong&gt;Deployments&lt;/strong&gt; tab shows the rollback status along with the target service revision.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fp0ahnc0921jb3hv33jv5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fp0ahnc0921jb3hv33jv5.png" alt=" " width="799" height="316"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This view is particularly useful for confirming that the circuit breaker worked as expected.&lt;/p&gt;




&lt;h2&gt;
  
  
  Sending Deployment Alerts to Slack
&lt;/h2&gt;

&lt;p&gt;To ensure deployment failures are noticed immediately, ECS deployment events can be routed to Slack using EventBridge and Lambda.&lt;/p&gt;

&lt;p&gt;The overall flow looks like this:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;ECS → EventBridge → Lambda → Slack&lt;/strong&gt;&lt;/p&gt;




&lt;h3&gt;
  
  
  Lambda Handler Example
&lt;/h3&gt;

&lt;p&gt;The Lambda function listens for ECS deployment state changes and sends notifications when a deployment fails or rolls back:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;lambda_handler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;detail_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;detail-type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;detail_type&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ECS Deployment State Change&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;event_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;detail&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{}).&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;eventName&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;event_name&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SERVICE_DEPLOYMENT_FAILED&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SERVICE_DEPLOYMENT_ROLLBACK_COMPLETED&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;]:&lt;/span&gt;
            &lt;span class="n"&gt;detail&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;detail&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{})&lt;/span&gt;
            &lt;span class="n"&gt;resources&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;resources&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[])&lt;/span&gt;
            &lt;span class="n"&gt;service_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resources&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;resources&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unknown&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
            &lt;span class="n"&gt;reason&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;detail&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;reason&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Unknown&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;updated_at&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;detail&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;updatedAt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Unknown&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

            &lt;span class="nf"&gt;send_slack_notification&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="n"&gt;service&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;service_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;reason&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;event_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;event_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;timestamp&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;updated_at&lt;/span&gt;
            &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h3&gt;
  
  
  EventBridge Rule (Terraform)
&lt;/h3&gt;

&lt;p&gt;The following EventBridge rule filters ECS deployment events and forwards them to the Lambda function:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="nx"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_cloudwatch_event_rule"&lt;/span&gt; &lt;span class="s2"&gt;"ecs_deployment"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;name&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"ecs-deployment-events"&lt;/span&gt;

  &lt;span class="nx"&gt;event_pattern&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;jsonencode&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="s2"&gt;"source"&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"aws.ecs"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="s2"&gt;"detail-type"&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"ECS Deployment State Change"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="s2"&gt;"detail"&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="s2"&gt;"eventName"&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="s2"&gt;"SERVICE_DEPLOYMENT_FAILED"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"SERVICE_DEPLOYMENT_ROLLBACK_COMPLETED"&lt;/span&gt;
      &lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_cloudwatch_event_target"&lt;/span&gt; &lt;span class="s2"&gt;"lambda"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;rule&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;aws_cloudwatch_event_rule&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ecs_deployment&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt;
  &lt;span class="nx"&gt;arn&lt;/span&gt;  &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;aws_lambda_function&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;notification&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;arn&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Final Outcome
&lt;/h2&gt;

&lt;p&gt;After enabling the ECS deployment circuit breaker and adding Slack notifications:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Failed deployments automatically roll back&lt;/li&gt;
&lt;li&gt;Silent service failures are eliminated&lt;/li&gt;
&lt;li&gt;Deployment issues become visible in real time&lt;/li&gt;
&lt;li&gt;ECS services are safer by default&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;By combining automated rollback with real-time alerts, you can significantly reduce operational risk and increase confidence in your ECS deployments.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>ecs</category>
      <category>terraform</category>
      <category>devops</category>
    </item>
    <item>
      <title>Deploying and Customizing AWS ParallelCluster Service (PCS) for HPC Workloads</title>
      <dc:creator>gokcedemirdurkut</dc:creator>
      <pubDate>Sun, 26 Oct 2025 09:15:02 +0000</pubDate>
      <link>https://dev.to/gokcedemirdurkut/deploying-and-customizing-aws-parallelcluster-service-pcs-for-hpc-workloads-km1</link>
      <guid>https://dev.to/gokcedemirdurkut/deploying-and-customizing-aws-parallelcluster-service-pcs-for-hpc-workloads-km1</guid>
      <description>&lt;p&gt;I recently worked on a project involving AWS ParallelCluster Service (PCS).&lt;/p&gt;

&lt;p&gt;The main goal was to build an &lt;a href="https://aws.amazon.com/hpc" rel="noopener noreferrer"&gt;HPC&lt;/a&gt; cluster that meets our specific requirements such as using an image with &lt;a href="https://www.python.org/downloads/release/python-3100" rel="noopener noreferrer"&gt;Python 3.10&lt;/a&gt;, installing the necessary dependencies and deploying a PCS cluster.&lt;br&gt;
To achieve this, I built a custom AMI(Amazon Machine Image) using Packer, then launched a PCS cluster based on that image.&lt;br&gt;
Throughout the process, I automated the workflow using Terraform, GitHub Actions and shell scripts.&lt;br&gt;
After deploying the cluster, I verified that everything was working correctly by running several SLURM job commands, which confirmed that our setup was operational.&lt;/p&gt;

&lt;p&gt;In this article, I’ll walk you through the entire process from building the custom AMI to running jobs on PCS.&lt;/p&gt;
&lt;h2&gt;
  
  
  What is AWS PCS?
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://aws.amazon.com/hpc/parallelcluster" rel="noopener noreferrer"&gt;AWS ParallelCluster Service&lt;/a&gt; (PCS) is a managed service that enables high-performance computing (HPC) on AWS. It’s designed for running parallel workloads such as simulations, ML training, or large-scale data analysis.&lt;br&gt;
Using PCS, you can deploy and manage HPC clusters without manually configuring compute nodes, networking or schedulers. It integrates seamlessly with services like Amazon S3 and AWS Batch, supporting complex workloads efficiently.&lt;/p&gt;
&lt;h2&gt;
  
  
  Why Use AWS PCS?
&lt;/h2&gt;

&lt;p&gt;Traditionally, deploying and managing HPC clusters required deep expertise in cluster configuration, job scheduling, and infrastructure management.&lt;br&gt;
AWS PCS abstracts much of this complexity by offering:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Fully managed cluster orchestration&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Integration with SLURM scheduler&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Elastic scaling based on job demand&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Infrastructure as Code (IaC) support with &lt;a href="https://developer.hashicorp.com/terraform" rel="noopener noreferrer"&gt;Terraform&lt;/a&gt; and &lt;a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html" rel="noopener noreferrer"&gt;CloudFormation&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This makes AWS PCS an excellent choice for researchers, data scientists, and DevOps engineers who want to focus on workloads rather than infrastructure.&lt;/p&gt;
&lt;h2&gt;
  
  
  How Was the PCS Cluster Created?
&lt;/h2&gt;

&lt;p&gt;The PCS cluster was provisioned using Terraform with the &lt;a href="https://registry.terraform.io/providers/hashicorp/awscc/latest" rel="noopener noreferrer"&gt;awscc&lt;/a&gt;&lt;br&gt;
provider. The infrastructure is defined as code and includes the cluster, login/compute node groups and job queue.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; The awscc provider was required because PCS resources are not yet supported by the standard AWS provider. It uses the AWS Cloud Control API to manage newer services like PCS.&lt;/p&gt;

&lt;p&gt;📢 &lt;a href="https://aws.amazon.com/about-aws/whats-new/2025/03/announcing-terraform-parallel-computing-service" rel="noopener noreferrer"&gt;Official Announcement — Terraform Support for AWS ParallelCluster Service (March 2025)&lt;/a&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  PCS Cluster Components
&lt;/h2&gt;

&lt;p&gt;A PCS setup typically includes three main components:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cluster:&lt;/strong&gt; Defines general settings, scheduler configuration, and networking.&lt;br&gt;
&lt;strong&gt;Node Groups:&lt;/strong&gt; Specifies login and compute nodes.&lt;br&gt;
&lt;strong&gt;Queue:&lt;/strong&gt; Manages job scheduling and execution PCS Setup with Terraform.&lt;/p&gt;

&lt;p&gt;Below is a simplified example of how a PCS cluster can be created using Terraform and the awscc provider:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create PCS Cluster using awscc provider&lt;/span&gt;
&lt;span class="k"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"awscc_pcs_cluster"&lt;/span&gt; &lt;span class="s2"&gt;"this"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;count&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kd"&gt;var&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;enable_cluster&lt;/span&gt; &lt;span class="err"&gt;?&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="err"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
  &lt;span class="nx"&gt;name&lt;/span&gt;  &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kd"&gt;var&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;cluster_name&lt;/span&gt;
  &lt;span class="nx"&gt;size&lt;/span&gt;  &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kd"&gt;var&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;cluster_size&lt;/span&gt;

  &lt;span class="nx"&gt;scheduler&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;type&lt;/span&gt;    &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"SLURM"&lt;/span&gt;
    &lt;span class="nx"&gt;version&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kd"&gt;var&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;cluster_slurm_version&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="nx"&gt;networking&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;security_group_ids&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;aws_security_group&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="nx"&gt;subnet_ids&lt;/span&gt;         &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kd"&gt;var&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;subnet_ids&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Variables like &lt;em&gt;cluster_name&lt;/em&gt;, &lt;em&gt;subnet_ids&lt;/em&gt;, and &lt;em&gt;cluster_slurm_version&lt;/em&gt; can be parameterized to adapt across environments.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cost Awareness &amp;amp; Usage Control&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Running a PCS cluster can become expensive depending on your configuration.&lt;br&gt;
Each compute node in the cluster uses Amazon EC2 instances, and costs increase while those instances are running.&lt;/p&gt;

&lt;p&gt;To manage this, we added a Terraform variable that controls whether the PCS cluster should be deployed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Control cluster deployment&lt;/span&gt;
&lt;span class="nx"&gt;enable_pcs_cluster&lt;/span&gt; &lt;span class="err"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;   &lt;span class="c1"&gt;# create the cluster&lt;/span&gt;
&lt;span class="c1"&gt;# enable_pcs_cluster = false  # skip deployment&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This allows you to enable or disable the PCS cluster as needed.&lt;br&gt;
For example, during development or testing, you can set &lt;em&gt;enable_pcs_cluster = false&lt;/em&gt; to avoid unnecessary charges.&lt;/p&gt;
&lt;h2&gt;
  
  
  Creating Custom AMIs with Packer
&lt;/h2&gt;

&lt;p&gt;Custom AMIs let you pre-install software and dependencies on compute nodes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Steps:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Create a Packer template based on a base AMI (e.x. Ubuntu or Amazon Linux 2).&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Install the required software packages.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Build the AMI and use it in your PCS cluster.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Example Packer Template:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"builders"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"amazon-ebs"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"{{user `aws_region`}}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"source_ami"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"{{user `source_ami`}}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"instance_type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"{{user `instance_type`}}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"ssh_username"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"{{user `ssh_username`}}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"ami_name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"{{user `ami_name_prefix`}}-{{user `distribution`}}-{{user `architecture`}}-{{isotime `2006.01.02-15.04`}}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"ami_description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"{{user `ami_description`}}"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"provisioners"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"shell"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"inline"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="s2"&gt;"sudo yum update -y"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="s2"&gt;"sudo yum install -y gcc make python3"&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="err"&gt;...&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;em&gt;This is just an example. You can add other dependencies as needed.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;We used SLURM as the job scheduler within PCS to manage HPC workloads efficiently.&lt;br&gt;
It handles queueing, job submission, and node allocation allowing multiple users or jobs to share compute resources dynamically.&lt;/p&gt;

&lt;p&gt;During our setup, we required Python 3.10, but the default Amazon Linux 2 AMI provided only Python 3.7.&lt;br&gt;
To solve this, we created a custom AMI using Packer, based on the following Ubuntu Marketplace image:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Ubuntu Server 22.04 LTS (arm64)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;AMI ID: ami-0f45e2f16611e3139&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Source: AWS Marketplace&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This custom AMI included:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Python 3.10 preinstalled&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Common HPC dependencies (gcc, make, python3-pip)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Additional Python libraries required for our workloads&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;By using a custom AMI, we ensured compatibility with our codebase and reduced setup time for new compute nodes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; Custom AMIs are especially helpful when your HPC workloads require specific compiler versions, Python environments, or third-party libraries.&lt;/p&gt;
&lt;h2&gt;
  
  
  Testing the SLURM Cluster
&lt;/h2&gt;

&lt;p&gt;Once the PCS cluster was up and running, we validated the environment by executing a few basic SLURM commands:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Check the cluster and nodes&lt;/span&gt;
sinfo

&lt;span class="c"&gt;# Submit a test job&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"echo Hello from SLURM"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; test.sh
sbatch test.sh

&lt;span class="c"&gt;# View the job queue&lt;/span&gt;
squeue
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These simple tests confirmed that the SLURM scheduler was active, compute nodes were responding, and jobs were successfully executed across the PCS cluster.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;AWS PCS simplifies HPC cluster management with a managed, scalable, and elastic environment.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Terraform (awscc) enables modern Infrastructure as Code deployment.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;SLURM provides flexible and familiar scheduling for HPC users.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Custom AMIs and bootstrap scripts enable deep customization and reproducibility.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Ideal for research, AI/ML training, and data-intensive simulations.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Thanks for reading!&lt;/p&gt;

&lt;p&gt;#terraform #aws #devops #cloud #hpc #slurm #packer #ami #python&lt;/p&gt;

</description>
      <category>aws</category>
      <category>terraform</category>
      <category>devops</category>
      <category>hpc</category>
    </item>
  </channel>
</rss>
