DEV Community: Oleksandr Kuryzhev

Terraform Vault Secrets: Keep Them Out of State and Git

Oleksandr Kuryzhev — Sat, 17 Oct 2026 07:01:59 +0000

Originally published on kuryzhev.cloud

The scenario

A team's Terraform repo needs a database password. The quickest way to get one into the config is a terraform.tfvars file that someone pasted into a CI variable. Handling Terraform Vault secrets properly means the password lives in HashiCorp Vault, Terraform fetches it at apply time, and nothing sensitive is committed. That is the goal, but it has one catch worth understanding up front.

The classic approach is the vault_kv_secret_v2 data source. It works, but values read through data sources are persisted in Terraform state in plain text, and this is documented behavior. Marking a variable sensitive only hides it from CLI output. So the secret leaves Vault and ends up in whatever backend stores your state.

Newer Terraform and Vault provider releases close that gap with ephemeral resources and write-only arguments. These values exist during a run and are never written to plan or state. This guide builds that pattern end to end. It also shows the fallback when your versions or target resources don't support it yet.

Prerequisites

You need a reachable Vault server with a KV version 2 secrets engine mounted (the dev-mode default is secret/) and a way to authenticate non-interactively.

Ephemeral resources require Terraform 1.10 or later, and write-only arguments require 1.11 or later. The Vault provider needs a release that ships the vault_kv_secret_v2 ephemeral resource. Pinning the current 5.x line is a safe default, but confirm the exact minimum in the provider changelog. Verify the details in the Terraform ephemeral resources documentation and the Vault provider registry page, because they move between releases.

You also need a target resource that exposes a write-only argument for the secret. Many providers have added these gradually, for example password_wo on the AWS provider's database instance resource. Check your provider's resource documentation rather than assuming support.

Terraform 1.11+ (or 1.10+ if you only use ephemeral values in provider blocks and other ephemeral contexts)
A KV v2 secret, for example at secret/app/db with a password key
A Vault auth method for CI, such as AppRole or JWT/OIDC, not a long-lived root token
A remote state backend with encryption and restricted access, as a baseline regardless

If you already run keyless CI, the same idea applies here. The DevOps_DayS archive covers OIDC federation for CI, which pairs well with Vault's JWT auth.

Step 1: Configure the provider without hard-coded credentials

Keep the provider block free of tokens. The Vault provider reads VAULT_ADDR and VAULT_TOKEN from the environment. It also supports auth_login* blocks for AppRole, JWT, and other methods, which are listed in the provider documentation on developer.hashicorp.com.

The following provider block sets only the address and one behavior flag worth understanding.

terraform {
  required_version = ">= 1.11"
  required_providers {
    vault = {
      source  = "hashicorp/vault"
      version = "~> 5.0" # confirm the ephemeral-capable version in the registry
    }
  }
}

provider "vault" {
  address = var.vault_address # or set VAULT_ADDR in the environment

  # By default the provider requests a child token, which needs
  # permission on auth/token/create. This example skips that and uses
  # the supplied token directly; remove this line if your policy
  # allows token creation and you want a short-lived child token.
  skip_child_token = true
}

Watch out for the child token default. A typical failure is a 403 on auth/token/create when the CI policy is tightly scoped. You have two documented options. You can grant that capability and drop the flag, or keep skip_child_token = true as shown, in which case the CI token's own TTL is what limits exposure.

Step 2: Write a least-privilege Vault policy

The identity Terraform uses should read only the paths the configuration needs. With KV v2, the API path differs from the CLI path. The policy must target secret/data/..., not secret/....

# terraform-ci.hcl: apply with: vault policy write terraform-ci terraform-ci.hcl
path "secret/data/app/db" {
  capabilities = ["read"] # KV v2 reads go through /data/
}

path "secret/metadata/app/db" {
  capabilities = ["read"] # only needed if you read version metadata
}

Watch out for the missing data/ segment. A policy written for secret/app/db looks correct but still returns permission denied on KV v2. Bind the policy to your AppRole or JWT role. Keep token TTLs short enough that a leaked CI token expires quickly.

Step 3: Read Terraform Vault secrets as ephemeral values

An ephemeral block fetches the secret during plan and apply but never records it. You can reference it only in places that accept ephemeral values: write-only arguments, provider configuration, and other ephemeral contexts. Terraform rejects references from ordinary attributes at validation time, which is the safety net you want.

ephemeral "vault_kv_secret_v2" "db" {
  mount = "secret"
  name  = "app/db"
}

resource "aws_db_instance" "app" {
  identifier        = "app-db"
  engine            = "postgres"
  instance_class    = "db.t4g.micro"
  allocated_storage = 20
  username          = "app_admin"

  # Write-only: sent to the API, never stored in plan or state
  password_wo         = ephemeral.vault_kv_secret_v2.db.data["password"]
  password_wo_version = 1 # bump this number to trigger a password update
}

Because write-only values are not stored, Terraform cannot detect when they change. The companion _version argument is the documented mechanism: increment it when the secret rotates in Vault. Confirm the attribute names for your provider versions, since the ephemeral resource's output shape (data versus data_json) is version-dependent.

Step 4: Plan the fallback for unsupported resources

Not every resource has a write-only argument yet. When you must use the vault_kv_secret_v2 data source, the secret will be in state, so treat state as a secret store:

Use an encrypted remote backend.
Limit read access to the state to the same people who can read the secret.
If the backend is an object store such as S3 or GCS, enable versioning and access logging on the bucket.

Prefer dynamic secrets where possible. Vault's database and AWS secrets engines issue short-lived credentials, so a value captured in state expires on its own. That limits the damage even when the pattern isn't ideal.

# Fallback: value WILL be stored in state. Protect the backend accordingly.
data "vault_kv_secret_v2" "legacy" {
  mount = "secret"
  name  = "app/legacy"
}

# Reference as data.vault_kv_secret_v2.legacy.data["api_key"]

Also avoid managing the secret's value itself with vault_kv_secret_v2 as a resource unless you accept it in state. Recent provider releases offer a write-only data_json_wo argument for that case; check the resource documentation for your version.

Verify and test

Start by proving that the identity can read the path and nothing broader. Then confirm the secret is absent from state. Use a throwaway secret value for this test so a failure costs nothing.

# 1. Confirm the policy grants what you expect (note the /data/ path)
vault token capabilities secret/data/app/db

# 2. Validate and plan; a reference from a normal attribute fails here
terraform validate
terraform plan -out=tfplan

# 3. Apply, then search state and plan for the secret value
terraform apply tfplan
terraform state pull | grep -c "your-test-password"   # expect 0 with write-only args
terraform show -json tfplan | grep -c "your-test-password"

With the ephemeral and write-only path, both counts should be zero. A non-zero count means something is still reading the value through a data source or a regular attribute. Run terraform state list and look for data.vault_kv_secret_v2 entries.

Rotate the secret in Vault, bump password_wo_version, and confirm the plan shows an in-place update.
Revoke the CI token and confirm the run fails with a clear permission error rather than a silent default.
If an audit device is enabled, check Vault's audit log for the read. It gives you evidence of who fetched what and when.
Grep the repository and CI variables for leftover .tfvars files with the old password.

Where this leaves you

Getting Terraform Vault secrets right comes down to three habits:

Authenticate with short-lived identities.
Scope policies to exact KV v2 paths.
Prefer ephemeral resources with write-only arguments so values never reach state.

When a resource doesn't support that yet, treat the state backend as part of your secret store and favor dynamic credentials that expire. Ephemeral support and attribute names change between Terraform and provider releases. Before you upgrade, check the official Terraform documentation and the Vault provider registry page, and rerun the state-grep check after every provider bump.

Ansible server bootstrap locks you out of new hosts: fix

Oleksandr Kuryzhev — Fri, 16 Oct 2026 07:01:57 +0000

Originally published on kuryzhev.cloud

Failure scenario: an Ansible server bootstrap that locks you out

A typical Ansible server bootstrap goes wrong like this. A team writes one playbook that runs against a fresh cloud VM as root. It creates a deploy user, turns on the firewall, and disables root and password logins in sshd. On the first host it appears to work. On the next one, the play stops halfway with UNREACHABLE, and every later attempt fails with Permission denied (publickey).

The first attempt on a new host behaves differently from every re-run. The root session that started the play is already authenticated, so it survives the sshd restart. A fresh connection is rejected, because root is now disabled and the new user's key may never have been installed correctly.

The playbook below is the shape that produces this failure. It is a composite for illustration, not a recording of a real incident.

---
# ANTI-PATTERN: one play, one connection user, hardening mixed with account setup
- name: Bootstrap server
  hosts: new_servers
  remote_user: root          # inventory also says ansible_user=root, forever
  tasks:
    - name: Create deploy user
      ansible.builtin.user:
        name: deploy
        groups: sudo

    - name: Disable root and password login
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: "^#?{{ item.key }}"
        line: "{{ item.key }} {{ item.value }}"
      loop:
        - { key: PermitRootLogin, value: "no" }
        - { key: PasswordAuthentication, value: "no" }
      notify: restart ssh

    - name: Enable firewall
      community.general.ufw:
        state: enabled          # default policy may deny port 22 if no allow rule exists yet

  handlers:
    - name: restart ssh
      ansible.builtin.service:
        name: sshd
        state: restarted

Nothing in this file is syntactically wrong, which is why it passes review. The problems are in the ordering and in what is never checked.

Why it happens

Several independent causes combine here, and any one of them is enough to lock you out.

The authorized key is never installed. The deploy user is created, but no task places a public key in its ~/.ssh/authorized_keys. Once root login and passwords are off, no valid credential remains.

The firewall can cut off port 22. With the UFW module, enabling the firewall with a default deny incoming policy and no allow rule for SSH drops new connections. The module's documented behavior is to apply exactly what you ask, so a missing allow rule is your mistake, not a bug.

Watch out for drop-in precedence. Current Debian and Ubuntu images ship an Include /etc/ssh/sshd_config.d/*.conf line at the top of sshd_config. For most sshd keywords the first value read wins. A cloud-init file such as 50-cloud-init.conf that sets PasswordAuthentication yes can therefore override your edit at the bottom of the main file. You may think you hardened the host when you did not. Verify the effective values with sshd -T.

There are also three smaller problems:

Handlers run at the end of the play. If any later task fails, the restart is skipped and the host sits in an inconsistent state until you rerun.
Minimal images may have no Python, so fact gathering fails before your first task.
The service is named ssh on Debian-family systems and sshd on RHEL-family ones. Some recent Ubuntu releases also use socket activation, so check how your target restarts the daemon.

The fix: split the bootstrap into verified phases

The principle is simple: never remove an access path until you have proven the replacement works. The playbook below uses one play to create access. A second play connects as the new user and does the hardening.

Phase 1 connects with the image's initial credentials and creates the admin user, its key, and sudo rights.

---
- name: Phase 1 - create admin access
  hosts: new_servers
  gather_facts: false
  vars:
    ansible_user: root                      # play vars override inventory for this play only
  tasks:
    - name: Ensure Python exists (minimal Debian/Ubuntu images; use dnf on RHEL-family)
      ansible.builtin.raw: test -x /usr/bin/python3 || (apt-get update && apt-get install -y python3)
      changed_when: false

    - name: Gather facts now that Python is present
      ansible.builtin.setup:

    - name: Create deploy user
      ansible.builtin.user:
        name: deploy
        shell: /bin/bash
        create_home: true

    - name: Install public key
      ansible.posix.authorized_key:
        user: deploy
        key: "{{ lookup('ansible.builtin.file', 'files/deploy.pub') }}"
        exclusive: true                     # removes stray keys; remove this line if you manage keys elsewhere

    - name: Passwordless sudo, validated before it lands
      ansible.builtin.copy:
        dest: /etc/sudoers.d/90-deploy
        content: "deploy ALL=(ALL) NOPASSWD:ALL\n"
        mode: "0440"
        validate: /usr/sbin/visudo -cf %s   # a broken sudoers file is its own lockout

Phase 2 connects as deploy. If the key or sudo is wrong, the play fails here, while root access still works.

- name: Phase 2 - verify access, then harden
  hosts: new_servers
  become: true
  vars:
    ansible_user: deploy
  tasks:
    - name: Prove key login and sudo work (fails before anything is locked down)
      ansible.builtin.command: id -u
      register: uid
      changed_when: false
      failed_when: uid.stdout != "0"

    - name: Allow SSH BEFORE enabling the firewall
      community.general.ufw:
        rule: allow
        port: "22"
        proto: tcp

    - name: Enable firewall
      community.general.ufw:
        state: enabled
        policy: deny
        direction: incoming

    - name: SSH hardening drop-in (00- sorts before 50-cloud-init.conf)
      ansible.builtin.copy:
        dest: /etc/ssh/sshd_config.d/00-hardening.conf
        content: |
          PermitRootLogin no
          PasswordAuthentication no
        mode: "0644"
      notify: reload ssh

    - name: Test full sshd config before any reload
      ansible.builtin.command: /usr/sbin/sshd -t
      changed_when: false

    - name: Apply now, not at end of play
      ansible.builtin.meta: flush_handlers

    - name: Drop the multiplexed session so the next task opens a fresh SSH login
      ansible.builtin.meta: reset_connection

    - name: Confirm a new key-based connection still works
      ansible.builtin.wait_for_connection:
        timeout: 30

  handlers:
    - name: reload ssh
      ansible.builtin.service:
        name: "{{ 'ssh' if ansible_os_family == 'Debian' else 'sshd' }}"
        state: reloaded

The reset_connection step matters. Ansible normally reuses a persistent SSH control connection, so without it wait_for_connection could succeed over the old session and prove nothing. Also note that if sshd -t fails, the play stops before the reload, but the bad drop-in stays on disk. Fix or remove it before anything restarts sshd or reboots the host.

Once the host is hardened, rerunning is safe if you target only the second play, for example with a tag or a separate inventory group. Keep the root-based play restricted to hosts that have never been bootstrapped. The module names above are from the current ansible.builtin, ansible.posix, and community.general collections. Check the Ansible collections documentation for your installed versions and parameter details.

Prevention checklist

Before the bootstrap playbook goes near a real fleet, run through the points below. Most of them are cheap to automate in CI against a throwaway VM.

[ ] Access first, hardening second: separate plays, separate connection users
[ ] A task proves key login AND sudo work as the new user before hardening
[ ] Firewall: allow 22/tcp (or your SSH port) before enabling; default deny after
[ ] sshd changes live in a drop-in named to sort before cloud-init files
[ ] sshd -t runs before every reload; sudoers uses validate: visudo -cf %s
[ ] Handlers flushed explicitly (meta: flush_handlers) when order matters
[ ] meta: reset_connection before verifying reconnection, so the check uses a fresh login
[ ] Service name handled per OS family (ssh vs sshd); socket activation checked
[ ] Python bootstrap via raw for minimal images, with changed_when: false
[ ] Effective config verified with: sshd -T | grep -Ei 'permitrootlogin|passwordauth'
[ ] Provider console / serial access or a rescue snapshot confirmed before the first run
[ ] Tested on a disposable host: real run first, then a second run (--check --diff is only
    meaningful after the first real run, since Phase 2 needs the deploy user to exist)

Watch out for the second run. A bootstrap playbook that only succeeds once is a script, not automation. Run it twice against the same disposable host. The second pass of the hardening play should report no changes. If it cannot even connect, the inventory still assumes root access that no longer exists.

Keep one out-of-band recovery route, such as the provider's web console or a rescue mode. Confirm it works before you automate anything that touches SSH or the firewall. It will not prevent the mistake, but it turns a rebuild into a quick repair. For more infrastructure automation patterns, browse the rest of kuryzhev.cloud.

A reliable Ansible server bootstrap is mostly about ordering and proof: create access, test it from a fresh connection, and only then close the old doors.

Helm Chart Values for Staging vs Production Done Right

Oleksandr Kuryzhev — Thu, 15 Oct 2026 07:02:05 +0000

Originally published on kuryzhev.cloud

A deploy goes green in staging, then production comes up with one replica, a debug log level, and an ingress host that points at the wrong environment. Nobody changed the chart. The Helm chart values were assembled differently, and nobody could say exactly how. This is a common way environment drift shows up. It is usually a misunderstanding of how Helm merges values rather than a Helm bug.

What Helm chart values actually do

Values are the input to the chart's Go templates. Helm builds a single values object from several sources and renders the templates against it. The documented precedence, lowest to highest, is:

the chart's own values.yaml
values from a parent chart
files passed with -f / --values (later files win)
--set, --set-string, and --set-file flags

Maps are merged key by key, so an environment file only needs to contain what differs. Lists are not merged. A list in a later source replaces the earlier list entirely. That one rule explains many "why did my tolerations vanish" tickets.

Setting a key to null in an override file removes the default key. This is useful for dropping a default resource limit or annotation.

Helm stores the user-supplied values with each release, alongside the chart.

helm get values shows the values you supplied.
helm get values --all shows the full merged result.

These are the first commands to run when two environments disagree. Details are in the official Helm values documentation. Verify the exact behavior against the Helm version you run, since Helm 3 and Helm 4 differ in some details.

How people use Helm chart values wrong

The first pattern is copying the entire values.yaml into values-staging.yaml and values-production.yaml. Within a few releases, someone adds a key to one file and not the others. The files stop being overrides and become three divergent configurations.

The second is environment logic inside templates, such as {{ if eq .Values.env "production" }}. It looks convenient, but the chart now encodes knowledge about your environments. Staging can no longer exercise the same code path as production, and every new environment needs a template change.

The third is pipeline-side --set flags. A CI job with a dozen --set arguments is configuration that lives outside version control review.

Watch out for type coercion here:

--set enabled=true becomes a boolean.
--set image.tag=20240101 becomes an integer. Use --set-string when the value must stay a string.
Values files have the same trap through YAML parsing. An unquoted version: 1.10 is read as the float 1.1, so quote version-like strings.

Other common failure modes:

Overriding a list (hosts, env entries, tolerations) and losing the defaults.
Putting secrets in plain values files committed to Git.
Using image.tag: latest in either environment. This makes a rollback unreliable, because the same tag can point at different images.
Skipping helm template or helm lint and discovering a missing key only at install time.

The correct approach to Helm chart values

Treat values.yaml as safe, production-shaped defaults, and keep each environment file as a small diff on top. Staging should differ from production in scale and endpoints, not in structure. The templates stay identical across environments.

Here is a minimal layout. The base file holds defaults, and the environment files carry only differences:

# values.yaml (chart defaults, safe and production-shaped)
replicaCount: 3
image:
  repository: registry.example.com/shop/api
  tag: ""            # set at deploy time, never "latest"
resources:
  requests: { cpu: 250m, memory: 256Mi }
  limits: { memory: 512Mi }
ingress:
  enabled: true
  host: api.example.com
logLevel: info

# values-staging.yaml (only what differs)
replicaCount: 1
resources:
  requests: { cpu: 50m, memory: 128Mi }
ingress:
  host: api.staging.example.com
logLevel: debug

Deploy by stacking files in a fixed order. The pipeline selects the environment file and passes the image reference, and nothing else:

# render first; fails fast on template and schema errors
helm template shop ./chart -f values-staging.yaml --set-string image.tag="$IMAGE_TAG" > /dev/null

# chart values.yaml is applied automatically; the -f file overrides it,
# and any additional -f files would override in the order given
helm upgrade --install shop ./chart \
  --namespace shop-staging --create-namespace \
  -f values-staging.yaml \
  --set-string image.tag="$IMAGE_TAG" \
  --atomic --wait --timeout 5m   # roll back automatically on failure

In Helm 3, --atomic already implies --wait. In Helm 4 it is renamed to --rollback-on-failure, so adjust the flag to match your client version.

Add a values.schema.json to the chart. Helm validates merged values against it on install, upgrade, lint, and template.

A string where an integer belongs fails before anything reaches the cluster.
A typo like replicaCout is only rejected if the schema sets "additionalProperties": false for that object.

Advanced patterns

Once the basic layering works, a few patterns reduce risk further. Which one fits depends on how many environments and teams you have.

Layered files per dimension. Instead of one file per environment, split by concern: values-common.yaml, values-prod.yaml, values-eu-west.yaml. Pass them in order of increasing specificity. This scales to multiple clusters and regions without duplicating shared settings.

Convert lists to maps where you control the chart. Since lists are replaced wholesale, a map such as env: { LOG_FORMAT: json } can be overridden one key at a time. Templates can iterate the map with range $k, $v := .Values.env. Map iteration in Go templates is sorted by key, so output stays deterministic.

Secrets outside values. Reference Kubernetes Secrets by name in values and populate them with an external secrets operator. Alternatively, encrypt value files with SOPS through a plugin such as helm-secrets. Check the plugin's current maintenance status before adopting it.

GitOps tools. Argo CD supports multiple value files per application source. Flux's HelmRelease supports valuesFrom and inline values. In both, the environment is declared in Git and reviewed like code. Verify the field names against your installed version.

Review the diff, not the values. The helm diff upgrade plugin shows the rendered manifest change for a release. A one-line values change that rewrites a StatefulSet's volume template is far easier to catch there than in a values pull request.

Another gotcha: subchart values live under the subchart's name. A typo in that key is silently ignored by Helm unless a schema rejects unknown keys. Use global: only for values that genuinely apply to every subchart.

Performance notes

Helm rendering itself is rarely the bottleneck. Template rendering is local and usually finishes quickly even for large charts. Very heavy use of tpl, nested include, or huge values trees can slow it. Measure with time helm template before optimizing.

The costs that matter are at runtime and in release storage:

Release size. Helm stores each release revision in a Secret by default, and Kubernetes caps a single Secret at 1 MiB. Helm compresses the release data, but large embedded files in values or templates can still hit that limit. Use --history-max (default 10 on helm upgrade) to cap retained revisions.
Rollout time. --wait blocks until resources are ready. Your --timeout should reflect readiness probe settings and image pull times, not a guess.
Resource requests. Cutting staging requests saves cost, but extreme differences hide problems. A service that is never CPU-throttled in staging may be throttled in production. See the Kubernetes resource management docs for how requests and limits affect scheduling and throttling.
Replica parity. One staging replica is unlikely to reveal rolling-update or PodDisruptionBudget problems. If those matter, run at least two replicas in staging.

The practical rule is to keep environment differences few, explicit, and reviewable. For more infrastructure patterns, browse the rest of kuryzhev.cloud.

ECR Image Scan Gate in GitLab CI Passes Vulnerable Images

Oleksandr Kuryzhev — Wed, 14 Oct 2026 07:02:09 +0000

Originally published on kuryzhev.cloud

A merge request goes green, the image is deployed, and a day later someone opens the ECR console and finds a critical CVE on that exact tag. The pipeline had an ECR image scan gate, and it reported success. This runbook covers why a gate like that passes when it should not, and how to rebuild it so it fails closed.

Symptoms

These are the typical signs of a gate that fails open. You may see one or several.

The gate job is green, but aws ecr describe-image-scan-findings run by hand shows CRITICAL or HIGH findings for the same image.
The gate passes on fresh pushes and only "works" on re-runs, when the scan has had time to finish.
The job log shows an empty or None severity count, and the script treats that as zero.
Intermittent ScanNotFoundException or LimitExceededException errors appear, and someone added || true to quiet them.
The scanned tag (latest, or a branch name) points at a different image by the time the deploy job runs.

Watch out for: a green job with no printed counts. If the log never shows a number, the gate did not check anything.

Root cause

ECR scans are asynchronous. A push triggers a scan (if scan-on-push is enabled), but the results arrive later. If the gate queries findings immediately, the response either errors (typically ScanNotFoundException) or returns a status such as IN_PROGRESS with no findingSeverityCounts at all.

The severity counts map also omits severities with zero findings. A naive query for .CRITICAL therefore returns null (or None with --output text) in two very different situations: the scan is clean, or the scan has not finished. A script that treats "empty" as "zero" cannot tell these apart, so it passes both.

Three secondary causes make this worse:

Swallowed errors. || true or a missing set -e turns an API failure into a pass.
Mutable tags. The gate scans repo:latest, but the deploy job pulls whatever that tag points at later.
Scan type differences. Basic scanning (on push or manual) and enhanced scanning (Amazon Inspector, continuous) report different statuses and have different manual-scan behavior. The applicable limits and status values depend on how scanning is configured for your registry, so check the ECR image scanning documentation for your setup.

Fixing the ECR image scan gate means removing every path where "no data" becomes "no findings".

Fix #1: Wait for a terminal scan status, then fail closed

Poll the scan status by image digest, and exit non-zero on anything that is not a completed scan. The job below uses only the AWS CLI's built-in --query, so it needs no jq in the image. Only "scan not found yet" is treated as retryable. Any other API error, such as AccessDenied, fails the job immediately.

ecr-scan-gate:
  stage: verify
  image:
    name: amazon/aws-cli:latest   # pin a specific version tag in real pipelines
    entrypoint: [""]              # the image's default entrypoint is "aws", which breaks script lines
  script:
    - set -eu
    - DIGEST=$(cat image-digest.txt)
    - |
      STATUS="NOT_STARTED"
      for i in $(seq 1 30); do
        if OUT=$(aws ecr describe-image-scan-findings \
              --repository-name "$ECR_REPO" \
              --image-id imageDigest="$DIGEST" \
              --query 'imageScanStatus.status' --output text 2>&1); then
          STATUS="$OUT"
        else
          case "$OUT" in
            *ScanNotFoundException*) STATUS="NOT_STARTED" ;;    # scan not registered yet: retry
            *) echo "$OUT"; echo "API error, failing closed"; exit 1 ;;
          esac
        fi
        echo "attempt $i: scan status = $STATUS"
        case "$STATUS" in
          COMPLETE|ACTIVE) break ;;                              # ACTIVE appears with enhanced scanning
          FAILED|UNSUPPORTED_IMAGE|SCAN_ELIGIBILITY_EXPIRED|FINDINGS_UNAVAILABLE)
            echo "scan unusable: $STATUS"; exit 1 ;;
        esac
        sleep 10
      done
      case "$STATUS" in COMPLETE|ACTIVE) ;; *) echo "scan never completed"; exit 1 ;; esac

The loop ends in a failure unless a terminal good status was seen. Timing out is a failure, not a pass. Verify the exact status values your registry returns against the AWS documentation before relying on the case list.

Fix #2: Gate on explicit severity counts with a default

Once the scan is complete, a missing severity key genuinely means zero findings. That default is safe only after Fix #1 has confirmed the status. Append this to the same script, and make the threshold a variable so teams can tune it. Anything other than a number or None, including an empty result from a failed API call, stops the job.

    - |
      count() {
        # prints a number; maps the CLI's "None" (key absent) to 0 -- safe ONLY after status is COMPLETE/ACTIVE
        V=$(aws ecr describe-image-scan-findings \
              --repository-name "$ECR_REPO" \
              --image-id imageDigest="$DIGEST" \
              --query "imageScanFindings.findingSeverityCounts.$1" --output text) || return 1
        case "$V" in
          None) echo 0 ;;
          ''|*[!0-9]*) echo "unexpected count for $1: '$V'" >&2; return 1 ;;
          *) echo "$V" ;;
        esac
      }
      CRIT=$(count CRITICAL) || { echo "could not read CRITICAL count"; exit 1; }
      HIGH=$(count HIGH)     || { echo "could not read HIGH count"; exit 1; }
      echo "CRITICAL=$CRIT HIGH=$HIGH (max allowed HIGH=${MAX_HIGH:-0})"
      [ "$CRIT" -eq 0 ] || { echo "blocked: critical findings"; exit 1; }
      [ "$HIGH" -le "${MAX_HIGH:-0}" ] || { echo "blocked: too many high findings"; exit 1; }

Gotcha: the log must print the counts. When a reviewer later asks why a build passed, "CRITICAL=0 HIGH=0" in the job output is the evidence. Also note that a gate on severity counts alone cannot express an exception for a known, accepted CVE. Handle that with a reviewed allowlist file rather than by loosening the threshold.

Fix #3: Pin to the digest and handle scan triggers deliberately

Scan the digest that was pushed, and hand that same digest to the deploy job. This closes the mutable-tag gap. Capture the digest at push time and publish it as an artifact. The job image must provide both Docker (for example via a Docker-in-Docker service) and the AWS CLI.

build-push:
  stage: build
  script:
    - set -eu
    - aws ecr get-login-password --region "$AWS_REGION" | docker login --username AWS --password-stdin "${ECR_URI%%/*}"
    - docker build -t "$ECR_URI:$CI_COMMIT_SHA" .
    - docker push "$ECR_URI:$CI_COMMIT_SHA"
    - |
      aws ecr describe-images --repository-name "$ECR_REPO" \
        --image-ids imageTag="$CI_COMMIT_SHA" \
        --query 'imageDetails[0].imageDigest' --output text > image-digest.txt
    - echo "IMAGE_DIGEST=$(cat image-digest.txt)" > build.env
  artifacts:
    paths: [image-digest.txt]
    reports:
      dotenv: build.env     # deploy jobs read $IMAGE_DIGEST and deploy by digest, never by tag

Next, decide who triggers the scan. With scan-on-push enabled on the repository, the gate only reads results. If you rely on a manual aws ecr start-image-scan call instead, basic scanning limits how often one image can be rescanned, so a retried job can hit LimitExceededException. Treat that specific error as "scan already requested, keep polling", and fail on every other error. With enhanced scanning, manual scans are not the mechanism at all, so check the AWS docs for your configuration.

Finally, scope IAM per job. The gate job needs only ecr:DescribeImageScanFindings on that repository. The build job needs ecr:GetAuthorizationToken (which applies to all resources), the push actions (ecr:BatchCheckLayerAvailability, ecr:InitiateLayerUpload, ecr:UploadLayerPart, ecr:CompleteLayerUpload, ecr:PutImage), and ecr:DescribeImages to read the digest. Obtain these roles through GitLab's ID token authentication rather than stored access keys. A missing permission otherwise surfaces as an AccessDenied that an old || true would hide.

Prevention

Make the failure mode part of the pipeline design, not a one-time patch. Use this checklist when you add or review the gate.

[ ] Scan-on-push is enabled on the repository (or enhanced scanning is configured at registry level)
[ ] Gate queries by imageDigest, not by mutable tag
[ ] Gate waits for a terminal status; a timeout exits non-zero
[ ] FAILED / UNSUPPORTED_IMAGE / SCAN_ELIGIBILITY_EXPIRED / FINDINGS_UNAVAILABLE exit non-zero
[ ] Only ScanNotFoundException is retried; other API errors exit non-zero
[ ] Missing severity key is read as 0 ONLY after a completed status
[ ] Non-numeric or empty counts exit non-zero
[ ] No "|| true" and "set -eu" is active in the script
[ ] Counts are printed in the job log
[ ] Deploy job consumes the same digest via dotenv artifact
[ ] IAM roles are least-privilege per job and obtained via OIDC, not static keys
[ ] A known-vulnerable test image proves the gate actually fails

The last item matters most. Push an old base image with known critical CVEs to a throwaway repository and confirm the job goes red. A gate that has never failed has never been shown to work.

Also plan for drift. A clean image today can have new findings next week, and a one-time gate at build time will not see them. Enhanced scanning re-evaluates images continuously. You can combine it with a scheduled GitLab pipeline that re-runs the digest check against what is currently deployed. For more pipeline and AWS runbooks, browse kuryzhev.cloud.

Content Workflow Automation Failing in n8n or Bedrock? Fixes

Oleksandr Kuryzhev — Mon, 12 Oct 2026 07:02:21 +0000

Originally published on kuryzhev.cloud

A content workflow automation built in n8n passes every test with one topic, then fails the first time an editor submits twenty. Some drafts never arrive, a few arrive twice, and one is cut off mid-sentence and sits in WordPress as a "ready" post. This runbook covers the failure modes that show up when n8n orchestrates Amazon Bedrock calls, and the changes that remove them.

Symptoms

Several symptoms often appear together, but each points to a different cause.

ThrottlingException or HTTP 429 from Bedrock when a batch starts, often within the first few items depending on your quota and model.
ValidationException mentioning the model identifier or saying on-demand throughput is not supported for that model. The exact wording is version- and model-dependent.
AccessDeniedException on a model that works in the console, commonly because the IAM role, the region, or the account's model access differs.
504 or "Gateway Timeout" returned to the caller of an n8n webhook while the execution keeps running in the background.
Duplicate posts after a retry, a restart, or a client that resent the webhook.
Broken or truncated output: invalid JSON in a "parse" step, missing closing sections, or markdown fences inside the post body.

If only the first symptom is present, jump to Fix #1. If posts are duplicated, start with Fix #2.

Root cause

These failures usually come from treating an LLM call like an ordinary fast API call inside a content workflow automation. Three assumptions break under real load.

Assumption 1: capacity is unlimited. Bedrock enforces per-model, per-region quotas on requests and tokens per minute. n8n's default behavior is to push items through a node as fast as it can, so a batch of items becomes a burst. The documented behavior is a throttling error, and quotas differ by model and account. Verify yours in the Bedrock quotas documentation and the Service Quotas console.

Assumption 2: the call finishes before the caller gives up. Long generations can take tens of seconds, depending on the model and output length. A webhook that waits for the whole chain hits the proxy or load balancer timeout first, the client retries, and the workflow runs again.

Assumption 3: the output is trustworthy. A model can stop at the token limit, wrap JSON in prose, or return a different structure than requested. If the next node publishes whatever arrives, the defect lands in production.

Watch out for the interaction between 2 and 3: a retry after a timeout is non-idempotent by default, so each retry produces a fresh, slightly different article.

Fix #1: Control concurrency and retry throttling

Slow the burst down and let retries absorb the remainder. In n8n, put the Bedrock call inside a Loop Over Items (Split In Batches) node with a batch size of 1 to 3, and add a Wait node when quotas are tight. On the node that makes the Bedrock request (for example HTTP Request with AWS authentication), enable Retry On Fail with a wait between tries of several seconds. n8n caps the number of tries, so treat this as a complement to batching, not a replacement.

If the call runs in a Lambda worker instead of an n8n node, let the AWS SDK for JavaScript v3 do the backoff, and set the function timeout well above the expected generation time plus retries. The sketch below uses the Converse API, which gives one request shape across supported models.

import { BedrockRuntimeClient, ConverseCommand } from "@aws-sdk/client-bedrock-runtime";

const client = new BedrockRuntimeClient({
  region: process.env.AWS_REGION,
  maxAttempts: 8,         // SDK retries throttling errors with backoff
  retryMode: "adaptive",  // adds client-side rate limiting on throttles
});

export const handler = async (event) => {
  const res = await client.send(new ConverseCommand({
    modelId: process.env.MODEL_ID,   // inference profile ID/ARN if the model requires one
    messages: [{ role: "user", content: [{ text: event.prompt }] }],
    inferenceConfig: { maxTokens: 2000, temperature: 0.3 },
  }));
  return {
    text: res.output.message.content[0].text,
    stopReason: res.stopReason,      // "max_tokens" means the draft was cut off
  };
};

Two gotchas here. Some newer models are only invokable through a cross-region inference profile, so the plain model ID triggers the ValidationException above; check the model's page in the Bedrock documentation. Also, the IAM policy must allow bedrock:InvokeModel on the profile and on the underlying models in each region it routes to. If throttling persists at low volume, request a quota increase rather than adding more retries.

Fix #2: Acknowledge fast and make runs idempotent

Stop the webhook from waiting on the generation. Set the Webhook node's response mode to respond immediately, return a job ID, and let the rest of the workflow run asynchronously. For sustained volume, run n8n in queue mode so workers pull executions from Redis and a concurrency cap protects Bedrock.

# docker-compose.yml excerpt (no top-level "version" key needed)
services:
  n8n-worker:
    image: docker.n8n.io/n8nio/n8n:latest   # pin an exact tag in real deployments
    command: worker --concurrency=3         # parallel executions per worker
    environment:
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=redis
      - EXECUTIONS_TIMEOUT=600              # kill runaway executions (seconds)
      - NODE_FUNCTION_ALLOW_BUILTIN=crypto  # lets Code nodes hash input for dedupe

Variable names and worker flags change between n8n releases, and workers also need the same database and encryption key settings as the main instance. Confirm everything in the n8n environment variable reference.

Then add an idempotency key. Hash the topic, target keyword, and requester into a stable ID, and claim it atomically before generating anything, for example by inserting it into a Postgres table with a unique constraint. If the insert fails because the key exists, exit the run. A separate lookup followed by a write lets two concurrent duplicates both pass. Do not use the execution ID as the key: a resent webhook or a restart starts a new execution with a new ID, which defeats the check.

Fix #3: Validate output before anything publishes

Add a gate between generation and publishing. Check the stop reason, parse the structure, and enforce basic rules. Anything that fails goes to a review queue, not to the CMS.

This n8n Code node sketch assumes the previous step returns text and stopReason.

// n8n Code node, "Run Once for Each Item"
const { text, stopReason } = $json;
const errors = [];

// plain-text generation should end with "end_turn"; tool-use flows return other reasons
if (stopReason !== "end_turn") errors.push(`unexpected stop: ${stopReason}`);

let post;
try {
  // strip accidental markdown fences before parsing
  post = JSON.parse(text.replace(/^

```(?:json)?\s*|\s*```

$/g, ""));
} catch (e) {
  errors.push("invalid JSON");
}

if (post) {
  if (!post.title || post.title.length > 70) errors.push("bad title");
  if (!post.body || post.body.split(/\s+/).length < 600) errors.push("body too short");
  if (post.body && /lorem ipsum|as an ai/i.test(post.body)) errors.push("placeholder text");
}

return { json: { ok: errors.length === 0, errors, post } };

Route on ok with an IF node. Passing items should be created as drafts via the WordPress node, never straight to published. Where the model supports it, tool use or structured output in the Converse request can reduce parse failures further, though support varies by model. Verify with the Bedrock documentation.

Prevention

Build these guardrails in once so later batches do not depend on manual checks.

Create a CloudWatch alarm on Bedrock's InvocationThrottles and client-error metrics, so throttling shows up before editors report it.
Attach an n8n Error Trigger workflow that posts failed execution IDs and node names to a chat channel.
Keep model IDs and inference profiles in configuration, not hard-coded in nodes, so a deprecation is a one-line change.
Export workflows as JSON to Git, and review changes like code.
Set an AWS Budgets alert on Bedrock spend. A looping retry bug is also a billing bug.

Pre-launch checklist for every new workflow:

[ ] Batch size and Wait set; tested with 20+ items, not 1
[ ] Retry On Fail enabled; SDK maxAttempts/backoff configured
[ ] Webhook responds immediately; job ID returned
[ ] Idempotency key claimed atomically before generation
[ ] stopReason and schema validated before publish
[ ] Output lands as draft; human approval required
[ ] IAM scoped to the specific model/profile ARNs
[ ] Alarms: throttles, errors, spend
[ ] Failure test: kill a worker mid-run, resend webhook, confirm no duplicates

The last item matters most. Duplicate and half-finished runs only appear when something fails on purpose. For more runbooks on automation and cloud tooling, browse kuryzhev.cloud, and check the Amazon Bedrock quotas documentation whenever you add a model or region.

Kubernetes Ingress Manifest or Gateway API: Which to Ship

Oleksandr Kuryzhev — Sun, 11 Oct 2026 07:01:20 +0000

Originally published on kuryzhev.cloud

A team has a working Deployment and a ClusterIP Service, and now needs a public hostname with TLS. The familiar route is a Kubernetes Ingress manifest, but the ecosystem has been moving toward Gateway API, and the most widely copied Ingress controller has been announced for retirement. The Deployment and Service stay almost identical either way. What changes is the object that exposes them, and that choice is worth making deliberately.

When this choice matters

For a single app on a single cluster with one hostname, both approaches work, and the difference is mostly about future maintenance. The choice starts to matter when several teams share a cluster, when you need weighted traffic splits or header-based routing, or when your current controller is going out of support.

The ingress-nginx project announced its retirement, with best-effort maintenance planned to end in March 2026. Verify the current status in the project repository before building on it. A manifest full of controller-specific annotations is the part that hurts most during a migration, because those annotations are not portable.

Whichever you pick, the first two objects are the same. Here is the shared base: a Deployment and a Service using stable APIs.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web              # must match the pod template labels exactly
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: registry.example.com/web:1.4.2   # pin a tag or digest, not :latest
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:    # gates Service endpoints; without it traffic hits cold pods
            httpGet:
              path: /healthz
              port: http
          resources:
            requests: { cpu: 100m, memory: 128Mi }
            limits: { memory: 256Mi }
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: ClusterIP
  selector:
    app: web                # a typo here yields a Service with zero endpoints
  ports:
    - name: http
      port: 80
      targetPort: http      # refers to the named container port above

Watch out for: a Service selector that does not match the pod labels fails silently. The Service exists, DNS resolves, and every request errors. Verify with kubectl get endpointslices -l kubernetes.io/service-name=web.

Option A: A classic Kubernetes Ingress manifest

Ingress (networking.k8s.io/v1) is the established path. One object maps a host and path to a Service, and a controller you install separately implements it. The documented behavior is that the API is stable but frozen; the Kubernetes docs point new feature work to Gateway API.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
spec:
  ingressClassName: public      # replaces the old kubernetes.io/ingress.class annotation
  tls:
    - hosts: [app.example.com]
      secretName: app-example-tls   # kubernetes.io/tls Secret in the same namespace
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix    # required field in v1
            backend:
              service:
                name: web
                port:
                  name: http

Pros:

Small and widely understood. Most tutorials, Helm charts, and cert-manager examples assume it.
One object per app is easy to review in a pull request.
Mature support across managed cloud load balancer controllers and self-hosted proxies.

Cons:

The spec covers only host and path routing. Everything else, such as rewrites, timeouts, and canaries, lives in controller-specific annotations that do not carry over to another controller.
Roles are blurred. The app team that writes the Ingress also effectively touches shared edge behavior.
No new features are planned for the API itself.

Watch out for: an Ingress with a missing or wrong ingressClassName is simply ignored by every controller. It looks valid, kubectl apply succeeds, and nothing is routed.

Option B: Gateway API with HTTPRoute

Gateway API (gateway.networking.k8s.io/v1) splits the problem into objects owned by different roles. A platform team defines a GatewayClass and a Gateway (listeners, ports, TLS); app teams attach HTTPRoutes to it. The Deployment and Service above do not change.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web
spec:
  parentRefs:
    - name: shared-gateway       # Gateway owned by the platform team; its allowedRoutes must permit this namespace
      namespace: infra
  hostnames: [app.example.com]
  rules:
    - matches:
        - path: { type: PathPrefix, value: / }
      backendRefs:
        - name: web
          port: 80
          weight: 100            # add a second backendRef to split traffic natively

Pros:

Header matching, redirects, request header modification, and weighted backends are part of the standard API, not annotations.
Role separation: the Gateway can restrict which namespaces may attach routes.
Portable across conformant implementations, though implementation-specific extensions still exist.

Cons:

The CRDs are not built into Kubernetes. You install them, pick a channel (standard or experimental), and must keep versions compatible with your controller.
More objects to understand, and cross-namespace references need explicit grants such as ReferenceGrant.
Implementation coverage varies. Check your controller's conformance report for the features you need.

Watch out for: an HTTPRoute can be created successfully yet not be attached to the Gateway, for example when the Gateway's allowedRoutes excludes your namespace or a backend reference does not resolve. Always read status.parents[].conditions on the HTTPRoute for Accepted and ResolvedRefs instead of assuming apply means working.

Decision matrix

The table compares the two approaches on the factors that usually decide the question. It describes classes of approach, not specific products.

Factor	Ingress	Gateway API
One app, one host, simple TLS	Fits well	Works, more setup
Multiple teams on shared edge	Weak isolation	Designed for it
Canary or header routing	Annotations, non-portable	Standard fields
Cluster add-ons required	Controller only	Controller plus CRDs
Lock-in on migration	High if annotation-heavy	Lower, extensions aside
Long-term API direction	Frozen	Actively developed

A quick checklist to run against your own cluster before deciding:

# 1. What implements edge traffic today? (gatewayclass errors if the CRDs are absent)
kubectl get ingressclass,gatewayclass

# 2. How many lines of controller-specific annotations are you carrying?
kubectl get ingress -A -o yaml | grep -c 'nginx.ingress.kubernetes.io'

# 3. Are Gateway API CRDs installed, and which version?
kubectl get crd gateways.gateway.networking.k8s.io \
  -o jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}'

A high count in step 2 is a rough indicator of how much annotation-specific work a migration will involve, since each one needs an equivalent or a deliberate replacement.

Evidence-based recommendation

For new clusters, start with Gateway API if your chosen controller documents conformance for the features you need. The API is where development is happening, and it keeps routing logic out of annotations. Platform teams serving several app teams get the most benefit, because the Gateway/HTTPRoute split matches how responsibility already divides.

Keep a plain Kubernetes Ingress manifest when the cluster is small, the routing is host and path only, and your controller's Gateway API support is immature or unverified. It remains valid and supported as an API. Just keep annotations to a minimum and write down each one, so a later move is a mechanical task. The Gateway API project provides an ingress2gateway tool for converting existing resources; verify its coverage of your controller and annotations before relying on it.

If you already run an Ingress controller that is retiring or unmaintained, treat that as the deciding factor. Plan the move to a maintained controller, and use the moment to adopt Gateway API instead of swapping one annotation dialect for another. Migrate one low-risk hostname first, compare behavior, then shift the rest.

Whatever you choose, keep the Deployment and Service boring: pinned image, readiness probe, named ports, matching selectors. Many "my Ingress is broken" problems trace back to those, not the edge object. For the authoritative field reference, see the Kubernetes Ingress documentation and the Gateway API overview. More practical Kubernetes and DevOps notes are on kuryzhev.cloud.

Terraform Modules for AWS VPC, IAM, and S3: Structure That Scales

Oleksandr Kuryzhev — Sat, 10 Oct 2026 07:02:15 +0000

Originally published on kuryzhev.cloud

The scenario

A second team asks for "the same network and storage setup as the first one," and suddenly your single 600-line main.tf needs to become Terraform modules that two environments can share. The VPC, the application bucket, and the role that reads from it are all tangled together, and copying the file means copying every hardcoded CIDR and bucket name with it.

This guide splits that setup into three small modules: vpc, s3, and iam. A thin root configuration then composes them per environment. The design rule is simple. Each module owns one concern, exposes a few typed inputs, and returns only the outputs that other modules need.

The IAM module does not create or look up the bucket. It receives a bucket ARN as an input, which keeps the dependency visible and one-directional. The VPC module stands alone. Nothing here is exotic, but the boundaries are where most module designs go wrong, and they are what make reuse possible later.

Prerequisites

You need Terraform 1.6 or newer (or a compatible OpenTofu release) and the AWS provider on the current 6.x line. Pin both, and check the provider changelog before bumping a major version. You also need an AWS account you can safely experiment in, with credentials supplied through SSO, an assumed role, or environment variables rather than static keys in code.

Use this layout. The modules/ directory holds reusable code, and envs/ holds one small root configuration per environment:

infra/
  modules/
    vpc/   (main.tf, variables.tf, outputs.tf)
    s3/    (main.tf, variables.tf, outputs.tf)
    iam/   (main.tf, variables.tf, outputs.tf)
  envs/
    dev/   (main.tf, providers.tf, backend.tf)

Watch out for: do not put provider blocks inside child modules. Modules inherit the provider from the caller, and embedded provider configuration makes removal and reuse painful. The HashiCorp guidance on module development covers this and the standard file structure.

Also decide on remote state now. An S3 backend with locking is common. Recent Terraform releases support native S3 lock files via use_lockfile = true, and DynamoDB-based locking is deprecated. The backend block belongs in the root configuration, never in a module.

Step 1: Build the VPC module

Keep the first Terraform modules boring. This VPC module takes a name, a CIDR, and a map of availability zones to fixed subnet indexes, then derives subnets with cidrsubnet. It deliberately omits NAT gateways, which cost money per hour and per GB, so you can add them behind an explicit flag later.

# modules/vpc/main.tf
variable "name" { type = string }

variable "cidr_block" {
  type    = string
  default = "10.0.0.0/16"
}

# AZ name => fixed subnet index, e.g. { "eu-central-1a" = 0 }
variable "azs" { type = map(number) }

resource "aws_vpc" "this" {
  cidr_block           = var.cidr_block
  enable_dns_support   = true
  enable_dns_hostnames = true
  tags                 = { Name = var.name }
}

# keyed by AZ name with a fixed index: adding or removing a zone
# does not renumber or recreate the others
resource "aws_subnet" "private" {
  for_each          = var.azs
  vpc_id            = aws_vpc.this.id
  availability_zone = each.key
  cidr_block        = cidrsubnet(var.cidr_block, 8, each.value)
  tags              = { Name = "${var.name}-private-${each.key}" }
}

resource "aws_subnet" "public" {
  for_each                = var.azs
  vpc_id                  = aws_vpc.this.id
  availability_zone       = each.key
  cidr_block              = cidrsubnet(var.cidr_block, 8, each.value + 100) # offset avoids overlap
  map_public_ip_on_launch = false
  tags                    = { Name = "${var.name}-public-${each.key}" }
}

output "vpc_id"             { value = aws_vpc.this.id }
output "private_subnet_ids" { value = [for s in aws_subnet.private : s.id] }

Gotcha: using count, or for_each with an index derived from list position, ties each subnet's CIDR to its position in the list. Removing the first zone shifts the indexes, changes the computed CIDRs, and proposes replacing real subnets. Keying for_each on the AZ name and giving each zone a fixed index avoids that. Add an internet gateway and route tables when you actually need public routing.

Step 2: Build the S3 module with safe defaults

Since the AWS provider v4 split, bucket settings live in separate resources rather than inline arguments. The module should make the secure choice the default: block public access, enable versioning, and set encryption explicitly even though S3 now encrypts new objects with SSE-S3 by default. Explicit config documents intent and lets you switch to KMS.

# modules/s3/main.tf
variable "bucket_name" { type = string }

variable "force_destroy" {
  type    = bool
  default = false
}

variable "kms_key_arn" {
  description = "KMS key ARN; null means SSE-S3"
  type        = string
  default     = null
}

resource "aws_s3_bucket" "this" {
  bucket        = var.bucket_name
  force_destroy = var.force_destroy
}

resource "aws_s3_bucket_public_access_block" "this" {
  bucket                  = aws_s3_bucket.this.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_versioning" "this" {
  bucket = aws_s3_bucket.this.id
  versioning_configuration { status = "Enabled" }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
  bucket = aws_s3_bucket.this.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = var.kms_key_arn == null ? "AES256" : "aws:kms"
      kms_master_key_id = var.kms_key_arn
    }
  }
}

output "bucket_arn" { value = aws_s3_bucket.this.arn }
output "bucket_id"  { value = aws_s3_bucket.this.id }

Watch out for: bucket names are global across all AWS accounts, so pass a name that includes account or environment context. Leave force_destroy false by default so a careless destroy cannot wipe data. See the AWS documentation on Block Public Access for what each of the four flags does.

Step 3: Build the IAM module around least privilege

The IAM module creates a role that EC2 can assume and attaches a policy scoped to one bucket. It takes the bucket ARN as input rather than reaching into the S3 module, which keeps it reusable for any resource that exposes an ARN. Policy documents built with the data source are easier to review than JSON heredocs.

# modules/iam/main.tf
variable "name"       { type = string }
variable "bucket_arn" { type = string }

data "aws_iam_policy_document" "assume" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

data "aws_iam_policy_document" "bucket_read" {
  statement {
    actions   = ["s3:ListBucket"]
    resources = [var.bucket_arn]          # bucket-level action
  }
  statement {
    actions   = ["s3:GetObject"]
    resources = ["${var.bucket_arn}/*"]   # object-level action needs /*
  }
}

resource "aws_iam_role" "this" {
  name               = var.name
  assume_role_policy = data.aws_iam_policy_document.assume.json
}

resource "aws_iam_role_policy" "bucket_read" {
  name   = "${var.name}-bucket-read"
  role   = aws_iam_role.this.id
  policy = data.aws_iam_policy_document.bucket_read.json
}

output "role_arn" { value = aws_iam_role.this.arn }

A typical failure here is granting s3:GetObject on the bucket ARN alone. The call is denied because object actions require the /* resource form, while ListBucket requires the bare bucket ARN. Both statements above are needed.

Step 4: Compose the modules in a root configuration

The root configuration is where environment decisions live. It passes values in and wires outputs to inputs, and Terraform infers the order from those references.

# envs/dev/main.tf
terraform {
  required_version = ">= 1.6"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 6.0" }
  }
}

provider "aws" { region = "eu-central-1" }

module "vpc" {
  source = "../../modules/vpc"
  name   = "dev"
  azs    = { "eu-central-1a" = 0, "eu-central-1b" = 1 }
}

module "data_bucket" {
  source      = "../../modules/s3"
  bucket_name = "example-dev-data-123456789012" # must be globally unique
}

module "app_role" {
  source     = "../../modules/iam"
  name       = "dev-app"
  bucket_arn = module.data_bucket.bucket_arn
}

output "app_role_arn" {
  value = module.app_role.role_arn
}

Local paths are fine while one repository owns everything. When several teams consume the modules, publish them to a registry or a Git tag and pin a version, so a change to modules/s3 cannot silently alter another team's plan.

Verify and test

Run static checks first, then read the plan critically before any apply. In a sandbox account, apply once and confirm the results with the AWS CLI rather than trusting the plan alone.

terraform fmt -check -recursive      # formatting gate for CI
terraform init
terraform validate                   # catches wrong types and missing arguments
terraform plan -out=tfplan           # review: expect only creates on a first run
terraform apply tfplan

# confirm the real bucket settings
aws s3api get-public-access-block --bucket example-dev-data-123456789012
aws s3api get-bucket-versioning   --bucket example-dev-data-123456789012

# confirm the role can read but not write
aws iam simulate-principal-policy \
  --policy-source-arn "$(terraform output -raw app_role_arn)" \
  --action-names s3:GetObject s3:PutObject \
  --resource-arns "arn:aws:s3:::example-dev-data-123456789012/test.txt"

The app_role_arn output defined in the root configuration supplies the role ARN for this check. The simulation should report s3:GetObject as allowed and s3:PutObject as implicitly denied. Verify the exact response fields against the current AWS CLI reference.

For regression checks, Terraform's native terraform test command can run plan-level assertions against each module. Check the HashiCorp documentation for the syntax in your installed version. Finish with terraform destroy in the sandbox so nothing lingers.

Where to go from here

Three small modules with typed inputs, secure defaults, and explicit wiring in the root are enough to stamp out new environments without copying files. Next, add NAT and routing behind a flag, an S3 bucket policy that enforces TLS, and a CI job that runs the checks above on every pull request. For more infrastructure patterns, browse the rest of kuryzhev.cloud, and keep the module interfaces small, because every extra variable is something you have to support later.

Bash backup script not running from cron: S3 sync fixes

Oleksandr Kuryzhev — Fri, 09 Oct 2026 07:02:21 +0000

Originally published on kuryzhev.cloud

You wrote a bash backup script, ran it by hand, and watched the files land in S3. A week later someone asks for a restore and the newest object in the bucket is nine days old. This is a typical failure: the bash backup script works in your terminal and quietly does nothing, or something different, when cron runs it. This runbook walks from symptom to cause to three fixes.

Symptoms

Match what you see against this list before changing anything.

The script succeeds interactively, but the S3 bucket has no new objects after the scheduled time.
The cron log shows the job started, yet no backup log is written. Where to look depends on the host:
- grep CRON /var/log/syslog on Debian or Ubuntu with rsyslog.
- /var/log/cron on RHEL-family systems.
- journalctl -u cron or journalctl -u crond where only the journal exists.
Errors such as aws: command not found, Unable to locate credentials, or The config profile (...) could not be found appear only in local mail or not at all.
The backup log grows without limit, or the disk fills with archives in /var/backups.
Two backup processes show up in ps at once, and the second one is still running when the next schedule fires.
The log says "done" but the S3 prefix is missing files, or files deleted at the source are still in the bucket.

If the cron daemon never logged the job at all, the problem is the crontab itself. Common causes:

The job is in the wrong user's crontab.
The schedule line has a syntax error.
A file in /etc/cron.d has a dot in its name, is writable by group or other, or lacks a trailing newline.

Check that first with crontab -l as the intended user, or inspect the /etc/cron.d file.

Root cause: cron is not your shell

Cron runs jobs with a minimal environment. The common default on Linux cron implementations is:

a short PATH (often /usr/bin:/bin);
SHELL=/bin/sh;
no TTY;
none of the exports from your ~/.bashrc or profile.

Anything your interactive session provided silently is missing. That includes a pip-installed or /usr/local/bin AWS CLI, AWS_PROFILE, SSO sessions, relative paths, and even the working directory.

Three further causes sit on top of that, and they explain the "ran but wrong" symptoms:

Swallowed exit codes. Without set -euo pipefail, a failed tar in a pipeline still lets the script continue and print "backup complete".
No overlap protection. A slow upload runs into the next schedule, and two jobs fight over the same archive or the same log.
Misread sync semantics. aws s3 sync compares size and modification time. It does not delete destination objects unless you pass --delete. It also returns exit code 2 when some files were skipped (for example, unreadable files), which a naive script treats as success or ignores.

Reproduce the cron environment before debugging anything else:

env -i HOME="$HOME" PATH=/usr/bin:/bin /bin/sh -c '/opt/backup/backup.sh'

Run it as the cron user. If it fails there, you have found the class of bug.

Fix #1: Give the bash backup script an explicit environment

Stop depending on anything cron does not provide:

Set PATH at the top of the script and use absolute paths.
Prefer an instance profile or IAM role over static keys.
If you must use a named profile, set AWS_PROFILE, and set AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE if the files are not in the cron user's home directory.
Do not rely on IAM Identity Center (SSO) sessions for unattended jobs. They expire and cannot be refreshed without a browser login.

See the AWS CLI credential options in the official authentication documentation.

Crontab entry with the environment declared and output captured:

# /etc/cron.d/backup  (system crontabs need a user field)
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=ops@example.com

# 02:15 daily; note the escaped percent signs if you use date formats here
15 2 * * * backup /usr/bin/flock -n /var/lock/backup.lock /opt/backup/backup.sh >> /var/log/backup/backup.log 2>&1

Watch out for the percent sign. In a crontab, an unescaped % is treated as a newline, so $(date +%F) directly in the cron line truncates the command. Put date logic inside the script, or write \%.

Also avoid cd-less relative paths. A script that does tar czf backup.tgz data/ resolves paths against cron's working directory. That is typically the cron user's home directory, but it is implementation-dependent, so cd explicitly or use absolute paths.

Fix #2: Fail loudly, lock, and clean up

Make every failure visible and stop overlapping runs. The flock -n wrapper in the cron line skips a run if the previous one still holds the lock. Inside the script, strict mode and a trap make failures non-silent and remove temporary archives.

A minimal skeleton that propagates errors and records the exit state:

#!/usr/bin/env bash
set -Eeuo pipefail                      # exit on error, unset vars, pipe failures
export PATH=/usr/local/bin:/usr/bin:/bin

SRC=/srv/app/data
BUCKET=s3://example-backups/app         # replace with your bucket/prefix
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
TMP=$(mktemp -d /var/tmp/backup.XXXXXX)

log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "$*"; }
cleanup() { rm -rf "$TMP"; }
trap cleanup EXIT                       # always remove the temp dir
trap 'log "FAILED at line $LINENO"' ERR

log "start $STAMP"
tar -C "$SRC" -czf "$TMP/app-$STAMP.tgz" .
aws s3 cp "$TMP/app-$STAMP.tgz" "$BUCKET/archives/app-$STAMP.tgz" --only-show-errors
log "done"

Watch out for set -e inside conditionals and || or && chains. It is deliberately disabled there, so a failure inside an if test is not fatal. That also applies inside functions called from those contexts. Check critical commands explicitly when they sit there.

Uploading a single archive with aws s3 cp gives you one clear exit status. That is easier to alert on than a directory sync when the goal is a point-in-time backup.

Fix #3: Correct S3 sync behavior and rotate the logs

If you do mirror a directory, decide deliberately what deletions mean:

Without --delete, aws s3 sync keeps removed files in the bucket forever.
With --delete, a mistakenly empty or unmounted source can wipe the destination prefix.

Guard against that and handle exit code 2:

# Refuse to sync from an unmounted or empty source
mountpoint -q /srv/app || { log "source not mounted, aborting"; exit 1; }   # if SRC lives on its own mount
[ -n "$(ls -A "$SRC")" ] || { log "source empty, aborting"; exit 1; }

rc=0
aws s3 sync "$SRC" "$BUCKET/mirror/" --delete --only-show-errors || rc=$?
# rc 1 = transfer failed, rc 2 = some files skipped (e.g. permission denied)
[ "$rc" -eq 0 ] || { log "sync exit code $rc"; exit "$rc"; }

The || rc=$? form captures the status without tripping set -e or the ERR trap.

For history, enable S3 versioning and a lifecycle rule on the bucket rather than relying on script-side retention. Versioning alone does not stop someone holding credentials that can delete object versions. For ransomware protection, add S3 Object Lock as well. Flag details can change between CLI releases, so verify with the aws s3 sync reference.

Then stop the log from growing. A short job that appends with >> works fine with a standard logrotate rule:

# /etc/logrotate.d/backup
/var/log/backup/backup.log {
    su backup backup
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 backup backup
}

The su line is needed because logrotate refuses to rotate in a directory writable by a non-root user.

If the writer is a long-running process that keeps the file descriptor open, add copytruncate. Otherwise it keeps writing to the renamed file.

Prevention

Backups fail silently by default, so build the checks into the process rather than trusting memory.

Test under cron conditions. Run the script with env -i after every change, and once as the actual cron user.
Alert on absence, not just errors. At the end of a successful run, send a ping to a dead-man's-switch service or write a success timestamp metric. A missing ping is the alarm, which also covers the case where cron never started.
Restore on a schedule. At least monthly, download the latest object and verify it with tar -tzf or a checksum. A backup never restored is an assumption.
Use least-privilege IAM. Scope the role to what each job needs:
- An archive-only job needs s3:PutObject on its prefix, plus s3:ListBucket on the bucket restricted with an s3:prefix condition.
- A mirror job using --delete also needs s3:DeleteObject, scoped to the mirror prefix only.
- With versioning on, deny s3:DeleteObjectVersion so a buggy run only creates delete markers.
Consider systemd timers. On systemd hosts, a timer with Persistent=true runs missed jobs after downtime. Its logs land in the journal with exit status.
Keep scripts in version control and lint them with ShellCheck in CI.

For more Bash and infrastructure runbooks, browse the rest of kuryzhev.cloud.

CI/CD Quality Gate Mistakes: Approvals and Rollback Pitfalls

Oleksandr Kuryzhev — Thu, 08 Oct 2026 07:02:36 +0000

Originally published on kuryzhev.cloud

A team adds a CI/CD quality gate after a bad release, bolts on a manual approval step for production, and writes "rollback: redeploy the previous version" in the runbook. Everyone feels safer. The pipeline is green, the approval button exists, and the rollback sentence is documented. This retrospective looks at how those three controls typically fail anyway. It draws on documented platform behavior and common design patterns rather than any specific incident.

Context: three controls, three different jobs

The three controls answer different questions, and mixing them up is where trouble starts. A quality gate answers "is this artifact technically acceptable?" It is automated, repeatable and should give the same verdict twice. A manual approval answers "is it acceptable to release this now?" That is a human judgment about timing, risk and business context. A rollback answers "how do we return to a known-good state when the first two were wrong?"

Each control is cheap to add as a checkbox and expensive to make meaningful. A gate that never fails, an approval nobody reads and a rollback nobody has run all look like working controls until the day they are needed. A common pattern is that the control exists in configuration but does not constrain anything.

The sections below cover the three most common ways these controls degrade, followed by a pattern that holds up better. The examples use GitHub Actions and Kubernetes because their behavior is well documented. The same ideas apply to GitLab CI, Jenkins or Argo CD.

Common failure 1: the gate that can be skipped, retried or ignored

The most frequent weakness is a CI/CD quality gate that is not actually required. A test job runs and fails, but the deploy job still starts. The cause is usually continue-on-error: true, a misconfigured needs, or a deploy workflow triggered separately from the test workflow. The dashboard shows red, but nothing stopped the release.

A second variant is the gate that measures the wrong thing. Coverage thresholds computed over the whole repository barely move when a risky change lands, so the gate passes. A better approach is to gate on new or changed code, plus a small set of meaningful checks: unit tests, a vulnerability scan with an explicit severity threshold, and a policy check on infrastructure changes. That is more honest than one large aggregate number.

A third variant is flakiness. When a test fails one run in ten, people rerun until green, and the gate trains the team to treat red as noise. Quarantine flaky tests visibly and track them, rather than letting reruns become the workflow.

Watch out for path filters on required checks. In GitHub, a workflow skipped because of path or branch filtering leaves its required check in a "Pending" state, which blocks the merge. Teams often "fix" this by removing the requirement. Check the current behavior in the GitHub Docs on required status checks, and design a lightweight always-run job instead.

Also verify that the gate runs against the exact commit being deployed. A gate on the pull request branch says little about the merge result on main if main has moved.

Common failure 2: the manual approval that approves nothing

Manual approval is usually added as a reaction to risk, and then it drifts into ceremony. The approver gets a notification saying "Deployment to production is waiting." It has no diff, no test summary, no link to the change and no indication of what differs from what is running now. Clicking "approve" becomes the path of least resistance, and approval fatigue sets in over time.

The deeper problem is approving a pipeline rather than an artifact. The approval may happen first with the build running afterward, or the deploy job may rebuild from source. Either way, the thing that was reviewed is not the thing that ships. A moving base image tag or an unpinned dependency can change the result between approval and deploy.

Watch out for self-approval and long waits. If the person who triggered the deployment can also approve it, the control is decorative. GitHub environments offer a "prevent self-review" option, and required reviewers are configured per environment. Waiting approvals also time out, and the exact limit is platform-dependent. Verify it in the docs before relying on a deploy that waits over a weekend.

Approvals also tend to be applied uniformly. Gating a documentation change and a database schema change identically teaches people that the gate carries no information. Reserve human approval for changes that are genuinely hard to reverse or time-sensitive. Let low-risk changes flow through the automated gate alone.

Common failure 3: rollback that was never exercised

"Roll back" often means "redeploy the previous commit." That is a rebuild, not a rollback. It depends on the build system working, registries being reachable and dependencies resolving the same way. It also depends on the quality gate (or an emergency bypass of it) running under pressure. A true rollback redeploys the previous immutable artifact, ideally by digest, without rebuilding anything.

Even then, rollback only reverts what the tool tracks. A Kubernetes Deployment rollback restores the previous pod template from the revision history. It does not undo changes to ConfigMaps, Secrets, CRDs or external resources. The retained history is bounded by revisionHistoryLimit, so check the value your manifests actually set. The Kubernetes Deployment documentation describes what a rollback does and does not revert.

The hardest case is state. Suppose release N+1 ran a destructive migration, such as dropping a column or rewriting data. Rolling the application back to N may then leave it running against a schema it does not understand. This is often discovered only during an attempted rollback.

Watch out for rollback paths that need the thing that just broke. The rollback may require the same pipeline, the same approver and the same registry. If so, a single outage can block both the release and the recovery. Rehearse the rollback in a non-production environment on a schedule, so the first real attempt is not the first attempt.

Safer operating pattern: build once, gate the artifact, verify after deploy

A more reliable CI/CD quality gate design follows a few principles:

Gate the exact commit that produces the artifact, and optionally scan the built image by digest.
Build the artifact once and identify it by digest.
Attach the approval to the environment, not to a workflow step.
Verify health after deploy, and undo automatically when verification fails.

The sketch below shows the shape in GitHub Actions; adapt names and secrets to your setup.

This workflow gates the commit, builds one image from that tree, deploys that exact digest to a protected environment, and reverts if rollout verification fails:

name: release
on:
  push:
    branches: [main]

jobs:
  build-and-gate:
    runs-on: ubuntu-24.04
    outputs:
      digest: ${{ steps.push.outputs.digest }}   # immutable reference reused downstream
    steps:
      - uses: actions/checkout@v5                # pin to a commit SHA in real pipelines
      - name: Unit tests and policy checks
        run: make test lint                      # no continue-on-error: failure stops the run
      - name: Set up Buildx
        uses: docker/setup-buildx-action@v3
      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: registry.example.com
          username: ${{ secrets.REGISTRY_USERNAME }}   # placeholder secret names
          password: ${{ secrets.REGISTRY_TOKEN }}
      - name: Build and push image
        id: push
        uses: docker/build-push-action@v6
        with:
          context: .                             # build the tree that was just tested
          push: true
          tags: registry.example.com/app:${{ github.sha }}

  deploy-prod:
    needs: build-and-gate
    runs-on: ubuntu-24.04
    environment: production                      # required reviewers + prevent self-review live here
    steps:
      - name: Configure cluster access
        env:
          KUBECONFIG_DATA: ${{ secrets.KUBECONFIG_PROD }}  # placeholder; prefer OIDC short-lived creds
        run: |
          mkdir -p ~/.kube
          printf '%s' "$KUBECONFIG_DATA" > ~/.kube/config
      - name: Deploy by digest
        run: |
          kubectl set image deployment/app \
            app=registry.example.com/app@${{ needs.build-and-gate.outputs.digest }}
      - name: Verify rollout
        id: verify
        run: kubectl rollout status deployment/app --timeout=180s
      - name: Roll back on failed verification
        if: failure() && steps.verify.outcome == 'failure'   # only undo a rollout that actually started
        run: kubectl rollout undo deployment/app

Environment protection rules, including required reviewers, are configured in repository settings rather than in the YAML. See the GitHub Docs on deployment environments for current options and plan limitations.

The rollback condition is deliberately narrow. If the deploy step itself fails, no new revision exists, and an unconditional rollout undo would revert to an even older release. Note also that rollout status only proves pods became ready, so add a real smoke test or metric check for anything user-facing.

The following checklist captures the decisions that matter before calling the pipeline production-ready:

GATE
[ ] Required check on the protected branch (not just "runs")
[ ] Always-run job so path filters cannot leave it pending
[ ] Thresholds apply to changed code; flaky tests quarantined, not retried

APPROVAL
[ ] Attached to the environment, shown with diff + test summary
[ ] Approver is not the deployer; timeout behavior known
[ ] Approves a digest, never a rebuild

ROLLBACK
[ ] Previous artifact still in registry (retention checked)
[ ] Migrations follow expand/contract; N-1 code works with N schema
[ ] Rollback rehearsed in staging; works without bypassing the gate
[ ] Post-deploy verification triggers it automatically

The migration line deserves emphasis. Expand/contract means three steps:

Add the new schema first.
Ship code that tolerates both shapes.
Remove the old shape in a later release.

That keeps the previous application version valid. It makes rollback a routine operation instead of a data-recovery project.

More pipeline design notes are available on the DevOps_DayS home page. None of this removes the need for judgment. It only ensures that the gate, the approval and the rollback each do the one job they were added for.

DataSync EventBridge Automation Checklist for S3 Transfers

Oleksandr Kuryzhev — Wed, 07 Oct 2026 07:02:34 +0000

Originally published on kuryzhev.cloud

A nightly copy from an on-premises NFS share into S3 works fine for weeks, then stops. Nobody notices until a downstream report is empty, because the task failed quietly and no alarm was wired to it. DataSync EventBridge automation exists to close that gap. It turns transfer state changes into events, and events into schedules, alerts, and follow-up jobs.

This checklist is for engineers who already have a DataSync task running manually and want it to run unattended. It assumes S3 as the destination. Details such as event field names and quotas change over time, so verify them against the DataSync user guide before you rely on them.

Why this checklist

Many unattended DataSync failures are not service problems. They come from missing glue: nothing starts the task on time, nothing tells anyone when it fails, and nothing reacts when the data lands. Each gap is small, and each is easy to close once you know where to look.

A checklist suits this topic better than a tutorial because the pieces are independent. You can set up scheduling, eventing, permissions, and cost controls in any order, and skipping one rarely causes an error on day one. The failure shows up later, usually on a night when something unusual happens. A source mount might go offline, or a task execution might overlap with the previous one.

The pattern covered here has three parts:

Trigger: EventBridge Scheduler starts the task, or an upstream event does.
Observe: DataSync task execution state changes flow to EventBridge.
React: S3 object events or a successful execution kick off processing, and failures page someone.

Even the observe step alone removes a significant blind spot.

The checklist

Decide who owns the schedule. DataSync has a built-in task schedule, and EventBridge Scheduler can call StartTaskExecution directly. Pick one. Running both produces overlapping executions that are hard to explain later. Scheduler gives you time zones and flexible windows, so it is often the better choice.
Create a dedicated Scheduler role. It needs datasync:StartTaskExecution on the specific task ARN and nothing broader. If you configure a dead-letter queue, it also needs sqs:SendMessage on that queue. Its trust policy must allow scheduler.amazonaws.com.
Create the schedule with the universal target. The example below uses the AWS SDK target, so you do not need a Lambda function just to start a task.
Enable EventBridge delivery on the destination bucket. S3 sends events to EventBridge only when the bucket is configured for it.
Write a failure rule first. Match DataSync task execution state changes that end in an error. Route them to SNS, a chat webhook, or an incident tool.
Write a success rule second. Use it to trigger post-processing, such as a validation job, a catalog update, or a manifest write.
Scope S3 object rules by prefix. Filter on bucket name and key prefix so unrelated uploads do not trigger your pipeline.
Turn on CloudWatch Logs for the task. Set the log level deliberately. Basic is usually enough for routine runs, while transfer-level detail costs more to store.
Enable task reports to an S3 location if you need per-file evidence of what was transferred, skipped, or failed.
Handle target failures explicitly. A dead-letter queue and retry policy on a rule or schedule target only catches events that could not be delivered. Throttling and missing permissions are typical causes. Errors inside your own code are a separate case. For Lambda, which EventBridge invokes asynchronously, configure an on-failure destination or a function-level dead-letter queue as well. A target that fails silently is the same blind spot you started with.

First, the failure rule. This event pattern matches DataSync task execution errors. Verify the exact detail-type and field names against the current EventBridge event reference for DataSync.

{
  "source": ["aws.datasync"],
  "detail-type": ["DataSync Task Execution State Change"],
  "detail": {
    "State": ["ERROR"]
  }
}

Next, the schedule. This CLI call starts a task nightly in a named time zone. It uses the SDK target, so no compute is involved. It also sends failed start attempts to an SQS dead-letter queue, for example when the call is rejected or throttled.

aws scheduler create-schedule \
  --name nightly-datasync-nfs-to-s3 \
  --schedule-expression "cron(0 2 * * ? *)" \
  --schedule-expression-timezone "Europe/Berlin" \
  --flexible-time-window Mode=OFF \
  --target '{
    "Arn": "arn:aws:scheduler:::aws-sdk:datasync:startTaskExecution",
    "RoleArn": "arn:aws:iam::111122223333:role/scheduler-datasync-start",
    "Input": "{\"TaskArn\":\"arn:aws:datasync:eu-central-1:111122223333:task/task-0example\"}",
    "RetryPolicy": {"MaximumRetryAttempts": 2, "MaximumEventAgeInSeconds": 3600},
    "DeadLetterConfig": {"Arn": "arn:aws:sqs:eu-central-1:111122223333:scheduler-datasync-dlq"}
  }'
# Input is a JSON string inside JSON, so the inner quotes must be escaped

Commonly missed items

These are documented behaviors and typical failure modes that tend to surface after the pipeline has been live for a while.

Watch out for bucket notification overwrites. The S3 API call that enables EventBridge delivery replaces the bucket's whole notification configuration. If the bucket already has Lambda or SQS notifications, sending only {"EventBridgeConfiguration":{}} removes them. Read the current configuration first and merge. Infrastructure-as-code tools can also fight each other over this one setting.

# Read existing config before changing anything
aws s3api get-bucket-notification-configuration --bucket landing-bucket

# Then write back the merged result, including the EventBridge key
aws s3api put-bucket-notification-configuration \
  --bucket landing-bucket \
  --notification-configuration file://merged-notifications.json

Watch out for event loops. If a rule triggers a job that writes back into the same bucket and prefix, each write emits another object event. Separate the input and output prefixes, or use a different bucket, and make the rule's prefix filter strict.

Other items that regularly get skipped:

Success does not mean complete data. A task can finish while skipping files, depending on its filters and options. Check the verification settings. Where it matters, compare counts from the task report.
Overlapping executions. A task generally runs one execution at a time. What happens when a second start is requested depends on current service rules. Check the documented quotas and queuing behavior rather than assuming.
Service-created objects. DataSync can write helper or metadata objects to destinations. Look at what actually lands in your bucket and exclude those objects from your S3 rules.
Storage class choice. Writing straight into an archival class carries minimum storage duration charges. If the data is reprocessed soon, that can cost more than a standard class plus a lifecycle rule.
Cross-account and KMS. A bucket in another account, or one encrypted with a customer managed key, needs grants for the DataSync location role. Grant access in both the bucket policy and the key policy.
Region mismatch. DataSync and S3 events are delivered to the default event bus in the Region where they occur. The rule that matches them must live in that Region. From there, you can forward events to a bus in another Region if your alerting is centralized.

Automation ideas

Once the basics are stable, a few small additions add value without growing the architecture.

Chain stages with events, not timers. Do not schedule a processing job at 03:00 and hope the transfer finished. Instead, match the success state change and start the next stage from that. This removes the guessing and the fragile time gaps. A Step Functions state machine works well as the target when the follow-up has several steps.

Use input overrides for ad-hoc runs. StartTaskExecution accepts include and exclude filters and option overrides. A second schedule can transfer only a recent prefix, such as the current day's folder, while the nightly run does the full pass. Stagger the two so they do not overlap, because both start executions of the same task. Verify the filter syntax in the DataSync documentation, since patterns are pipe-delimited and easy to get subtly wrong.

Build a freshness check. An event-driven alarm tells you about failures but not about a task that never started. Have the success rule's target publish a custom CloudWatch metric on each successful execution. Then alarm on that metric over your expected interval, with missing data treated as breaching. This catches disabled schedules, broken Scheduler permissions, and offline agents.

Define everything in code. Keep the task, schedule, role, rules, and bucket notification in one Terraform module or CloudFormation stack, so the notification overwrite problem has a single owner. For more AWS automation patterns, browse the rest of kuryzhev.cloud.

Tag and route by environment. Use rule input transformers so one alerting target can name the failed task and execution in its message. In DataSync execution events, the execution ARN appears in the event's resources array rather than in detail, so map it from there. Responders then see which job failed without opening the console. The EventBridge user guide covers input transformers, retry policies, and dead-letter queues in detail.

Start with the failure rule and the freshness check. Together they cover both a failed run and a run that never started, which are two of the easiest ways for this pipeline to fail unnoticed.

Cloudflare Tunnel Shows 502 or Inactive? Troubleshooting Fixes

Oleksandr Kuryzhev — Tue, 06 Oct 2026 07:02:39 +0000

Originally published on kuryzhev.cloud

The change window is open, the internal dashboard is supposed to be reachable at a public hostname with no inbound firewall rules, and the Cloudflare Tunnel you created an hour ago shows "Inactive" or returns a 502 to every browser. Nothing is listening on a public port, so there is no obvious place to look. This runbook covers the failure modes of Cloudflare Tunnel (formerly Argo Tunnel, still run by the cloudflared daemon) in the order they are usually diagnosed.

Symptoms

Match what you see to the list below before changing anything. Each symptom points at a different layer.

Error 1033 in the browser – Cloudflare resolved the hostname to a tunnel, but no cloudflared connector is attached to it.
Error 1016 (origin DNS error) – the DNS record points at a tunnel UUID that was deleted or never existed.
502 Bad Gateway – the tunnel is connected, but cloudflared could not get a valid response from the local service.
404 with an empty body – the request reached cloudflared but fell through to the catch-all ingress rule.
Tunnel flaps between Healthy and Degraded – connections to the edge drop and reconnect, often with timeout messages in the logs.

Start with the connector's own view of the world. The list and info subcommands need the account certificate (cert.pem) created by cloudflared tunnel login. On hosts that only have a tunnel token, check the tunnel's status and connectors in the dashboard instead.

# List tunnels and see which have active connections (requires cert.pem)
cloudflared tunnel list

# Show connector IDs, edge locations and origin IPs for one tunnel
cloudflared tunnel info my-internal-apps

# Read recent logs (systemd install)
journalctl -u cloudflared --since "30 min ago" --no-pager

If info shows no connectors, you have a connector problem. If it shows connectors but users still see errors, the problem is routing or the origin.

Root cause

A Cloudflare Tunnel has three independent links, and each can fail while the other two look fine.

Edge link. The outbound connection from cloudflared to Cloudflare's edge.
Hostname mapping. The mapping from a public hostname to a service. It lives either in the dashboard (remotely managed tunnels) or in a local config.yml (locally managed tunnels).
Origin link. The connection from cloudflared to the actual origin service.

Many failures trace back to one of these documented behaviors:

cloudflared only makes outbound connections. If egress to the edge is blocked (the documented port is 7844, TCP and UDP), it never registers and you get 1033.
localhost means "this process's own network namespace". Inside a container, it is not the host and not the neighboring container.
Ingress rules are evaluated top to bottom, and the last rule must be a catch-all. A hostname typo sends traffic to that fallback.
A service URL with https:// and a self-signed or mismatched certificate fails TLS verification and surfaces as 502.

Watch out for: mixing management modes. If a tunnel was created in the dashboard (token-based), the dashboard configuration is authoritative and a local config.yml ingress section is ignored. Editing the wrong one is a common reason a "fix" changes nothing. Verify the current behavior in the official Cloudflare Tunnel documentation.

Fix #1: Restore the connector (1033 and Inactive status)

First confirm the process is running and can reach the edge. A connector that never registers is usually a firewall, proxy, or stale-credential issue.

# Is the service up?
systemctl status cloudflared

# Can this host reach the edge on the tunnel port over TCP?
# (hostnames per Cloudflare's firewall docs; verify the current list)
# Note: this does not prove that UDP 7844 (QUIC) is open.
nc -vz region1.v2.argotunnel.com 7844
nc -vz region2.v2.argotunnel.com 7844

# Force HTTP/2 over TCP to rule out filtered UDP (QUIC)
# Locally managed tunnel:
cloudflared tunnel --protocol http2 run my-internal-apps
# Token-based (remotely managed) tunnel:
cloudflared tunnel --protocol http2 run --token <TOKEN>

The default protocol setting is auto. It tries QUIC, which uses UDP, first and can fall back to HTTP/2 over TCP. Some corporate firewalls and cloud security groups allow outbound TCP only. In those networks the connector may fail or reconnect repeatedly, so forcing http2 is a quick way to confirm that diagnosis. If it works, choose one of two permanent fixes:

Open outbound UDP 7844.
Set the protocol in your configuration, using protocol: http2 in config.yml or the --protocol flag in the service definition.

Other connector-level causes to rule out:

An expired or rotated token in the service unit. Run cloudflared service uninstall, then reinstall with cloudflared service install <TOKEN> using a fresh token from the dashboard.
A very old cloudflared binary. Cloudflare supports only recent releases, so update to the current one and check the release notes for your version.
Clock drift on the host, which can break TLS to the edge. Check NTP.

If 1016 appears instead, the DNS CNAME points at a deleted tunnel. For a remotely managed tunnel, re-create the public hostname in the dashboard. For a locally managed tunnel, run cloudflared tunnel route dns --overwrite-dns my-internal-apps app.example.com so the record targets the live tunnel ID.

Fix #2: Correct the ingress and the origin address (502 and unexpected 404)

When connectors are healthy, the next suspect is the service URL. For a locally managed tunnel, validate the file before restarting anything. The example below shows the pieces that most often go wrong.

# /etc/cloudflared/config.yml (locally managed tunnel)
tunnel: 6ff42ae2-0000-0000-0000-example     # tunnel UUID, placeholder
credentials-file: /etc/cloudflared/6ff42ae2-0000-0000-0000-example.json

ingress:
  - hostname: app.example.com
    service: http://localhost:8080          # plain HTTP to a local process
  - hostname: grafana.example.com
    service: https://localhost:3000
    originRequest:
      noTLSVerify: true                     # only for self-signed certs you accept; prefer caPool
  - hostname: ssh.example.com
    service: ssh://localhost:22
  - service: http_status:404                # mandatory catch-all, must be last

Then test how a specific URL would be routed, without sending real traffic:

cloudflared tunnel ingress validate
cloudflared tunnel ingress rule https://app.example.com

The second command prints which rule matches. If it prints the catch-all, the hostname in the rule does not match what the user typed. Fix the spelling or add a wildcard rule.

For 502 specifically, test from the same host (or container) running cloudflared: curl -v http://localhost:8080. If that fails, the problem is the application, not the tunnel. If you use HTTPS to the origin, set originServerName to match the certificate's name, or supply a CA with caPool rather than disabling verification permanently.

Watch out for: noTLSVerify: true silences the error but removes protection on the hop between cloudflared and the origin. Treat it as a temporary diagnostic, not a final setting.

Fix #3: Repair container networking and unstable connections

In Docker or Kubernetes, localhost in an ingress rule is a very common 502 cause. Point the service at the container or Kubernetes Service name instead, and keep both on the same network.

# compose.yaml (no top-level "version" key needed in current Compose)
services:
  cloudflared:
    image: cloudflare/cloudflared:${CLOUDFLARED_VERSION:-latest}   # pin CLOUDFLARED_VERSION in .env
    command: tunnel --no-autoupdate run      # token comes from the environment
    environment:
      TUNNEL_TOKEN: ${TUNNEL_TOKEN}          # keep in .env or a secret store, never in git
    restart: unless-stopped
    networks: [internal]
  app:
    image: ghcr.io/example/app:1.4.2         # placeholder application image
    networks: [internal]
networks:
  internal: {}

With a token-based tunnel, set the public hostname's service in the dashboard to http://app:8080. In Kubernetes, use http://my-svc.my-namespace.svc.cluster.local:8080. Set CLOUDFLARED_VERSION in .env to a specific current release tag from the official registry. The latest fallback only exists so the file runs as written, so do not rely on it in production.

For flapping connections, check three things:

Replicas. Run at least two replicas of cloudflared so a single restart does not take the route down.
Resource limits. Look for limits that cause OOM kills on small containers.
Metrics and readiness. Inspect the connector metrics endpoint (the --metrics flag sets the address) and its /ready path, which reports whether the connector currently has edge connections.

Prevention

Most of these failures are preventable with a few habits. Put them in your tunnel template once and reuse it across environments, the same way you would standardize other infrastructure patterns covered on kuryzhev.cloud.

Pick one management mode per tunnel. Prefer remotely managed tunnels with Terraform or the API so configuration is reviewable, and avoid editing both places.
Run two or more connectors on separate hosts or nodes for every production tunnel, and roll updates one at a time.
Allow egress explicitly. Document outbound port 7844 (TCP and UDP) in your firewall or security-group baseline, and decide on protocol deliberately.
Pin and update cloudflared. Use a pinned image tag, and review the release notes on a schedule so you never fall outside the supported window.
Alert on tunnel health. Use Cloudflare's tunnel health notifications, plus a probe of the public hostname, so you hear about 1033 before users do.
Protect the origin hop and the hostname. Put Cloudflare Access in front of internal apps, and keep TLS verification on between cloudflared and the origin.
Treat tokens as secrets. Store them in a secret manager, rotate them when staff change, and never bake them into images.

Before closing the ticket, run through a final checklist:

tunnel info (or the dashboard) shows connectors.
ingress rule matches the intended service on locally managed tunnels.
A local curl to the origin succeeds.
The public hostname returns the expected status.

Details can change between releases, so confirm flags and error-code meanings against the Cloudflare Tunnel documentation for your cloudflared version.

Cisco Config Backups with Ansible: ios_config vs ios_command

Oleksandr Kuryzhev — Mon, 05 Oct 2026 07:02:34 +0000

Originally published on kuryzhev.cloud

Cisco config backups usually start the same way: someone replaces a failed access switch, opens a ticket for the last known-good configuration, and finds a TFTP folder that stopped updating a year ago. Ansible can fix that in an afternoon, but the first design decision is how the configuration gets from the device to disk. There are two common approaches, and they behave differently once you add Git, compliance reviews, or a few hundred switches.

When Cisco config backups need a deliberate choice

For a dozen switches and a nightly job, either approach works and the choice barely matters. It starts to matter when you want meaningful diffs, when the backup files contain secrets, or when auditors ask who changed what and when.

Three questions decide it:

What do you do with the file afterward? Archiving timestamped copies is a different job from feeding a Git repository that alerts on drift.
How noisy is the raw output? IOS and IOS XE prepend lines such as the current-configuration byte count and the last-change and NVRAM-update timestamps. These change whenever the configuration is edited or saved. Some platforms also include lines such as ntp clock-period that drift on their own, so diffs can show changes nobody made intentionally.
Who maintains the playbook? A single module call is easy to hand over. A task chain with text processing needs tests and an owner.

Both options below use the cisco.ios collection over ansible.netcommon.network_cli. Install it with ansible-galaxy collection install cisco.ios and check the collection's documented minimum ansible-core version, since it changes between releases. This shared inventory setup works for both:

# group_vars/cisco_switches.yml
ansible_connection: ansible.netcommon.network_cli
ansible_network_os: cisco.ios.ios
ansible_user: "{{ vault_net_user }}"        # keep credentials in Ansible Vault
ansible_password: "{{ vault_net_pass }}"
ansible_become: true
ansible_become_method: enable               # privilege level 15 via enable
ansible_become_password: "{{ vault_enable_pass }}"

Option A: ios_config with backup: true

The cisco.ios.ios_config module has a documented backup parameter that fetches the running configuration and writes it to the controller. With no lines or src given, it changes nothing on the device. That is the whole playbook:

# backup_native.yml
- name: Back up Cisco IOS running configs
  hosts: cisco_switches
  gather_facts: false            # no Python on the switch, so skip fact gathering
  tasks:
    - name: Save running-config via the module
      cisco.ios.ios_config:
        backup: true
        backup_options:
          dir_path: "/srv/net-backups/{{ inventory_hostname }}"
          filename: "running-config.cfg"   # fixed name suits Git; omit for timestamped files

Pros

Minimal code, maintained with the collection, and the intent is obvious to anyone reading it.
Without backup_options, it writes timestamped files to a backup/ directory next to the playbook, a reasonable archive with no extra work.
Low risk of a bad regex or templating error corrupting a backup.

Cons

You get the configuration as the module returns it. Volatile header lines may still show up in diffs, so verify the actual output on your platform and release.
It covers the running configuration. Startup-config, VLAN databases, or other show output need separate tasks.
Watch out for the file contents. Backups can include hashed enable secrets, type 7 passwords, and SNMP community strings. Restrict directory permissions and treat the repository as sensitive.

Option B: ios_command plus your own file handling

Here you run show running-config through cisco.ios.ios_command and write the result yourself from the controller. It is more work, but you control exactly what reaches disk. For example, you can strip lines that change without an intentional edit so Git only records real changes. Unlike the module's backup option, copy does not create missing parent directories, so the playbook creates the target directory first:

# backup_custom.yml
- name: Back up and normalize Cisco configs
  hosts: cisco_switches
  gather_facts: false
  tasks:
    - name: Ensure backup directory exists on the controller
      ansible.builtin.file:
        path: /srv/net-backups
        state: directory
        mode: "0700"
      delegate_to: localhost
      run_once: true

    - name: Collect running-config
      cisco.ios.ios_command:
        commands: show running-config
      register: run_cfg

    - name: Write normalized file on the controller
      ansible.builtin.copy:
        dest: "/srv/net-backups/{{ inventory_hostname }}.cfg"
        mode: "0600"                         # configs contain secrets
        content: |
          {{ run_cfg.stdout[0] | regex_replace('(?m)^(Building configuration|Current configuration|!\s*Last configuration change|!\s*NVRAM config last updated|ntp clock-period).*$\n?', '') }}
      delegate_to: localhost                 # otherwise copy targets the switch

Stripping ntp clock-period is a judgment call. It is a real configuration line, but the device adjusts it on its own, so keeping it mostly adds noise to the diff.

Pros

Full control over normalization, file mode, naming, and which commands run. Adding show startup-config or show version means one more entry in commands plus a write task that reads the matching stdout index.
Clean, stable output makes Git history and drift alerts far more useful.
The same pattern extends to other platforms using ansible.netcommon.cli_command or their own collections.

Cons

You own the regex. Header lines differ across IOS, IOS XE, and release trains, so verify against real output before trusting it.
Watch out for delegate_to: localhost. Omit it and copy runs against the device over the network connection, where it may fail or behave unpredictably.
Over-aggressive normalization can hide a real change. Keep the pattern narrow and review it when platforms change.

Decision matrix

Match your situation to the row that hurts most. Neither option is wrong; they trade simplicity for control.

Situation	Option A: ios_config backup	Option B: ios_command + copy
Small estate, archive only	Best fit	Overkill
Git history with drift alerts	Works if output is already stable	Better: normalize before commit
Need startup-config or extra show output	Needs extra tasks anyway	Natural fit
Mixed-skill team	Easier to hand over	Needs an owner and tests
Strict file permissions and naming rules	Limited to module options	Full control
Multi-vendor estate	Similar backup options exist in other vendors' *_config modules, configured per platform	One pattern reusable across platforms

A quick checklist to run before you commit to either design:

# pre-flight checklist
[ ] Backup target directory is outside the playbook repo, or encrypted
[ ] Files are mode 0600 and owned by the automation user
[ ] One sample switch per platform family diffed across two runs with no changes
[ ] Volatile header lines confirmed absent (or stripped) in the second run
[ ] Credentials come from Ansible Vault or an external secrets backend
[ ] Job scheduled (AWX/AAP, cron, or CI schedule) with failure notifications

Evidence-based recommendation

Start with Option A. It is documented behavior, a few lines long, and it removes the real problem, which is that nobody is backing up at all. Run it on a schedule, store the output somewhere access-controlled, and confirm you can actually read a restored file.

Move to Option B when a specific problem forces it. The usual triggers are noisy diffs in Git, a requirement to capture more than the running configuration, or a file-handling rule the module cannot express. Do not start there because it feels more flexible. Every regex you write is a maintenance commitment, and a typical failure is a normalization pattern that silently stops matching after an OS upgrade.

A hybrid is also legitimate. Use Option A for the nightly archive and add a separate Option B job that produces normalized files for the Git repository. Keeping the raw archive means that if the normalizer ever hides something, the original is still on disk.

Whichever you pick, check the parameter list for your installed collection version in the official cisco.ios.ios_config documentation, since defaults and backup_options behavior can change between releases. For more automation patterns, browse the rest of kuryzhev.cloud.

DEV Community: Oleksandr Kuryzhev

Terraform Vault Secrets: Keep Them Out of State and Git

The scenario

Prerequisites

Step 1: Configure the provider without hard-coded credentials

Step 2: Write a least-privilege Vault policy

Step 3: Read Terraform Vault secrets as ephemeral values

Step 4: Plan the fallback for unsupported resources

Verify and test

Where this leaves you

Related

Ansible server bootstrap locks you out of new hosts: fix

Failure scenario: an Ansible server bootstrap that locks you out

Why it happens

The fix: split the bootstrap into verified phases

Prevention checklist

Related

Helm Chart Values for Staging vs Production Done Right

What Helm chart values actually do

How people use Helm chart values wrong

The correct approach to Helm chart values

Advanced patterns

Performance notes

Related

ECR Image Scan Gate in GitLab CI Passes Vulnerable Images

Symptoms

Root cause

Fix #1: Wait for a terminal scan status, then fail closed

Fix #2: Gate on explicit severity counts with a default

Fix #3: Pin to the digest and handle scan triggers deliberately

Prevention

Related

Content Workflow Automation Failing in n8n or Bedrock? Fixes

Symptoms

Root cause

Fix #1: Control concurrency and retry throttling

Fix #2: Acknowledge fast and make runs idempotent

Fix #3: Validate output before anything publishes

Prevention

Related

Kubernetes Ingress Manifest or Gateway API: Which to Ship

When this choice matters

Option A: A classic Kubernetes Ingress manifest

Option B: Gateway API with HTTPRoute

Decision matrix

Evidence-based recommendation

Related

Terraform Modules for AWS VPC, IAM, and S3: Structure That Scales

The scenario

Prerequisites

Step 1: Build the VPC module

Step 2: Build the S3 module with safe defaults

Step 3: Build the IAM module around least privilege

Step 4: Compose the modules in a root configuration

Verify and test

Where to go from here

Related

Bash backup script not running from cron: S3 sync fixes

Symptoms

Root cause: cron is not your shell

Fix #1: Give the bash backup script an explicit environment

Fix #2: Fail loudly, lock, and clean up

Fix #3: Correct S3 sync behavior and rotate the logs

Prevention

Related

CI/CD Quality Gate Mistakes: Approvals and Rollback Pitfalls

Context: three controls, three different jobs

Common failure 1: the gate that can be skipped, retried or ignored

Common failure 2: the manual approval that approves nothing

Common failure 3: rollback that was never exercised

Safer operating pattern: build once, gate the artifact, verify after deploy

Related

DataSync EventBridge Automation Checklist for S3 Transfers

Why this checklist

The checklist

Commonly missed items

Automation ideas

Related

Cloudflare Tunnel Shows 502 or Inactive? Troubleshooting Fixes

Symptoms