DEV Community: Amaan Ul Haq Siddiqui

End-to-End Supply Chain Security for a Go Project: TUF on CI, cosign, and SLSA L3

Amaan Ul Haq Siddiqui — Sat, 11 Apr 2026 21:25:52 +0000

Adding cosign sign to a CI pipeline and calling it "signed releases" is a bit like putting a lock on a glass door. The lock works. The glass does not. Signing the image proves a specific digest was signed by a specific identity at a specific time. It says nothing about whether the source commit matches what was built, whether the build environment was clean, or whether someone replaced the release asset after the fact.

I had been going deep on supply chain security for a while - reading through TUF specs, the Sigstore design docs, how Fulcio issues short-lived certificates, how Rekor works as an append-only transparency log. At some point I came across the OpenSSF Best Practices requirements and saw the full picture laid out as a checklist. I could have just signed the image and moved on. Instead I used oci-prometheus-sd-proxy - a project that does OCI Prometheus service discovery - as the thing to actually build it on. I wanted to understand each layer well enough to explain it, not just wire it up. What I ended up with: cosign keyless signing, a CycloneDX SBOM attestation, SLSA L3 build provenance, and TUF metadata distribution via tuf-on-ci published to GitHub Pages. No long-lived keys anywhere in the pipeline.

Why each layer exists

This is the question worth answering properly, because you can absolutely ship just cosign and be in a better position than 95% of projects. So why go further?

HTTPS protects against network-level tampering. It does not protect against a compromised GitHub account pushing a backdoored release, or a build that was modified after it completed, or an asset quietly swapped post-publication.

cosign on a container image proves that a specific digest was signed by a specific OIDC identity with the event recorded in Rekor's transparency log. That is genuinely useful - but it only covers the image. It says nothing about the source ref, the build environment, or the workflow that produced it. Someone could sign a backdoored binary and the cosign verification would pass.

SLSA L3 provenance fills that gap, and it is the layer I found most interesting to wire up. The provenance is generated in a separate, isolated signing job with its own OIDC identity, not in the main build job. That isolation is what makes L3 meaningful: an attacker who compromises the main build job cannot forge L3 provenance because they do not control the signing job's OIDC token. The provenance attests to the exact source ref, the exact workflow, and the exact runner environment. You can look at a SLSA L3 attestation and know that the image you are running came from that commit in that repo via that verified builder.

TUF adds something orthogonal to all of the above - it is about distribution trust, not just signing trust. It adds a role-based metadata layer where clients can verify that a release target was authorised by the project's key holders, that the metadata has not been rolled back to a previous version, and that the metadata is actually fresh. The design difference that matters: TUF survives key compromise in a way that a single cosign keypair does not. If my cosign key leaked tomorrow, every past signature would be under a cloud. With TUF, key rotation is a defined protocol. The damage is bounded and recoverable.

The point of having all four is that verification is fully independent. Verifying the image signature, the build provenance, and the TUF metadata chain are three separate operations against different data sources. Compromising any single one of them is not enough to ship a malicious release undetected. You need to compromise all of them simultaneously - and the transparency logs make doing that silently very hard.

Two repos, one trust boundary

The implementation lives across two repositories:

amaanx86/oci-prometheus-sd-proxy - the application, the build pipeline, the release workflow
amaanx86/oci-prometheus-sd-proxy-tuf-on-ci - TUF metadata, managed by tuf-on-ci, published to GitHub Pages

Keeping them separate was a deliberate trust boundary decision, not just organisation. The app CI can push a signing branch to the TUF repo, but it cannot merge that branch or sign targets.json. That step requires my OIDC identity authenticating via Sigstore in a browser - not the CI system's token. An attacker who steals the app repo's CI tokens hits a hard wall at the TUF signing step. They can push a branch, but they cannot authorise the release. The human is the last gate.

App repo CI (run #24290258383)
  └── builds image (linux/amd64, linux/arm64)
  └── pushes to GHCR
  └── cosign attest --type cyclonedx sbom.cyclonedx.json (SBOM)
  └── cosign sign (image signature, OIDC -> Fulcio cert -> Rekor)
  └── cosign attest --type  release-metadata.json
  └── slsa-github-generator -> SLSA L3 provenance (isolated job)
  └── pushes sign/release-1-4-2-rc-24290258383 branch -> TUF repo

TUF repo (tuf-on-ci) [PR #8]
  └── signing-event.yml detects branch push -> opens signing PR
  └── Maintainer: tuf-on-ci-sign (browser OIDC -> @amaanx86 identity)
  └── PR merged -> online-sign.yml refreshes snapshot + timestamp
  └── publish.yml -> GitHub Pages
  └── test.yml -> smoke-tests TUF client (scheduled)

Users / Policy Engines
  └── cosign verify - checks image signature against Rekor
  └── cosign verify-attestation --type cyclonedx - checks SBOM
  └── cosign verify-attestation --type  - checks release-metadata
  └── slsa-verifier verify-image - checks SLSA L3 provenance
  └── TUF client (python-tuf ngclient) - fetches metadata from GitHub Pages, verifies chain

The build pipeline

The workflow (docker-build-push.yml) triggers on release publication. After pushing the multi-arch image (linux/amd64 and linux/arm64) to GHCR it does five more things.

SBOM generation and attestation

First, Syft generates a CycloneDX SBOM for the pushed image, which gets attached as a cosign attestation:

cosign attest \
  --yes \
  --predicate sbom.cyclonedx.json \
  --type cyclonedx \
  ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd

The SBOM is also uploaded as a release asset (the 80.7 KB *.cyclonedx.json artifact visible on the workflow run).

cosign image signing

cosign sign \
  --yes \
  ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd

No --key flag. cosign uses the GitHub Actions OIDC token to get an ephemeral certificate from Fulcio, signs the digest, and records the operation in Rekor. The private key is generated in memory and never stored. The Rekor entry pins the workflow identity to the specific digest at a specific timestamp.

Note the image is signed by digest, not by tag. Tags are mutable; the digest is what the signature actually covers.

No secret to rotate. No key to leak.

Release-metadata attestation

The workflow generates a release-metadata.json with the image digest, source commit, release tag, and build timestamp, then attaches it as an attestation under a custom predicate type:

cosign attest \
  --yes \
  --predicate release-metadata.json \
  --type https://github.com/amaanx86/oci-prometheus-sd-proxy/release-metadata \
  ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd

Using a project-specific type URI keeps the attestation namespaced and lets cosign verify-attestation --type <uri> fetch exactly this attestation rather than every in-toto statement on the image.

SLSA L3 provenance

uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.0.0
with:
  image: ghcr.io/amaanx86/oci-prometheus-sd-proxy
  digest: ${{ needs.build-and-push.outputs.digest }}

slsa-github-generator runs as a reusable workflow with its own isolated OIDC identity. The provenance attestation is generated and signed there, not in the main build job. L3 specifically requires this isolation - the verified builder identity in the provenance is https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v2.0.0, distinct from the app workflow's identity.

Triggering the TUF signing event

Last step: clone the TUF repo and push a signing branch with a new target file at targets/oci-prometheus-sd-proxy/releases/v1.4.2-rc.json. The branch name embeds the run ID to avoid collisions: sign/release-1-4-2-rc-24290258383 (dots in the version become hyphens, run ID appended). This branch push is what kicks off the TUF side of the pipeline.

The Python tuf library (v5+) is used inline to update metadata/targets.json with the new target entry before committing - the targets metadata version is incremented and signatures cleared, ready for the human signing step.

tuf-on-ci

tuf-on-ci manages the TUF metadata lifecycle entirely within a GitHub repository. All online signing uses GitHub Actions OIDC. All offline signing uses Sigstore OIDC (browser-based). No key files anywhere.

The TUF repo has four workflows. signing-event.yml fires on any sign/** branch push, opens a PR, and annotates it with what needs signing. online-sign.yml runs after a signing PR is merged and refreshes snapshot.json and timestamp.json using the Actions OIDC token. publish.yml deploys everything to GitHub Pages. test.yml runs on a schedule and verifies the full metadata chain with a real TUF client to catch expiry or breakage before any user does.

Signing a release

When signing-event.yml opens PR #8 with title "Signing event: sign/release-1-4-2-rc-24290258383", I run:

tuf-on-ci-sign sign/release-1-4-2-rc-24290258383

This opens a browser window for Sigstore OIDC. I authenticate as @amaanx86 via GitHub. Fulcio issues an ephemeral certificate tied to that identity, the tool signs targets.json, and pushes the signature to the branch. Rekor gets an entry proving the person at @amaanx86 authorised this specific targets update at this specific time.

Worth being clear on what "offline" means here: it requires a human with a verified identity, not a CI token. It does not require an air-gapped machine. The private key is still ephemeral.

After merge, online-sign.yml takes over and refreshes snapshot and timestamp automatically using the Actions OIDC token. No human needed for that part.

What gets published

GitHub Pages at amaanx86.github.io/oci-prometheus-sd-proxy-tuf-on-ci/metadata/ serves:

root.json - signed by @amaanx86 via Sigstore OIDC; defines trusted key holders for all roles
targets.json - signed by @amaanx86; lists all authorised release targets with digests
snapshot.json - signed by GitHub Actions OIDC; prevents any metadata file from being swapped with an older version
timestamp.json - signed by GitHub Actions OIDC; has a short validity window to prevent freeze attacks

Each release gets a target file at targets/oci-prometheus-sd-proxy/releases/v1.4.2-rc.json. The TUF metadata key (used inside targets.json) is oci-prometheus-sd-proxy/releases/v1.4.2-rc.json - the path is relative to the targets directory, not the repo root.

Verifying v1.4.2-rc

All verification uses the digest, not the tag, since the tag is a mutable pointer.

Image signature:

cosign verify \
  --certificate-identity="https://github.com/amaanx86/oci-prometheus-sd-proxy/.github/workflows/docker-build-push.yml@refs/tags/v1.4.2-rc" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd

SBOM attestation:

cosign verify-attestation \
  --certificate-identity="https://github.com/amaanx86/oci-prometheus-sd-proxy/.github/workflows/docker-build-push.yml@refs/tags/v1.4.2-rc" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  --type cyclonedx \
  ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd

Release-metadata attestation:

cosign verify-attestation \
  --certificate-identity="https://github.com/amaanx86/oci-prometheus-sd-proxy/.github/workflows/docker-build-push.yml@refs/tags/v1.4.2-rc" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  --type "https://github.com/amaanx86/oci-prometheus-sd-proxy/release-metadata" \
  ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd

SLSA L3 provenance (requires digest reference - slsa-verifier rejects mutable tags):

slsa-verifier verify-image \
  "ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd" \
  --source-uri "github.com/amaanx86/oci-prometheus-sd-proxy" \
  --source-tag "v1.4.2-rc"
# Verified build using builder "...generator_container_slsa3.yml@refs/tags/v2.0.0"
# at commit a68d4cd5d29cc6b865c6804fe63adff14ac74b27
# PASSED: SLSA verification passed

TUF metadata chain verification is documented in detail - including the full client walkthrough and what each step validates - in the release verification docs. The short version: a compliant TUF client walks root trust, snapshot consistency, and timestamp freshness before fetching the target. If the metadata is expired, rolled back, or the signature chain is invalid, the client raises before returning anything. That freshness check is what separates TUF from static signature verification.

What this covers and where the gaps are

This pipeline secures the release artifact and its provenance chain. An operator fetching the image can independently verify who built it, from what source, in what environment, and that the release was authorised by a human identity. That is a lot more than most projects ship with.

But supply chain security has layers, and the release artifact is only one of them. A few honest gaps:

Go module dependencies. The SBOM shows what modules are in the binary, and go.sum pins their hashes. But govulncheck and a periodic dependency audit are what actually catch known-vulnerable transitive dependencies. The attestation proves the SBOM is authentic; it does not tell you the SBOM is safe.

Runtime enforcement. Signing and provenance only matter if someone checks them at deploy time. Right now verification is a manual step. The more interesting place this is going is integrating the cosign attestations and SLSA provenance into a Kyverno or OPA/Gatekeeper policy engine that enforces admission control in Kubernetes. Policy engines like Kyverno can query Sigstore and reject any image that lacks a valid SLSA L3 attestation from the correct workflow identity - automatically, at admission time, not as a manual verification step. That closes the loop between what we proved at build time and what is allowed to run in production.

TUF root key compromise. If the @amaanx86 GitHub account itself was compromised, an attacker could rotate the root TUF key in a way that would pass client verification. TUF supports threshold signatures across multiple root key holders to mitigate this, which becomes relevant as the project scales to multiple maintainers.

The dependency of your dependencies. None of this solves a compromised slsa-github-generator or a backdoored Syft release. That is a solved problem in theory (pin action hashes, verify the tools themselves) but it is worth naming.

What I am building toward

The next step that interests me most is closing the loop at runtime.

The next meaningful investment is runtime enforcement. Kyverno ClusterPolicies that require verified SLSA provenance before admission, OPA rules that check SBOM attestations against a known-safe package policy, and Sigstore-aware image admission are all achievable with what the pipeline already produces. The attestations are already there. The policy layer that consumes them is the missing piece.

After that, expanding to multi-maintainer TUF with hardware-backed root keys and threshold signatures would make the trust model genuinely robust at scale.

Full implementation details and the verification workflow are documented at oci-prometheus-sd-proxy.readthedocs.io/en/latest/releasing.html.

Built My Own Prometheus Service Discovery for Oracle Cloud Because a 3-Year-Old PR Never Got Merged

Amaan Ul Haq Siddiqui — Wed, 11 Mar 2026 23:20:32 +0000

There is a specific kind of frustration reserved for when you know a problem is solved, you can see the solution, and you still cannot use it. That is how this project started.

The Context: Setting Up Observability From Scratch

I was building out observability from scratch across Oracle Cloud Infrastructure - multiple tenancies, multiple regions, a decent number of compute instances spread across compartments. The goal was full coverage: every VM enrolled in monitoring, no gaps, no guesswork.

When you are starting from zero, one of the first things you ask is how Prometheus is going to discover what it needs to scrape. For AWS you have EC2 service discovery built right in. Same for GCP, Azure, Hetzner, DigitalOcean. You configure credentials, set some filters, and Prometheus takes care of the rest.

OCI is not on that list.

I searched. I found a pull request in the Prometheus repository opened by an engineer at Oracle. It was exactly what I needed. It was also three years old and had never been merged. Comments, reviews, back and forth, and then silence. The PR is still open today. If you have gone looking for OCI service discovery in Prometheus you have probably landed on that same page, felt a brief moment of hope, and then noticed the date.

So I built it myself.

What I Actually Needed

The requirement was simple: new VM comes up, Prometheus finds it and starts scraping. No manual steps in that loop. Not because I was fixing a broken process - there was no process yet. I was designing this from scratch and I was not going to design it with a gap where a human has to remember to update a config file.

The concern was blind spots. Infrastructure grows, VMs get provisioned, things change. If enrollment is manual, coverage is only as good as the last person who remembered to do it. I wanted observability that was structurally complete, not best-effort.

The workflow I landed on:

Tag the VM in OCI.
When provisioning a new instance, add a tag - something like prometheus:scrape = true. That is the only enrollment step.

Open the Prometheus port on the network security group.
Allow the Prometheus server IP to reach port 9100. One rule, specific source, no broad exposure.

Run the node exporter playbook.
An Ansible playbook installs and starts node exporter. Done.

That is the full enrollment flow. The VM is in monitoring. No touching the Prometheus config. No reloading Prometheus. No rollout restart of a Kubernetes deployment. No SSH back into the server to verify anything.

The proxy polls OCI, finds every instance with the right tag across all configured tenancies and compartments, and hands the target list to Prometheus via the HTTP service discovery API. Prometheus picks it up on its next refresh cycle. The whole thing is automatic by design, not patched in after the fact.

What I Built

oci-prometheus-sd-proxy is a lightweight Go service that implements the Prometheus HTTP Service Discovery API for OCI.

You point Prometheus at it like this:

scrape_configs:
  - job_name: oci_instances
    http_sd_configs:
      - url: 'http://oci-sd-proxy:8080/v1/targets'
        authorization:
          type: Bearer
          credentials: 'YOUR_TOKEN'
        refresh_interval: 60s
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    relabel_configs:
      - source_labels: [__meta_oci_instance_name]
        target_label: instance
      - source_labels: [__meta_oci_tenancy_name]
        target_label: tenancy
      - source_labels: [__meta_oci_compartment_name]
        target_label: compartment
      - source_labels: [__meta_oci_region]
        target_label: region
      - source_labels: [__meta_oci_availability_domain]
        target_label: availability_domain
      - source_labels: [__meta_oci_instance_shape]
        target_label: shape

The proxy handles the rest. It scans OCI, filters by tag, and returns targets with rich metadata: tenancy, compartment, region, availability domain, shape, fault domain, private IP, and all your custom OCI tags as Prometheus labels. Use them for relabeling, alerting rules, dashboards - whatever you need.

A few things I cared about when building it:

Multi-tenancy from day one. The proxy handles all tenancies in parallel from a single config file. One deployment, full coverage.

Rate limiting. OCI's API will return 429s if you push it too hard. The proxy uses a token bucket proactively and a retry policy reactively. Discovery does not silently fail under load.

Security. Bearer token auth, distroless container, read-only config mounts. In an HA setup it sits on the local network, only reachable by Prometheus. It does not need to be internet-facing and it should not be.

Caching. Discovery results are cached so Prometheus always gets a response, even if the OCI API is momentarily slow or rate limiting.

Battle Tested

This is running in production across 10+ Oracle Cloud tenancies - different regions, different compartments, different team setups. It has handled OCI API slowness, tenancy permission edge cases, and the general entropy that comes with real infrastructure at scale.

The thing that still satisfies me is watching a new VM appear in Grafana within a minute of it booting, with nobody doing anything to make that happen. That is what good observability infrastructure should feel like. You build it right once and it stays right.

Why This Did Not Exist Already

OCI is a smaller player compared to AWS or GCP and Prometheus contributors naturally prioritize the platforms most of their users are on. Oracle engineers clearly wanted to solve this - that PR is evidence of that - but getting a first-party integration merged upstream is a long road with no guarantees.

The HTTP SD API that Prometheus exposes is actually the right answer for this situation. It lets any platform plug in without touching the Prometheus codebase. I just had to build the other end of that interface.

Try It

Repository: https://github.com/amaanx86/oci-prometheus-sd-proxy
Documentation: https://oci-prometheus-sd-proxy.readthedocs.io/

The docs cover installation, the full configuration reference, OCI IAM permissions, and Docker Compose deployment examples. If you are building observability on OCI and want automatic enrollment from day one, this is what I use.

And if you were one of the people who commented on that PR hoping it would eventually land - same. This is what I built instead.

Original Post : https://amaanx86.github.io/blog/oci-prometheus-service-discovery

What Broke During Our AWS DMS Migration (And How We Fixed It)

Amaan Ul Haq Siddiqui — Sat, 24 Jan 2026 09:46:30 +0000

Let me tell you about the time I thought migrating a database would be straightforward. Spoiler alert: it wasn't.

The Setup

I was tasked with migrating our MySQL database from DigitalOcean's managed service to AWS RDS. Armed with confidence and the AWS DMS documentation, I dove in headfirst.

The First Roadblock: Serverless Seemed Like a Good Idea

Creating the source and target endpoints went surprisingly smooth. I felt like I was on a roll. Then came the task creation, and I thought, "Hey, let's use serverless DMS. Modern, scalable, perfect."

That's when everything came to a grinding halt.

The networking configuration for serverless DMS had me completely stumped. I spent way too long trying to figure out the right VPC setup, subnet configurations, and security group rules. Nothing seemed to work the way I expected. The documentation made sense in theory, but practice was a different story.

Eventually, I gave up on serverless and pivoted to the traditional EC2-based replication instance. Sometimes the old way is the right way.

Excluding the Noise

With my shiny new replication instance ready, I created the migration task. But I needed to make sure I wasn't migrating MySQL's system databases along with my actual data. Nobody needs that mess.

I configured table mappings to include all user databases while explicitly excluding the system ones:

{
  "rules": [
    {
      "rule-type": "selection",
      "rule-id": "1",
      "rule-name": "include-all-user-dbs",
      "object-locator": {
        "schema-name": "%",
        "table-name": "%"
      },
      "rule-action": "include"
    },
    {
      "rule-type": "selection",
      "rule-id": "2",
      "rule-name": "exclude-mysql",
      "object-locator": {
        "schema-name": "mysql",
        "table-name": "%"
      },
      "rule-action": "exclude"
    },
    {
      "rule-type": "selection",
      "rule-id": "3",
      "rule-name": "exclude-sys",
      "object-locator": {
        "schema-name": "sys",
        "table-name": "%"
      },
      "rule-action": "exclude"
    },
    {
      "rule-type": "selection",
      "rule-id": "4",
      "rule-name": "exclude-information_schema",
      "object-locator": {
        "schema-name": "information_schema",
        "table-name": "%"
      },
      "rule-action": "exclude"
    },
    {
      "rule-type": "selection",
      "rule-id": "5",
      "rule-name": "exclude-performance_schema",
      "object-locator": {
        "schema-name": "performance_schema",
        "table-name": "%"
      },
      "rule-action": "exclude"
    }
  ]
}

Clean and specific. I felt good about this.

The Premigration Checks Humbled Me

I ran the premigration assessment checks, expecting maybe a warning or two. Instead, I was greeted with a wall of failures. Major ones. The kind that make you question your life choices.

I spent the next few hours going through each failed check, cross-referencing with AWS documentation, and fixing issues one by one. Most of the critical warnings got resolved, though some minor ones remained. I figured those were acceptable and proceeded with the migration.

The Migration Itself: A False Sense of Security

This was our staging database, and it was massive. The migration kicked off, and surprisingly, it ran smoothly. Hours passed, data transferred, progress bars filled. Everything looked perfect.

We switched the application endpoint to the new RDS instance, deployed the changes, and waited for the green light.

Then the login feature stopped working entirely.

The Investigation That Nearly Broke Me

Cue several hours of frantic debugging. We checked connection strings, verified credentials, tested queries manually, checked network rules, and compared database structures. Everything looked identical.

Until we looked closer at the tables themselves.

Our foreign keys were gone. Primary keys were missing. Auto-increment sequences had reset. DMS had essentially eaten the structural integrity of our database.

Turns out, this is a known behavior. DMS focuses on moving data efficiently, and in doing so, it doesn't always preserve things like constraints and keys perfectly during the initial load.

The Actual Solution

After more digging through documentation and forums, we found the recommended approach: manually dump and restore the schema first, then let DMS handle just the data migration.

We also discovered that you can pass additional connection parameters to the DMS endpoint configuration to better preserve database objects during migration. We updated our endpoint settings with these parameters and ran the migration again.

This time, everything worked. Foreign keys intact, primary keys preserved, auto-increment sequences functioning as expected. The application came back to life, and logins worked perfectly.

Lessons Learned

First, serverless isn't always the answer, especially when you're still figuring out the networking intricacies of a new service.

Second, premigration checks exist for a reason. Those warnings are trying to save you from pain later.

Third, and most importantly, when migrating databases with DMS, take the time to migrate your schema separately. Don't rely on DMS to handle everything. It's a data migration service, not a complete database cloning tool.

Fourth, if you're planning to use CDC (Change Data Capture) for ongoing replication, make sure binary logging is enabled on your source database with the correct format. MySQL requires binlog_format set to ROW for DMS to capture changes properly. Without this, your CDC tasks will fail silently or miss updates entirely.

The whole experience was frustrating, time-consuming, and honestly a bit embarrassing. But it taught me more about AWS DMS in a few days than I would have learned in weeks of casual reading.

If you're planning a DMS migration, learn from my mistakes. Your future self will thank you.

Designing a Secure AWS Landing Zone with Control Tower (What Most Blogs Don’t Tell You)

Amaan Ul Haq Siddiqui — Sun, 04 Jan 2026 10:50:10 +0000

Intro

imagine ur the architect of this massive growing org right and ur job is basically to design a secure compliant scalable aws environment that wont fall apart when business needs change which they always do. u gotta ensure the right governance is there sensitive data is locked down and everything can scale up. sounds like a headache honestly but with aws control tower u actually have a shot at making this work without losing ur mind

lets go on a trip thru the cloud setting up a landing zone using control tower and seeing how organizational units aka OUs can be the backbone of ur security game

1. Building the foundation: the org management account

so every solid landing zone starts somewhere and that somewhere is the organization management account. think of this as the heart of ur aws world where policies access and the basic structure live. its where u define global security rules and where control tower actually does its magic

for me it all starts by making this org management account the master key to everything. first step i take is locking this thing down tight access control everything because if this account gets compromised its game over

2. The structure takes shape: OUs and governance

now that the management account is chillin i start carving out the structure. this is where control tower is actually super useful. using organizational units i make a hierarchy that actually makes sense for the business

i always gotta balance letting devs do their thing while keeping control. for big companies i usually set up separate OUs for security production and staging just to keep sanity

so i generally define some high level OUs

security ou – the gatekeeper making sure audits happen
production ou – where the money is made live services live here
staging ou – the playground where we break stuff before prod

3. Nested OUs: a growing ecosystem

as the org gets bigger the aws environment gets messy so to keep it clean i start nesting OUs. this is basically putting folders inside folders but for cloud accounts

now i gotta support multiple teams so i typically make groups like app-1 and app-2 each with their own prod and staging accounts. this ensures app-1 cant mess with app-2s stuff. nice and isolated

then the internal ops need love too so i make an internal operations OU with sub-OUs for finance hr and it departments so hr doesnt accidentally delete the finance database

4. Audit and compliance: the log archive

structure is done but u need visibility. audit logs compliance retention policies all that boring but super critical stuff needs to be set up so u dont fail an audit

i set up audit and log archive accounts immediately. these are like the black box of an airplane immutable records of everything that happens. logs go in they dont come out (unless u need to investigate something) and automated backups keep everything safe

5. The real magic: terraform and automation with AFT

ok so the structure is cool and all but clicking buttons in the console is for amateurs. we want speed and consistency. enter the aws control tower account factory for terraform or AFT for short. this is where things get super interesting because now we are treating our account vending machine as code

check this out u can use this module here
https://registry.terraform.io/modules/aws-ia/control_tower_account_factory/aws/latest

and the code lives here
https://github.com/aws-ia/terraform-aws-control_tower_account_factory

so instead of manually provisioning accounts i deploy AFT. now when a new team needs an account they just push a change to a terraform repo. AFT sees the change spins up the account using control tower and then—this is the best part—it automatically applies all the baseline customization. we are talking setting up security groups iam roles and connecting to the vpc automatically. no human error just pure automation pipelines running smooth

it basically lets u maintain a global account customization repo so every single account that gets birthed by control tower comes out pre-configured with my specific tooling and security baselines right out of the box. massive time saver

6. Conclusion: a secure scalable future

so i step back drink some coffee and look at the dashboard. i didnt just build some servers i built a foundation. the org is secure scalable and aligned with business goals. thanks to control tower and that sweet AFT automation the journey from a messy handful of accounts to a compliant enterprise grade environment was actually kinda smooth and easy access and control setup with IAM Identity Centre SSO & Directory Services which we can integrate with Azure EntraID as well :)

Why We Didn’t Move to EKS (Yet): Choosing ECS Over Kubernetes in Production

Amaan Ul Haq Siddiqui — Sun, 28 Dec 2025 09:47:19 +0000

In the cloud-native world, Kubernetes (EKS) is often treated as the default destination for container orchestration. It’s powerful, flexible, and industry-standard. But for many engineering teams, it’s also overkill.

We recently faced the classic "build vs. buy" decision for our infrastructure. The pressure to adopt EKS was there, but after evaluating our actual needs, we made a conscious choice to stick with Amazon ECS.

The result? We saved a handsome amount of money, months of engineering time, and avoided the operational tax that comes with managing Kubernetes clusters. Here is how we architected a robust, scalable production environment on ECS without the K8s complexity.

The "Kubernetes Tax" We Wanted to Avoid

Kubernetes is amazing, but it requires a significant investment in tooling and maintenance. To run EKS properly in production, you aren't just managing containers; you're managing a platform. You need:

GitOps tools: ArgoCD or FluxCD for deployments.
Observability: Fluentd or similar for log shipping.
Ingress Controllers: NGINX or ALB controllers.
Security: Constant patching of the control plane and worker nodes.

For our team, we wanted to focus 100% on shipping application code, not managing infrastructure plumbing.

Our Hybrid ECS Architecture

We designed a hybrid ECS strategy that leverages the best of both serverless and provisioned compute.

1. Fargate for Stateless Workloads

For our main application servers and Sidekiq background workers, we used ECS Fargate.

No Servers to Manage: We don't worry about OS patching or scaling instances.
Right-Sizing: We pay only for the vCPU and RAM the tasks actually use.
Scalability: Fargate handles the heavy lifting of launching thousands of containers if needed.

2. EC2 Launch Type for Cron Jobs

Interestingly, we didn't go 100% Fargate. For our scheduled Cron jobs, we stuck with the EC2 Launch Type.

Why? Cron jobs run frequently and often use the same base images.
The Cost Hack: By running these on EC2 instances, we can cache Docker layers locally on the host. This drastically reduces data transfer costs from ECR (Elastic Container Registry) and speeds up start times, something Fargate doesn't support as efficiently for frequent, short-lived tasks.

The Stack: Simple and Managed

We offloaded state management to AWS managed services to keep the compute layer purely ephemeral:

Database: Amazon RDS for PostgreSQL.
Caching: Amazon ElastiCache (Redis).

CI/CD: Skipping the Complexity

One of the biggest wins was avoiding the "GitOps" complexity of ArgoCD or Flux. Our pipeline is a straightforward GitHub Actions workflow:

Build: Create the Docker image.
Scan: Run security vulnerability scans.
Push: Upload to ECR.
Deploy: Update the ECS Task Definition and force a new deployment.

That’s it. No separate synchronization server, no complex CRDs (Custom Resource Definitions), and no managing Helm charts. The pipeline is robust, easy to debug, and requires zero maintenance.

The Verdict: Time is Money

By choosing ECS, we:

Skipped the Learning Curve: No need to train the team on kubectl, manifests, or cluster networking.
Reduced Operational Overhead: No node patching, no control plane upgrades.
Lowered Bill: We aren't paying for EKS control plane fees ($73/month per cluster) or the overhead of system pods running on worker nodes.

We might move to EKS one day if our requirements for custom networking or service mesh become complex enough to warrant it. But for now, ECS allows us to run a stable, high-performance production environment where the only thing we have to take care of is our application code.

Sometimes, the best engineering decision is the boring one.

Memory-Based Auto Scaling: Saving Our Sidekiq Jobs When CPU Metrics Lied to Us

Amaan Ul Haq Siddiqui — Fri, 26 Dec 2025 15:13:41 +0000

We usually just default to CPU-based scaling for our Auto Scaling Groups (ASGs). It’s the standard move. It’s easy, it’s familiar, and for web servers? It usually works fine.

But sometimes, CPU utilization lies.

We recently hit a wall where CPU scaling completely failed us. This is the story of how a critical background job kept crashing even though our dashboards said everything was "healthy," and how switching to memory-based metrics saved the day.

The Silent Failure

We run a Ruby on Rails app. It relies heavily on Sidekiq for background work. These workers run on EC2 instances in an Auto Scaling Group.

On paper, everything looked great.
CPU usage? A comfortable 20–30%.
Network? Normal.
Disk? Fine.
AWS said we were green.

But the app was on fire.

Critical jobs were timing out. The queues were piling up. Retries were spiking. Worst of all? Workers were just... vanishing. Processes were dying, but since CPU was low, the auto-scaler didn't care. It didn't launch new instances. It just let them die.

The Culprit: The OOM Killer

I dug into the logs, and the answer was right there. Memory.

Our Sidekiq jobs are hungry. As the Ruby processes chewed through heavy tasks, they ate up more and more RAM. The instances were running out of memory, and the Linux OOM (Out-of-Memory) Killer stepped in to save the server by killing our Sidekiq process.

The problem? EC2 doesn't send memory metrics to CloudWatch by default.

So, while our RAM was screaming for help, CloudWatch saw low CPU and thought, "Everything is chill."

Step 1: Making Memory Visible

You can't fix what you can't see.

First thing I did was install the CloudWatch Agent on our instances. I needed it to ship custom metrics—specifically mem_used_percent—to AWS.

As soon as we turned it on, the graphs confirmed it.
CPU was bored at 20%.
Memory? It was spiking over 85%.

Above: Finally seeing the truth. CPU was low, but RAM was maxed out.

Step 2: Changing the Rules

We stopped listening to CPU. I set up a Target Tracking scaling policy that looks strictly at memory.

I told the ASG: "Keep average memory at 40%."

Sounds low, right? But background workers are unpredictable. They need breathing room for sudden spikes. This setup does two things:

It adds new servers before we hit the danger zone (75%+).
It doesn't kill servers too fast, so we avoid "thrashing" (booting up and shutting down constantly).

Above: The new policy. If RAM goes up, we scale out.

The Result

Instant fix.

The scaling became predictive. Instead of waiting for a crash, the cluster sees the memory pressure building and adds more power before things break.

We haven't seen a single OOM kill since. The Sidekiq service is happy, the queues are empty, and I can finally sleep.

Above: Stable, boring, and running perfectly.

Why CPU Scaling Sucks for Workers

Here's the takeaway.

Web traffic is usually CPU-heavy. You get a request, you process it, you send a response. CPU spikes, you scale. Simple.

Background workers (like Sidekiq, Celery, Bull) are different. They load big files. They process heavy data objects. They eat RAM. Your CPU can be totally asleep while your memory is completely full.

Final Thoughts

If you're running background jobs in an ASG and you're only watching CPU... you might be one heavy job away from a silent outage.

Don't just use the default settings. Scale on what actually hurts your application.

Note: The screenshots above are from a test setup. I’ve hidden sensitive stuff like Account IDs.

My First AWS Community Day Dubai & that too as a Speaker :)

Amaan Ul Haq Siddiqui — Thu, 25 Dec 2025 08:21:38 +0000

My First AWS Community Day Dubai (And That Too as a Speaker!)

Speaking at an AWS event was always on my "someday" list.
But "someday" arrived a lot sooner than I expected.

This past Sunday, I attended AWS Community Day UAE 2025 in Dubai.
And I didn't just attend.
I gave my first-ever talk.

The Announcement: Is This Real?

When the speaker list dropped, seeing my name next to industry veterans felt surreal.

Session: The Winning Stack: Lessons from a DevOps Hackathon Winner
Who: Me (DevOps Engineer at SUDO Consultants)

Representing my team and sharing my own messy, real-world journey in front of the AWS community? Humbling. Terrifying. But mostly, exciting.

The Talk: No Fluff, Just Real DevOps

I didn't want to give a lecture on textbook theory. We have documentation for that.
I wanted to talk about what actually happens when things are on fire.

I focused on:

How winning a DevOps Hackathon changed my perspective.
Why time pressure forces you to build better architectures.
How to build a secure CI/CD pipeline that actually works in production.
The massive gap between "Best Practices" and "Real Reality."

My goal was simple:
Share what actually works.

A Sunday Well Spent

The event was on a Sunday, and the turnout was incredible.
The best part wasn't even the sessions—it was what happened in the hallways.

I saw students asking about cloud careers for the first time.
I saw senior engineers swapping war stories about production outages.
I saw a community that actually wants to help each other.

One attendee told me later that hearing my story—going from a student to a DevOps pro—helped them understand what skills actually matter.
That feedback alone? Worth all the prep time.

Check out this post from the event

Why You Should Go

Events like this aren't just about the tech.
They are about the reality check.

You get honest career stories you won't find in a whitepaper.
You get networking that feels human, not transactional.
For anyone starting in Cloud or DevOps, this is how you accelerate.

Big Thanks

I have to give a shoutout to the people who made this happen:

SUDO Consultants for trusting me and pushing me forward.
The AWS User Group UAE volunteers—you guys run a world-class show.
The audience for the questions and the energy.

Being part of a group that values sharing over competition is rare. I don't take it for granted.

To Students & First-Timers

If you're a student or a junior engineer wondering if you should go to these events... or if you're "ready" to speak at one:

Don't wait until you feel ready.

You will never feel 100% ready.
These events don't reward perfection. They reward showing up.
Every expert on that stage started exactly where you are right now.

What's Next?

This was my first AWS Community Day Dubai.
It definitely won't be my last.

Thank you to the UAE AWS community for the platform and the inspiration.
Here’s to more learning, more sharing, and better clouds.

Why AWS CodeBuild Can Replace Self-Hosted GitHub Actions Runners

Amaan Ul Haq Siddiqui — Sat, 25 Oct 2025 10:14:10 +0000

Building CI/CD pipelines with GitHub Actions is usually pretty smooth. But the moment you decide to manage your own runners? That is where the headache starts.

Recently i was trying to deploy a self-hosted runner on ECS Fargate and honestly... it was a pain. I ran into so many issues with Docker-in-Docker (DinD) and realized i was just burning money on idle resources.

So i switched to AWS CodeBuild. Here is why.

The Problem with Fargate Runners

I thought putting runners on Fargate would be "serverless" and easy. I was wrong.

1. The Docker-in-Docker Nightmare
Most of my workflows need to build Docker images. But Fargate doesn't support DinD natively. You have to use messy workarounds to get it running and it adds so much complexity to something that should be simple.

2. Paying for Air
A self-hosted runner consumes resources even when it's doing absolutely nothing. You are paying for CPU and RAM just to wait for a job. For a small team or a side project that creates a bill you don't need.

Why CodeBuild is Better

I decided to try running my GitHub Actions workflows directly through AWS CodeBuild as a Proof of Concept.

It just worked. Seamlessly.

The Wins:

Native GitHub Support: CodeBuild can run your GH pipeline jobs directly. You don't need complex connectors.
Pay-Per-Use: This is the biggest one. No idle costs. You only pay when a build is actually running.
Private Access: Since it's in your AWS account it easily connects to your VPCs and private subnets without extra hassle.
Scalability: You can run multiple builds in parallel and never worry about queueing or adding more runner instances.

How It Works

The setup was surprisingly simple

Create a CodeBuild project
Connect your GitHub repo (OIDC or Access Token)
Point your workflow to the CodeBuild project

That’s it.

I deployed my apps directly from there and it skipped all the drama i had with ECS.

Conclusion

Self-hosted runners give you control but they also bring operational overhead that most of us just don't have time for.

If you are already on AWS and struggling with DinD on Fargate or just tired of managing runner fleets... check out CodeBuild. It’s cleaner, cheaper and it just gets out of your way so you can ship code.