DEV Community: Kamal Namdeo

SLSA, Sigstore and Provenance — Complete Practical Guide

Kamal Namdeo — Mon, 20 Jul 2026 07:39:16 +0000

SLSA + Sigstore + Provenance — Complete Mental Model

1. The problem this whole ecosystem solves

Traditional security scanning checks what's in an artifact (CVEs in dependencies).
It says nothing about how the artifact was built or whether the thing you're
running is actually what the source code says it should be.

Real attacks this misses:

SolarWinds (2020): attacker compromised the build system, not the source repo. Code review would've shown clean code — the binary was tampered with during compilation.
xz backdoor (2024): malicious code was injected into the release tarball, not the git repo. git log looked fine. The tarball didn't match the source.
event-stream npm (2018): a new maintainer with no history pushed a malicious version. No build-system compromise needed — just trust exploitation.

None of these are "vulnerable dependency" problems. They're supply chain integrity
problems: can you prove a binary came from a specific source, built by a specific
trusted process, without tampering in between?

That's what SLSA (framework/checklist) + Sigstore (tooling) + in-toto (attestation format) exist to solve.

Think of it like Go modules with go.sum — go.sum proves the content hasn't
changed since you first fetched it (hash pinning). SLSA provenance is go.sum's
older cousin: it proves not just "the bytes are what I saw before" but "here is
cryptographic evidence of the exact build process that produced these bytes."

2. SLSA — Supply-chain Levels for Software Artifacts

SLSA (pronounced "salsa") is not a tool. It's a specification/checklist —
like PCI-DSS for supply chains. It defines maturity levels for your build process.

Level	Requirement	Real-world meaning
L0	Nothing	No guarantees
L1	Provenance exists, build is scripted	You have a record of how it was built, but it could be forged (e.g., generated by the same machine that could tamper with the build)
L2	Provenance is generated by a hosted build service, signed	A trusted third party (GitHub Actions, GitLab CI, Cloud Build) attests to the build, not you. Harder to forge
L3	Build runs in an isolated/hardened environment; provenance is non-forgeable even by build service admins	Ephemeral, hermetic build environments; source and dependencies fully tracked

Key artifact SLSA produces: "provenance." This is a signed attestation
(metadata document) that says: *"Artifact X (identified by its SHA256 digest)
was built from source Y (specific commit), using builder Z (specific CI system

workflow file), with these exact inputs, at this time."*

Provenance answers: who built it, from what source, using what process, with
what inputs — and makes tampering with that record detectable.

What provenance actually looks like (in-toto format)

SLSA provenance is encoded using the in-toto attestation framework. Structure:

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [
    {
      "name": "ghcr.io/kamal/myapp",
      "digest": { "sha256": "a1b2c3...actual build output hash" }
    }
  ],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": {
    "buildDefinition": {
      "buildType": "https://github.com/actions/workflow@v1",
      "externalParameters": { "workflow": { "ref": "refs/heads/main", "repository": "github.com/kamal/myapp" } },
      "resolvedDependencies": [ { "uri": "git+https://github.com/kamal/myapp", "digest": { "sha1": "commit-hash" } } ]
    },
    "runDetails": {
      "builder": { "id": "https://github.com/actions/runner" },
      "metadata": { "invocationId": "run-12345", "startedOn": "...", "finishedOn": "..." }
    }
  }
}

Three things bolted together:

subject — the digest of what got produced (immutable, content-addressed — same idea as an OCI image digest or Go module checksum)
predicate — the claim being made about the subject (here: SLSA provenance; could also be a vuln scan result, SBOM, test results, etc. — in-toto is a generic envelope)
signature (added by Sigstore, wraps the whole thing) — proof of who's making this claim

This is the critical mental separation: in-toto = format of the claim,
SLSA = what claims you need to make and how strong they must be, Sigstore = how
you sign/verify/timestamp the claim.

3. Sigstore — the signing infrastructure

Sigstore solves: how do you sign artifacts without the nightmare of long-lived
private key management (rotation, HSMs, leaked keys, org-wide key distribution)?

Three components, each solving one piece:

3a. Fulcio — the Certificate Authority for "keyless signing"

Traditional signing: you generate a keypair, guard the private key forever,
distribute the public key out-of-band. Losing/leaking the key = disaster.

Fulcio flips this: instead of long-lived keys, you get a short-lived
certificate (valid ~10 minutes) bound to your OIDC identity (your GitHub
login, Google account, or a CI workflow's OIDC token — not a password, an
identity assertion).

Flow:

Your CI job (or you locally) generates an ephemeral keypair in memory.
It authenticates to Fulcio using an OIDC token (e.g., "I am github.com/kamal/myapp running workflow .github/workflows/release.yml on commit abc123" — GitHub's OIDC provider vouches for this).
Fulcio issues a 10-minute X.509 certificate binding that ephemeral public key to that verified identity.
You sign the artifact with the ephemeral private key.
You throw away the private key immediately. There's nothing to leak, nothing to rotate, nothing to steal.

This is the single biggest mental shift from traditional PKI: the identity
is the credential, not a stored secret. Verification later checks "was this
cert issued to github.com/kamal/myapp's release workflow?" — not "does this
public key match a file I have on disk."

3b. Rekor — the transparency log

Problem: if certs are only valid for 10 minutes, how do you verify a signature
years later after the cert has expired? And how do you stop someone from
secretly minting a cert for your identity?

Rekor is an append-only, tamper-evident transparency log (same design
lineage as Certificate Transparency logs for HTTPS). Every signing event gets
recorded: artifact digest, public key/cert used, timestamp, signature — all
inserted into a Merkle tree.

Why Merkle tree specifically:

Every leaf = one log entry (hashed)
Every parent node = hash of its two children
The root hash changes if any single entry, anywhere in the tree, is altered
Rekor publishes signed tree heads (STHs) periodically — anyone can verify the root hash didn't silently change (this is "gossip"/log monitoring)
You get an inclusion proof: a small set of sibling hashes (O(log n)) that let you cryptographically prove "entry X is definitely in this tree," without downloading the whole log

This gives you:

Non-repudiation: signer can't later claim "I didn't sign that" — it's in a public, immutable log
Detectability: if Fulcio (or an attacker) mis-issues a cert for your identity, it shows up in the public log where you (or automated monitors) can catch it
Timestamping: proves the signature existed at time T, so it remains valid even after the 10-minute cert expires — this is the trick that makes ephemeral certs workable long-term

3c. Cosign — the CLI that ties it together

cosign is the tool you actually run. It:

Talks to Fulcio to get the ephemeral cert (keyless mode) or uses a static keypair (traditional mode)
Signs the artifact (typically a container image by digest, or any blob)
Pushes the signature to Rekor
Stores the signature alongside the image in the OCI registry itself, using the OCI Referrers API / the old sha256-<digest>.sig tag convention — so the signature travels with the image without needing a separate system
Also handles attestations (cosign attest) — signing arbitrary in-toto statements (SBOM, SLSA provenance, vuln scan results) and attaching them to the image the same way

3d. `cosign sign` vs `cosign attest` — what's actually different

Both commands lean on the exact same Fulcio/Rekor/OIDC machinery under the
hood. What changes is what document actually gets signed.

A useful analogy: sign is putting a wax seal directly on a package. That's
the whole transaction — later, anyone can check "is this seal genuine, and
whose seal is it?" But the seal itself makes no claims beyond "I vouch for
this exact package." attest is writing a signed letter and attaching it to
the package. The letter makes an actual, structured claim — "this was built
from commit abc123 by GitHub Actions run #456" or "here is the complete
SBOM" — and it's that letter, not just the package, that gets signed.
Anyone reading it later can verify both "is this letter genuine?" and "what
does it actually say?", and make a decision based on the content.

	`cosign sign`	`cosign attest`
What's signed	The bare artifact digest (a minimal "simple signing" payload)	A structured in-toto `Statement` (SBOM, SLSA provenance, vuln scan, custom predicate), wrapped in a DSSE envelope
Claim being made	"I vouch that this exact digest is legitimate"	"I vouch for this specific, structured claim about this digest"
Verification	`cosign verify` — signed / not signed, by whom	`cosign verify-attestation --type <predicate>` — same identity check, plus the decoded predicate can be fed into a policy engine (Rego/CUE) to reason about content, e.g. "block if SLSA level < 3" or "block if SBOM has a banned license"
Storage in registry	Signature object referencing the image manifest	DSSE-wrapped attestation, also attached via the same referrer mechanism, tagged separately (`.att`) from a plain signature (`.sig`)

One important point that's easy to gloss over: there's no such thing as an
unsigned attestation. cosign attest doesn't attach a document and then
sign it as a second step — signing is the action. The DSSE envelope exists
specifically to pin down the exact bytes being signed, so the moment you run
attest, you get a signed claim or nothing at all. A predicate document with
no signature is just a JSON file anyone could have written; the signature is
what gives it any evidentiary value.

One-liner to keep: sign proves identity vouched for a digest.
attest proves identity vouched for a signed, structured claim about that
digest. An image can carry one signature and many attestations (SBOM +
provenance + vuln scan), each independently signed and independently logged
in Rekor.

4. How it all fits together — end to end flow

 Developer pushes code
        │
        ▼
 CI builds artifact (container image)
        │
        ▼
 CI generates SLSA provenance (in-toto statement describing the build)
        │
        ▼
 Cosign requests ephemeral keypair + cert from Fulcio
   (proves identity via CI's OIDC token — no stored secrets)
        │
        ▼
 Cosign signs: (a) the image itself, (b) the provenance attestation, (c) optionally the SBOM
        │
        ▼
 Signatures + cert + timestamp recorded in Rekor (Merkle tree, publicly auditable)
        │
        ▼
 Signature + attestations pushed to registry, attached to image via OCI Referrers API
        │
        ▼
 Consumer (you, a k8s admission controller, etc.) pulls image
        │
        ▼
 cosign verify checks:
   1. Signature is valid for this exact image digest
   2. Cert chains back to Fulcio root, and cert's identity matches an allow-list
      (e.g., "must be signed by github.com/kamal/myapp release workflow")
   3. Rekor has a valid inclusion proof for this signature at a plausible timestamp
        │
        ▼
 Policy engine (e.g., Kyverno, OPA/Gatekeeper) allows or blocks deployment based on
 the SLSA level / provenance content, not just "is it signed"

This is exactly the shape of what you'd want feeding a FedRAMP-grade SBOM
pipeline: SBOM (CycloneDX) as one attestation, SLSA provenance as another,
both signed and logged, both queryable by policy before deploy.

5. Hands-on: do the whole thing locally

Assumes Docker + a registry you can push to (use ghcr.io with a GitHub PAT, or spin up a local registry:2 container). Install cosign first:

# macOS
brew install cosign

# Linux
curl -O -L "https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64"
chmod +x cosign-linux-amd64 && sudo mv cosign-linux-amd64 /usr/local/bin/cosign

cosign version

5a. Local registry for practice

docker run -d -p 5000:5000 --name registry registry:2

docker pull alpine:3.19
docker tag alpine:3.19 localhost:5000/myapp:v1
docker push localhost:5000/myapp:v1

5b. Key-pair signing (offline mode — no Fulcio, simplest to grok first)

cosign generate-key-pair
# produces cosign.key (encrypted private key, you set a password) and cosign.pub

cosign sign --key cosign.key localhost:5000/myapp:v1
# signs by DIGEST, not tag — cosign resolves the tag to a digest first,
# because tags are mutable and digests aren't

cosign verify --key cosign.pub localhost:5000/myapp:v1

Inspect what actually got pushed — the signature lives as a separate OCI
artifact in the same repo, referencing your image's digest:

docker exec registry ls /var/lib/registry/docker/registry/v2/repositories/myapp/_manifests/tags

5c. Keyless signing (the real Sigstore flow — needs internet + browser/OIDC)

COSIGN_EXPERIMENTAL=1 cosign sign localhost:5000/myapp:v1

This pops a browser to authenticate (Google/GitHub/Microsoft OIDC), requests
a short-lived cert from the public Fulcio instance, signs, and pushes the
signature + writes the entry to the public Rekor log. Note there's no
.key file anywhere — nothing to lose, nothing to leak.

Verify, pinning to an identity (this is the important part — verifying who
signed it, not just that it's signed):

COSIGN_EXPERIMENTAL=1 cosign verify \
  --certificate-identity=your-email@example.com \
  --certificate-oidc-issuer=https://accounts.google.com \
  localhost:5000/myapp:v1

5d. Look at the actual Rekor entry

cosign verify localhost:5000/myapp:v1 --certificate-identity=... --certificate-oidc-issuer=... -o json | jq '.[0].rekorBundle'

# or query Rekor directly by artifact hash
rekor-cli search --sha $(docker inspect --format='{{index .RepoDigests 0}}' localhost:5000/myapp:v1 | cut -d@ -f2)

You'll see the log index, inclusion proof, signed tree head — this is the
Merkle-tree machinery from section 3b made concrete.

5e. Attach a SLSA provenance attestation

cosign attest mirrors cosign sign exactly — it supports the same two
modes. Don't let the flag choice below fool you into thinking attestation
needs a key; it's just the simpler path to demo first.

cat > provenance.json << 'EOF'
{
  "buildType": "https://example.com/local-manual-build",
  "builder": { "id": "kamal-local" },
  "invocation": { "configSource": { "uri": "git+https://github.com/kamal/myapp", "digest": {"sha1": "deadbeef"} } }
}
EOF

# Key-pair mode (reuses the keypair from 5b)
cosign attest --key cosign.key \
  --type slsaprovenance \
  --predicate provenance.json \
  localhost:5000/myapp:v1

cosign verify-attestation --key cosign.pub \
  --type slsaprovenance \
  localhost:5000/myapp:v1

Keyless equivalent (same Fulcio/OIDC/Rekor flow as 5c — a fresh ephemeral
cert is minted, the attestation gets its own Rekor entry, no key file
involved):

COSIGN_EXPERIMENTAL=1 cosign attest \
  --type slsaprovenance \
  --predicate provenance.json \
  localhost:5000/myapp:v1

COSIGN_EXPERIMENTAL=1 cosign verify-attestation \
  --type slsaprovenance \
  --certificate-identity=your-email@example.com \
  --certificate-oidc-issuer=https://accounts.google.com \
  localhost:5000/myapp:v1

verify-attestation decodes the in-toto envelope, checks the signature, and
prints the predicate — this is what a policy engine does automatically before
allowing deploy.

5f. Attach your SBOM the same way (ties directly to your day job)

cosign attest --key cosign.key \
  --type cyclonedx \
  --predicate sbom.cyclonedx.json \
  localhost:5000/myapp:v1

Now the image carries three linked, signed claims: the image itself, its
build provenance, and its SBOM — all independently verifiable, all queryable
via the OCI Referrers API without a side database.

6. Where GitHub Actions automates all of this

In real CI (not manual like above), you don't hand-roll provenance JSON — you
use slsa-github-generator, a reusable workflow that:

Runs your build in an isolated GitHub-hosted runner
Generates the provenance itself (SLSA L3, since GitHub's OIDC + isolated runner makes the provenance non-forgeable by you)
Signs via keyless cosign automatically (GitHub's OIDC token is the identity presented to Fulcio — no secrets needed in the workflow at all)

jobs:
  build:
    permissions:
      id-token: write   # <- this is what lets the job mint an OIDC token for Fulcio
      contents: read
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.0.0
    with:
      image: ghcr.io/kamal/myapp
      digest: ${{ needs.build.outputs.digest }}

permissions: id-token: write is the whole trick — it's GitHub minting a
short-lived OIDC token scoped to this exact job run, which Fulcio trusts
without you ever touching a private key.

7. The mental model to keep forever

in-toto = envelope format for "here's a claim about an artifact" (generic — SBOM, provenance, test results all use it)
SLSA = the checklist/spec for how trustworthy your build process needs to be, and what the provenance claim must contain
Sigstore (Fulcio + Rekor + Cosign) = infrastructure to sign any of these claims without managing long-lived keys, and to make every signature publicly, permanently auditable
OCI Referrers API = the plumbing that lets signatures/attestations travel with the artifact in a normal registry, instead of needing a separate metadata database

The Go analogy that'll stick: go.sum pins content (hash of what you fetched).
This whole stack pins provenance — not just "these are the bytes I saw
before" but "here's cryptographic, timestamped, publicly-logged proof of the
exact process, identity, and inputs that produced these bytes."

Strangler Fig Pattern Implementation - Part 2

Kamal Namdeo — Sat, 18 Jul 2026 08:35:31 +0000

Strangler Fig Data Migration: Orders Service Extraction

The core problem: two systems (monolith DB + new Orders microservice DB) need to represent the same truth during the transition, with zero downtime and zero data loss, until you can cut over completely.

Here's the depth version, phase by phase.

Phase 1: Dual Writes (Synchronization begins)

Before the new service takes any real traffic, you need both databases populated and kept in sync.

Pattern: Event-driven sync via CDC (Change Data Capture), not application-level dual writes.

Why not just write to both DBs from application code?

// DON'T do this — the classic dual-write trap
func CreateOrder(o Order) error {
    if err := monolithDB.Save(o); err != nil {
        return err
    }
    if err := newOrdersDB.Save(o); err != nil {
        // Now monolith has it, new service doesn't. Split brain.
        return err
    }
    return nil
}

This has no atomicity across two databases — a crash between the two writes leaves you inconsistent, and there's no distributed transaction that saves you here without introducing 2PC (which nobody wants across service boundaries).

Instead: CDC off the monolith's DB.

Use Debezium (or AWS DMS, or a custom binlog/WAL tailer) attached to the monolith's orders table.
Every INSERT/UPDATE/DELETE on orders gets captured from the transaction log and published to a Kafka topic (orders.cdc).
A consumer (part of your new Orders service, or a dedicated sync worker) reads this stream and applies changes to the new Orders DB.

This gives you:

No dual-write race conditions — the monolith DB remains the single writer during this phase.
Ordering guarantees — CDC preserves the commit order from the WAL/binlog, so you don't apply an UPDATE before its INSERT.
Backfill + streaming in one pipeline — Debezium supports an initial snapshot (bulk copy of existing rows) followed by seamless handoff to streaming changes, so you don't need a separate backfill script that risks missing a window of writes.

Monolith DB (writer) --WAL--> Debezium --> Kafka (orders.cdc) --> Sync Consumer --> New Orders DB (read replica of truth)

At this stage, the new Orders DB is a shadow copy — not yet authoritative, not yet serving real requests.

Phase 2: Shadow Traffic / Verification (Read-path validation)

Before switching any real read/write traffic, validate correctness:

Shadow reads: for every read request to the monolith's order API, asynchronously fire the same read against the new service and diff the responses (log mismatches, don't serve them to users). Tools like GitHub's scientist library formalize this "science experiment" pattern.
Reconciliation job: periodic batch job comparing row counts / checksums / hashes between monolith orders and new-service orders table, alerting on drift. This catches CDC consumer bugs, lag, or dropped events.
Lag monitoring: track CDC consumer lag (Kafka consumer group lag) — if new-service data is 30s behind monolith, that matters for correctness guarantees later.

You stay here until mismatch rate is effectively zero over a sustained period.

Phase 3: Cutover of Writes (the actual dangerous part)

This is where "no downtime" gets hard, because now you need to flip who owns writes without losing anything mid-flight.

Strangler fig at the write path uses a facade/proxy layer — typically an API gateway or a thin routing layer in front of both systems:

Client --> Facade/Router --> [routes based on rule] --> Monolith Orders API
                                                    \--> New Orders Service

Common approach — write cutover via feature-flagged dual-write with new service as source of truth:

Stop the CDC-based one-way sync.
Flip writes to go to the new Orders service first (new service becomes authoritative writer).
New service now dual-writes back to monolith DB (or publishes an event the monolith consumes) — this is the reverse sync, needed because other still-unmigrated parts of the monolith (invoicing, inventory, whatever hasn't been strangled yet) still read orders directly from the monolith DB.
This reverse sync is temporary — it exists only until every consumer of monolith orders data has itself been migrated to call the new Orders service (via API or its own CDC stream off the new DB).

This is the key insight of the strangler fig pattern: you don't migrate one big-bang cutover — you flip the direction of the sync arrow, and the "old" system becomes the dependent, then eventually gets its dependency removed entirely.

Phase 4: Handling In-Flight Writes During Cutover

The truly tricky moment is the write cutover instant. Techniques used:

Dual-write window with idempotency keys: during a short window, both old and new API paths accept writes, each write carries an idempotency key (order ID + version/timestamp), so replays or races from CDC catch-up don't create duplicates.
Version/vector clock on rows: each order row carries a monotonic version number. The sync consumer only applies an update if incoming version > current version — this makes the sync idempotent and order-independent (protects against Kafka redelivery or out-of-order application).
Brief write-freeze (seconds, not downtime): some teams accept a sub-second write lock on the specific resource being cut over (e.g., quiesce writes to orders table, drain CDC lag to zero, then flip router) rather than a fully lock-free cutover — this is often simpler and the "downtime" is only a few hundred ms of increased latency, not an outage.
Outbox pattern on the new service: once the new service is the writer, it uses a transactional outbox (write to its own DB + outbox table in the same local transaction) to reliably publish events for the reverse sync, avoiding its own dual-write problem.

Phase 5: Read Cutover

Reads are far less risky than writes since they're idempotent by nature:

Router shifts a percentage of read traffic to the new service (1% → 10% → 50% → 100%), canary-style.
Compare latency/error rates at each step.
Because you validated correctness in Phase 2, this is mostly a confidence/load-bearing exercise now, not a data-integrity one.

Phase 6: Decommission

Once 100% of reads and writes go through the new Orders service, and all other monolith modules that referenced orders have been updated to call the new service's API (or consume its event stream) instead of hitting the shared DB directly:
Turn off the reverse sync.
Drop the monolith's orders table (or archive it).
Remove the facade routing rule — it's no longer a fork, just a straight call.

Summary of the sync topology over time

Phase 1-2:  Monolith DB --(CDC, one-way)--> New Orders DB          [monolith authoritative]
Phase 3:    New Orders DB --(reverse sync)--> Monolith DB          [new service authoritative]
Phase 4-5:  Facade gradually shifts read/write traffic to new service
Phase 6:    Monolith DB orders table retired, sync removed          [new service sole owner]

The general principle: never have two systems both accepting independent writes to the same entity without a defined, single source of truth and a reconciliation mechanism. Every "dual write" period in a strangler fig migration should really be "one authoritative writer + one propagated-to replica," with the authority handed off deliberately, not two peers writing simultaneously.

Master the Strangler Fig Pattern - Part 1

Kamal Namdeo — Sat, 18 Jul 2026 07:56:55 +0000

Stop Doing "Big Bang" Rewrites: Master the Strangler Fig Pattern

This is two parts series, here is the part - 1, you can refer the part 2 here, I would suggest reading sequentially.

If you’ve been in software engineering long enough, you’ve likely heard a developer say, "This legacy codebase is a mess. We just need to stop feature work for six months and rewrite it from scratch."

This is the "Big Bang" rewrite. And historically, it is one of the riskiest maneuvers a software team can attempt.

To safely modernize legacy systems, we need a better approach. Enter the Strangler Fig Pattern, a concept coined by Martin Fowler (and heavily associated with Thoughtworks) that allows teams to incrementally migrate a legacy system without halting business value.

Here is a breakdown of how it works, the architectural mechanics behind it, and when you should (and shouldn't) use it.

Why Big-Bang Rewrites Fail

The naive approach to modernizing a legacy monolith is to freeze the old system, build a complete replacement in a vacuum, and then cut over all at once. This usually ends in disaster for a few reasons:

The Parallel Maintenance Trap: Rewrites take time—often years. During this time, the old system still needs bug fixes and security patches. You end up maintaining two systems in parallel with zero incremental payoff.
Maximum Risk on Day One: The new system inevitably has edge-case bugs that only surface under full production load. "Day one" of the cutover becomes the highest-risk moment of the entire project.
Moving Targets: Business requirements keep changing while you rewrite. By the time the "final" target is finished, it is already outdated.
The Second-System Effect: As Fred Brooks noted in The Mythical Man-Month, teams tend to over-engineer the replacement. Because they have a blank slate, they try to fix every architectural complaint about the old system at once, severely bloating the scope.

What is the Strangler Fig Pattern?

In software, it means putting a facade in front of your legacy system and gradually replacing functionality piece by piece until the legacy system can be safely decommissioned.

How it Works (Step-by-Step)

Deploy a Facade: Place an API Gateway or reverse proxy in front of the legacy system. Initially, it routes 100% of incoming traffic to the old system. The user experience remains completely unchanged.
Pick a Slice: Choose one small piece of functionality—like a specific endpoint or a distinct business capability (e.g., "Order History").
Build and Route: Build that single piece in the new architecture. Update the facade to route traffic for only that specific capability to the new system, while everything else falls back to the legacy monolith.
Verify: Test the new slice in production. Teams often use shadow traffic (sending a copy of real traffic to the new system silently to compare outputs) or canary releases before fully committing.
Rinse and Repeat: Pick the next piece. Build, reroute, verify. Do this until the facade routes 100% of traffic to the new microservices.
Decommission: Turn off the legacy monolith.

The core benefit: At every single step, your system is in a fully working, shippable state. You are never stuck in a "half-built, nothing works" limbo.

The Architectural Mechanics

How do you actually build this? There are two main technical challenges you have to solve: routing and data synchronization.

1. The Facade Layer (Routing)

This is usually an API Gateway, an AWS Application Load Balancer (ALB), or a reverse proxy like NGINX using path-based routing.

Here is what that looks like in a basic NGINX configuration:

server {
    listen 80;
    server_name api.myapp.com;

    # 1. The "Strangled" Piece: Route to the new microservice
    location /api/v2/orders {
        proxy_pass http://new-orders-microservice:8080;
    }

    # 2. The Fallback: Everything else goes to the legacy monolith
    location / {
        proxy_pass http://legacy-monolith:80;
    }
}

2. The Data Synchronization Challenge

Routing web requests is the easy part. The hard part is state.

If the legacy system and the new system both need access to the same underlying data during the multi-month transition, they need to stay in sync. There are two common ways to handle this:

Dual Writes: The application layer is configured to temporarily write data to both the old database and the new database.
Change Data Capture (CDC): A much more robust approach. You stream database changes from the legacy system's database into the new system in near-real-time. Tools like Debezium (often paired with Kafka) are industry standards for this. It allows the new system to have its own pristine data model while staying seamlessly updated without touching the legacy code.

Where Do You Start?

When deciding which piece of the monolith to strangle first, you generally have two choices:

Lowest Risk, Highest Confidence: Pick a simple, read-only feature first. This proves the routing infrastructure works, builds developer confidence, and earns trust from stakeholders without risking critical business flows.
Highest Active Pain: Pick the part of the monolith that fails the most, scales the worst, or requires the most active feature development. This delivers massive business value upfront, though it carries higher technical risk.

Pro tip: Start with one low-risk component to test the pipes, then immediately target a high-pain component.

When to Use (and When to Avoid) Strangler Fig

✅ When it's the right call:

The system is business-critical. You cannot afford extended downtime or a highly risky cutover.
The system is massive. A full rewrite would genuinely take months or years.
The business cannot freeze feature development. You need to keep shipping value while the migration happens in the background.

❌ When it's overkill:

The system is small enough that a team could completely rewrite and test it in a few weeks.
The system isn't mission-critical.
Why? Strangler Fig introduces complex, temporary infrastructure (the facade, dual-routing, CDC pipelines). If a careful rewrite-and-cutover is genuinely feasible, don't take on the overhead of maintaining two systems at once.

Conclusion

Modernizing a legacy system isn't just a technical challenge; it's a risk management exercise. By using the Strangler Fig pattern, you trade the heart-stopping panic of a midnight Big Bang cutover for a series of boring, predictable, incremental deployments. And in software engineering, boring is exactly what we want.

The Thundering Herd Problem (Cache Stampede) — and How to Tame It

Kamal Namdeo — Fri, 17 Jul 2026 08:18:10 +0000

1. What Is the Thundering Herd Problem?

The thundering herd problem (also called a cache stampede or dogpile effect) happens when a popular cache key expires or is missing, and a large number of concurrent requests all miss the cache at the same instant. Instead of one request hitting the database, hundreds or thousands do — simultaneously.

Picture a hot product page on an e-commerce site. The cache entry for product:12345 expires at exactly midnight. If that product gets 2,000 requests/second, all 2,000 requests in that window see a cache miss and stampede toward the database at once. The DB, which was only ever meant to serve the occasional cache-refill query, gets hammered with duplicate work — often enough to tip over under load, which then causes more timeouts, more retries, and a feedback loop that can take down the whole service.

The core issue isn't that the cache missed — misses are normal. The issue is redundant, uncoordinated work: every request tries to regenerate the same value independently, when only one regeneration was ever necessary.

2. The Solution Strategy: Locking (a.k.a. "Cache Mutex" / Request Coalescing)

The code below solves this with a distributed lock pattern built on Redis's SETNX (set-if-not-exists). The idea:

On a cache miss, only the first goroutine to notice gets to regenerate the value. Everyone else waits for that one regeneration to finish, then reads the freshly-populated cache.

This turns an N-way stampede into effectively one database query, no matter how many concurrent requests miss at once.

3. Walking Through the Code

func getProduct(id string) (Product, error) {
    val, err := redis.Get(ctx, "cache:product:"+id)
    if err == nil {
        return val, nil // cache hit, done
    }

Step 1 — Try the cache first. This is the fast path. Most requests, most of the time, exit right here. No locking overhead is paid unless there's actually a miss.

    acquired, _ := redis.SetNX(ctx, "lock:product:"+id, "1", 5*time.Second)
    if acquired {
        val := fetchFromDB(id)
        redis.Set(ctx, "cache:product:"+id, val, 300*time.Second)
        redis.Del(ctx, "lock:product:"+id)
        return val, nil
    }

Step 2 — Race for the lock. SETNX is atomic in Redis: only one caller among all the concurrent racers will get acquired == true. That caller becomes the designated "regenerator":

It queries the DB (the expensive operation).
It repopulates the cache with a fresh TTL (300*time.Second).
It deletes the lock, signaling "I'm done, cache is fresh."

The lock itself has its own TTL (5*time.Second). This is a safety net: if the regenerating goroutine crashes, panics, or the process dies before it can Del the lock, the lock self-expires instead of being held forever — preventing a permanent deadlock where nobody can ever regenerate that key again.

    for i := 0; i < 5; i++ {
        time.Sleep(50 * time.Millisecond)
        val, err := redis.Get(ctx, "cache:product:"+id)
        if err == nil {
            return val, nil
        }
    }

Step 3 — Everyone else waits and polls. All the losing goroutines (those that didn't get the lock) don't go to the DB themselves. Instead they poll the cache a handful of times with a short sleep in between, waiting for the winner to finish and populate cache:product:*. As soon as they see the key exists, they return it — they get a fresh value without ever touching the database.

    return fetchFromDB(id), nil
}

Step 4 — Bounded fallback. If the winner is unusually slow (5 × 50ms = 250ms of waiting with no result), the losers give up polling and fetch from the DB directly. This is a deliberate trade-off: it sacrifices some stampede protection in the worst case in exchange for bounded latency — no request waits forever, no matter what happens to the winner.

For reference, complete example:

package main

import "time"

func getProduct(id string) (Product, error) {
    val, err := redis.Get(ctx, "cache:product:"+id)
    if err == nil {
        return val, nil // cache hit, done
    }

    // cache miss — try to become the one who regenerates
    acquired, _ := redis.SetNX(ctx, "lock:product:"+id, "1", 5*time.Second)

    if acquired {
        // I won the lock — I go to the DB myself
        val := fetchFromDB(id)
        redis.Set(ctx, "cache:product:"+id, val, 300*time.Second)
        redis.Del(ctx, "lock:product:"+id)
        return val, nil
    }

    // I did NOT win the lock — someone else is already fetching
    for i := 0; i < 5; i++ {
        time.Sleep(50 * time.Millisecond) // <-- THIS is the sleep. My own goroutine, sleeping itself.
        val, err := redis.Get(ctx, "cache:product:"+id)
        if err == nil {
            return val, nil // the winner finished, I can read it now
        }
    }

    // gave up waiting — fall back to hitting DB myself anyway
    return fetchFromDB(id), nil
}

4. Why This Works

Without locking	With locking
N concurrent misses → N DB queries	N concurrent misses → 1 DB query + (N-1) cache reads
DB load scales with traffic spikes	DB load stays flat regardless of concurrency
Cache expiry moments are dangerous	Cache expiry moments are absorbed by the lock

The lock converts a thundering herd into a queue of one worker, many waiters — a classic instance of the broader technique known as request coalescing or singleflight (Go's golang.org/x/sync/singleflight package implements the same idea in-process, without Redis, for a single-node case).

5. Trade-offs and Failure Modes to Be Aware Of

Polling adds latency for losers. In the worst case, a loser waits up to 250ms before falling back to the DB itself. If your DB call is itself slow, tune the sleep/retry count accordingly, or switch to a pub/sub notification instead of polling (Redis Pub/Sub or a condition-variable-like mechanism) so losers are woken up immediately rather than polling blindly.
Lock TTL vs. DB latency. If fetchFromDB can legitimately take longer than 5 seconds, the lock could expire while the winner is still working, letting a second goroutine "acquire" the lock and duplicate the fetch. Size the lock TTL comfortably above your p99 DB latency.
Fallback defeats the purpose under sustained slow DB. If the DB is consistently slow (not just a one-off), many losers will time out their poll loop and hit the DB anyway, partially reintroducing the herd. This pattern protects best against short stampedes (cache expiry moments), not sustained DB degradation.
Stale reads during regeneration. Everyone except the winner is blocked or reading old-then-new data; if you need to avoid ever serving a stale value while it's mid-regeneration, this pattern needs pairing with logic that serves the old value during the lock window (stale-while-revalidate) rather than making losers wait.

6. Related/Alternative Strategies

Probabilistic early expiration (XFetch): recompute the value slightly before it actually expires, with a probability that increases as expiry nears — spreads regeneration out over time instead of concentrating it at a single instant.
Stale-while-revalidate: serve the old (slightly stale) value immediately while one background goroutine refreshes it, so no request ever blocks.
In-process singleflight + distributed lock combo: dedupe requests within a single node using singleflight first (cheap, no Redis round-trip), and only use the Redis lock across nodes — reduces Redis load further in high-QPS services.

This pattern — SETNX lock + bounded poll + DB fallback — is a solid, production-common baseline. The main dial to tune is the poll interval/count vs. lock TTL vs. your DB's actual latency profile.

Nginx Event Loop — Complete Lifecycle Reference

Kamal Namdeo — Sat, 27 Jun 2026 15:21:43 +0000

nginx Event Loop — Complete Lifecycle Reference

A precise, bottom-up reference covering every buffer, syscall, interrupt, and data movement from the moment a TCP packet hits the NIC to the moment a response is sent back. Two concurrent users are used throughout as a concrete example.

Foundations — fd and Socket
Hardware Layer — NIC, DMA, Interrupts
Kernel Structures and All Buffers
epoll — How the Worker Waits Efficiently
nginx Startup Sequence
Complete Request Lifecycle — Two Concurrent Users
What Happens While Worker is Busy
All Buffers — Master Reference
All Syscalls — Master Reference
Failure Modes

1. Foundations

1.1 Everything is a File

Linux's core philosophy: every I/O resource — files on disk, network connections, pipes, terminals, devices — is represented as a file. This means one unified API (read, write, close) works on all of them. The kernel manages the actual resource. Your process holds a token.

1.2 File Descriptor (fd)

A file descriptor is just an integer. It is a per-process token that refers to a kernel-managed resource. The kernel maintains a table per process called the fd table — a simple array where the index is the fd and the value is a pointer into the kernel.

Process fd table:
┌─────┬───────────────────────────────┐
│ fd  │ points to                     │
├─────┼───────────────────────────────┤
│  0  │ stdin                         │
│  1  │ stdout                        │
│  2  │ stderr                        │
│  3  │ listen socket (nginx)         │
│  5  │ User A client connection      │
│  6  │ User B client connection      │
│ 12  │ backend connection for User A │
│ 13  │ backend connection for User B │
└─────┴───────────────────────────────┘

0, 1, 2 are always pre-assigned. Application fds start from 3 upward. The fd is meaningless on its own. It only means something when passed to a syscall — the kernel uses it to look up the real resource.

1.3 Socket

A socket is the kernel's internal data structure representing one end of a network connection. Created when your process calls socket(). Lives entirely in kernel RAM. Your process never touches it directly.

Socket struct (kernel RAM):
┌────────────────────────────────────┐
│ local_ip:    192.168.1.10          │
│ local_port:  80                    │
│ remote_ip:   203.0.113.5           │
│ remote_port: 54231                 │
│ state:       ESTABLISHED           │
│ recv_buffer: [ ...incoming bytes ] │
│ send_buffer: [ ...outgoing bytes ] │
│ tcp_state_machine: ...             │
│ timers, seq numbers, window size   │
└────────────────────────────────────┘

1.4 How fd and Socket Relate

PROCESS (user space)            KERNEL (kernel space)

fd table                        Open file table         Socket struct
┌──────┐                        ┌─────────────┐         ┌──────────────┐
│ fd=5 │ ─────────────────────► │ file entry  │ ──────► │ socket {     │
└──────┘                        │ (flags,     │         │  recv buffer │
                                │  offset)    │         │  send buffer │
                                └─────────────┘         │  state ...   │
                                                        └──────────────┘

This two-level indirection exists so that two processes (parent/child after fork) can share the same socket with different fd numbers.

2. Hardware Layer

2.1 NIC (Network Interface Card)

A physical chip on the motherboard (or PCIe slot). It receives electrical/optical signals from the network cable, decodes them into bytes, and writes them into RAM. The CPU has no involvement in receiving the bytes themselves — the NIC does it autonomously.

2.2 DMA (Direct Memory Access)

DMA is the mechanism that lets the NIC write bytes directly into RAM without involving the CPU. The kernel sets up a region of RAM called the NIC Ring Buffer at boot time and tells the NIC its address. The NIC writes incoming packets straight there. Zero CPU cycles spent moving bytes.

2.3 NIC Ring Buffer

A circular array of fixed-size slots in kernel RAM, allocated at NIC driver initialization.

NIC Ring Buffer (in kernel RAM):
┌──────────┬──────────┬──────────┬──────────┐
│ slot 0   │ slot 1   │ slot 2   │ slot 3   │  ← NIC writes here via DMA
│ pkt data │ pkt data │ (empty)  │ (empty)  │
└──────────┴──────────┴──────────┴──────────┘
      ▲                     ▲
   NIC writes            kernel reads (via interrupt)

Size: 256 to 4096 slots typically. Each slot holds one packet (up to 1500 bytes for standard Ethernet MTU).
Who writes: NIC hardware via DMA.
Who reads: Kernel interrupt handler (ISR).
What happens when full: Packets are dropped. Visible via ethtool -S eth0 | grep drop or ip -s link.

2.4 Hardware Interrupt (IRQ — Interrupt Request)

When the NIC finishes writing one or more packets into the Ring Buffer via DMA, it sends an electrical signal on a dedicated wire to the CPU. This signal is an IRQ (Interrupt Request).

The CPU responds by:

Finishing the current instruction (not mid-instruction).
Saving its register state (so it can resume later).
Looking up the interrupt number in the IDT (Interrupt Descriptor Table) — a kernel table mapping interrupt numbers to handler functions.
Jumping to the handler function.
Restoring state and resuming what it was doing before.

This entire sequence takes microseconds. The user process (nginx worker) does not know it happened.

2.5 ISR (Interrupt Service Routine)

The kernel function registered to handle NIC interrupts. When called:

Reads packet(s) from the NIC Ring Buffer.
Parses the Ethernet frame, IP header, TCP header.
Identifies which socket this packet belongs to (by matching src_ip:src_port:dst_ip:dst_port).
Copies payload bytes into that socket's recv buffer in kernel RAM.
Updates TCP sequence numbers, ACKs, window size.
Marks that socket's fd as ready (EPOLLIN) in the epoll ready list.
If the process was sleeping in epoll_wait, wakes it up.
Returns. CPU resumes what it was doing.

3. Kernel Structures and All Buffers

3.1 SYN Queue (Incomplete Connections Queue)

When a client sends a SYN packet to initiate a TCP connection:

Kernel puts this half-open connection into the SYN Queue.
Sends back SYN-ACK.
Waits for the client's final ACK to complete the handshake.

Tunable:  /proc/sys/net/ipv4/tcp_max_syn_backlog
Default:  128 (older kernels) / 1024 (newer)

Who writes: Kernel TCP stack on receiving SYN.
Who reads: Kernel TCP stack on receiving final ACK — moves entry to Accept Queue.
What happens when full: Kernel drops incoming SYNs silently. Client retries. Appears as connection timeout.

SYN flood attacks fill this queue deliberately. tcp_syncookies is the mitigation.

3.2 Accept Queue (Complete Connections Queue)

After the 3-way handshake completes, the connection moves from SYN Queue to the Accept Queue. These connections are fully established and waiting for your process to call accept().

Effective size = min(listen() backlog argument, net.core.somaxconn)

net.core.somaxconn default: 128 (kernel < 5.4) / 4096 (kernel >= 5.4)
nginx default backlog:      511

So on modern Linux: min(511, 4096) = 511 slots

Who writes: Kernel TCP stack when handshake completes.
Who reads: Your process via accept() syscall.
What happens when full: Kernel drops incoming SYNs. Client sees connection timeout.

3.3 Socket Receive Buffer

Per-connection buffer in kernel RAM. Incoming payload bytes (from ISR) land here after the TCP header is stripped.

Default size: ~208 KB  (/proc/sys/net/core/rmem_default = 212992)
Maximum size: 128 MB   (/proc/sys/net/core/rmem_max = 134217728)
Auto-tuning:  Linux adjusts size dynamically between rmem_default and rmem_max

Who writes: Kernel ISR (on hardware interrupt).
Who reads: Your process via read() or recv() syscall.
What happens when full: Kernel stops sending TCP ACKs to the sender. Sender's TCP stack sees its window shrink to zero and stops sending. This is TCP flow control. No data loss — sender just waits.

3.4 Socket Send Buffer

Per-connection buffer in kernel RAM. When your process calls write(), bytes go here. Kernel TCP stack drains this buffer by constructing TCP packets and sending them out via the NIC.

Default size: ~208 KB  (/proc/sys/net/core/wmem_default = 212992)
Maximum size: 128 MB   (/proc/sys/net/core/wmem_max = 134217728)

Who writes: Your process via write() or send() syscall.
Who reads: Kernel TCP stack (constructs packets and hands to NIC).
What happens when full: write() blocks (in blocking mode) or returns EAGAIN (in non-blocking mode). nginx uses non-blocking — it registers EPOLLOUT on the fd and retries when kernel signals the buffer has drained.

3.5 nginx Worker Memory (User Space)

When the worker calls read(fd, buf, size), the kernel copies bytes from the socket recv buffer (kernel RAM) into buf (process RAM). This is the boundary crossing — from kernel space to user space.

The worker parses HTTP in this memory and builds request_context objects:

request_context {
  client_fd:   5           ← incoming connection fd
  backend_fd:  12          ← outgoing connection fd
  state:       WAITING_BACKEND
  method:      GET
  path:        /api/users
  headers:     { host, auth, content-type ... }
  body:        (if POST)
  response:    (filled later)
}

This is the only place that links a client fd to its backend fd. The kernel has no concept of this pairing.

4. epoll

4.1 What is epoll

epoll is a Linux kernel subsystem that lets a single process monitor thousands of fds simultaneously without scanning all of them. It is the engine behind nginx's ability to handle 10,000+ concurrent connections in a single thread.

Three kernel structures make up epoll:

4.2 epoll Instance

Created via epoll_create1(). Returns an fd that represents the epoll instance itself. The instance contains:

Interest List — which fds to watch and what events.
Ready List — which fds currently have events pending.

4.3 Interest List

A red-black tree (O(log n) insertion/deletion) inside the kernel. Each node represents one watched fd and what event to watch for.

Interest list (red-black tree in kernel RAM):
┌─────────────────────────────────┐
│ fd=3,  events=EPOLLIN           │  ← new client connections
│ fd=5,  events=EPOLLIN           │  ← User A client data
│ fd=6,  events=EPOLLIN           │  ← User B client data
│ fd=12, events=EPOLLIN|EPOLLOUT  │  ← backend for User A
│ fd=13, events=EPOLLIN|EPOLLOUT  │  ← backend for User B
└─────────────────────────────────┘

Modified by: epoll_ctl(epfd, EPOLL_CTL_ADD/MOD/DEL, fd, event) — always called by worker.

4.4 Ready List

A doubly-linked list inside the kernel. When the ISR marks an fd as having data, it adds that fd to this list. epoll_wait() reads and clears this list.

Written by: Kernel ISR (on hardware interrupt) — completely independent of worker state.
Read by: Worker via epoll_wait().

This separation is critical: the ready list is updated even when the worker is busy. Nothing is lost.

4.5 Edge Triggered vs Level Triggered

nginx uses Edge Triggered (EPOLLET).

Mode	epoll_wait returns fd when	Risk if you don't drain fully
Level Triggered (default)	fd has data (keeps returning until empty)	None — will re-notify
Edge Triggered (EPOLLET)	fd transitions from no-data → has-data	Will NOT re-notify until new data arrives

Edge triggered means nginx must loop read() until it gets EAGAIN on every wakeup. If it reads only once and goes back to epoll_wait, remaining data in the buffer will never trigger a new event and that connection stalls.

Same applies to accept() — nginx must loop accept() until EAGAIN to drain the entire accept queue every time fd=3 fires.

4.6 epoll Syscalls

Syscall	What it does
`epoll_create1(flags)`	Creates epoll instance, returns epfd
`epoll_ctl(epfd, op, fd, event)`	ADD/MOD/DEL an fd in the interest list
`epoll_wait(epfd, events[], maxevents, timeout)`	Sleep until events ready; returns array of ready fds

timeout = -1 means sleep indefinitely. The process is removed from the CPU scheduler run queue. Zero CPU consumed while sleeping.

5. nginx Startup Sequence

Master Process — Before Workers Start

nginx has two process roles: one master process and one or more worker processes.

Master process:

Runs as root.
Reads nginx.conf. Determines worker_processes count — if set to auto, nginx reads the number of CPU cores on the machine (nproc) and spawns that many workers.
Calls socket(), bind(), listen() on port 80 once, before forking. This is deliberate — binding requires root privileges. Workers never need to bind themselves.
Calls fork() once per worker. Each forked child inherits fd=3 (the listen socket) automatically. This is how all workers share the same port without any of them needing root.
After forking, master drops into a supervision loop — watches workers, restarts any that crash, handles signals (nginx -s reload, nginx -s stop).
Master itself never handles a single HTTP request.

After fork, each worker independently:

Calls epoll_create1() — gets its own epoll instance.
Calls epoll_ctl(ADD, fd=3, EPOLLIN) — registers the inherited listen socket.
Calls epoll_wait() — sleeps, waiting for connections.

Master Process (root)
  socket() + bind() + listen() on fd=3
        │
        ├── fork() → Worker 1 (inherits fd=3) → own epoll → epoll_wait
        ├── fork() → Worker 2 (inherits fd=3) → own epoll → epoll_wait
        ├── fork() → Worker 3 (inherits fd=3) → own epoll → epoll_wait
        └── fork() → Worker 4 (inherits fd=3) → own epoll → epoll_wait

All workers call accept() on the same fd=3.
Kernel distributes incoming connections across workers.
Master just watches. Restarts crashed workers.

When multiple workers call accept() on the same fd=3 simultaneously, the kernel ensures only one worker gets each connection — no duplication. nginx also sets accept_mutex (older versions) or relies on EPOLLEXCLUSIVE flag (modern Linux) to avoid all workers waking up for every single connection — known as the thundering herd problem.

This runs once when nginx starts. The worker does not handle any requests until this is complete.

1.  socket(AF_INET, SOCK_STREAM, 0)
    → kernel creates socket struct
    → returns fd=3 (listen socket)

2.  setsockopt(fd=3, SO_REUSEADDR, 1)
    → allows reuse of port 80 after restart
    → avoids "Address already in use" error

3.  bind(fd=3, {ip=0.0.0.0, port=80})
    → registers this socket as owner of port 80 with kernel
    → kernel now routes port 80 packets to fd=3

4.  listen(fd=3, backlog=511)
    → kernel creates SYN Queue and Accept Queue for this socket
    → accept queue limit = min(511, somaxconn)
    → kernel begins accepting TCP handshakes even before accept() is called

5.  epoll_create1(EPOLL_CLOEXEC)
    → creates epoll instance
    → returns epfd=4

6.  epoll_ctl(epfd=4, EPOLL_CTL_ADD, fd=3, {EPOLLIN|EPOLLET})
    → registers listen socket in interest list
    → "wake me up when a new connection is ready to accept"

7.  epoll_wait(epfd=4, events[], 512, -1)
    → worker sleeps. 0 CPU consumed. Waiting for first knock.

6. Complete Request Lifecycle

Cast

Symbol	What
`fd=3`	nginx listen socket
`fd=5`	User A ↔ nginx TCP connection
`fd=6`	User B ↔ nginx TCP connection
`fd=12`	nginx ↔ backend TCP connection for User A
`fd=13`	nginx ↔ backend TCP connection for User B
`epfd=4`	epoll instance

Phase 1 — TCP Handshake (Kernel Only, Worker Sleeping)

Who acts: Kernel TCP stack + NIC. Worker is untouched.

User A browser                    Kernel (port 80)
      │── SYN ──────────────────► NIC receives packet
      │                           ISR runs: puts in SYN Queue
      │◄── SYN-ACK ─────────────  Kernel sends SYN-ACK via NIC
      │── ACK ──────────────────► ISR runs: moves to Accept Queue
      │                           Kernel assigns fd=5 internally
      │                           Marks fd=3 as EPOLLIN in ready list

User B browser (simultaneously)
      │── SYN ──────────────────► Same process
      │◄── SYN-ACK ─────────────
      │── ACK ──────────────────► fd=6 assigned internally
                                  fd=3 still marked EPOLLIN

Buffers touched:

SYN Queue: written (User A SYN), read (on ACK received), written again (User B SYN)
Accept Queue: written twice (one entry per completed handshake)
NIC Ring Buffer: written by NIC DMA, read by ISR

Interrupts: Hardware IRQ fires per packet. ISR runs. Worker untouched.

Phase 2 — Worker Wakes, Accepts Both Connections

Syscalls: epoll_wait, accept, epoll_ctl, fcntl

epoll_wait(epfd=4, events[], 512, -1)
→ returns: [{fd=3, EPOLLIN}]
→ worker wakes up

Worker enters accept loop (must drain fully because EPOLLET):

Iteration 1:
  accept(fd=3)
  → kernel dequeues User A's connection from Accept Queue
  → creates fd=5 for this connection
  → returns fd=5

  fcntl(fd=5, F_SETFL, O_NONBLOCK)
  → sets fd=5 to non-blocking mode

  epoll_ctl(epfd=4, EPOLL_CTL_ADD, fd=5, {EPOLLIN|EPOLLET})
  → fd=5 added to interest list

Iteration 2:
  accept(fd=3)
  → dequeues User B's connection
  → returns fd=6

  fcntl(fd=6, F_SETFL, O_NONBLOCK)
  epoll_ctl(epfd=4, EPOLL_CTL_ADD, fd=6, {EPOLLIN|EPOLLET})

Iteration 3:
  accept(fd=3)
  → returns -1, errno=EAGAIN
  → Accept Queue is empty. Stop looping.

Buffers touched:

Accept Queue: read (two entries dequeued)

Interest list now:

fd=3  EPOLLIN  ← always watching for new clients
fd=5  EPOLLIN  ← User A
fd=6  EPOLLIN  ← User B

Worker calls epoll_wait again. Sleeps.

Phase 3 — HTTP Request Bytes Arrive

Who acts: NIC + Kernel ISR. Worker sleeping.

User A's browser sends GET /api/users HTTP/1.1\r\nHost: ...\r\n\r\n

NIC receives TCP segment
→ DMA writes to NIC Ring Buffer
→ NIC fires hardware IRQ
→ CPU saves state, jumps to ISR

ISR:
  reads packet from NIC Ring Buffer
  parses IP + TCP headers
  matches src_ip:src_port:dst_ip:dst_port → identifies fd=5's socket
  copies HTTP payload into fd=5's recv buffer (kernel RAM)
  updates TCP: ACK sent back to User A's browser
  marks fd=5 as EPOLLIN in epoll ready list
  checks: is worker sleeping in epoll_wait? YES → wakes worker
  returns

User B's GET /api/orders HTTP/1.1 arrives simultaneously or milliseconds later. Same ISR sequence for fd=6.

Buffers touched:

NIC Ring Buffer: written by DMA, read by ISR
fd=5 recv buffer: written by ISR (~hundreds of bytes, the HTTP request)
fd=6 recv buffer: written by ISR

Interrupts: One IRQ per packet per user. ISR runs in microseconds each time.

Phase 4 — Worker Reads and Parses Both Requests

Syscalls: epoll_wait, read

epoll_wait(epfd=4, events[], 512, -1)
→ returns: [{fd=5, EPOLLIN}, {fd=6, EPOLLIN}]
→ both events in one batch

Processing fd=5 (User A):

Worker enters read loop (must drain, EPOLLET):

  Iteration 1:
    read(fd=5, buf, 4096)
    → kernel copies bytes from fd=5 recv buffer (kernel RAM)
      into buf (worker process RAM)
    → returns n=312 (bytes read, size of HTTP request)

  Iteration 2:
    read(fd=5, buf, 4096)
    → returns -1, errno=EAGAIN
    → recv buffer empty. Stop.

Worker parses buf:
  method  = GET
  path    = /api/users
  headers = { Host: ..., Authorization: Bearer xyz }

Worker creates:
  request_context_A {
    client_fd  = 5
    backend_fd = -1       ← not yet
    state      = PARSED
    request    = "GET /api/users..."
  }

Processing fd=6 (User B): Same sequence. request_context_B created with client_fd=6.

Buffers touched:

fd=5 recv buffer: read by worker (now empty)
fd=6 recv buffer: read by worker (now empty)
Worker heap memory: written (two request_context objects)

Kernel boundary crossing: Every read() call copies bytes from kernel RAM → process RAM.

Phase 5 — Worker Opens Backend Connections

Syscalls: socket, fcntl, connect, epoll_ctl

For User A:

socket(AF_INET, SOCK_STREAM, 0)
→ kernel creates new socket struct
→ returns fd=12

fcntl(fd=12, F_SETFL, O_NONBLOCK)
→ makes fd=12 non-blocking

connect(fd=12, {backend_ip, port=8080})
→ kernel initiates TCP handshake with backend (sends SYN)
→ returns immediately with errno=EINPROGRESS
   (handshake happening asynchronously in kernel)

epoll_ctl(epfd=4, EPOLL_CTL_ADD, fd=12, {EPOLLOUT|EPOLLET})
→ "tell me when fd=12 is writable" = when handshake completes

request_context_A.backend_fd = 12
request_context_A.state      = CONNECTING_BACKEND

For User B: Same. Creates fd=13, connects, registers EPOLLOUT.

Interest list now:

fd=3   EPOLLIN          ← new clients
fd=5   EPOLLIN          ← User A client
fd=6   EPOLLIN          ← User B client
fd=12  EPOLLOUT         ← backend for User A (connecting)
fd=13  EPOLLOUT         ← backend for User B (connecting)

Worker calls epoll_wait. Sleeps. 5 fds watched. 0 CPU consumed.

Phase 6 — Backend TCP Handshakes Complete

Who acts: Kernel. Worker sleeping.

Backend sends SYN-ACK for User A's connect (fd=12)
→ NIC receives, IRQ fires, ISR runs
→ Kernel completes handshake, fd=12 state = ESTABLISHED
→ fd=12 is now writable
→ ISR marks fd=12 EPOLLOUT in ready list
→ Worker is sleeping → kernel wakes it

Same for User B (fd=13)

Phase 7 — Worker Forwards Requests to Backend

Syscalls: epoll_wait, write, epoll_ctl

epoll_wait returns: [{fd=12, EPOLLOUT}, {fd=13, EPOLLOUT}]

Handling fd=12 (User A):

Look up context: fd=12 → request_context_A → request = "GET /api/users..."

write(fd=12, "GET /api/users HTTP/1.1\r\n...", len)
→ kernel copies bytes from worker RAM → fd=12 send buffer (kernel RAM)
→ kernel TCP stack will send these bytes to backend as TCP packets
→ returns immediately (non-blocking write)

epoll_ctl(epfd=4, EPOLL_CTL_MOD, fd=12, {EPOLLIN|EPOLLET})
→ switch from watching EPOLLOUT to EPOLLIN
→ "now tell me when backend sends a response back"

request_context_A.state = WAITING_BACKEND

Handling fd=13 (User B): Same. Forwards GET /api/orders.

Buffers touched:

fd=12 send buffer: written by worker (the forwarded HTTP request)
fd=13 send buffer: written by worker
Kernel TCP stack reads send buffers, constructs packets, hands to NIC

Interrupt on outgoing side: When NIC finishes sending packets from send buffer, it fires an IRQ to tell kernel "send buffer slots are free now." No worker involvement.

Worker calls epoll_wait. Sleeps.

Phase 8 — Backend Responds (Possibly Out of Order)

Who acts: Kernel ISR. Worker sleeping.

Backend processes /api/orders faster. Sends response for User B first.

Backend TCP segment for User B arrives at NIC
→ DMA → NIC Ring Buffer
→ IRQ fires → ISR runs
→ Payload copied into fd=13 recv buffer
→ fd=13 marked EPOLLIN in ready list

Later:
Backend TCP segment for User A arrives
→ Same path → fd=12 recv buffer filled
→ fd=12 marked EPOLLIN in ready list

Worker was sleeping → kernel wakes it

Buffers touched:

NIC Ring Buffer: written by DMA, read by ISR
fd=13 recv buffer: written by ISR (User B response body)
fd=12 recv buffer: written by ISR (User A response body)

Phase 9 — Worker Reads Backend Response, Writes to Client

Syscalls: epoll_wait, read, write

epoll_wait returns: [{fd=13, EPOLLIN}, {fd=12, EPOLLIN}]

Note: User B's response came first even though User A connected first. epoll returns what is ready, not what came first.

Handling fd=13 (User B's backend response):

read loop on fd=13:
  read(fd=13, buf, 65536)
  → kernel copies backend response from fd=13 recv buffer → worker RAM
  → returns n bytes

  read(fd=13, buf, 65536)
  → returns EAGAIN (buffer empty, done)

Look up context: fd=13 → request_context_B → client_fd = 6

write(fd=6, buf, n)
→ kernel copies bytes from worker RAM → fd=6 send buffer
→ kernel TCP stack sends packets to User B's browser

epoll_ctl(epfd=4, EPOLL_CTL_DEL, fd=13, NULL)
→ remove backend fd from interest list

close(fd=13)
→ kernel tears down backend connection, frees socket struct and buffers

request_context_B.state = DONE

Handling fd=12 (User A): Same. Reads fd=12 → writes to fd=5 → closes fd=12.

Buffers touched:

fd=13 recv buffer: read by worker (emptied)
fd=12 recv buffer: read by worker (emptied)
fd=6 send buffer: written by worker (User B response going to browser)
fd=5 send buffer: written by worker (User A response going to browser)

Kernel does the rest: TCP stack drains send buffers → NIC → wire → browsers.

Phase 10 — Connection Teardown

Depends on: HTTP version and Connection header.

HTTP/1.0 or Connection: close:

close(fd=5)   → kernel sends FIN to User A, tears down connection, frees socket
close(fd=6)   → same for User B
epoll_ctl(epfd=4, EPOLL_CTL_DEL, fd=5, NULL)
epoll_ctl(epfd=4, EPOLL_CTL_DEL, fd=6, NULL)

HTTP/1.1 keep-alive (default):

fd=5 and fd=6 remain open and registered in epoll interest list
Worker goes back to epoll_wait
If same client sends another request → fd=5 fires EPOLLIN → handled immediately
Kernel has idle timers: if no data for N seconds → close

Worker returns to epoll_wait. Sleeping again. Ready for next event.

7. What Happens While Worker is Busy

Scenario: Worker is processing User A's request. At the same moment, Users C, D, E connect and send requests.

Worker processing [User A]
        │
        │  ← IRQ fires (User C SYN arrives)
        │     ISR runs in microseconds
        │     Kernel completes C's handshake
        │     Adds C to Accept Queue
        │     Marks fd=3 EPOLLIN in ready list
        │     Worker is NOT sleeping → no wakeup sent. That's fine.
        │
        │  ← IRQ fires (User D SYN arrives)
        │     Same. D added to Accept Queue. fd=3 already marked ready.
        │
        │  ← IRQ fires (User E data arrives on existing connection)
        │     Bytes go into fd=E's recv buffer. fd=E marked EPOLLIN.
        │
Worker finishes [User A]
        │
epoll_wait()
        │
→ returns: [{fd=3, EPOLLIN}, {fd=E, EPOLLIN}]
  (fd=3 fires once for all queued connections — edge triggered fires on transition)

Worker drains Accept Queue:
  accept() → User C → fd=7 → register with epoll
  accept() → User D → fd=8 → register with epoll
  accept() → EAGAIN → done

Worker reads fd=E
Worker calls epoll_wait. Sleeps.

Key principle: The ready list is written by ISR independently of worker state. The worker sees everything accumulated while it was busy, on its very next epoll_wait call.

8. All Buffers — Master Reference

#	Buffer	Location	Default Size	Written By	Read By	Full Behaviour
1	NIC Ring Buffer	NIC / kernel RAM (DMA region)	256–4096 packet slots	NIC hardware via DMA	Kernel ISR	Packets dropped, visible via ethtool
2	SYN Queue	Kernel RAM	~128–1024 slots	Kernel TCP stack (on SYN)	Kernel TCP stack (on ACK)	SYN drops, connection timeout
3	Accept Queue	Kernel RAM	min(backlog, somaxconn) ≈ 511	Kernel TCP stack (post-handshake)	Process via `accept()`	SYN drops, connection timeout
4	Socket Recv Buffer (client fd)	Kernel RAM	208 KB default, 128 MB max	Kernel ISR (per interrupt)	Process via `read()`	TCP flow control, sender throttled
5	Socket Send Buffer (client fd)	Kernel RAM	208 KB default, 128 MB max	Process via `write()`	Kernel TCP stack → NIC	`write()` returns EAGAIN (non-blocking)
6	Worker Heap (request_context)	Process RAM	Unbounded (heap)	Process (after `read()` + parse)	Process (during response handling)	OOM if too many large requests
7	Socket Recv Buffer (backend fd)	Kernel RAM	208 KB default, 128 MB max	Kernel ISR (backend response)	Process via `read()`	TCP flow control
8	Socket Send Buffer (backend fd)	Kernel RAM	208 KB default, 128 MB max	Process via `write()`	Kernel TCP stack → NIC	`write()` returns EAGAIN

Data movement order for one request:

NIC Ring Buffer
  → (ISR) → Socket Recv Buffer [client fd]
  → (read syscall) → Worker Heap
  → (write syscall) → Socket Send Buffer [backend fd]
  → (kernel TCP) → NIC → wire → backend

Backend response:
NIC Ring Buffer
  → (ISR) → Socket Recv Buffer [backend fd]
  → (read syscall) → Worker Heap
  → (write syscall) → Socket Send Buffer [client fd]
  → (kernel TCP) → NIC → wire → client browser

9. All Syscalls — Master Reference

Syscall	Called By	When	What Kernel Does
`socket()`	Worker	Startup + per backend conn	Allocates socket struct, returns fd
`setsockopt()`	Worker	Startup	Sets socket options (SO_REUSEADDR etc.)
`bind()`	Worker	Startup	Claims port 80 for this process
`listen()`	Worker	Startup	Creates SYN + Accept queues
`accept()`	Worker	On fd=3 EPOLLIN	Dequeues one entry from Accept Queue, returns new fd
`fcntl(F_SETFL, O_NONBLOCK)`	Worker	After accept/socket	Sets fd to non-blocking mode
`epoll_create1()`	Worker	Startup	Creates epoll instance
`epoll_ctl(ADD)`	Worker	After accept/socket	Adds fd to interest list
`epoll_ctl(MOD)`	Worker	After connect/read	Changes watched events (EPOLLIN ↔ EPOLLOUT)
`epoll_ctl(DEL)`	Worker	Before close	Removes fd from interest list
`epoll_wait()`	Worker	After each batch	Sleeps until events; returns ready fd list
`connect()`	Worker	Per upstream conn	Initiates TCP handshake (non-blocking, returns EINPROGRESS)
`read()`	Worker	On EPOLLIN	Copies bytes from socket recv buffer → process RAM
`write()`	Worker	After parsing/response	Copies bytes from process RAM → socket send buffer
`close()`	Worker	After response sent	Sends FIN, frees socket struct and all buffers

10. Failure Modes

10.1 Accept Queue Full

Cause: Connections arriving faster than worker calls accept().
Symptom: Client sees Connection timed out (kernel drops SYN silently).
Fix: Increase net.core.somaxconn, increase nginx backlog, add more workers.

10.2 Event Loop Starvation

Cause: Worker doing heavy CPU computation or blocking syscall (sync file I/O, blocking DB call) for one connection.
Symptom: All other connections queue up. Latency spikes for everyone.
Fix: Never block the event loop. Offload CPU-heavy work to thread pool (aio threads in nginx). Use async database drivers.

10.3 Send Buffer Full (Slow Client)

Cause: Client reading response slowly. fd=5 send buffer fills up.
Symptom: write(fd=5) returns EAGAIN. Nginx registers EPOLLOUT on fd=5 and comes back to write more when buffer drains.
Fix: nginx handles this internally. Tune send_timeout to close connections from very slow clients.

10.4 Recv Buffer Full (Slow Worker)

Cause: Data arriving faster than worker reads it.
Symptom: TCP flow control kicks in. Sender throttled. No data loss.
Fix: Usually not a problem. If persistent, check for event loop blocking.

10.5 fd Exhaustion

Cause: Too many open connections. Each costs one fd.
Symptom: accept() returns EMFILE (too many open files).
Fix: ulimit -n 65536 in OS. worker_connections 10240 in nginx config.

Quick Mental Model

INTERNET
   │ raw bytes
   ▼
NIC Ring Buffer  ← DMA (no CPU)
   │ IRQ fires
   ▼
Kernel ISR  ← runs in microseconds, interrupts everything
   │ copies bytes
   ▼
Socket Recv Buffer  ← kernel RAM, per fd
   │ waits here until worker is ready
   ▼
epoll Ready List  ← kernel marks fd ready
   │ wakes worker if sleeping
   ▼
Worker (epoll_wait returns)
   │ read() → copies to process RAM
   │ parse HTTP
   │ write() to backend fd → send buffer
   │
   ▼
Socket Send Buffer  ← kernel RAM, per fd
   │ kernel drains via TCP
   ▼
NIC → wire → backend

Backend response takes same path in reverse.
Worker links client fd ↔ backend fd via request_context in its own memory.
Kernel has no concept of this pairing.
Everything the worker does is non-blocking.
When waiting, worker is off the CPU run queue. Zero cycles consumed.

Bulkhead Pattern - Go

Kamal Namdeo — Sun, 14 Jun 2026 15:22:59 +0000

Go Bulkhead Pattern — Complete Reference

What is the Bulkhead Pattern
Three Mechanisms
HTTP Transport Parameters — Deep Dive
Little's Law — Sizing Semaphores Correctly
Full Production Implementation
Wiring It Up
Correctly Sized Real-World Configs
Mental Model Summary

1. What is the Bulkhead Pattern

Named after watertight compartments in ship hulls. If one compartment floods, the others stay dry.

In software: isolate resources per dependency so a failure or slowdown in one cannot exhaust shared resources and take down unrelated parts of the system.

Without bulkheads:

Auth service slows down
  → goroutines pile up waiting
  → shared thread/connection pool exhausted
  → Payment service also fails (never touched auth)
  → Entire service down

With bulkheads:

Auth service slows down
  → only auth's semaphore/pool fills up
  → auth calls get fast 503s
  → Payment service unaffected
  → Ship stays afloat

2. Three Mechanisms

Mechanism 1 — Semaphore (Concurrency Bulkhead)

Caps how many concurrent in-flight calls exist to a dependency at any moment.

sem := semaphore.NewWeighted(20) // max 20 concurrent calls

func call(ctx context.Context) error {
    // TryAcquire = instant rejection if full (true bulkhead)
    // Acquire    = waits up to QueueTimeout for a slot
    acquireCtx, cancel := context.WithTimeout(ctx, 50*time.Millisecond)
    defer cancel()

    if err := sem.Acquire(acquireCtx, 1); err != nil {
        return fmt.Errorf("bulkhead full: %w", err) // fast 503
    }
    defer sem.Release(1)

    // make the actual call
}

TryAcquire vs Acquire:

Method	Behaviour	Use when
`TryAcquire(1)`	Instant reject if full	Strict load shedding
`Acquire(ctx, 1)`	Wait up to ctx deadline	Brief queuing acceptable

Each dependency gets its own independent semaphore — they do not share state.

Mechanism 2 — Connection Pool (TCP Bulkhead)

Caps how many TCP connections exist to each host. Without this, all services share http.DefaultTransport and one slow host monopolises all connections.

transport := &http.Transport{
    MaxConnsPerHost:     20, // hard ceiling on total connections
    MaxIdleConnsPerHost: 20, // warm connections kept alive
    MaxIdleConns:        20, // global idle pool
}

Mechanism 3 — Timeouts (Blast Radius Bulkhead)

Even if concurrency is high, tight timeouts ensure slots are released quickly when a dependency misbehaves. This is your blast radius control knob.

client := &http.Client{
    Transport: transport,
    Timeout:   600 * time.Millisecond, // absolute end-to-end deadline
}

3. HTTP Transport Parameters — Deep Dive

transport := &http.Transport{
    MaxIdleConnsPerHost:   N,
    MaxConnsPerHost:       N,
    MaxIdleConns:          N,
    DialContext:           ...,
    TLSHandshakeTimeout:   ...,
    ResponseHeaderTimeout: ...,
    IdleConnTimeout:       ...,
}
client := &http.Client{
    Transport: transport,
    Timeout:   ...,
}

`MaxConnsPerHost` — Hard Ceiling

What:    Maximum total TCP connections (active + idle) to a single host.
         New requests BLOCK when this is hit (until a connection is freed or context times out).
Effect:  This IS the TCP-level bulkhead. Beyond this number, no new connections are opened.

How to set it: Match your semaphore's MaxConcurrent. There is no point allowing 50 concurrent calls if you only have 10 TCP connections — requests would block waiting for a free connection anyway.

MaxConnsPerHost: cfg.MaxConcurrent // keep these equal

`MaxIdleConnsPerHost` — Warm Connection Cache

What:    How many idle (reusable) connections to keep open to a single host after a request completes.
Effect:  Avoids TCP+TLS handshake overhead on the next request. Higher = faster reuse, more FDs held open.

How to set it:

Set equal to or slightly below MaxConnsPerHost
For internal services (fast, high RPS): match MaxConnsPerHost exactly — you want all connections warm
For external APIs (slow, low RPS): can be lower since connections are held for longer and fewer are needed warm simultaneously

MaxIdleConnsPerHost = MaxConnsPerHost       // internal, high-throughput
MaxIdleConnsPerHost = MaxConnsPerHost / 2   // external, slow/bursty

`MaxIdleConns` — Global Idle Pool

What:    Total idle connections across ALL hosts combined.
Effect:  If this is too low, connections for one host get evicted to make room for another,
         causing unexpected TCP reconnects even though per-host limits haven't been hit.

How to set it: Sum of all your per-host idle limits across all bulkheads in the process.

// If you have 3 bulkheads: auth(20) + payment(10) + inventory(15)
MaxIdleConns: 20 + 10 + 15, // = 45, avoids cross-host eviction

Common mistake: Leaving this at Go's default of 100 when MaxIdleConnsPerHost is also 100. The global cap then silently limits what you think are independent pools.

`ResponseHeaderTimeout` — First Byte Timeout

What:    Time allowed between sending the request body and receiving the first byte of response headers.
Effect:  The most important timeout for catching slow/hung servers.
         Does NOT include time to read the response body.

How to set it: Slightly above your p99 latency for this dependency. This is the "server is stuck" detector.

External API p99 = 500ms → ResponseHeaderTimeout = 600ms
Internal svc p99 = 5ms   → ResponseHeaderTimeout = 20ms
Database p99     = 20ms  → ResponseHeaderTimeout = 50ms

`http.Client.Timeout` — Absolute End-to-End Deadline

What:    Time from request initiation to reading the last byte of response body.
         Includes: DNS + TCP dial + TLS + write request + wait for headers + read body.
Effect:  The outer hard deadline. Cancels the entire request if exceeded.

How to set it: ResponseHeaderTimeout + estimated body read time. Always set this — without it you can leak goroutines indefinitely.

ResponseHeaderTimeout = 600ms  (detect slow server)
body read estimate    = 200ms  (for your expected payload size)
Client.Timeout        = 800ms  (slightly above their sum)

`DialContext.Timeout` — TCP Handshake Limit

What:    Time allowed to establish the TCP connection itself.
Effect:  Catches unreachable hosts fast. Completely independent of request latency.

How to set it: 1–3 seconds universally. TCP handshake should be near-instant on healthy networks; longer means routing problem, not slowness.

DialContext: (&net.Dialer{
    Timeout:   2 * time.Second,  // TCP dial hard limit
    KeepAlive: 30 * time.Second, // TCP keepalive probes
}).DialContext,

`TLSHandshakeTimeout` — TLS Negotiation Limit

What:    Time allowed for TLS handshake after TCP is established.
Effect:  Catches TLS issues (expired certs, overloaded TLS terminator) separately from request latency.

How to set it: 3–5 seconds. TLS handshake involves a round trip plus crypto; 5s is generous but safe.

TLSHandshakeTimeout: 5 * time.Second,

`IdleConnTimeout` — Stale Connection Eviction

What:    How long an idle connection is kept in the pool before being closed.
Effect:  Prevents holding open TCP connections that the server has already closed (common after 60–90s).

How to set it: 60–90 seconds. Most servers close idle connections at ~90s. Setting this below their limit prevents "connection reset" errors.

IdleConnTimeout: 90 * time.Second, // safe default

`QueueTimeout` — Semaphore Wait Budget

What:    How long a request waits for a semaphore slot before being rejected.
Effect:  Controls the queue depth in time. Even if a slot frees in 10ms, you might not want to wait.

How to set it: Based on your own SLA minus downstream latency.

Your SLA:              500ms total
Downstream p99:        200ms
Budget for queuing:    500 - 200 - 50(overhead) = 250ms
QueueTimeout:          ~100ms (conservative, leaves room for retries)

For slow external APIs: lower QueueTimeout (fail fast, don't queue) For fast internal services: even lower (if pool is full, something is wrong)

All Parameters at a Glance

Parameter	Controls	Rule of Thumb
`MaxConnsPerHost`	TCP connection ceiling	= `MaxConcurrent`
`MaxIdleConnsPerHost`	Warm connection cache	= `MaxConnsPerHost` (internal), ÷2 (external)
`MaxIdleConns`	Global idle pool	= sum of all per-host idle limits
`ResponseHeaderTimeout`	"Server is slow" detector	p99 latency × 1.2
`Client.Timeout`	Absolute end-to-end limit	`ResponseHeaderTimeout` + body read estimate
`DialContext.Timeout`	TCP dial limit	1–2s universally
`TLSHandshakeTimeout`	TLS negotiation limit	3–5s universally
`IdleConnTimeout`	Stale connection eviction	60–90s (below server's idle timeout)
`QueueTimeout`	Semaphore wait budget	Your SLA − p99 − overhead

4. Little's Law — Sizing Semaphores Correctly

L = λ × W

L = concurrent requests in-flight (what your semaphore controls)
λ = throughput (requests per second you want to sustain)
W = average time per request (latency in seconds)

Key Insight

Slower dependency = MORE semaphore slots needed to sustain the same RPS.

Target: 100 rps

Auth svc  p99=100ms → L = 100 × 0.1 = 10 concurrent  (fast, slots free quickly)
Payment   p99=500ms → L = 100 × 0.5 = 50 concurrent  (slow, slots held longer)

This is counterintuitive but correct: a slow service holds each semaphore slot for longer, so you need more slots to keep throughput flowing.

What Happens When Semaphore Is Too Small

Payment svc p99=500ms, target 100rps, but MaxConcurrent=10:

Max throughput = 10 / 0.5 = 20 rps   ← you're throttling yourself to 20%

What Happens When Semaphore Is Too Large

Blast radius grows: more goroutines can pile up waiting on a misbehaving dep.
Blast radius is controlled by TIMEOUT, not concurrency.

The Correct Formula

min_concurrent = target_rps × p99_latency_seconds
semaphore_limit = min_concurrent × 1.3   // +30% headroom for bursts

NumCPU() is Wrong for IO-Bound Bulkheads

runtime.NumCPU() is the correct pool size only for CPU-bound work (hashing, compression, image processing) where goroutines burn CPU the entire time.

For IO-bound calls (HTTP, DB, Kafka): goroutines spend ~99% of their time parked waiting on the network. The Go scheduler puts them to sleep and runs other goroutines on the same OS thread. You can sustain thousands of concurrent IO goroutines on a 4-core machine.

IO call timeline:
[send ~50µs] [====== blocked waiting ~200ms ======] [read ~50µs]
                        ↑
              goroutine is PARKED here, OS thread runs other work

→ semaphore limit is a downstream capacity question, not a CPU question

5. Full Production Implementation

package bulkhead

import (
    "context"
    "fmt"
    "net"
    "net/http"
    "sync/atomic"
    "time"

    "golang.org/x/sync/semaphore"
)

// Config defines the bulkhead parameters for one downstream dependency.
type Config struct {
    Name           string
    MaxConcurrent  int64         // semaphore size — derived from Little's Law: target_rps × p99_latency
    MaxConnections int           // TCP connection ceiling — keep equal to MaxConcurrent
    RequestTimeout time.Duration // absolute end-to-end HTTP deadline (blast radius knob)
    QueueTimeout   time.Duration // how long to wait for a semaphore slot before rejecting
}

// Metrics holds live counters. All fields use atomic ops — safe to read from any goroutine.
type Metrics struct {
    Accepted atomic.Int64 // requests that acquired a semaphore slot
    Rejected atomic.Int64 // requests rejected because bulkhead was full
    Errors   atomic.Int64 // requests that acquired a slot but the HTTP call failed
}

// Bulkhead combines a per-dependency semaphore and an isolated HTTP client.
// Each instance is independent — a saturated bulkhead has zero effect on others.
type Bulkhead struct {
    cfg     Config
    sem     *semaphore.Weighted // Gate 1: concurrency cap
    client  *http.Client        // Gate 2: connection pool cap + timeouts
    Metrics Metrics
}

// New creates a Bulkhead. Call once per downstream dependency at startup.
func New(cfg Config) *Bulkhead {
    transport := &http.Transport{
        // --- Gate 2: TCP-level bulkhead ---
        MaxConnsPerHost:     cfg.MaxConnections,        // hard ceiling: new reqs block when hit
        MaxIdleConnsPerHost: cfg.MaxConnections,        // warm connections: avoids TCP+TLS overhead
        MaxIdleConns:        cfg.MaxConnections,        // global pool: set >= sum of all per-host

        // --- TCP dial ---
        DialContext: (&net.Dialer{
            Timeout:   2 * time.Second,  // TCP handshake hard limit (unreachable host detection)
            KeepAlive: 30 * time.Second, // TCP keepalive probe interval
        }).DialContext,

        // --- TLS ---
        TLSHandshakeTimeout: 5 * time.Second, // TLS negotiation limit

        // --- Request lifecycle ---
        // ResponseHeaderTimeout: time between sending request and receiving first response byte.
        // This is the "server is hung" detector. Set to p99 × 1.2.
        ResponseHeaderTimeout: cfg.RequestTimeout,

        // Evict idle connections before the server closes them (~90s on most servers).
        IdleConnTimeout: 90 * time.Second,
    }

    return &Bulkhead{
        cfg: cfg,
        sem: semaphore.NewWeighted(cfg.MaxConcurrent),
        client: &http.Client{
            Transport: transport,
            // Absolute deadline covering DNS+TCP+TLS+write+headers+body.
            // Always set — without this, goroutines can leak indefinitely.
            Timeout: cfg.RequestTimeout,
        },
    }
}

// Do executes an HTTP request through the bulkhead.
// Returns an error immediately if the semaphore is full (after QueueTimeout).
func (b *Bulkhead) Do(ctx context.Context, req *http.Request) (*http.Response, error) {
    // --- Gate 1: Semaphore ---
    // Use a child context with QueueTimeout so we don't wait forever for a slot.
    acquireCtx, cancel := context.WithTimeout(ctx, b.cfg.QueueTimeout)
    defer cancel()

    if err := b.sem.Acquire(acquireCtx, 1); err != nil {
        b.Metrics.Rejected.Add(1)
        return nil, fmt.Errorf("bulkhead [%s] rejected (semaphore full after %s): %w",
            b.cfg.Name, b.cfg.QueueTimeout, err)
    }
    defer b.sem.Release(1) // always release, even on HTTP error

    b.Metrics.Accepted.Add(1)

    // --- Gate 2: HTTP client (connection pool + timeouts) ---
    resp, err := b.client.Do(req.WithContext(ctx))
    if err != nil {
        b.Metrics.Errors.Add(1)
        return nil, fmt.Errorf("bulkhead [%s] call failed: %w", b.cfg.Name, err)
    }

    return resp, nil
}

// Stats returns a human-readable snapshot of metrics.
func (b *Bulkhead) Stats() string {
    return fmt.Sprintf("[%s] accepted=%d rejected=%d errors=%d",
        b.cfg.Name,
        b.Metrics.Accepted.Load(),
        b.Metrics.Rejected.Load(),
        b.Metrics.Errors.Load(),
    )
}

Why a Separate Semaphore per Dependency is Correct

authBulkhead.sem   ──► controls only auth calls
paymentBulkhead.sem ──► controls only payment calls

Auth saturated:
  authBulkhead.sem full → auth calls rejected
  paymentBulkhead.sem unaffected → payment calls proceed normally

A shared semaphore across all dependencies defeats the purpose entirely.

6. Wiring It Up

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "time"

    "yourmodule/bulkhead"
)

// Declare one bulkhead per downstream dependency at package level.
// These are long-lived, safe to use concurrently.
var (
    authBulkhead    = bulkhead.New(authConfig())
    paymentBulkhead = bulkhead.New(paymentConfig())
)

func HandleOrder(ctx context.Context) error {
    // Auth call — isolated behind its own semaphore + connection pool
    req, _ := http.NewRequest("GET", "https://auth.internal/verify", nil)
    resp, err := authBulkhead.Do(ctx, req)
    if err != nil {
        // Bulkhead full or auth down — fail this request fast.
        // Payment bulkhead is completely unaffected.
        return fmt.Errorf("auth unavailable: %w", err)
    }
    defer resp.Body.Close()
    io.Copy(io.Discard, resp.Body) // always drain body to return connection to pool

    // Payment call — completely isolated from auth
    req2, _ := http.NewRequest("POST", "https://payments.internal/charge", nil)
    resp2, err := paymentBulkhead.Do(ctx, req2)
    if err != nil {
        return fmt.Errorf("payment unavailable: %w", err)
    }
    defer resp2.Body.Close()
    io.Copy(io.Discard, resp2.Body)

    return nil
}

Important: Always io.Copy(io.Discard, resp.Body) before closing. If you close without draining, Go cannot reuse the TCP connection — it gets thrown away, causing unnecessary TCP handshakes on the next request.

7. Correctly Sized Real-World Configs

Sizing Formula

min_concurrent  = target_rps × p99_latency_seconds
semaphore_limit = min_concurrent × 1.3   // +30% burst headroom
MaxConnections  = semaphore_limit        // keep aligned
RequestTimeout  = p99_latency × 1.2     // slightly above p99, blast radius knob
QueueTimeout    = your_SLA - p99 - overhead_budget

External HTTP API (slow, p99=500ms, target 30 rps)

// Little's Law: 30 × 0.5 = 15, +30% = 20
// Blast radius: controlled by tight RequestTimeout (600ms), NOT by low concurrency
// Slower dep → more slots needed to sustain throughput
func externalAPIConfig() bulkhead.Config {
    return bulkhead.Config{
        Name:           "external-auth-api",
        MaxConcurrent:  20,                       // L = 30rps × 0.5s = 15, +30% = 20
        MaxConnections: 20,
        RequestTimeout: 600 * time.Millisecond,   // p99(500ms) × 1.2 — blast radius knob
        QueueTimeout:   100 * time.Millisecond,   // don't queue long for slow dep
    }
}

Internal Microservice (fast, p99=5ms, target 500 rps)

// Little's Law: 500 × 0.005 = 2.5 → floor at ~10 (connection overhead)
// Faster dep → naturally fewer slots needed, slots free in milliseconds
func internalServiceConfig() bulkhead.Config {
    return bulkhead.Config{
        Name:           "inventory-internal",
        MaxConcurrent:  10,                       // L = 500rps × 0.005s = 2.5, floor at 10
        MaxConnections: 10,
        RequestTimeout: 20 * time.Millisecond,    // tight: it's internal, same DC
        QueueTimeout:   5 * time.Millisecond,     // fail fast: if pool busy, something is wrong
    }
}

PostgreSQL (p99=20ms, target 200 rps, 5 app instances)

// Little's Law: 200 × 0.02 = 4 per instance
// Hard constraint: Postgres max_connections=100, shared across 5 instances → 20 per instance
// Use: max(Little's Law result × 1.3, safety floor) but never exceed hard cap
func postgresConfig() bulkhead.Config {
    return bulkhead.Config{
        Name:           "postgres",
        MaxConcurrent:  10,                       // L=4, ×1.3=5.2, safety floor=10, hard cap=20 ✓
        MaxConnections: 10,
        RequestTimeout: 100 * time.Millisecond,   // DB should be fast; slow DB = problem
        QueueTimeout:   20 * time.Millisecond,
    }
}

Kafka Producer (p99=2ms, target 2000 rps)

// Little's Law: 2000 × 0.002 = 4 concurrent
// Kafka batches internally — keep TCP connections low, semaphore has headroom for bursts
func kafkaConfig() bulkhead.Config {
    return bulkhead.Config{
        Name:           "kafka-producer",
        MaxConcurrent:  10,                       // L=4, headroom for burst
        MaxConnections: 3,                        // Kafka multiplexes internally; few TCP conns enough
        RequestTimeout: 50 * time.Millisecond,
        QueueTimeout:   5 * time.Millisecond,
    }
}

Summary Table

Dependency	p99	Target RPS	L=λ×W	+30%	MaxConcurrent	RequestTimeout	QueueTimeout
External API	500ms	30	15	20	20	600ms	100ms
Internal svc	5ms	500	2.5	4*	10*	20ms	5ms
Postgres	20ms	200	4	5**	10**	100ms	20ms
Kafka	2ms	2000	4	5	10	50ms	5ms

floor applied: connection establishment overhead makes sub-10 impractical ** floor applied, hard cap from Postgres max_connections / num_instances

8. Mental Model Summary

Two Laws, Two Knobs

Little's Law          → sets MaxConcurrent
  slow dep = MORE slots (slots held longer)
  fast dep = FEWER slots (slots released quickly)

Blast Radius Control  → sets RequestTimeout + QueueTimeout
  NOT concurrency — tight timeouts release slots fast
  even if MaxConcurrent is high, a 600ms timeout limits damage

The Failure Isolation Guarantee

Normal:                         Auth saturated:

User → Auth    ✓               User → Auth    ✗ (semaphore full → instant 503)
User → Payment ✓               User → Payment ✓ (own semaphore, unaffected)
User → S3      ✓               User → S3      ✓ (own semaphore, unaffected)

What Each Layer Protects Against

Layer                   | Protects against
────────────────────────┼──────────────────────────────────────────
Semaphore               | Goroutine pile-up from slow/hung deps
Connection pool         | TCP connection exhaustion from one host
ResponseHeaderTimeout   | Server that accepts connection but never responds
Client.Timeout          | Goroutine leak from infinite response body reads
QueueTimeout            | Cascading slowdowns from queue buildup

Bulkhead vs Circuit Breaker

These are complementary:

Pattern	Question it answers	Action
Bulkhead	"How much capacity do I allocate?"	Limits concurrent slots per dep
Circuit Breaker	"Should I even try this call?"	Opens when error rate exceeds threshold

Use both together: bulkhead limits blast radius, circuit breaker stops calling a dep that's known to be down.

Reference covers: golang.org/x/sync/semaphore, net/http.Transport, Little's Law (L=λW), Go scheduler IO parking, per-dependency isolation.

Note

So the minimal complete config for a single-host (not shared amont multiple deps) bulkhead transport is:

transport := &http.Transport{
    MaxConnsPerHost:     20,               // bulkhead ceiling
    MaxIdleConnsPerHost: 20,               // full connection reuse
    IdleConnTimeout:     90 * time.Second, // evict before server does

    DialContext: (&net.Dialer{
        Timeout:   2 * time.Second,
        KeepAlive: 30 * time.Second,
    }).DialContext,
    TLSHandshakeTimeout:   5 * time.Second,
    ResponseHeaderTimeout: cfg.RequestTimeout,
}

Everything else (MaxIdleConns, MaxConnsPerHost without PerHost suffix) is either irrelevant for single-host transports or has safe defaults for this use case.

False Sharing: The Silent Performance Killer in Concurrent Go

Kamal Namdeo — Sat, 11 Apr 2026 13:20:19 +0000

False Sharing: The Silent Performance Killer in Concurrent Go

Your goroutines never touch each other's data. Your atomics are correct. Your mutexes are in place. And yet your program slows down as you add more cores. Here is why — and how to fix it.

In this article, I would go through

The hardware reality: why memory access is never just one variable
What a cache line actually is
False sharing: correctness without performance
Three patterns that silently kill your throughput
The fix: padding and struct layout
Cache line sizes across architectures
How the Go standard library handles this
Rules to wire into your design instincts

THE HARDWARE REALITY MOST DEVELOPERS IGNORE

When you write a concurrent Go program, you think in variables: goroutine A writes to hits, goroutine B writes to misses. They are different variables. There is no data race. Your atomics are correct. Everything should be fine.

But your program gets slower as you add more cores. You profile it. There is no lock contention. No goroutine is blocked. The CPU utilisation looks high. And yet, throughput does not scale.

The bug is not in your code. It is in your mental model of how memory works.

The CPU does not fetch individual variables. It fetches 64-byte chunks of memory. Always.

This single fact — which most developers never internalise — is the root cause of an entire class of concurrency bugs called false sharing. Understanding it changes how you design every concurrent data structure you will ever write.

WHAT A CACHE LINE ACTUALLY IS

Modern CPUs have a memory hierarchy. Reading from main RAM costs roughly 100–300 clock cycles. Reading from L1 cache costs 3–5 cycles. This 60–100x difference is why caches exist — and why programs that use them well run dramatically faster than programs that do not.

But the cache does not work at the byte level. It works at the cache line level. A cache line is a 64-byte contiguous chunk of memory. Whenever the CPU needs any byte, it loads the entire 64-byte chunk that contains it into cache. You never get just the byte you asked for; you always get the whole 64-byte neighborhood.

RAM (simplified view):

Address:  0    8    16   24   32   40   48   56   64   72 ...
          |    |    |    |    |    |    |    |    |    |
          [────────────── cache line 1 ──────────────][── cache line 2 ──...
                         (64 bytes)                        (64 bytes)

Each core gets its own independent copy of any cache line it needs. If Core 1 and Core 2 both need data from the same 64-byte block, they both hold a copy. Now the hardware has a problem to solve: what happens when those copies diverge?

The coherence protocol

CPUs solve this with a cache coherence protocol (MESI is the most common variant). The rule is simple and absolute: when any core modifies a cache line, every other core's copy of that cache line is immediately marked invalid. Any core that subsequently tries to read or write that cache line must discard its invalidated copy and fetch a fresh one — from main memory, at full main-memory cost (100–300 cycles).

FALSE SHARING: WHEN INDEPENDENCE IS AN ILLUSION

False sharing occurs when two goroutines running on different cores are writing to different variables — logically unrelated variables — but those variables happen to sit within the same 64-byte cache line.

The defining characteristic of false sharing: your code is correct, your synchronisation is right, and you are still paying main-memory prices because of how your data happens to be laid out in RAM.

THREE PATTERNS THAT SILENTLY KILL YOUR THROUGHPUT

1. A slice of counters written by N goroutines

You have a slice of integers, one per goroutine, and each goroutine only ever writes to its own index.

counters := make([]int64, 8)

for i := 0; i < 8; i++ {
    go func(i int) {
        for {
            counters[i]++  // only touches index i
        }
    }(i)
}

In memory, eight int64 elements (8 bytes each) sit contiguously:

Address:  0        8        16       24       32       40       48       56
          [ c[0] ][ c[1] ][ c[2] ][ c[3] ][ c[4] ][ c[5] ][ c[6] ][ c[7] ]
          |────────────────────────────────────────────────────────────────|
                          one single cache line (64 bytes)

Every increment of c[0] by Core 0 invalidates the cache line for all seven other cores, because of the same copy of the cache line onto all eight cores, and changes by one core invalidate all other copies. Eight cores are now bouncing a single cache line through main memory like a hot potato.

2. A mutex and the data it protects on the same cache line

type SafeCounter struct {
    mu    sync.Mutex  // 8 bytes at offset 0
    value int64       // 8 bytes at offset 8
}

The act of contending for a lock invalidates the very cache line that holds the protected data. The goroutine that just won the lock pays a main memory penalty (100+ cycles) just to read the data it just unlocked.

Here is the step-by-step breakdown of the performance collapse:

Core 1 wants the lock: It executes an instruction to acquire mu. To do this, it pulls the cache line containing mu into its own L1 cache and marks it as Modified.
Core 2 tries to get the lock: It looks for mu. It sees that Core 1 owns the cache line. Core 2’s hardware must now wait for Core 1 to "give up" that line.
Core 1 updates the value: Since value is on the same cache line that Core 1 already "owns" for the mutex, this part is fast—initially.
Core 1 releases the lock: It writes to mu to unlock it.
Core 2 finally gets the line: The hardware intercepts.
Core 2’s request. Because Core 1 modified that line, the hardware invalidates Core 1's copy and forces the data to be synchronised and fetched from RAM.

This is the "Silent Killer" part: When Core 2 finally successfully acquires the mutex, it thinks it's ready to work. But because the entire 64-byte line was marked invalid during the "handover" from Core 1, the value variable is also gone from Core 2's cache.

Even though Core 2 now "owns" the lock, the very first time it tries to read SafeCounter.value, it hits a Cache Miss. It has to reach all the way out to the slow Main RAM to get the current value, wasting roughly 100–300 clock cycles.
The Result

You are paying a "RAM Tax" twice:

Once to fight for the Mutex.
Once to read the Data the mutex was supposed to protect.

3. Atomic counters declared side by side

var (
    hits   atomic.Int64   // offset 0
    misses atomic.Int64   // offset 8
)

atomic.Int64 guarantees correctness, but it says nothing about cache lines. The atomic instruction itself takes nanoseconds; the cache miss that precedes it takes 100+ cycles.

Let's understand with an example -

As shown in the snippet above, the vars hits and misses are on the same cache line.

Core 1 is responsible for incrementing hits.
Core 2 is responsible for incrementing misses.

Even though these two variables are logically unrelated, they are physically roommates. When Core 1 performs an atomic increment on hits:

It must lock the entire cache line to ensure atomicity.
It increments the value and marks the cache line as Modified (M).

Because the line is modified, Core 2’s cache copy of that entire 64-byte block is instantly marked Invalid (I). Now, Core 2 wants to increment misses.

Core 2 looks in its cache. It sees the line containing misses is Invalid.
The Penalty: Core 2 must wait for the hardware to fetch the updated line from Core 1 (or main memory). This takes ~100–300 cycles.
Core 2 performs its atomic increment, which in turn invalidates Core 1’s copy.

They are now "ping-ponging" the cache line back and forth across the CPU bus.

THE FIX: PADDING AND STRUCT LAYOUT

The solution is to force hot fields onto separate cache lines using padding.

Fix 1: Padded slice elements

type PaddedInt struct {
    Val int64
    _   [56]byte  // 8 + 56 = 64 bytes total = one full cache line
}

counters := make([]PaddedInt, 8)

Fix 2: Padding between mutex and protected data

type SafeCounter struct {
    mu    sync.Mutex
    _     [56]byte  // pad mu to its own cache line
    value int64
    _     [56]byte  // pad value to its own cache line
}

Field order is load-bearing

Go lays out struct fields in declaration order. Padding must go between the fields you want to isolate.

// CORRECT: padding acts as a wall between variables, hits and misses

type Good struct {
    hits     atomic.Int64
    _        [56]byte   // wall after hits
    misses   atomic.Int64
    _        [56]byte   // wall after misses
}

CACHE LINE SIZES ACROSS ARCHITECTURES

Architecture	Cache line size	Notes
x86-64 (Intel, AMD)	64 bytes	Standard server hardware
ARM64 (AWS Graviton)	64 bytes	Cloud ARM servers
Apple M1/M2/M3	64 / 128 bytes	L2 uses 128-byte chunks
IBM POWER9/10	128 bytes	128-byte lines throughout

Pro Tip: For portable code, pad to 128 bytes. It wastes a bit of RAM but ensures safety across all modern high-performance architectures.

HOW THE GO STANDARD LIBRARY HANDLES THIS

The Go team uses this sparingly but effectively.

runtime.p: The processor struct is explicitly padded to prevent adjacent p structs in the internal array from invalidating each other's run queues.
sync.Pool: The poolChain struct pads its head and tail pointers. Since producers write to the head and consumers read from the tail, padding prevents these two roles from fighting over the same cache line.

RULES TO WIRE INTO YOUR DESIGN INSTINCTS

Slices written by N goroutines: Use a padded wrapper struct.
Embedded Mutexes: Pad the mutex if the protected data is "hot."
Atomic Counters: Pad them apart if they are logically independent.
Scaling issues: If performance degrades as you add cores, suspect false sharing and profile cache miss rates.
Field Order: Remember that padding is a wall, not a footer. Place it physically between the fields you need to separate.

In distributed systems, you think about network round trips. In concurrent code, the equivalent cost is: will this memory access hit L1 cache, or main memory?

Getting this right is what separates code that scales from code that merely passes the race detector.

DEV Community: Kamal Namdeo

SLSA, Sigstore and Provenance — Complete Practical Guide

SLSA + Sigstore + Provenance — Complete Mental Model

1. The problem this whole ecosystem solves

2. SLSA — Supply-chain Levels for Software Artifacts

What provenance actually looks like (in-toto format)

3. Sigstore — the signing infrastructure

3a. Fulcio — the Certificate Authority for "keyless signing"

3b. Rekor — the transparency log

3c. Cosign — the CLI that ties it together

3d. cosign sign vs cosign attest — what's actually different

4. How it all fits together — end to end flow

5. Hands-on: do the whole thing locally

5a. Local registry for practice

5b. Key-pair signing (offline mode — no Fulcio, simplest to grok first)

5c. Keyless signing (the real Sigstore flow — needs internet + browser/OIDC)

5d. Look at the actual Rekor entry

5e. Attach a SLSA provenance attestation

5f. Attach your SBOM the same way (ties directly to your day job)

6. Where GitHub Actions automates all of this

7. The mental model to keep forever

Strangler Fig Pattern Implementation - Part 2

Strangler Fig Data Migration: Orders Service Extraction

Phase 1: Dual Writes (Synchronization begins)

Phase 2: Shadow Traffic / Verification (Read-path validation)

Phase 3: Cutover of Writes (the actual dangerous part)

Phase 4: Handling In-Flight Writes During Cutover

Phase 5: Read Cutover

Phase 6: Decommission

Summary of the sync topology over time

Master the Strangler Fig Pattern - Part 1

Stop Doing "Big Bang" Rewrites: Master the Strangler Fig Pattern

Why Big-Bang Rewrites Fail

What is the Strangler Fig Pattern?

How it Works (Step-by-Step)

The Architectural Mechanics

1. The Facade Layer (Routing)

2. The Data Synchronization Challenge

Where Do You Start?

When to Use (and When to Avoid) Strangler Fig

Conclusion

The Thundering Herd Problem (Cache Stampede) — and How to Tame It

1. What Is the Thundering Herd Problem?

2. The Solution Strategy: Locking (a.k.a. "Cache Mutex" / Request Coalescing)

3. Walking Through the Code

4. Why This Works

5. Trade-offs and Failure Modes to Be Aware Of

6. Related/Alternative Strategies

Nginx Event Loop — Complete Lifecycle Reference

nginx Event Loop — Complete Lifecycle Reference

Table of Contents

1. Foundations

1.1 Everything is a File

1.2 File Descriptor (fd)

1.3 Socket

1.4 How fd and Socket Relate

2. Hardware Layer

2.1 NIC (Network Interface Card)

2.2 DMA (Direct Memory Access)

2.3 NIC Ring Buffer

2.4 Hardware Interrupt (IRQ — Interrupt Request)

2.5 ISR (Interrupt Service Routine)

3. Kernel Structures and All Buffers

3.1 SYN Queue (Incomplete Connections Queue)

3.2 Accept Queue (Complete Connections Queue)

3.3 Socket Receive Buffer

3.4 Socket Send Buffer

3.5 nginx Worker Memory (User Space)

4. epoll

4.1 What is epoll

4.2 epoll Instance

4.3 Interest List

4.4 Ready List

4.5 Edge Triggered vs Level Triggered

4.6 epoll Syscalls

5. nginx Startup Sequence

Master Process — Before Workers Start

6. Complete Request Lifecycle

Cast

Phase 1 — TCP Handshake (Kernel Only, Worker Sleeping)

3d. `cosign sign` vs `cosign attest` — what's actually different

`MaxConnsPerHost` — Hard Ceiling

`MaxIdleConnsPerHost` — Warm Connection Cache

`MaxIdleConns` — Global Idle Pool

`ResponseHeaderTimeout` — First Byte Timeout

`http.Client.Timeout` — Absolute End-to-End Deadline

`DialContext.Timeout` — TCP Handshake Limit

`TLSHandshakeTimeout` — TLS Negotiation Limit

`IdleConnTimeout` — Stale Connection Eviction

`QueueTimeout` — Semaphore Wait Budget