DEV Community: Oleksandr Kuryzhev

Kubectl Alias Pitfalls You Should Fix Before They Bite

Oleksandr Kuryzhev — Sun, 06 Sep 2026 07:04:44 +0000

Originally published on kuryzhev.cloud

Context: how kubectl alias pitfalls creep into daily workflows

An engineer running one cluster, one AWS account, and one Terraform workspace rarely notices anything wrong with a short alias like alias k=kubectl. The problems around kubectl alias pitfalls show up later, once that same engineer is context-switching between staging, production, and a client's isolated namespace multiple times an hour.

The root cause isn't a bug. kubectl, the AWS and gcloud CLIs, and Terraform workspaces all default to whatever state was last set, not what the operator currently has in mind. kubectl config current-context reflects the last use-context call, not the cluster the reader assumes they're targeting. AWS CLI profile resolution follows a fixed order — --profile flag, then AWS_PROFILE env var, then the [default] entry in ~/.aws/config — and that resolution happens silently, per the official AWS CLI configuration documentation.

Tools like kubectx, kubens, aws-vault, and direnv exist specifically because this default behavior gets error-prone once the number of environments grows. They don't fix the underlying design; they add friction and visibility back into a workflow that convenience aliases had stripped away.

What follows are three well-documented failure patterns tied to context-switching shortcuts, plus a pattern that reduces exposure without banning aliases outright. None of this assumes a specific incident happened — it's a survey of how these tools are known to behave under normal, heavy multi-environment use.

Common failure 1 — the silent context carryover

Kubeconfig state persists across terminal sessions in a way that's easy to forget. If a previous session ran kubectl config use-context or a kubectx switch and never reverted it, the next terminal — even a brand-new one — inherits that context by default. This follows from kubectl's config merge behavior: ~/.kube/config is read fresh on every invocation, but it reflects whatever was last written to disk, not what any particular session "remembers."

Aliases compound this. alias k=kubectl is convenient precisely because it removes friction, but that friction is often the moment someone would have paused to check kubectl config current-context before running apply or delete. Typing the full command occasionally triggers a second thought; the two-letter alias rarely does.

Multi-terminal and tmux-heavy workflows make this worse. Each pane or split reads the same ~/.kube/config file, so switching context in one pane silently changes what every other pane targets next. Verify current state before any destructive verb with:

kubectl config current-context
kubectl config get-contexts

These two commands are the documented way to confirm state — not something a reader is expected to intuit from the shell prompt alone.

Common failure 2 — destructive aliases without guardrails

Some aliases go further than saving keystrokes; they bake in flags that bypass confirmation entirely. A pattern like alias tfd='terraform destroy -auto-approve' or alias kdel='kubectl delete' removes exactly the pause that manual typing, tab-completion, or a plan review would otherwise provide.

Docker's cleanup commands follow the same logic and are frequently misunderstood. docker system prune -a --volumes removes all unused containers, networks, images, and volumes on the host — not just the ones tied to the current project — according to the official Docker CLI reference. An alias that runs this without the operator reading the confirmation prompt can quietly wipe images that another project on the same machine depends on.

Watch out for aliases that hardcode --force, -auto-approve, or -y. Tool authors added those confirmation steps deliberately; baking the bypass into a shortcut disables a safety check without the person running the command necessarily remembering it's gone.

There's a second trap here that's easy to miss even for people who know better: bash expands aliases while it parses a script, before the rest of the file — including a function definition with the same name — is ever evaluated. Defining kdel() { ... } right after alias kdel='kubectl delete' in the same shell session doesn't override the alias the way you'd expect; the alias can still shadow the function depending on how and when the shell reads it. The alias has to be removed first. A safer version of a delete wrapper looks like this:


# Example of how "convenience" aliases erase safety checks
# Documented CLI behavior — flags below bypass built-in confirmation prompts

# Risky: hardcoded auto-approve, no context echo
alias tfd='terraform destroy -auto-approve'

# Risky: kubectl alias with no context check
alias kdel='kubectl delete'

# Bash expands aliases at parse time, so a function defined with the
# same name won't reliably shadow it unless the alias is cleared first
unalias kdel 2>/dev/null

# Safer: wrapper that forces visibility before destructive action
kdel() {
  echo "Target context: $(kubectl config current-context)"
  read -p "Continue delete? [y/N] " confirm
  [[ "$confirm" == "y" ]] && kubectl delete "$@"
}

# Safer: explicit workspace/profile scoping instead of relying on global state
terraform -chdir=envs/staging destroy   # no auto-approve, explicit path
AWS_PROFILE=staging-readonly aws ec2 describe-instances --region us-east-1

The wrapper doesn't remove the alias's convenience for read operations — it just refuses to let a destructive verb run without showing where it's about to run.

Common failure 3 — profile/region mismatch in cloud CLIs

The same class of kubectl alias pitfalls shows up in AWS and GCP tooling, often with a larger blast radius because cloud accounts, not just clusters, are involved. AWS CLI falls back to the [default] profile or a previously exported AWS_PROFILE env var. In a long-lived shell — one left open for days across a laptop's sleep cycles — a stale exported variable silently redirects every subsequent command to the wrong account.

gcloud config set project behaves differently from a per-command flag: it mutates a persistent local configuration file, not a session-scoped variable. Unlike --project, which applies only to one invocation, a forgotten set call from an earlier task keeps affecting every gcloud command run afterward until someone explicitly changes it back.

There's a cost dimension worth naming honestly, too. Running bulk list or describe operations against the wrong account or region — iterating over EC2 instances across every region in a script, for example — can inflate API request volume. For most read APIs this has no direct billing impact, but for metered services or accounts with request-based throttling, unexpected volume against the wrong target is at minimum a monitoring and quota nuisance, and in some services carries real cost.

Watch out for assuming the shell prompt reflects the current profile or project. Without an explicit indicator plugin installed, most default prompts show nothing about cloud context at all — the operator is relying on memory, which is exactly the failure mode context-switching tools were built to eliminate.

Safer operating pattern for kubectl alias pitfalls

None of this requires abandoning aliases. It requires putting visibility and friction back where destructive verbs are involved, while keeping shortcuts for anything read-only.

Context-aware shell prompts — starship, kube-ps1, or running commands through aws-vault exec — surface the active cluster, profile, or account continuously instead of requiring a manual check. This turns an invisible default into something visible on every prompt line. One caveat worth flagging: these prompts show the context and namespace, not the RBAC scope behind them. A context can be perfectly valid and still allow a delete in a namespace nobody meant to touch, if the role bound to that context is broader than it needs to be.

Prefer explicit per-command scoping over persistent global state, especially for destructive operations: --context for kubectl, --profile for AWS CLI, -chdir for Terraform. These flags override whatever was left over from a previous task, which matters most exactly when memory of "what did I set earlier" is least reliable.

Reserve short aliases for inspection commands — get, describe, plan, logs — and keep destructive verbs typed in full or wrapped in a script that echoes the target and asks for confirmation, as shown above. direnv or per-project .envrc files can also auto-scope AWS_PROFILE and KUBECONFIG per directory, reducing cross-project bleed without any manual step.

Scoping IAM policies and Kubernetes RBAC tightly to specific contexts — least-privilege roles rather than broadly-permissioned defaults — limits the damage even when a command is accidentally aimed at the wrong target. That's the real backstop for the caveat above: prompts and aliases reduce how often you make the mistake, but tight RBAC and IAM scoping determine how much a mistake actually costs. For more on scoping cloud credentials safely, see the DevOps_DayS archive on IAM and access patterns.

Before running anything destructive from muscle memory, a short checklist catches most of the failure modes above:


Quick decision checklist before running a destructive shortcut:

[ ] Does my prompt/plugin show current context, profile, or workspace?
[ ] Did I just switch contexts/profiles in this shell session?
[ ] Does this alias contain -auto-approve, --force, or -y baked in?
[ ] Is this command scoped with --context/--profile/-chdir, or relying on defaults?
[ ] Would a dry-run (--dry-run=client, terraform plan) catch a mismatch first?

If any box is unchecked → type the full command manually.

Postmortems that trace back to "wrong cluster" or "wrong account" mistakes usually point to stale terminal state rather than a tool defect. If you're setting up guardrails today, start with the two cheapest fixes: strip -auto-approve/--force/-y out of every alias you already have, and install a context-aware prompt plugin before adding anything more elaborate. Everything else in this pattern — wrapper scripts, direnv scoping, tighter RBAC — is worth doing, but those two changes alone close most of the gap for the least effort. Full documentation on kubectl configuration behavior is available in the Kubernetes CLI reference for readers building their own guardrail scripts.

Bedrock IAM Least Privilege: Stopping Runaway Bot Costs

Oleksandr Kuryzhev — Sat, 05 Sep 2026 07:03:41 +0000

Originally published on kuryzhev.cloud

The 3 a.m. Bedrock Bill

Bedrock IAM least privilege sounds like a compliance checkbox until a CI job proves otherwise. A common setup: a pull-request summarizer or changelog bot runs on every push, calling bedrock:InvokeModel through a shared IAM role someone created months ago "to get the demo working." Nobody has touched the policy since.

The role's trust and permission policy use "Resource": "*" and "Action": "bedrock:*". That's fine for a demo. It becomes a liability the day a malformed prompt or a broken rate-limit backoff sends the pipeline into a retry loop — hundreds of invocations in a short window, often against the most expensive model tier the account has access to, because nothing in the policy stops the call from reaching for it.

There's no per-model spending guardrail and no CloudTrail alert wired to Bedrock data events. The first signal isn't a failed pipeline step — the job might even report success on retry. The first signal is an AWS Budgets email a day or two later, or the invoice itself. By then the retry loop has already run its course, and reconstructing what happened means digging through CloudTrail logs that, in a lot of accounts, were never configured to capture InvokeModel calls at all.

This isn't an edge case someone got unlucky with. It's the predictable outcome of a permission model scoped for "make it work" and never revisited for "safe to leave running unattended."

Why Bedrock IAM Ends Up Over-Permissioned

This pattern repeats across teams for structural reasons, not because any one engineer got careless. The Bedrock quickstart documentation and the console's "test invoke" wizard both default to broad bedrock:* on Resource: "*". Scoping to a specific model ARN or inference profile is an extra step most getting-started guides skip entirely, so the copy-paste path leads straight to a wildcard.

CI and bot roles also tend to get provisioned once and reused indefinitely. A role created for one project quietly becomes "the AI role" for three others. As new models or regions get enabled on the account, nobody circles back to narrow the original policy — permission creep happens through convenience, not through any single bad decision.

The third factor is credential hygiene. Many pipelines still authenticate to AWS with a long-lived IAM user's access key pair stored as a CI secret, instead of short-lived OIDC federation. A static key has no built-in expiry and no binding to a specific repository, branch, or job. If that key leaks — through a misconfigured fork PR, a logging bug, or a compromised dependency — the blast radius is whatever the role can do, indefinitely, until someone notices and rotates it by hand.

None of these three shortcuts looks dangerous on its own. Stacked together they produce a role with wildcard model access, no regional boundary, and a credential that never expires — exactly the setup a runaway retry loop needs to turn into a five-figure bill.

Scoping the Policy — Model, Region, and Identity

The fix has three parts: scope the actions and resources explicitly, constrain region and identity with condition keys, and replace static keys with short-lived OIDC-federated credentials.

Start by restricting the Bedrock actions to only what the bot needs — typically bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream — and point Resource at an explicit foundation-model ARN instead of a wildcard. Model ARNs follow the pattern arn:aws:bedrock:<region>::foundation-model/<model-id>. If the application uses cross-region inference profiles rather than direct model invocation, those carry a different ARN shape (arn:aws:bedrock:<region>:<account-id>:inference-profile/<profile-id>) — a policy written only for direct model ARNs will silently deny, or fall back in unexpected ways, if the app later switches to profiles. Verify with the Bedrock inference profile documentation which ARN format the SDK call actually targets before you write the policy.

Layer in condition keys: aws:RequestedRegion blocks cross-region invocation when Bedrock is enabled in more than one region on the account, and aws:PrincipalTag binds the policy to a specifically tagged bot identity rather than any role that happens to look similar. Pair this with GitHub or GitLab OIDC federation so the CI job assumes a role scoped to a specific repo, branch, and job — which removes long-lived keys from CI secrets entirely.

The Terraform below defines a GitHub Actions bot role trusted only for pushes to main, with an inline policy scoped to one model family, one region, and one principal tag.


# Terraform: least-privilege Bedrock invoke role for a GitHub Actions bot
resource "aws_iam_role" "bedrock_ci_bot" {
  name = "ci-bedrock-summarizer-bot"

  # OIDC federation trust — scoped to one repo, one branch, one job
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = { Federated = aws_iam_openid_connect_provider.github.arn }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:sub" = "repo:org/repo:ref:refs/heads/main"
        }
      }
    }]
  })

  tags = { Purpose = "bedrock-ci" }
}

resource "aws_iam_role_policy" "bedrock_invoke_scoped" {
  name = "bedrock-invoke-scoped"
  role = aws_iam_role.bedrock_ci_bot.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ]
      # Model-family scoped ARN, not an account-wide wildcard
      Resource = "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-*"
      # Both keys belong in ONE StringEquals block — a second block with
      # the same operator name silently overwrites the first in jsonencode
      Condition = {
        StringEquals = {
          "aws:RequestedRegion"        = "us-east-1"
          "aws:PrincipalTag/Purpose"   = "bedrock-ci"
        }
      }
    }]
  })
}

Before attaching a policy like this to a production role, dry-run it. aws iam simulate-principal-policy checks whether specific actions and resources resolve to allow or deny without granting anything, which makes it usable as a policy-as-code test step in CI.


# Dry-run the policy before attaching it — catches accidental over-scope
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/ci-bedrock-summarizer-bot \
  --action-names bedrock:InvokeModel bedrock:ListFoundationModels \
  --resource-arns "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-*" \
  --region us-east-1

# Expect: InvokeModel -> allowed, ListFoundationModels -> implicitDeny
# If ListFoundationModels comes back "allowed," the policy is broader than intended.

Watch out for one specific overreach: don't grant model-access management actions like bedrock:PutFoundationModelEntitlement to CI roles. Those control which models an account can reach at all, and belong to human admin roles, not automated pipelines — a bot role that can grant itself access to new models effectively has no ceiling left to hit.

Prevention Checklist

Scoping one policy fixes one bot. Auditing every existing Bedrock role against a checklist is what stops the next retry-loop bill before it starts. Each item below closes a specific way permission creep or credential exposure re-enters an account over time.

One role per bot or pipeline. A shared "AI automation" role across projects means a bug in one pipeline spends budget attributed to an unrelated one, and makes the blast radius of a leak impossible to scope.
Deny-by-default on high-cost model tiers. Require an explicit allow plus a matching tag for anything above a defined cost class, so a retry loop can't silently escalate to the priciest model available.
OIDC trust scoped to repo, branch, and job. No static access keys in CI secrets — a token leaked from a fork PR shouldn't be able to assume a production-scoped role.
CloudTrail data events enabled for bedrock:InvokeModel*. Management events alone won't log invocation calls. Pair the log with a metric filter and alarm, not just a cost-based Budgets alert that fires days late.
Quarterly policy simulation. Run IAM Access Analyzer or aws iam simulate-principal-policy against every bot role to catch drift as new models or regions get enabled.
Permission boundaries on bot roles. A boundary caps what the role can ever be granted, even if someone later attaches a broader inline policy by mistake.
Bedrock Guardrails as a second, independent layer. IAM scoping controls what the bot can call; Guardrails controls what it can generate. Scoped IAM alone does nothing against prompt injection or unsafe output, so treat the two controls as separate problems.

None of this requires exotic tooling — most items are a Terraform diff and a CloudTrail trail away. What actually prevents the 3 a.m. bill is treating Bedrock IAM least privilege as a recurring audit rather than a one-time setup step, especially in accounts where new models and regions get switched on faster than anyone remembers to revisit the roles that reach them. For broader patterns on locking down AWS automation identities, see the related guidance on kuryzhev.cloud.

3 boto3 and argparse Mistakes That Break Python AWS Scripts

Oleksandr Kuryzhev — Fri, 04 Sep 2026 07:04:15 +0000

Originally published on kuryzhev.cloud

Context

A Python AWS script built on boto3 argparse logging seems trivial to write correctly — until it runs against the wrong account, exits with a traceback CI can't parse, or writes signed credentials to a log file. This trio has become the de-facto skeleton for internal automation: snapshot cleanup jobs, tag auditors, cost reports, one-off migration scripts. Each library "just works" out of the box, which is exactly why the defaults are dangerous.

All three fail silently in different ways. botocore resolves region and credentials through a chain that doesn't always match what the operator expects. argparse exits the process via SystemExit instead of raising a normal exception, which breaks naive error handling. logging.basicConfig() is a no-op if a handler already exists on the root logger, so a "fix" to increase verbosity can be silently ignored.

None of this is exotic. It's documented behavior in the argparse docs and the boto3 credentials guide. The failure patterns below come from reading how these libraries are built, not from a specific outage — but they recur often enough in internal tooling to be worth documenting as a checklist rather than tribal knowledge.

Common failure: wrong account, wrong region, no warning

boto3 resolves credentials through a chain: environment variables, then shared config/profile files, then container or instance metadata. If a script assumes AWS_PROFILE is always set and doesn't validate it, an empty environment falls through to whatever identity is available next — which might be a different account than the one intended.

Region resolution is stricter than credential resolution, but it can still surprise. boto3.client() raises NoRegionError immediately at construction time if no region can be resolved from an explicit argument, an environment variable, or the shared config file — for services that require a region. The catch is that this failure surfaces wherever the client happens to get built, which in a larger script can be buried inside a helper function called conditionally, well after argument parsing and logging setup have already run and looked fine.

A common mistake pattern: a script accepts --profile as optional with a "sensible default," meant to save typing during testing. Months later, that same default silently points at a production account when someone forgets to pass the flag from a cron job or CI runner. Deregistering AMIs or deleting snapshots against the wrong account isn't a boto3 bug — it's the documented credential chain doing exactly what it's designed to do, just not what the operator expected.

Watch out for: instance and container metadata credentials silently taking priority in environments where AWS_PROFILE isn't explicitly exported, especially inside containers that inherit an IAM task role.

Common failure: argparse errors that vanish into broad except blocks

When parser.parse_args() encounters invalid input, it doesn't raise a normal exception — it calls sys.exit(2), which raises SystemExit. This is a subclass of BaseException, not Exception. A try/except Exception: block wrapped around the whole entrypoint won't catch it, which is correct behavior. But scripts that go one step further and wrap everything in try/except BaseException: "for clean output" will swallow it too — masking real usage errors and turning a clear --help message or validation exit into a silently continuing script.

A second, quieter issue: argparse's type=int or type=str only coerce type, not business rules. A negative --retention-days value or an empty --resource-id string passes parsing cleanly and fails much later, deep inside a boto3 call, with a stack trace that has nothing to do with the actual root cause.

Verify with python script.py --help (exit code 0) and then with a deliberately invalid flag (exit code 2) — in both cases, argparse's SystemExit should propagate untouched, not be caught anywhere in the code path. Custom validation logic for value ranges belongs in a post-parse check or a custom type= callable, not left implicit.

Common failure: logging that hides the problem or leaks it

logging.basicConfig() only takes effect if the root logger has no handlers attached yet — a widely-cited gotcha in the Python logging docs. If any other imported module has already called basicConfig() or attached its own handler to the root logger, a later call in the entrypoint script is silently ignored. This means a developer trying to raise the log level to DEBUG for troubleshooting sees no change at all, and assumes the bug they're chasing doesn't produce log output.

The opposite failure is more serious. Setting the root logger to DEBUG to "see what's happening" also enables botocore's internal debug logging, which in some SDK versions includes full request and response bodies — including signed headers and, in specific cases, temporary session credentials. Verbose CI logs are a documented and common accidental credential-leak vector; this is worth checking against the boto3 logging documentation for the SDK version in use.

A third pattern shows up as multiple modules each calling logging.getLogger(__name__), each attaching its own handler, with propagate=True left at its default. The result is duplicated log lines for every event, which makes troubleshooting slower rather than faster — exactly the opposite of the intended effect.

Safer operating pattern

The fix for all three failure classes is the same shape: make defaults explicit, and validate identity before mutation. Construct the session directly with required --profile and --region flags, and call sts.get_caller_identity() immediately afterward — it's a cheap, read-only call that confirms account and region before anything destructive runs.


# main.py — pattern for combining boto3 + argparse + logging safely
import argparse
import logging
import sys
import boto3
from botocore.exceptions import ClientError, NoRegionError

def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description="Snapshot cleanup utility")
    parser.add_argument("--profile", required=True, help="AWS profile to use")
    parser.add_argument("--region", required=True, help="AWS region, e.g. us-east-1")
    parser.add_argument("--retention-days", type=int, default=30)
    parser.add_argument("--dry-run", action="store_true")
    parser.add_argument("--verbose", action="store_true", help="Enable debug logging for this script's logger only")
    return parser

def configure_logging(verbose: bool) -> logging.Logger:
    logger = logging.getLogger("cleanup_tool")  # named logger, not root
    handler = logging.StreamHandler()
    handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(message)s"))
    logger.addHandler(handler)
    logger.setLevel(logging.DEBUG if verbose else logging.INFO)
    logger.propagate = False  # avoid duplicate lines via root logger
    # Keep the SDK quiet even if our app logger is DEBUG
    logging.getLogger("botocore").setLevel(logging.WARNING)
    logging.getLogger("boto3").setLevel(logging.WARNING)
    return logger

def main() -> int:
    args = build_parser().parse_args()  # SystemExit here is intentional, don't swallow it
    log = configure_logging(verbose=args.verbose)

    session = boto3.Session(profile_name=args.profile, region_name=args.region)
    try:
        identity = session.client("sts").get_caller_identity()
        log.info("Running as %s in %s", identity["Arn"], args.region)
    except NoRegionError:
        log.error("Region not resolved — check --region or profile config")
        return 1
    except ClientError as e:
        log.error("Failed to verify identity: %s", e)
        return 1

    if args.retention_days < 1:
        log.error("--retention-days must be >= 1")
        return 1

    # ... cleanup logic using session.client("ec2") once, reused, not per-loop ...
    return 0

if __name__ == "__main__":
    sys.exit(main())

For tools with more than one action, argparse's add_subparsers() scales better than accumulating mutually exclusive flags. Explicit sys.exit(1) on every handled failure path — rather than relying on an unhandled traceback for a nonzero exit code — is what lets a CI pipeline or cron wrapper reliably detect failure. Testing boto3 calls with botocore.stub.Stubber or the moto library catches parameter and logic errors before a script ever touches real infrastructure.

A checklist form of this pattern is easier to apply consistently across a team than a single reference script:


Pre-flight checklist before writing another boto3 CLI script:

[ ] --profile / --region are explicit CLI args, not silently defaulted
[ ] sts.get_caller_identity() called once, logged, before any mutating action
[ ] argparse validation errors are never caught by a broad except block
[ ] custom type= or post-parse checks cover value ranges, not just types
[ ] logging configured once, in main(), on a named logger (not root)
[ ] botocore/boto3 loggers explicitly capped at WARNING even in verbose mode
[ ] one boto3 client per resource type, created once, reused across calls
[ ] AWS_RETRY_MODE / max_attempts set explicitly, not left on legacy defaults
[ ] sys.exit(1) on every handled failure path for CI/cron detection
[ ] no credentials passed as CLI args or printed in logs

Two smaller details are worth adding to that list. First, creating a new boto3 client inside a loop instead of reusing one instance adds per-call connection setup overhead that's easy to miss in a short script but adds up at scale. Second, switching retry mode from the legacy default to standard or adaptive via AWS_RETRY_MODE improves resilience against throttling, but without an explicit max_attempts cap it can multiply API calls — and cost — during sustained throttling events. Details vary by botocore version, so verify current behavior against the botocore retries documentation before relying on it in production tooling.

None of these boto3 argparse logging fixes require a framework or a rewrite — they're a handful of explicit checks added to a skeleton that most internal AWS scripts already share. For broader patterns on structuring AWS automation and CI pipelines, see the DevOps_DayS archive.

How We Learned to Harden Jenkins Agents: 3 Mistakes That Cost Us

Oleksandr Kuryzhev — Wed, 02 Sep 2026 07:01:47 +0000

Originally published on kuryzhev.cloud

Context: agents are the attack surface, not the controller

We had to harden Jenkins agents after a routine security review turned into something less routine. A pull request from an external contributor triggered a build that, for a brief moment, had a live network path to an internal metadata endpoint it had no business touching. Nothing was exfiltrated. But it was close enough that we stopped what we were doing and audited everything.

For years our mental model was simple: the Jenkins controller has RBAC, matrix-based security, folder permissions — so the controller is the thing we lock down. Agents were "just workers." They ran the build, printed some logs, and got recycled. We assumed that whatever restrictions we set on the controller UI somehow extended down to what code running on an agent could actually do at the OS level.

That assumption is wrong, and it's a common one. Agents execute arbitrary code — your own pipelines, sure, but also third-party fork-PR builds, community plugins pulled during a build step, and whatever a compromised dependency decides to do at build time. If you don't treat that execution environment as hostile by default, you're one bad Jenkinsfile away from a real incident.

What follows is what our audit found, not a theoretical checklist we wrote from a blog post. Three mistakes, each embarrassing in hindsight, and what we changed after.

Mistake 1: We treated agents as trusted extensions of the controller

Our agents connected over standard JNLP/Remoting, and we never restricted what Groovy code running on an agent could call back into the controller JVM. In practice this meant a compromised or malicious build step had a plausible path toward controller-side objects — credentials store included — depending on which plugins were loaded and what libraries a pipeline pulled in.

The bigger problem was label reuse. We had a single pool of agents labeled linux-docker that served both our internal, trusted repositories and external fork-PR builds. Same credentials binding scope, same agent images, same everything. A label in Jenkins is just a scheduling hint — it is not an isolation boundary unless you build one underneath it.

So untrusted code from a stranger's pull request was scheduled onto the same agent pool that had access to deploy credentials for internal services. It never got exploited that we know of, but the exposure was there the entire time, and nobody had explicitly decided to accept that risk. It was just an accident of convenience — one pool was easier to maintain than two.

The fix direction we landed on (details below) was splitting trust tiers before touching anything else: internal builds and PR builds needed to stop sharing infrastructure entirely, not just share it with tighter permissions.

Mistake 2: We mounted docker.sock for build convenience

This one stings the most because we knew better and did it anyway. Several build jobs needed to build and push Docker images, and the fastest way to get Docker-in-Docker working was mounting /var/run/docker.sock into the agent container. It worked immediately. It also meant any build running on that agent had a straightforward path to root on the host node — mounting the socket is functionally equivalent to giving the container root access to the underlying machine.

We didn't get hit by this through an incident. A security review flagged it, and once we understood the blast radius, it was hard to unsee. A single compromised build dependency — a malicious npm package, a poisoned base image, anything running arbitrary code during the build — could pivot from "container" to "node" with almost no extra effort.

Making it worse: our agent containers ran as root by default, with no dropped capabilities and no seccomp or AppArmor profile applied. There was nothing standing between a build process and full control of the container, and from there, the host.

Watch out for this specifically: it's incredibly common in Jenkins-on-Kubernetes setups because docker.sock mounts are the first result in every "how to build Docker images in Jenkins" tutorial. Treat any docker.sock mount on a build node as a critical finding, not a shortcut, per the guidance in Docker's own security documentation.

Mistake 3: We kept long-lived "pet" agents around

Our agent fleet was a mix of static VMs that had been running for months, sometimes years. They were provisioned once from a base image and then patched in place — or, more honestly, patched whenever someone remembered to. Over time they drifted hard from whatever the "declared" image was supposed to look like: manually installed CLI tools, leftover SDK versions, cached credentials from debugging sessions nobody cleaned up.

Workspace hygiene was inconsistent too. Some older pipelines never called cleanWs() at the end of a job, assuming the next build would just overwrite whatever was there. It mostly did — except when it didn't. We found at least one case where a previous job's temporary credentials file survived in the workspace and showed up, unmasked, in a completely unrelated job's build artifacts. Masked console output doesn't help if the secret was written to disk instead of printed to the log.

Patching cadence for these static agents lagged for months at a time because ownership was fuzzy. Nobody's job description said "rebuild the Jenkins agent AMI," so it just didn't happen until a review forced the question. That's the real cost of pet infrastructure — not that it's insecure on day one, but that it quietly gets more insecure every week nobody touches it.

What we do differently now

The single biggest architectural change was moving to ephemeral, pod-per-build agents using the Kubernetes plugin. Every build gets a fresh pod, and the pod is destroyed the moment the build finishes. No leftover workspace, no drifted image, no cached credentials from three jobs ago.

Here's roughly what a hardened pod template looks like for our fork-PR pool today:


# Hardened Kubernetes pod template for a Jenkins build agent
apiVersion: v1
kind: Pod
metadata:
  labels:
    jenkins/agent-pool: "pr-untrusted"   # separate pool from internal builds
spec:
  automountServiceAccountToken: false     # no implicit cluster API access
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: build
      image: registry.internal/jenkins-agent:2026-week12   # rebuilt weekly
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
      resources:
        limits:
          cpu: "2"
          memory: "4Gi"
      volumeMounts:
        - name: workspace
          mountPath: /home/jenkins/agent
        # deliberately no docker.sock, no hostPath volumes
  volumes:
    - name: workspace
      emptyDir: {}   # wiped automatically when pod is destroyed post-build
  restartPolicy: Never

Internal and fork-PR builds now run in entirely separate agent pools, enforced through Kubernetes namespaces with a restricted Pod Security Standard profile — no privileged containers, no hostPath, no hostNetwork, full stop. Fork-PR builds get zero long-lived cloud credentials by default; anything they need is a short-lived OIDC or Vault token scoped tightly to that specific job and expired within minutes.

We also cut off unrestricted internet access from build agents with a NetworkPolicy that only allows egress to our package registries and internal artifact store. If a compromised dependency tries to phone home during a build, it has nowhere to go. Agent images are rebuilt on a fixed weekly schedule instead of patched in place, and the Script Console is locked down to a two-person admin group — agents should never need controller-side script execution rights in the first place.

Yes, ephemeral pods add cold-start latency — image pulls and pod scheduling aren't free, and some teams notice the extra 20-40 seconds per build. For anything that touches secrets or third-party code, I think that tradeoff is an easy call. We wrote up a broader take on similar rollout tradeoffs in our DevOps notes on kuryzhev.cloud if you want more context on how we sequenced this migration without blocking releases.

Here's the checklist we now run through before onboarding any new Jenkins agent pool:


Agent hardening checklist (post-incident version):

[ ] No docker.sock or hostPath mounts on any build pod
[ ] Separate agent pools for internal vs. fork-PR / external builds
[ ] Fork-PR builds get zero long-lived cloud credentials
[ ] runAsNonRoot + dropped capabilities on every pod template
[ ] cleanWs() (or pod destruction) guaranteed after every build
[ ] Egress NetworkPolicy limits agents to registries + artifact store
[ ] Agent images rebuilt on a fixed schedule, not patched in place
[ ] Script Console / Groovy approval restricted to a small admin group
[ ] Secrets injected as short-lived tokens (Vault/OIDC), not static creds
[ ] Quarterly review of which labels/pools can reach which credentials

None of this is exotic. It's the same trust-boundary thinking you'd apply to any multi-tenant compute — we just hadn't applied it to Jenkins agents because we'd mentally filed them under "internal tooling" instead of "runs arbitrary third-party code." If you run fork-PR builds, or share agent pools across teams with different trust levels, it's worth doing this audit before something forces you to.

Rolling Out Kubernetes Pod Security Standards Without Breaking Prod

Oleksandr Kuryzhev — Tue, 01 Sep 2026 07:01:36 +0000

Originally published on kuryzhev.cloud

Someone on the platform team wrote a script that labeled every namespace with enforce: restricted in one PR, merged it on a Friday, and by Monday half the deploy pipelines were failing with cryptic admission errors. No pods were down — nothing was running yet — but nothing new could ship either. That's the story behind most botched Kubernetes Pod Security Standards rollouts: the policy itself is fine, the sequencing is the problem.

Pod Security Standards (PSS), enforced through the Pod Security Admission (PSA) controller, is one of those Kubernetes features that looks trivial in the docs and then bites you in production because nobody read past the "apply this label" example. Let's go through what it actually checks, where teams get the rollout wrong, and how to do it without a 2am incident.

What Pod Security Standards actually does

PSS defines three built-in profiles — privileged, baseline, and restricted — that describe increasingly strict constraints on pod specs. These aren't runtime rules. There's no sidecar watching your containers, no daemon inspecting syscalls. It's pure admission-time validation inside kube-apiserver, stable and built-in since Kubernetes 1.25, replacing the deprecated PodSecurityPolicy. Nothing to install, nothing to upgrade separately — it ships with the control plane.

Each namespace gets three independent mode labels: enforce, audit, and warn. Each can point to a different profile and even a different pinned version. This independence is the entire rollout lever most teams ignore — you can run warn: restricted for weeks without ever blocking a single deploy, just to see what would break.

What it does not do matters just as much. PSS has zero effect on RBAC, network policy, or image provenance. It doesn't scan running containers for drift. And critically — it only evaluates pods on create or update. A pod that was already running when you tightened the policy stays exactly as it was, non-compliant or not, until the next rollout touches it. That gap is where "we're compliant" turns into "we thought we were compliant."

How people use it wrong

The most common failure is skipping straight to enforcement. A bulk label sweep that sets enforce: restricted across every namespace, with no prior audit/warn signal, breaks CI/CD the moment any Helm chart sets privileged: true, runs as root, or simply omits seccompProfile. And a lot of upstream charts still do exactly that. You find out during a deploy, not during planning.

Second mistake: treating exemptions as permanent. exemptions.usernames and exemptions.runtimeClasses exist for legitimate edge cases — but teams add an exemption to unblock a deploy under pressure and never revisit it. Six months later nobody remembers why it's there, and the policy has quietly stopped protecting the thing it was meant to protect. Every exemption should come with a written reason and an expiry date, full stop.

Third: assuming a label change retroactively fixes existing pods. It doesn't. Watch out — if you flip a namespace to enforce: restricted and nothing in the audit log complains, that just means no new pod was created yet. Old ReplicaSets can sit there indefinitely, non-compliant, invisible to the policy until the next rollout or node reschedule triggers a re-check. Teams report "we're secure now" based on the label existing, not on actual pod state.

The correct approach

Start every namespace in observation mode: enforce at baseline (or unset), audit and warn at restricted. This lets policy violations surface in events and audit logs without blocking anything. Run it for at least one full release cycle before touching enforce.

# Safe rollout pattern: audit + warn first, enforce stays permissive
# until violations are reviewed. Apply per-namespace, not cluster-wide.
apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    # Pin the version during migration so cluster upgrades don't
    # silently change what "restricted" means mid-rollout.
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/enforce-version: v1.31
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.31
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.31
---
# Minimal pod spec that actually passes "restricted" —
# use this as the checklist when fixing violating workloads.
apiVersion: v1
kind: Pod
metadata:
  name: compliant-example
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      image: registry.internal/app:stable
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
        readOnlyRootFilesystem: true

Pin enforce-version explicitly during migration instead of leaving it at latest. If you leave it floating and the cluster gets upgraded mid-rollout, the definition of "restricted" can shift under you — new checks get added between minor versions, and a workload that passed last week can suddenly fail after a control plane bump nobody on your team even noticed.

Fix workloads at the source rather than exempting them. Drop hostNetwork, add seccompProfile: RuntimeDefault, set a non-root runAsUser, drop all capabilities and re-add only what's needed (usually nothing, sometimes NET_BIND_SERVICE). This is more work than adding a namespace to an exemption list, but it's the difference between a policy that does something and one that's theater.

Advanced patterns

PSS gives you three fixed profiles. It doesn't know your org requires images from an internal registry, or that every deployment needs a team label, or that resource limits are mandatory. For that, layer a policy engine — Kyverno or OPA/Gatekeeper — on top. Keep the division clean: PSS handles the universal security baseline, the policy engine handles business rules. Don't reimplement "no privileged containers" in Kyverno when PSS already does it for free.

Some workloads are legitimately privileged — CNI plugins, CSI storage drivers, certain monitoring agents that need hostPath or host networking. Don't handle these with broad cluster-wide exemptions. Isolate them into a dedicated namespace labeled enforce: privileged, and lock down who can create pods there with RBAC scoped to that namespace only. The exemption becomes a wall, not a hole.

For GitOps-managed clusters, the real risk is new namespaces appearing without any PSS labels at all — someone self-serves a namespace through Argo CD or Flux and it defaults to whatever the cluster-wide baseline is, which might be nothing. Bake PSS labels into your namespace-provisioning template or enforce them via a mutating admission policy so the gap never reopens as teams create namespaces on their own.

# Rollout checklist — do this in order, per namespace, not all at once

1. Set enforce=baseline (or leave unset), audit=restricted, warn=restricted
2. Deploy normally for one full release cycle
3. Grep audit logs / kubectl get events for "violates PodSecurity"
4. Triage violations:
   - Fixable (missing seccompProfile, root user) -> patch chart/manifest
   - Genuinely privileged (CNI, storage driver) -> move to dedicated
     namespace labeled enforce=privileged, lock down via RBAC
5. Only after violation count hits zero: flip enforce=restricted
6. Set an expiry date on any remaining exemptions.usernames entries
7. Re-run step 3 after next cluster minor upgrade (policy behavior
   can shift even with enforce-version pinned, if you bump it)

Performance notes

PSS itself is nearly free — it's one admission controller check inside kube-apiserver, no external webhook round trip, negligible per-request latency. The cost shows up when you stack third-party admission webhooks on top of it. Kyverno and Gatekeeper both add real per-pod-create latency, and if failurePolicy: Fail is set and the webhook pod goes down, every deployment in the cluster stops. Test with failurePolicy: Ignore in staging before you ever flip it to Fail in prod.

Audit mode isn't free either. Turning on cluster-wide audit: restricted on day one in a large cluster generates a real spike in audit log and event volume — enough to move your log ingestion bill if you're shipping everything to a centralized store. Scope audit logging to the namespaces you're actively migrating first, not the whole cluster at once.

One more thing worth saying plainly: baseline is not a security posture, it's a floor. It blocks the obviously dangerous stuff — hostPID, hostNetwork, privileged containers — but it still permits capability sets and configurations that a security team would flag in a real audit. If your compliance checklist says "PSS enabled" and stops there at baseline, you have a checkbox, not a control. restricted — non-root, no privilege escalation, dropped capabilities, seccomp required — is the actual target for anything handling sensitive data.

PSS also won't save you from a bad network path or an over-permissioned service account. Pair any Kubernetes Pod Security Standards rollout with NetworkPolicy and least-privilege RBAC; on its own, PSS is one layer, not a complete posture. We've written more on locking down namespace-level traffic over on kuryzhev.cloud if you're doing both at once.

The short version: don't enforce before you audit, don't exempt without an expiry, and don't confuse a namespace label with a compliant fleet of running pods. Get the sequencing right and PSS becomes background noise — invisible until it catches something real.

WireGuard Key Rotation: Static Configs vs Control-Plane Mesh

Oleksandr Kuryzhev — Mon, 31 Aug 2026 07:03:09 +0000

Originally published on kuryzhev.cloud

When you face this choice

Rotate a WireGuard key wrong and your tunnel doesn't error out — it just goes quiet. No log line, no alert, nothing. You spend an hour debugging "the network" before realizing the handshake never happened, because the peer on the other end is still holding the old public key. That's usually the moment WireGuard key rotation stops being a checkbox in a compliance doc and becomes a real operational problem.

The trigger is almost always the same story. You started with two or three hand-edited wg0.conf files, maybe checked into a private repo, and it worked fine for months. Then peer count climbed past 15-20 — contractors joining, ephemeral CI runners spinning up, autoscaled nodes coming and going — and suddenly every rotation is a fan-out exercise across machines you don't all remember exist.

Here's the thing people get wrong up front: the decision isn't "WireGuard or something else." WireGuard is staying either way — it's fast, it's auditable at the protocol level, and the Noise-based handshake is genuinely solid. The real decision is who owns peer state and rekeying: your git repo plus a config management tool, or a coordination service that pushes live state to every node.

The failure mode that forces this decision is almost always the same: a rotation that "worked" on the box you were staring at, but left two or three peers with stale keys because nobody had a reliable list of who needed the update. No alarm fires. You just get silent, un-alarmed packet loss until someone notices a service timing out.

Option A: Static configs + GitOps rotation

This is the approach most teams start with, and for good reason — it's simple to reason about. Peer definitions and keypairs live in version control, applied via Ansible, Salt, or Terraform, and pushed with wg syncconf so you get a zero-downtime reload instead of tearing the interface down.

# Generate a keypair — never commit the private key in plaintext
wg genkey | tee privatekey | wg pubkey > publickey

# Apply a new config without dropping existing sessions
wg syncconf wg0 <(wg-quick strip wg0)

Pros: everything is auditable through git history — you can point to a commit and say exactly when a key rotated and who approved it. There's no third-party trust anchor to compromise. It works fine in air-gapped or regulated environments where "call an external API to manage network identity" is a non-starter. And there's no extra infrastructure to patch, back up, or secure.

The problem is rotation itself is a fan-out problem. Every peer needs the new public key before the old one is dropped, and that coordination scales linearly (badly) with peer count. Race conditions creep in — someone reruns the playbook against a subset of hosts, someone forgets a peer that's technically still in inventory but not tagged right, and you're back to silent packet loss. There's also no built-in discovery or ACL layer; you're managing AllowedIPs by hand, which gets error-prone past a few dozen entries.

I still like this option a lot for small, stable meshes. It's the boring choice, and boring is good in network identity management.

Option B: Dynamic control-plane / mesh orchestrator

The alternative is a coordination layer — something in the headscale/Netmaker family, or a custom control API — that holds peer identity and state centrally and pushes configuration out, including automatic key or PSK rotation on a schedule.

The pitch is obvious: automatic peer discovery, real ACLs instead of hand-maintained AllowedIPs, and live rekeying without touching every node by hand. Onboarding a new contractor or scaling out a fleet of autoscaled workers stops being a git PR and manual apply — it's a policy the control plane enforces continuously. This scales cleanly into the hundreds of peers, which static configs frankly don't.

The catch — and it's a real one — is that the control plane becomes a trust and availability dependency. Existing WireGuard tunnels don't need a heartbeat to a server to keep working (that's a nice property of the protocol), but new joins and new rotations absolutely do. If your control plane is down during a scheduled rotation window, that rotation just doesn't happen, and you need a runbook that accounts for that distinction explicitly — "tunnels stay up" is not the same as "rotation succeeded."

You're also adding attack surface: a service that can push identity changes to your entire mesh is a very attractive target. There's tool lock-in to consider, and a real learning curve for whoever ends up owning that service in production. SaaS-hosted control planes are frequently disallowed outright in regulated or air-gapped environments, which pushes serious teams toward self-hosting the coordination layer if they want the automation at all.

Decision matrix

Here's the version I actually use when a client asks me to make this call. Weigh peer count and churn, required rotation cadence, regulatory constraints, and whether the team has the appetite to run and secure another piece of infrastructure.

Criterion                     | Static config + GitOps      | Control-plane orchestrator
------------------------------|------------------------------|-----------------------------
Peer count                    | Comfortable < 20-30          | Scales to 100s cleanly
Peer churn rate                | Low (stable topology)        | High (autoscale, contractors)
Rotation cadence               | Manual/scripted, quarterly+   | Automatic, can go monthly
Air-gapped / regulated env     | Strong fit                   | Needs self-hosted variant
Ops team maturity needed       | Low-medium (git + Ansible)    | Medium-high (own the control plane)
Extra infra to secure          | None                         | Yes (control plane HA + auth)
Audit trail                    | Git history = built-in       | Depends on tool's logging
Failure blast radius           | Push errors only, no SPOF     | Control plane outage blocks new joins/rotations

Somewhere in the 10-30 stable peers with quarterly rotation range, it's genuinely a toss-up — pick whichever your team already knows how to operate well. Below that, static configs win clearly. Above it, or with high churn, the control plane starts paying for itself fast.

One pattern I've seen work well as a middle ground: run the control plane for day-to-day onboarding/offboarding, but keep a git-based source of truth as an audit trail and disaster-recovery fallback if the control plane itself gets wiped out.

My pick

Under roughly 20 peers with low churn and any regulatory pressure at all: static config plus GitOps rotation, full stop. The automation a control plane gives you isn't worth the added trust surface at that scale — you're trading a manageable manual process for a new system you now have to secure, patch, and keep highly available.

Past that scale, or with frequent peer churn from autoscaling and contractor access, I go with a self-hosted control plane every time, even accounting for the extra component to run. The operational sanity of automatic rekeying and real ACLs outweighs the added surface, and honestly, doing large-scale WireGuard key rotation manually is where most of the silent-failure incidents I've cleaned up actually came from.

Regardless of which camp you're in, a few things aren't optional. Rotate on a schedule — 30 to 90 days is the standard that survives audit review in 2026, anything longer tends to get flagged. Use a proper rotation script with an overlap window instead of dropping the old key immediately:

#!/usr/bin/env bash
# rotate-peer-key.sh — zero-downtime key rotation for one WireGuard peer
set -euo pipefail
IFACE="wg0"
PEER_NAME="$1"
OLD_PUBKEY=$(vault kv get -field=pubkey "wg/peers/${PEER_NAME}")

# Generate new keypair — private key never touches disk unencrypted
NEW_PRIVKEY=$(wg genkey)
NEW_PUBKEY=$(echo "$NEW_PRIVKEY" | wg pubkey)

# Push new keypair to Vault before touching any live config
vault kv put "wg/peers/${PEER_NAME}" \
  privkey="$NEW_PRIVKEY" pubkey="$NEW_PUBKEY"

# Overlap window: both old and new keys are valid at once
wg set "$IFACE" peer "$NEW_PUBKEY" allowed-ips "10.10.0.7/32" persistent-keepalive 25

# Wait for a confirmed handshake on the NEW key before removing the OLD one
for i in {1..30}; do
  if wg show "$IFACE" dump | grep -q "$NEW_PUBKEY"; then
    echo "New key handshake confirmed for ${PEER_NAME}"
    break
  fi
  sleep 2
done

# Safe cutover: only remove the old peer once the new one is live
wg set "$IFACE" peer "$OLD_PUBKEY" remove
echo "Rotation complete for ${PEER_NAME}: ${OLD_PUBKEY} -> ${NEW_PUBKEY}"

Two gotchas I've hit personally: dropping the old key immediately instead of running an overlap window kills in-flight sessions mid-rotation, and rotating the key while forgetting the firewall rules or monitoring tied to the old peer identity breaks alerting quietly — the exact opposite of what you want from a security control. Also stagger rotations across a large mesh; forcing simultaneous rekeys causes a handshake storm and real CPU spikes on hub or relay nodes.

Last thing: never treat "no error" as proof a rotation succeeded. Verify with wg show wg0 dump and check last-handshake timestamps before and after. Store keys the way you'd store TLS private keys — in Vault or SOPS-encrypted secrets, injected at runtime, with access logged. Rotate preshared keys on their own, often shorter, cadence as a cheap second defense layer. If you're weighing this against broader infra automation choices, it's worth reading our notes on infrastructure automation tradeoffs before committing either way.

For the protocol-level details on handshakes and key exchange, the official WireGuard protocol documentation is worth a re-read, and if you're going the control-plane route, check the headscale documentation before you commit to self-hosting one in production.

Fix Nginx Cache-Control Misconfig Serving Stale or Uncached Assets

Oleksandr Kuryzhev — Sun, 30 Aug 2026 07:01:55 +0000

Originally published on kuryzhev.cloud

Symptoms

Nginx cache-control tuning is one of those things that looks fine in the config file and completely falls apart in DevTools. You open the Network tab, reload the page, and every single JS, CSS, and font file shows 200 instead of 304 or (disk cache). The browser is re-downloading assets it should already have.

Zoom out to the CDN dashboard and the story gets worse. Cache hit ratio sits at 40-60% when it should be north of 90% for static assets. Bandwidth costs are creeping up month over month, and nobody on the team can point to a config change that caused it — because there wasn't one, this was baked in from day one.

Then there's the deploy problem. You ship a UI fix, and support tickets come in for two more days from users staring at the old broken layout. Or the opposite happens: users get a blank white screen right after deploy because the HTML shell got cached for a year and now references JS hashes that no longer exist on the server.

Running curl -I against an asset gives you a mixed bag: no Cache-Control header at all, or one that contradicts itself — no-cache on a fingerprinted file that should be cached forever, or a long max-age on a file with no hash in the name. Something is actively fighting your caching strategy, and it's usually Nginx itself, not the CDN.

Root cause

The most common trap: add_header in Nginx does not merge across nested blocks. If your server {} block sets Cache-Control and X-Content-Type-Options, and then a location block inside it adds even one more header, Nginx drops every header the parent block defined for that location. It's not additive — it's block-level replacement. This bites almost everyone once, usually in production, usually silently.

Second issue: teams conflate "browser cache" with "CDN cache" as if they're the same lever. Cache-Control: public, max-age=86400 controls the browser. Whether your CDN respects that same value depends on the provider — Cloudflare, Fastly, and CloudFront each have their own rules around s-maxage, and some will happily double-cache or ignore your directive entirely if Vary isn't set correctly.

Third: no cache-busting strategy. Without content-hashed filenames like app.a1b2c3.js, you can't safely set a long max-age — doing so means users are stuck on broken code for up to a year with no way out except a hard refresh. So teams under-cache everything defensively, which tanks hit ratio and inflates origin bandwidth.

Last one, and it's sneaky: testing headers with curl -I straight against the origin IP or internal hostname. That tells you what Nginx sends. It tells you nothing about what the CDN or WAF in front of it does to that header on the way out. I've debugged "Nginx isn't caching" tickets for an hour before realizing the actual problem was Cloudflare stripping Cache-Control because a page rule was set to bypass cache on that path.

Fix #1 — Set explicit, layered Cache-Control by asset type

Stop using one location / block with one Cache-Control value for everything. Fingerprinted assets, non-fingerprinted assets, and the HTML entrypoint all need different policies, and they need to live in separate location blocks (or a centralized map) so nothing gets silently overridden.

Here's the config we run in front of a typical SPA. The map directive keeps the logic in one place instead of duplicating it across a dozen location blocks — cleaner, and much harder to break during a refactor.


# /etc/nginx/conf.d/static-cache.conf
# Map content-type to cache policy so we don't repeat logic per location block
map $sent_http_content_type $cache_policy {
    default                         "public, max-age=3600";
    ~*text/css                      "public, max-age=31536000, immutable";
    ~*application/javascript        "public, max-age=31536000, immutable";
    ~*image/                        "public, max-age=2592000, stale-while-revalidate=86400";
    ~*font/                         "public, max-age=31536000, immutable";
}

server {
    listen 443 ssl;
    server_name example.com;

    # HTML entrypoint: always revalidate, it references hashed asset filenames
    location = /index.html {
        add_header Cache-Control "no-cache, must-revalidate";
        add_header Vary "Accept-Encoding";
    }

    # Fingerprinted static assets — long-lived, immutable
    location ~* \.(js|css|woff2?|svg|png|jpg|jpeg|gif|ico)$ {
        add_header Cache-Control $cache_policy always;
        add_header Vary "Accept-Encoding";
        gzip_static on;          # serve pre-compressed .gz files, skip runtime gzip CPU cost
        access_log /var/log/nginx/static_access.log combined;
    }

    # Non-fingerprinted misc files — short max-age, no forced immutability
    location = /favicon.ico {
        add_header Cache-Control "public, max-age=3600";
    }
}

Gotcha: that always flag on add_header matters — without it, Nginx only sends the header on 2xx/3xx responses, and error pages served through the same location silently lose caching headers, which breaks some CDN edge-case handling.

Fix #2 — Fix cache-busting via filename fingerprinting, not header tricks

The real fix for deploy staleness isn't a smarter header — it's making the filename itself unique per build. Configure your build tool (Vite, webpack, esbuild, whatever) to emit content-hashed output: app.a1b2c3.js instead of app.js. Now a long max-age is safe, because a new deploy produces a new filename, not a new version of an old one.

Only the entry HTML — the file that references those hashed filenames — needs short-lived headers. That's the file browsers must always re-check. Everything it points to can be cached for a year without risk.

If you're stuck on a legacy setup without fingerprinting, you can decouple staleness partially: serve the HTML shell with no-cache, must-revalidate while keeping long max-age on assets, then bump a query string or path prefix on deploy. It's not as clean as real content hashing, but it stops the worst of the "users stuck on old JS" problem.

Gotcha: if you're not fingerprinting and relying on a CDN, you must add a cache purge step to your CI/CD pipeline on every deploy. Forgetting this means your origin serves fresh code but the CDN edge keeps handing out yesterday's build for however long the TTL says — sometimes a full day. If your CDN supports surrogate keys or cache tags, use them instead of full purges; it's far cheaper and doesn't nuke unrelated cached content.

Fix #3 — Align Nginx, CDN, and browser cache layers

Once single-server caching works, the harder problem shows up: three layers (browser, CDN, Nginx-as-proxy) disagreeing with each other. This is where most "it works locally but not in prod" caching bugs live.

Add Vary: Accept-Encoding everywhere you serve compressed content, and add Vary: Accept if you're conditionally serving AVIF/WebP based on request headers. Without it, a CDN edge node can cache the gzip response and then hand it to a client that sent no Accept-Encoding at all — broken rendering, hard to reproduce.

If Nginx itself sits as a caching reverse proxy in front of an app server, set proxy_cache_valid and proxy_no_cache explicitly for any route touching cookies or auth. Otherwise Nginx can cache a response meant for user A and serve it to user B — that's not a performance bug, that's a data leak.


# Quick verification checklist — run after any Nginx cache config change

# 1. Confirm headers survive through CDN, not just origin
curl -sI https://example.com/assets/app.a1b2c3.js | grep -i cache-control
# Expect: Cache-Control: public, max-age=31536000, immutable

# 2. Confirm HTML entrypoint is NOT long-cached
curl -sI https://example.com/index.html | grep -i cache-control
# Expect: Cache-Control: no-cache, must-revalidate

# 3. Check actual Nginx-level cache hit ratio
tail -n 500 /var/log/nginx/access.log | awk '{print $NF}' | sort | uniq -c
# Look for ratio of HIT vs MISS/EXPIRED (upstream_cache_status)

# 4. Confirm no auth/user-specific route is cached at shared layer
curl -sI https://example.com/api/user-avatar | grep -i cache-control
# Expect: Cache-Control: private, no-store

# 5. Verify Vary header is present for compressed variants
curl -sI https://example.com/assets/app.a1b2c3.js | grep -i vary
# Expect: Vary: Accept-Encoding

Test all of this through the production CDN domain, never directly against the origin. If a WAF or edge rule strips Cache-Control, you'll only catch it by checking the full path end-to-end.

Prevention

Cache misconfig regressions come back quietly — someone touches an unrelated location block six months later and breaks header inheritance again. Build guardrails so it fails loudly instead.

Add a CI check that fails the build if new static assets ship without content hashes in the filename. This closes the door on the "someone forgot to configure the build tool" class of bugs before it reaches production.

Add a synthetic smoke test after every deploy that asserts Cache-Control on a known asset path and on /index.html. This is cheap to write and catches Nginx config drift immediately instead of three days later when a support ticket shows up.

Document the caching policy per asset class — README or an ADR, doesn't matter which — so future engineers don't "fix" the CDN layer and accidentally break the Nginx layer, or vice versa. This is a distributed system with three independent caches; treat it like one.

Finally, actually look at your hit ratio numbers, not just your config. Configured headers mean nothing if $upstream_cache_status in your access logs still shows mostly MISS. We've seen teams go from a 60% to a 95% cache hit ratio after fixing nginx cache-control tuning properly, which cut origin bandwidth bills by 3-5x — that's the number that actually gets a fix prioritized by whoever owns the cloud bill. If you're also chasing bandwidth costs on the AWS side, our notes on AWS cost and infrastructure tuning cover related territory worth checking after this fix ships.

For the header semantics themselves, the official Nginx headers module docs and the MDN Cache-Control reference are worth bookmarking — the directive list changes slowly but the interaction between immutable, stale-while-revalidate, and CDN-specific extensions is easy to get wrong twice.

PostgreSQL VACUUM Monitoring: Catch Bloat Before It Causes Downtime

Oleksandr Kuryzhev — Sat, 29 Aug 2026 07:01:57 +0000

Originally published on kuryzhev.cloud

The scenario

PostgreSQL vacuum monitoring became a priority for us after a very specific scare: a 400GB OLTP database on managed Postgres started throwing slow queries and eating disk space, even though the actual dataset hadn't grown in weeks. No new customers, no big import job, nothing in the changelog that explained it. Classic symptoms, in hindsight — dead tuples piling up faster than autovacuum could clean them.

We didn't catch it early. We caught it when someone ran a diagnostic query out of curiosity and found dead-tuple counts in the millions on a handful of hot tables. Worse, when we checked age(datfrozenxid) against autovacuum_freeze_max_age, we were closer to transaction ID wraparound than anyone was comfortable admitting out loud in a Slack thread titled "quick question."

Wraparound is the nightmare scenario nobody explains until it's almost too late: if a database's transaction ID counter wraps before old rows get frozen, Postgres refuses new writes entirely. Disk usage climbing is annoying. Wraparound is an outage. That gap between "annoying" and "outage" is exactly what proactive vacuum monitoring is supposed to close — catching bloat and freeze risk weeks before they show up in a slow-query postmortem, not during one.

Prerequisites

Before touching any settings, you need a few things in place:

Access: superuser or the pg_monitor role, plus the ability to install extensions like pgstattuple. On managed Postgres — RDS, Cloud SQL — extension installs and some parameter changes go through a parameter group, and some require a reboot or "apply immediately" flag. Plan for that lead time.
Metrics pipeline: postgres_exporter (or an equivalent) feeding Prometheus and Grafana, with support for custom metric queries. The default exporter metrics don't cover dead-tuple ratio or freeze age — you'll need to add those yourself, covered below.
A maintenance window: some remediation, like repacking a badly bloated table, needs a low-traffic period. Identify one before you start, because you will find at least one table that needs it.

One security note: don't hand out blanket superuser access for monitoring. Grant pgstattuple execution and read access to a dedicated read-only monitoring role. It's a smaller attack surface and it's easier to justify in an access review.

Step 1: Baseline the current bloat and autovacuum health

Before changing anything, know where you actually stand. This baseline query set is the first thing we run on any database we're asked to look at:

-- baseline_vacuum_health.sql
-- Run this first: ranks tables by bloat risk and shows last autovacuum activity

-- 1. Dead tuple ratio per table (top offenders)
SELECT
  schemaname,
  relname,
  n_live_tup,
  n_dead_tup,
  ROUND(n_dead_tup::numeric / GREATEST(n_live_tup, 1) * 100, 2) AS dead_pct,
  last_autovacuum,
  last_autoanalyze
FROM pg_stat_user_tables
WHERE n_live_tup > 0
ORDER BY dead_pct DESC
LIMIT 20;

-- 2. Transaction ID wraparound risk per database
SELECT
  datname,
  age(datfrozenxid) AS xid_age,
  round(age(datfrozenxid)::numeric /
    (SELECT setting::numeric FROM pg_settings WHERE name = 'autovacuum_freeze_max_age') * 100, 1
  ) AS pct_of_freeze_max_age
FROM pg_database
WHERE datallowconn
ORDER BY xid_age DESC;

-- 3. Any vacuum currently in progress and its phase
SELECT
  p.pid,
  s.relname,
  p.phase,
  p.heap_blks_total,
  p.heap_blks_scanned,
  p.heap_blks_vacuumed
FROM pg_stat_progress_vacuum p
JOIN pg_stat_user_tables s ON s.relid = p.relid;

-- 4. Sessions that could be blocking vacuum's cleanup horizon
SELECT pid, state, xact_start, now() - xact_start AS duration, query
FROM pg_stat_activity
WHERE state = 'idle in transaction'
ORDER BY xact_start ASC;

Query #2 is the one that matters most and gets ignored most. Disk usage is a lagging, noisy signal — age(datfrozenxid) is the real wraparound-risk metric. We've seen teams stare at disk graphs for weeks while freeze age quietly crept past 70% of the default 200-million threshold.

Watch out: if you have tables with large text or jsonb columns, check their TOAST tables separately. TOAST bloats independently of the parent table and won't show up cleanly in the same dashboard view — we lost an afternoon once assuming a table was healthy because its main relation looked fine, while its TOAST relation was the actual problem.

Step 2: Tune autovacuum per table, not globally

The default autovacuum_vacuum_scale_factor of 0.2 means a table gets vacuumed once 20% of its rows are dead. That's fine for a 50-row config table. It's disastrous for a 50-million-row hot table, because 20% is 10 million dead tuples before autovacuum even triggers — and by then you're fighting a much bigger job with the same default cost limits.

Instead of touching global config, identify your top 5-10 tables by write churn and dead-tuple ratio (Step 1 gives you this list) and tune them individually:

ALTER TABLE orders SET (
  autovacuum_vacuum_scale_factor = 0.01,
  autovacuum_vacuum_cost_limit = 2000
);

-- Lower fillfactor on update-heavy tables to enable HOT updates
ALTER TABLE orders SET (fillfactor = 90);

Lowering fillfactor leaves free space in each page for updated rows to stay put instead of migrating and touching every index. That's a Heap-Only Tuple (HOT) update — it generates less bloat at the source and reduces index churn, which is cheaper than cleaning bloat up after the fact.

Gotcha: don't crank autovacuum_vacuum_cost_limit too aggressively across many tables at once. More aggressive vacuuming means more I/O, and on a busy primary that can worsen replication lag. Tune incrementally and watch replica lag metrics as you go — we've caused exactly this problem by being too enthusiastic with a "fix it once and for all" config push.

Also worth knowing: autovacuum_max_workers defaults to 3. On a cluster with hundreds of actively written tables, those workers become a bottleneck — tables can sit well past their scale_factor threshold simply waiting for a free worker. If Step 1's baseline keeps showing stale last_autovacuum timestamps despite reasonable per-table settings, check worker saturation before blaming your scale_factor math.

Step 3: Wire up continuous monitoring and alerts

One-time diagnostics are useful for a fire drill. They don't prevent the next one. We added custom queries to postgres_exporter for dead-tuple ratio, time since last autovacuum, and freeze age percentage — none of which are exposed by the exporter's default metric set.

With those metrics flowing into Prometheus, the alert rules are straightforward:

# prometheus_alerts_vacuum.yaml
# Custom alert rules built on postgres_exporter custom-query metrics
groups:
  - name: postgres_vacuum
    rules:
      - alert: HighDeadTupleRatio
        expr: pg_stat_user_tables_dead_pct > 15
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Table {{ $labels.relname }} dead tuple ratio > 15%"

      - alert: FreezeAgeCritical
        expr: pg_database_xid_age_pct_of_max > 75
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.datname }} nearing transaction ID wraparound"

      - alert: AutovacuumStalled
        expr: time() - pg_stat_user_tables_last_autovacuum_timestamp > 86400
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.relname }} not autovacuumed in 24h"

We page on freeze age crossing 75% of autovacuum_freeze_max_age and only warn on dead-tuple ratio, because dead tuples are a performance problem while wraparound is an availability problem — the severity should reflect that gap.

Also set log_autovacuum_min_duration = 0 (or a low threshold in ms) in your Postgres config. It's the cheapest observability win in this whole setup — every autovacuum run, its duration, and how many tuples it removed shows up directly in the Postgres logs, which you can correlate against your alerts without extra tooling. Check the official PostgreSQL vacuuming documentation for the full parameter list before changing defaults on a production cluster.

Step 4: Handle the cases autovacuum can't fix alone

Some bloat problems aren't config problems. The most common one we run into is an idle-in-transaction session — an app connection that opened a transaction and never committed or rolled back. That session holds back the xmin horizon, which silently prevents dead-tuple removal no matter how well you've tuned scale factors. Query #4 from Step 1 catches these; alert on any session idle-in-transaction for more than a few minutes and kill it.

For tables that are already badly bloated, resist the urge to run VACUUM FULL on a live production table. It rewrites the entire table and takes an ACCESS EXCLUSIVE lock for the whole operation — nothing reads or writes until it finishes. We've seen this used as an "emergency fix" that turned into a longer outage than the bloat itself. Use pg_repack instead — it rebuilds the table with much lighter locking.

A manual VACUUM (VERBOSE, ANALYZE) off-hours is still sometimes the pragmatic call, especially right after a bulk delete or migration, rather than waiting for tuned autovacuum settings to catch up on their own schedule.

Common mistake we've seen more than once: disabling autovacuum entirely on a hot table after it caused a lock contention scare during business hours. It feels like a fix in the moment. It guarantees catastrophic bloat and an eventual emergency VACUUM FULL a few months later — usually at a worse time than the original scare.

Verify and test

Don't call this done until you've confirmed it actually works. Re-run the Step 1 baseline queries a few days after applying per-table tuning and check that dead-tuple ratios on your tuned tables are trending down, not just stable.

Trigger a synthetic alert by temporarily lowering a threshold — drop HighDeadTupleRatio to 1% for a few minutes — and confirm it actually fires in Grafana and pages whoever it's supposed to page. An alert rule that's never fired in staging is an alert rule you can't trust in production.

Finally, confirm last_autovacuum timestamps on your key tables are recent, and that pg_total_relation_size on any repacked table has actually shrunk. Disk usage stabilizing — or dropping — after remediation is the clearest sign the whole loop is working, not just the dashboards.

PostgreSQL vacuum monitoring isn't a setting you configure once and forget — it's an ongoing feedback loop between your workload and autovacuum's settings, and workloads change constantly as tables grow, write patterns shift, and new features add hot paths you didn't tune for. Skipping that feedback loop doesn't save time; it just moves the cost from a boring dashboard check to an actual outage, measured in downtime rather than gigabytes. We treat these alerts the same way we treat disk-space or replication-lag alerts: routine, boring, and exactly the reason nothing dramatic happens at 3am. If you're building out a broader observability stack, our DevOps_DayS archive has more on wiring Prometheus and Grafana into production databases the same way.

7 Fixes for Python Lambda Cold Start Latency in 2026

Oleksandr Kuryzhev — Fri, 28 Aug 2026 07:01:48 +0000

Originally published on kuryzhev.cloud

Your Python Lambda's p99 latency graph has a spike that looks random. It isn't. Pull up the REPORT log line in CloudWatch and check Init Duration — that's your lambda cold start latency, and it almost always traces back to one of a handful of fixable causes. We've chased this on a handful of production APIs, and the fixes below are what actually moved the needle, not what we assumed would.

Right-size memory before you touch anything else

Lambda allocates CPU proportionally to memory. Below roughly 1024MB, your function is often CPU-starved during init, and that shows up directly as slow cold starts — not just slow execution. Before rewriting any code, run AWS Lambda Power Tuning, a Step Functions state machine that benchmarks your function across memory sizes and gives you an actual cost/latency curve.

Watch out for the common overcorrection here: teams see cold starts, max out memory to 3GB "just to be safe," and end up paying for GB-seconds they didn't need. We did this on one function for three months before someone ran the power tuning report and found 1024MB was the sweet spot — 1536MB bought us nothing but a bigger bill.

Move imports and clients outside the handler

This is the fix that surprises people the most because it affects warm invocations too, not just cold ones. If you instantiate a boto3 client or open a DB connection inside the handler function, you're re-paying that setup cost on every single call — cold or warm.

Initialize SDK clients, DB connections, and parsed config at module scope so the execution environment reuses them across invocations. Heavy libraries you only need occasionally — pandas, numpy, an ML SDK — should be imported lazily inside the specific code path that needs them, not at the top of the file.


# --- BAD: re-created on every invocation, adds latency to warm calls too ---
def handler(event, context):
    import boto3  # heavy import inside handler
    s3 = boto3.client("s3")  # new client every call
    conn = create_db_connection()  # new connection every call
    return s3.get_object(Bucket="my-bucket", Key=event["key"])

# --- GOOD: init once at module scope, reused across warm invocations ---
import boto3

s3 = boto3.client("s3")  # created once per execution environment
_db_conn = None

def get_db_connection():
    global _db_conn
    if _db_conn is None:
        _db_conn = create_db_connection()
    return _db_conn

def handler(event, context):
    # heavy, rarely-used dependency imported only when actually needed
    if event.get("needs_analytics"):
        import pandas as pd  # lazy import, skipped on most invocations
        return process_with_pandas(pd, event)

    conn = get_db_connection()
    return s3.get_object(Bucket="my-bucket", Key=event["key"])

One more gotcha: warm environments persist between invocations, so don't cache long-lived secrets or credentials in global scope without expiry logic. Use the Secrets Manager caching client instead of hardcoding a token that quietly goes stale.

Trim the deployment package aggressively

Package size directly affects how fast Lambda unpacks and initializes your code. We've seen zips creep past 60MB unzipped just from stray test fixtures and docs that never got cleaned out of the build.

Strip test files, documentation, and __pycache__ directories from your deployment artifact. Use pip install --no-cache-dir --target with a minimal requirements file instead of dragging in a full framework's extras. For container-image functions, multi-stage Docker builds with slim base images cut both image size and cold start time — check the Lambda container image docs for the current base image tags.

Try SnapStart where it's available for Python

SnapStart restores a pre-initialized, cached execution environment instead of running your init code from scratch. For functions with a heavy import graph, this is often a bigger win than any code trimming, because you're skipping the init phase entirely rather than optimizing it.

Here's the gotcha that bites people: SnapStart takes a snapshot of your initialized environment, so anything non-deterministic set up at init time — random seeds, unique IDs, cached short-lived tokens — needs to be refreshed after restore, not baked into the snapshot. And SnapStart is mutually exclusive with Provisioned Concurrency per function, so this is a strategy choice, not an add-on.

Reserve Provisioned Concurrency for latency-critical, predictable traffic

Provisioned Concurrency guarantees warm environments, but you're billed per allocated concurrency-hour whether it's invoked or not. This is a cost decision as much as a performance one, and treating it as a free cold-start eraser is how bills quietly balloon.

Pair it with Application Auto Scaling schedules so concurrency ramps up before known traffic spikes — business-hours APIs, scheduled batch triggers — and scales down overnight. The mistake we made early on was applying it blanket across every function "just in case." Half of those functions got a handful of invocations a day; the Provisioned Concurrency cost dwarfed the actual compute cost.


# Example: scheduled scaling target for a business-hours API function
Resources:
  ScalableTarget:
    Type: AWS::ApplicationAutoScaling::ScalableTarget
    Properties:
      MinCapacity: 5
      MaxCapacity: 5
      ResourceId: !Sub "function:${FunctionName}:${FunctionAlias}"
      ScalableDimension: lambda:function:ProvisionedConcurrency
      ServiceNamespace: lambda
      ScheduledActions:
        - ScheduledActionName: scale-up-morning
          Schedule: "cron(0 8 * * ? *)"  # 8am UTC, before traffic ramps
          ScalableTargetAction:
            MinCapacity: 5
            MaxCapacity: 5
        - ScheduledActionName: scale-down-evening
          Schedule: "cron(0 20 * * ? *)"  # 8pm UTC, off-hours drop
          ScalableTargetAction:
            MinCapacity: 0
            MaxCapacity: 0

Skip the VPC unless you truly need one

Hyperplane ENIs shrank the VPC cold-start penalty considerably, but VPC-attached functions still initialize slower than public ones — especially when outbound calls route through a NAT gateway. If you don't need RDS or ElastiCache directly, don't attach a VPC just because it feels "more secure."

If you do need one, add VPC endpoints for the AWS services you call — S3, DynamoDB, Secrets Manager — so traffic doesn't take a NAT round trip. And don't use "no VPC" as an excuse to skip least-privilege IAM; the network boundary disappearing doesn't mean the resource policy boundary should too.

Choose Graviton (ARM64) as the default architecture

Switching a Python function to arm64 is usually a one-line change in your IaC, and it's typically around 20% cheaper per GB-second while matching or beating x86 on init time for pure Python workloads. There's close to no reason not to default to it on new functions.

The gotcha is compiled dependencies. numpy, cryptography, and similar packages need arm64-compatible wheels, and if one's missing, the function won't fail at deploy — it'll fail silently at import time in production. Test the switch in a staging environment first, not directly on the function that's paging you at 2am.


# Terraform: switching architecture is a one-line change
resource "aws_lambda_function" "api" {
  function_name = "orders-api"
  architectures  = ["arm64"]  # was ["x86_64"] — verify compiled deps have arm64 wheels first
  runtime        = "python3.13"
  memory_size    = 1024
  handler        = "app.handler"
}

None of these fixes work as a substitute for measuring first. Keep-warm cron pings every 5 minutes are a common workaround we still see, but they don't guarantee the same execution environment gets reused once concurrency scales out — they're an unreliable, wasteful stand-in for actual Provisioned Concurrency. Use the decision checklist below to match the fix to your traffic pattern instead of applying all seven blindly.


Cold start mitigation decision checklist:

Traffic pattern              -> Recommended strategy
------------------------------------------------------
Spiky, unpredictable          -> Right-size memory + trim package + arm64
Predictable schedule (biz hrs)-> Provisioned Concurrency + Auto Scaling schedule
Heavy import graph, steady    -> SnapStart (if supported) over Provisioned Concurrency
Rare/low-traffic internal job -> Do nothing extra; cold start cost is negligible
VPC-required (RDS/ElastiCache)-> VPC endpoints + connection reuse, avoid NAT hops
Compiled deps (numpy/crypto)  -> Verify arm64 wheels before switching architecture

If you're debugging lambda cold start latency on a serverless API right now, start with Init Duration in CloudWatch, not the total Duration metric — it tells you exactly which phase is slow before you touch memory, packaging, or Provisioned Concurrency. For more serverless patterns and AWS troubleshooting notes, check the DevOps_DayS archive.

Setting Up Jenkins AWS OIDC Authentication to Replace Static Keys

Oleksandr Kuryzhev — Thu, 27 Aug 2026 07:01:33 +0000

Originally published on kuryzhev.cloud

Every static AWS key sitting in your Jenkins credential store is a standing invitation. It works from anywhere, forever, until someone remembers to rotate it — which is rarely. Jenkins AWS OIDC authentication fixes this by making Jenkins prove who it is on every single run instead of handing it a permanent secret to carry around.

I've migrated three separate Jenkins fleets off static IAM users onto OIDC federation, and the pattern of mistakes is remarkably consistent across teams. This post covers what's actually happening under the hood, where people get it wrong, and the setup that holds up in production.

What this actually does

Jenkins, through an OIDC-capable plugin or a custom step, mints a signed JWT that describes the job: repo, branch, build ID, whatever claims you configure. AWS STS trusts that token — not because "Jenkins can access AWS," but because AWS has registered Jenkins' issuer URL as a trusted OIDC provider and will only accept tokens matching specific claim conditions. The exchange happens through AssumeRoleWithWebIdentity, which trades that signed token for temporary credentials.

This is a fundamentally different trust model than an IAM user's access key. A static key is a bearer credential — no expiry, no scoping to a specific pipeline, no cryptographic proof of *who* is asking. If it leaks, it works everywhere until someone manually revokes it. An OIDC token is short-lived, tied to a specific subject claim, and expires on its own even if nobody notices the leak.

No secret ever crosses the wire in this model. Jenkins doesn't store an AWS credential at rest — it generates a signed assertion at runtime, and AWS validates that assertion against a trust policy before issuing anything. If the assertion doesn't match, there's nothing to steal because nothing was ever stored.

How people use it wrong

The most common failure I see is an overly broad trust policy. Teams condition the trust only on aud, or worse, wildcard the sub claim as repo:my-org/*:*. That means any job, in any repo, on any branch, in the entire org can assume a role meant for production deploys. You've replaced a static key with a slightly more elaborate static key — same blast radius, extra YAML.

Second gotcha: treating the Jenkins issuer URL as permanent. If Jenkins moves behind a new load balancer, gets a new domain, or the box gets rebuilt with a fresh cert, the OIDC discovery URL and JWKS endpoint change. Every trust policy that references the old issuer silently stops matching. Pipelines start failing with opaque AccessDenied errors on AssumeRoleWithWebIdentity, and nobody connects it to the infra change from two weeks ago.

Third: one shared IAM role "to keep things simple." Deploy jobs and read-only lint jobs get identical permissions. A compromised low-trust job — say, a PR build with a tampered pipeline script — now has the same reach as your production deploy. That defeats the entire point of scoping trust in the first place.

Watch out for: Jenkins must be reachable over valid HTTPS for AWS to fetch the JWKS/discovery document. Internal-only Jenkins instances or self-signed certs fail token validation silently — no error at setup time, just mysterious auth failures the first time a pipeline actually tries to assume a role.

The correct approach

Start by registering Jenkins as an IAM OIDC identity provider using its public HTTPS discovery URL. AWS fetches and caches the JWKS thumbprint from that endpoint — this is a one-time registration per AWS account, not per role.

Then build the trust policy around specific claims, not just audience. Pin sub to a concrete job path and aud to the exact identifier your plugin issues. Use StringEquals, never StringLike with wildcards, for anything touching production:


# --- IAM trust policy: scoped to a specific Jenkins job, not wildcarded ---
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/jenkins.example.com/oidc"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          # exact audience your Jenkins OIDC plugin issues
          "jenkins.example.com/oidc:aud": "sts.amazonaws.com",
          # exact subject — pinned to repo + branch, NOT a wildcard
          "jenkins.example.com/oidc:sub": "repo:my-org/infra:ref:refs/heads/main"
        }
      }
    }
  ]
}

Inside the pipeline, use the Jenkins OIDC plugin (or a manual aws sts assume-role-with-web-identity call) to mint the token and exchange it. Credentials live only in the job's environment for the duration of the stage — they never touch Jenkins' credential store:


# --- Pipeline step: exchange Jenkins-issued JWT for short-lived AWS creds ---
pipeline {
  agent any
  stages {
    stage('Deploy') {
      steps {
        script {
          // Plugin mints a JWT scoped to this job/branch
          def idToken = oidcIdToken(audience: 'sts.amazonaws.com')

          // Exchange it directly with STS — no static key ever touches Jenkins
          sh """
            aws sts assume-role-with-web-identity \
              --role-arn arn:aws:iam::123456789012:role/jenkins-deploy-main \
              --role-session-name jenkins-${env.BUILD_ID} \
              --web-identity-token ${idToken} \
              --duration-seconds 900 > creds.json
          """
        }
      }
    }
  }
}

One role per trust boundary is the rule I don't bend. Plan-only vs apply, staging vs prod, read-only vs deploy — each gets its own role, its own trust policy, its own narrow permission set. A compromised job should never have more reach than its narrowest legitimate need.

Advanced patterns

For multi-account setups, register the OIDC provider once per account that needs to trust Jenkins — don't try to hack a single cross-account provider. Each account's role trust policy still scopes sub to a specific job path, so Jenkins itself never holds account-wide access; it's just trusted differently by each account it touches.

Branch and PR scoping is where OIDC really earns its keep over static keys. Encode the branch or tag into the sub claim so only merges to main can assume the deploy role, while PR builds get a distinctly scoped, usually read-only role. This has to be enforced by the trust policy, not by conditionals inside the pipeline script — a PR can edit the pipeline script itself, but it can't edit an IAM trust policy in another account.

Session tagging closes an audit gap static keys always had. Pass job name, requester, and git SHA as session tags on the assume-role call, and CloudTrail logs show exactly which pipeline execution made which API call instead of an anonymous jenkins-role entry every time.

Layer permission boundaries on top of OIDC-assumed roles as defense-in-depth. If trust policy review lags behind team growth — and it always does — a boundary caps the damage even when a scope gets misconfigured. We cover related guardrail patterns for CI pipelines over on kuryzhev.cloud if you want more on locking down automated deploy paths.

Performance notes

AssumeRoleWithWebIdentity adds one extra network round trip per pipeline run — minting the JWT and exchanging it with STS — typically 100 to 300 milliseconds. That's negligible against most build and deploy times, but worth flagging if you have latency-sensitive smoke-test pipelines that fire dozens of times an hour.

STS calls themselves are free, but poorly scoped roles running across thousands of nightly builds generate real CloudTrail volume. Budget log retention and storage accordingly, especially if you're centralizing logs for compliance.

Token lifetime is typically 15 minutes to an hour by default, configurable up to the role's max session duration. Long-running jobs — multi-hour ETL pipelines, slow Terraform applies across dozens of modules — need to re-assume or chain roles rather than relying on one token for the whole job. "Credentials expired mid-deploy" is a very common failure mode the first time a team hits this wall.

Finally, AWS caches JWKS thumbprints and doesn't refetch instantly on rotation. If you're migrating Jenkins to a new domain or rotating its cert, update the IAM OIDC provider registration in every trusting account before or simultaneously with the DNS cutover — not after. Cutting over DNS first and fixing IAM later is how Jenkins AWS OIDC authentication breaks silently for every pipeline in the fleet at once.

Loki Retention Tuning: 7 Fixes for Runaway Log Storage Costs

Oleksandr Kuryzhev — Wed, 26 Aug 2026 07:01:31 +0000

Originally published on kuryzhev.cloud

Your Loki bill isn't growing because of log volume — it's growing because someone added a trace_id label back in 2024 and nobody ever removed it. We spent two sprints on Loki retention tuning last quarter, and the actual fix had almost nothing to do with the retention_period setting we started with. Here's what we changed, in the order it actually moved the needle.

Stop treating retention_period as one global dial

A single global retention setting either wastes storage on low-value debug logs or violates compliance requirements for audit trails — you can't win with one number. Loki supports per-tenant overrides in runtime_config, and they reload without restarting ingesters, so there's no excuse for lumping everything together.

We had payments audit logs and frontend debug logs under the same 30-day policy for over a year. Splitting them into separate tenant overrides cut storage for the noisy tenant by 70% while actually extending retention for the compliance-sensitive one.

Cardinality, not volume, is what bankrupts your index

Everyone tunes retention_period first. Almost nobody checks cardinality first, and that's backwards. Every unique combination of label values creates a new stream, and every stream gets its own chunk — so a label like request_id or pod IP doesn't just add data, it multiplies your index size.

Before touching any retention setting, run a cardinality audit:

# find labels driving the most unique streams
logcli series --analyze-labels --since=24h '{}'

Gotcha: if this command returns thousands of unique values for a single label, retention tuning won't save you — you're solving the wrong problem.

Let the compactor do the retention work, then tune it

Setting retention_period without enabling the compactor is a silent no-op — Loki will happily accept the config and delete nothing. We learned this after "shortening" retention for a tenant and watching storage stay flat for two weeks.

compactor:
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: aws   # required for the Delete API
  compaction_interval: 10m

The delete delay is intentional — it gives you a window to cancel a deletion request — but it also means "expired" data still occupies (and costs) storage until the compactor actually runs.

Chunk size and age settings decide your object storage bill

At high ingestion volume, object storage cost is often driven by request count, not raw bytes. Small chunks mean more frequent PUT/GET calls, and S3-style pricing punishes that hard. Bumping chunk_target_size toward 1.5–4MB reduced our PUT volume by roughly a third.

ingester:
  chunk_target_size: 2097152   # ~2MB, fewer object storage PUTs
  max_chunk_age: 1h            # balance flush frequency vs memory

The tradeoff: larger max_chunk_age means fewer flushes but more memory pressure on ingesters. Watch ingester OOMs after this change — we saw a couple during a traffic spike before we adjusted memory limits.

Use structured metadata instead of new labels

Loki 2.9+ and 3.x support structured metadata, which lets you attach high-cardinality fields like trace_id or request_id to log lines without putting them in the label index. This is the single change that had the biggest cardinality impact for us.

Migration isn't free — pipelines pushing everything through label relabeling in Promtail or Grafana Alloy need a rewrite to route those fields into structured metadata instead. It's worth the afternoon it takes; check the Grafana Loki structured metadata docs for the exact pipeline stage syntax.

Guardrail queries against your own retention window

Long retention is pointless — and expensive — if queries against old data time out or hammer cold storage every time someone opens a dashboard. We found a Grafana panel silently scanning 90 days of cold S3-backed chunks on every 30-second auto-refresh. That one panel was responsible for a noticeable chunk of our egress cost.

overrides:
  team-payments:
    retention_period: 2160h        # 90 days, compliance requirement
    ingestion_rate_mb: 20
    per_stream_rate_limit: 5MB
  team-web:
    retention_period: 168h         # 7 days, high-volume/low-value logs
    max_query_length: 72h          # block accidental 90-day scans

Pair max_query_length and split_queries_by_interval with per-tenant ingestion limits — otherwise one noisy team forces retention and index scaling decisions on everyone else sharing the cluster. See the official Loki retention documentation for the full list of tunables.

Remember logs are data too — secure them like it

Retention tuning conversations tend to skip security entirely, which is a mistake once you're storing PII or audit trails in log lines. Restrict IAM and bucket policies on your chunk and index storage separately from application data buckets — don't inherit broad defaults just because it's "only logs."

Gotcha: retention_period is not the same as guaranteed erasure. Compaction lag and backup snapshots can keep PII around long after it should be gone. For right-to-erasure requests, use the Delete API directly instead of waiting on a scheduled retention cycle.

Before you change a single retention setting, run through this checklist — it's saved us from at least three wasted afternoons:

Decision checklist — before you touch retention_period:

[ ] Have you measured stream cardinality? (logcli series --analyze-labels)
[ ] Are trace/request IDs in labels instead of structured metadata? -> fix first
[ ] Is compactor.retention_enabled actually true? (silent no-op otherwise)
[ ] Do different log sources need different retention (audit vs debug)?
[ ] Is query_timeout set to match your longest allowed retention window?
[ ] Do storage bucket IAM policies match your data sensitivity, not just app defaults?
[ ] Have you priced object storage requests, not just GB stored?

Loki retention tuning isn't one setting — it's cardinality control, compactor configuration, chunk sizing, and query guardrails working together. Get the cardinality and compactor pieces right first, and the retention_period number itself becomes almost trivial to set. For more log pipeline debugging, check the monitoring category on kuryzhev.cloud.

Argo CD Canary Deployment: Gating Rollouts with Real SLIs

Oleksandr Kuryzhev — Tue, 25 Aug 2026 07:01:52 +0000

Originally published on kuryzhev.cloud

The scenario

Argo CD said the deploy was "Healthy" — it just meant the manifests applied, not that checkout stopped returning 500s. We pushed a checkout service change, CI passed, Argo CD synced clean, dashboard all green. Ninety seconds later, our alerting channel lit up with 500s from real customers hitting the checkout flow.

The root cause wasn't subtle once we looked: Argo CD's health check for a plain kubectl.io/Deployment only confirms pods are running and ready. It has no opinion on whether the app is returning correct responses. We had auto-sync on, no canary step, no automated rollback — a bad image went from zero to 100% of pods in a single sync cycle. By the time a human noticed the error rate spike, every replica was already serving the broken build.

This is the setup we ended up with instead: swap the Deployment for an Argo Rollouts canary that ships to a small traffic slice first, gated by real Prometheus SLIs — error rate and p99 latency — before it's allowed to progress further. Argo CD's job goes back to what it's actually good at: keeping the cluster in sync with Git. Health judgment moves to Argo Rollouts, where it belongs.

Prerequisites

Before touching the Rollout spec, make sure these pieces are actually in place — skipping one of these is the fastest way to get stuck halfway through.

Argo CD (a recent 2.x release) plus the Argo Rollouts controller and the kubectl argo rollouts plugin installed in-cluster. The Rollout CRD needs to be registered before you apply anything referencing kind: Rollout.
A metrics backend reachable from inside the cluster for AnalysisTemplates. We use Prometheus below, but Datadog and CloudWatch providers follow the same pattern — same fields, different provider block.
A traffic-splitting layer that supports weighted routing: Istio VirtualService, NGINX ingress canary annotations, or Gateway API HTTPRoute. Pick one before you write the Rollout spec — the trafficRouting block is provider-specific and you can't mix and match mid-tutorial.
An Argo CD Application already pointing at the app's manifests (Helm or Kustomize), currently deploying a plain Deployment. This gets swapped out for a Rollout, not layered on top of it — running both will just confuse the selector.

Step 1: Replace the Deployment with a Rollout

The pod spec doesn't change at all. Same containers, same labels, same selectors. The only structural change is kind: Deployment becoming kind: Rollout (from the argoproj.io/v1alpha1 API group), and spec.strategy turning into a canary block with explicit steps.

Start conservative. We use setWeight: 10 followed by a two-minute pause, then setWeight: 50 with another pause, then 100%. Tune the step count and weights to your blast radius tolerance, not to gut feeling — a service handling payment traffic should sit at 10% longer than an internal reporting tool.

# rollout.yaml — canary Rollout replacing a plain Deployment
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-service
spec:
  replicas: 6
  selector:
    matchLabels:
      app: checkout-service
  template:
    metadata:
      labels:
        app: checkout-service
    spec:
      containers:
        - name: checkout-service
          image: registry.example.com/checkout-service:__TAG__
          ports:
            - containerPort: 8080
  strategy:
    canary:
      # traffic split target — must match your mesh/ingress resource
      trafficRouting:
        istio:
          virtualService:
            name: checkout-service-vs
            routes:
              - primary
      steps:
        - setWeight: 10
        - pause: {duration: 2m}
        - analysis:
            templates:
              - templateName: checkout-slo-check
        - setWeight: 50
        - pause: {duration: 3m}
        - setWeight: 100

Watch out for this one: if your Argo CD Application has syncPolicy.automated.selfHeal: true, Argo CD will try to "fix" the Rollout mid-canary. A paused Rollout's scaled-down replica count looks like drift to Argo CD, and selfHeal will happily fight the canary controller for control of spec.replicas. We got bitten by this the first time — the canary kept snapping back to full replicas before the pause finished. Fix it by adding ignoreDifferences scoped to the Rollout kind, or disable selfHeal during release windows.

Step 2: Add health gates with an AnalysisTemplate

A pause step by itself just buys you time — someone still has to decide whether to promote. An AnalysisTemplate turns that pause into an automated go/no-go decision based on actual metrics instead of a person staring at a dashboard.

Define the template with a Prometheus query for error rate and one for p99 latency. Set successCondition and failureCondition as PromQL thresholds — not raw request counts, which don't normalize across traffic volume. Reference the template from the Rollout's step list so it runs automatically at the pause point.

# analysistemplate.yaml — health gate using real SLIs, not CPU/memory
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: checkout-slo-check
spec:
  metrics:
    - name: error-rate
      interval: 30s
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="checkout-service",status=~"5.."}[2m]))
            /
            sum(rate(http_requests_total{app="checkout-service"}[2m]))
      successCondition: result[0] < 0.02
      failureCondition: result[0] >= 0.02
    - name: p99-latency
      interval: 30s
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.99,
              rate(http_request_duration_seconds_bucket{app="checkout-service"}[2m]))
      successCondition: result[0] < 0.8
      failureCondition: result[0] >= 0.8

The most common mistake I see teams make here is gating promotion on CPU or memory instead of a business-facing SLI. A canary can look perfectly healthy on resource metrics while quietly returning wrong responses or timing out downstream calls. Gate on what users actually feel — error rate, latency, business events like completed checkouts — not infrastructure vitals that don't correlate with correctness.

Also tune interval deliberately. Set it too short and you'll get flapping on naturally noisy metrics, aborting good deploys for no reason. Set it too long and you delay rollback, which widens the blast radius exactly when you're trying to shrink it. There's no universal default — tune against your metric's actual variance.

Step 3: Wire in traffic splitting and Argo CD health awareness

None of this matters if the canary pods aren't receiving real traffic. Configure trafficRouting in the Rollout spec to match your mesh or ingress choice — Istio VirtualService/DestinationRule names, NGINX canary annotations, or a Gateway API HTTPRoute. This is what makes setWeight steps actually shift live requests, not just replica counts sitting idle.

Current Argo CD versions ship built-in resource health logic for argoproj.io/Rollout. Confirm it's active — it's what makes the Argo CD UI correctly show "Progressing" during a canary window instead of a misleading "Healthy" the moment pods come up. Without it, Argo CD reads Rollout status the same way it reads a Deployment, and you're back to the original problem.

Finally, add an Argo CD Notifications trigger on on-degraded and on-analysis-run-failed for the Rollout resource. A failed AnalysisRun should page someone, not sit silently paused waiting for a person to check the dashboard. We wired ours into the same Slack channel as our other alerts — no separate tooling needed.

One more thing worth budgeting for: canary windows mean stable and canary replica sets run simultaneously for the entire pause duration. If you're running multiple sequential steps, that overlap compute cost adds up — factor it into capacity planning, especially for services with tight resource quotas.

Verify and test

Don't trust this setup until you've watched it fail on purpose. Start by running a live deploy and watching the rollout in real time:

# live view of weight, revision, and AnalysisRun status together
kubectl argo rollouts get rollout checkout-service --watch

The test that actually matters isn't a clean deploy — it's a deliberately broken one. Ship an image with an injected 500-error handler or artificial latency and confirm the AnalysisRun trips failureCondition, aborts automatically, and scales traffic back to the stable version. If your gate can't catch a deploy you know is broken, it can't be trusted to catch one you don't know about.

Also exercise manual override at least once, so operators know the escape hatch works when automation gets it wrong:

kubectl argo rollouts promote checkout-service   # force progression past a pause
kubectl argo rollouts abort checkout-service     # roll back to stable immediately

Check that Argo CD reflects the abort as expected sync state, not as an error condition — if ignoreDifferences is scoped correctly from Step 1, this should be quiet. The kubectl argo rollouts dashboard command gives a local UI for rollout history and analysis runs if you want a visual trail after the fact.

Canary deployments with AnalysisTemplates add real operational overhead — longer deploy windows, extra compute running two replica sets side by side, and another CRD your team has to reason about during incidents. For a low-traffic internal tool, a plain rolling update with a decent post-deploy smoke test is often the more honest choice; you're not paying for machinery you don't need. But for something like checkout, where an incident costs real revenue, catching a bad release automatically at 10% traffic instead of manually at 100% is worth every bit of that overhead. We haven't had a full-blast bad deploy reach checkout since we made this switch, and that alone paid for the setup time. If you're evaluating this for your own stack, start with one high-traffic service, get the AnalysisTemplate gates right, and expand from there — don't roll it out cluster-wide on day one. For more on rollout strategies and rollback tradeoffs, see our DevOps_DayS archive.

DEV Community: Oleksandr Kuryzhev

Kubectl Alias Pitfalls You Should Fix Before They Bite

Context: how kubectl alias pitfalls creep into daily workflows

Common failure 1 — the silent context carryover

Common failure 2 — destructive aliases without guardrails

Common failure 3 — profile/region mismatch in cloud CLIs

Safer operating pattern for kubectl alias pitfalls

Related

Bedrock IAM Least Privilege: Stopping Runaway Bot Costs

The 3 a.m. Bedrock Bill

Why Bedrock IAM Ends Up Over-Permissioned

Scoping the Policy — Model, Region, and Identity

Prevention Checklist

Related

3 boto3 and argparse Mistakes That Break Python AWS Scripts

Context

Common failure: wrong account, wrong region, no warning

Common failure: argparse errors that vanish into broad except blocks

Common failure: logging that hides the problem or leaks it

Safer operating pattern

Related

How We Learned to Harden Jenkins Agents: 3 Mistakes That Cost Us

Context: agents are the attack surface, not the controller

Mistake 1: We treated agents as trusted extensions of the controller

Mistake 2: We mounted docker.sock for build convenience

Mistake 3: We kept long-lived "pet" agents around

What we do differently now

Related

Rolling Out Kubernetes Pod Security Standards Without Breaking Prod

What Pod Security Standards actually does

How people use it wrong

The correct approach

Advanced patterns

Performance notes

Related

WireGuard Key Rotation: Static Configs vs Control-Plane Mesh

When you face this choice

Option A: Static configs + GitOps rotation

Option B: Dynamic control-plane / mesh orchestrator

Decision matrix

My pick

Related

Fix Nginx Cache-Control Misconfig Serving Stale or Uncached Assets

Symptoms

Root cause

Fix #1 — Set explicit, layered Cache-Control by asset type

Fix #2 — Fix cache-busting via filename fingerprinting, not header tricks

Fix #3 — Align Nginx, CDN, and browser cache layers

Prevention

Related

PostgreSQL VACUUM Monitoring: Catch Bloat Before It Causes Downtime

The scenario

Prerequisites

Step 1: Baseline the current bloat and autovacuum health

Step 2: Tune autovacuum per table, not globally

Step 3: Wire up continuous monitoring and alerts

Step 4: Handle the cases autovacuum can't fix alone

Verify and test

Related

7 Fixes for Python Lambda Cold Start Latency in 2026

Right-size memory before you touch anything else

Move imports and clients outside the handler

Trim the deployment package aggressively

Try SnapStart where it's available for Python

Reserve Provisioned Concurrency for latency-critical, predictable traffic

Skip the VPC unless you truly need one

Choose Graviton (ARM64) as the default architecture

Related

Setting Up Jenkins AWS OIDC Authentication to Replace Static Keys

What this actually does

How people use it wrong

The correct approach

Advanced patterns

Performance notes

Related

Loki Retention Tuning: 7 Fixes for Runaway Log Storage Costs

Stop treating retention_period as one global dial

Cardinality, not volume, is what bankrupts your index

Let the compactor do the retention work, then tune it

Chunk size and age settings decide your object storage bill