DEV Community: Peter Brau

A Successful Docker Build Does Not Mean You Built a Runnable Release

Peter Brau — Thu, 24 Sep 2026 00:16:14 +0000

Problem

A container image can build successfully and still be unusable as a release artifact.

A successful image build proves that Docker could assemble the image filesystem and metadata. It does not prove that the image's configured process can start, import the application correctly, remain alive, or answer the application's minimum health contract.

The observed failure class was exactly that gap: the image existed and could be retrieved, but the application process failed because Uvicorn was pointed at the wrong application target.

Failure Signature

The build and registry stages succeed, but the deployment never becomes healthy because the container's primary process exits or cannot initialize the application.

At that point the defect is surrounded by infrastructure noise:

the registry already contains the candidate image;
an orchestrator pulls it;
health checks begin failing;
deployment stabilization waits consume time;
rollback or replacement may start;
operators have to distinguish an application-startup defect from networking, load balancing, IAM, or orchestration problems.

If the same image would have failed immediately on a local runner, discovering it after publication is unnecessary delay.

What We Expected

The release pipeline implicitly treated:

source -> docker build -> registry push

as evidence that the artifact was deployable.

The stronger release contract should have been:

source
  -> build image
  -> start that exact image
  -> wait for application readiness
  -> call a minimal health endpoint
  -> stop the container
  -> publish that exact image

Root Cause

The CI contract validated packaging, not runtime startup.

In the observed case, the Uvicorn command did not match the application's factory-style startup contract. Uvicorn's current documentation still distinguishes the factory form: --factory tells Uvicorn to treat the referenced application target as a callable that returns an ASGI application.

A generic factory-style startup command looks like:

uvicorn <APPLICATION_MODULE>:create_app --factory --host 0.0.0.0 --port 8000

The module and callable are application-specific. The reusable lesson is to make that startup contract executable in CI before the artifact becomes a release candidate.

Why the Obvious Fix Was Wrong

Fixing the command in the deployment environment alone repairs one incident but leaves the detection gap intact.

The next malformed entry point, missing runtime dependency, import failure, or startup configuration error can still travel through image publication and reach the orchestrator before anyone discovers it.

A second weak fix is to build one image for the smoke test and then rebuild another image for publication. That proves one artifact and ships another.

Minimal Correction

Run the exact built image as a container before publication and require a minimal health check to pass.

Docker's current CLI documentation defines docker run as creating and running a container from an image. A generic CI preflight can therefore use the same image reference that will later be pushed:

container_name="app-under-test"

cleanup() {
  docker rm -f "$container_name" >/dev/null 2>&1 || true
}
trap cleanup EXIT

docker run -d \
  --name "$container_name" \
  -p 127.0.0.1:8080:8000 \
  "$IMAGE_ID"

for attempt in $(seq 1 30); do
  if curl --fail --silent http://127.0.0.1:8080/health >/dev/null; then
    exit 0
  fi
  sleep 1
done

docker logs "$container_name"
exit 1

The details should match the application:

use the real startup command baked into the image unless the test is explicitly validating an override;
use a health endpoint that proves the application initialized far enough to serve its minimum contract;
keep host port exposure loopback-only for the CI check;
capture container logs when startup fails;
guarantee cleanup on success and failure.

After the preflight passes, publish the same image identity. Do not rebuild between validation and push.

Verification

The correction proves the relevant failure class when CI fails before publication if the configured process cannot start or the health endpoint never becomes ready.

A regression test should deliberately exercise the startup contract so that an invalid application target or equivalent startup defect is caught without a registry push, orchestrator rollout, or rollback.

The positive path should demonstrate that the exact image which passed the startup check is the image subsequently published.

What Should Have Caught This Earlier

Container publication should have a release preflight with three explicit assertions:

Process assertion: the configured primary process remains running long enough to initialize.
Health assertion: the application responds successfully on its minimum local health contract.
Artifact-identity assertion: the image published is the same immutable artifact that passed the first two checks.

This is a good example of moving failure left without inventing a large test system. A short local container run can eliminate an entire class of slow deployment-time diagnosis.

Reusable Rule

A Docker build proves that an image can be assembled. A release check must prove that the intended process starts and the application can answer a minimal health contract.

Build it, run it, health-check it, then publish that exact artifact.

Sources

CloudFormation Generated Role Names Can Break Least-Privilege IAM Twice

Peter Brau — Thu, 24 Sep 2026 00:15:01 +0000

Problem

Least-privilege CloudFormation execution roles are often scoped to the IAM resources a stack is expected to create. That is a sound approach until the policy assumes a physical role-name pattern that CloudFormation does not actually use.

AWS currently documents that when RoleName is omitted from an AWS::IAM::Role, CloudFormation generates a unique physical ID and uses that value as the role name. The logical resource name is therefore not a reliable policy boundary by itself.

A mismatch can break the forward deployment and then break rollback against the same generated role family.

Failure Signature

The stack begins creating an IAM role, then an IAM operation fails because the execution policy's resource ARN pattern does not match the physical role name CloudFormation generated.

In the observed failure class, one generated role retained the expected naming shape while another generated role used a shorter physical name than the policy anticipated. The first visible denial occurred during tagging. When the stack rolled back, cleanup exposed the same resource-boundary mistake again.

That sequence is a useful signal: if both forward work and rollback are failing against the same IAM resource family, inspect the physical identity before adding more actions.

What We Expected

The execution policy was intended to authorize a bounded family of stack-created IAM roles, not every IAM role in the account.

The expectation was that the generated physical role names would continue to match a longer prefix inferred from the stack and logical-resource naming convention.

Root Cause

The policy was written from the intended naming convention rather than the observed physical resource identity.

A resource scope such as:

arn:aws:iam::<AWS_ACCOUNT_ID>:role/<EXPECTED_BOUNDED_PREFIX>*

is only useful if <EXPECTED_BOUNDED_PREFIX> actually matches the role names CloudFormation creates.

The lesson is not that generated names are unsafe. The lesson is that generated physical IDs are an implementation boundary that least-privilege policy must account for. CloudFormation's current AWS::IAM::Role documentation explicitly says that it generates a unique physical ID for the role name when RoleName is not supplied.

Why the Obvious Fix Was Wrong

The tempting correction is:

arn:aws:iam::<AWS_ACCOUNT_ID>:role/*

That makes a naming mismatch disappear by allowing the execution role to operate against every role in the account. It also destroys the resource boundary the policy was meant to preserve.

A second tempting mistake is to add only the action that failed during the forward path. That can get the deployment farther while leaving rollback unable to detach, untag, or delete the role if a later resource fails.

Least privilege must cover the intended lifecycle, not just the happy-path create call.

Minimal Correction

Use CloudFormation evidence to derive a bounded physical-name contract:

Read the stack event containing the denied IAM action and resource ARN.
Inspect the actual physical role name CloudFormation created or attempted to use.
Compare that name with the resource ARN pattern in the execution policy.
Replace the incorrect assumed prefix with the smallest verified prefix that covers only the intended generated role family.
Review the IAM actions needed for both forward operations and rollback/cleanup.
Keep iam:PassRole separately constrained to the intended role resources and destination service.
Add a policy-contract test for the bounded physical-name pattern.

The exact lifecycle actions depend on the template. Depending on what the stack manages, the forward path may need operations such as role creation, tagging, policy attachment, and role passing; rollback may need the corresponding detach, untag, and delete operations.

For iam:PassRole, AWS documents the iam:PassedToService condition key as a way to restrict which service may receive the role. For an ECS task role, a generic condition shape is:

{
  "Condition": {
    "StringEquals": {
      "iam:PassedToService": "ecs-tasks.amazonaws.com"
    }
  }
}

That condition should complement, not replace, a bounded Resource ARN for the roles that may be passed.

Verification

The forward-path correction is proven when CloudFormation can perform the intended IAM operations against the generated role without broadening the policy to all roles.

The rollback contract should be verified separately. A deployment permission model is incomplete if a later stack failure can leave CloudFormation unable to detach policies, remove tags, or delete resources that the same execution role was allowed to create.

For protected environments, the safest proof is a synthetic or otherwise controlled lifecycle exercise that demonstrates both creation/update and complete rollback under the same bounded policy.

What Should Have Caught This Earlier

Three controls can catch this class before it consumes a long protected deployment:

Physical-name preflight: compare expected IAM ARN patterns with generated physical names observed in a safe change or prior synthetic stack.
Forward/rollback policy matrix: for each stack-managed IAM resource family, map create/update operations to their cleanup counterparts.
Static policy contract: test that the execution role permits only the verified bounded prefixes and that iam:PassRole retains both resource and service-principal constraints.

If a naming assumption is not backed by an observed physical resource or current CloudFormation behavior, treat it as an unverified dependency.

Reusable Rule

Least-privilege IAM for CloudFormation must be written against what AWS actually creates, not what a logical resource name makes you expect.

And deployment permission is not complete until rollback can undo what the forward path was allowed to create.

Sources

GitHub Actions OIDC AccessDenied May Be a Trust-Policy Problem, Not a Permission-Policy Problem

Peter Brau — Thu, 24 Sep 2026 00:01:14 +0000

Problem

When a GitHub Actions job cannot assume an AWS role through OpenID Connect, the first instinct is often to inspect or widen the role's attached AWS permissions. That can target the wrong layer.

sts:AssumeRoleWithWebIdentity is an identity-and-trust decision. AWS evaluates whether the incoming OIDC identity is allowed to obtain the role before the resulting session receives the role's service permissions.

Failure Signature

The workflow reaches AWS role assumption and fails with an authorization error before it can call the AWS service the job was intended to use.

That is materially different from a workflow that assumes the role successfully and then receives AccessDenied from an AWS service API.

The first failure class points to OIDC identity and trust. The second points to the permissions granted after assumption.

What We Expected

A GitHub Actions workload with id-token: write should request an OIDC token whose issuer, audience, and subject satisfy the AWS role's trust policy. AWS STS should then issue a short-lived role session, after which the role's normal permission policy governs service access.

Root Cause

The failure class we diagnosed came from a mismatch between the workload identity GitHub presented and the identity the AWS trust policy permitted.

The important trust inputs are:

the AWS OIDC provider for token.actions.githubusercontent.com;
the audience expected by AWS STS, commonly sts.amazonaws.com when using the standard AWS credentials flow;
the exact GitHub sub claim authorized by the trust policy.

The subject deserves special attention because it depends on workflow context. A job that references a GitHub Environment has an environment-oriented subject rather than the branch or pull-request subject a copied example may use.

There is also a newer compatibility boundary. GitHub documents that repositories created after July 15, 2026 use an immutable default subject format that includes owner and repository IDs. Repositories created earlier keep the previous name-based format unless they opt in, while later renames or transfers also move to the immutable format. A trust policy should therefore be derived from the subject format the repository actually uses, not from a hard-coded historical example.

Why the Obvious Fix Was Wrong

Adding s3:*, cloudformation:*, or another AWS service permission to the role cannot make a failed trust decision succeed. Those permissions are available only after AWS has issued the role session.

Widening them during an OIDC failure increases privilege without fixing the failing control.

Minimal Correction

Keep the trust policy narrow and make its claims match the actual workflow identity.

A generic shape is:

{
  "Effect": "Allow",
  "Principal": {
    "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/token.actions.githubusercontent.com"
  },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub": "<GITHUB_SUBJECT>"
    }
  }
}

<GITHUB_SUBJECT> is deliberately a placeholder. Determine it from the repository's current OIDC subject mode and the job context. If the job uses a protected GitHub Environment, keep the trust relationship aligned with that environment and use the environment's protection rules as another control.

A useful diagnostic sequence is:

Confirm the job has permissions: id-token: write.
Confirm the AWS account has the intended GitHub OIDC provider.
Confirm the role trust policy names that provider as the federated principal.
Confirm the aud condition matches the audience requested by the credentials flow.
Confirm the sub condition matches the repository's current GitHub subject format and the job context.
Only after role assumption succeeds, diagnose service-level permissions.

Verification

The correction is validated when the workflow can obtain the intended AWS role session under the expected GitHub context while a context outside the trust boundary remains unauthorized.

An identity-only preflight can stop after role assumption and an STS identity check. That cleanly proves the trust layer before any deployment or mutation is attempted.

What Should Have Caught This Earlier

Treat OIDC trust as a testable contract.

Before a deployment workflow is allowed to execute protected operations, verify:

the repository's current OIDC subject format;
the expected audience;
the environment, branch, or event context encoded in the subject;
the exact federated provider ARN;
a zero-change role-assumption preflight.

For long-lived repositories, record whether they use the previous or immutable GitHub subject format so a rename, transfer, or opt-in does not become a surprise production failure.

Reusable Rule

If GitHub Actions fails at AssumeRoleWithWebIdentity, debug identity and trust first. Do not widen the role's AWS service permissions until the role can actually be assumed.

Sources

CloudFormation Least Privilege Can Fail Outside the Service You Think You Are Deploying

Peter Brau — Thu, 24 Sep 2026 00:01:13 +0000

Problem

A least-privilege CloudFormation execution role can look correct when compared with the resources in a template and still fail during deployment.

The reason is that CloudFormation does not merely translate each resource type into one same-service API call. The resource implementation can perform supporting lookups, associations, tagging, or cleanup through APIs in adjacent AWS services.

If the execution role was designed only from the visible service names in the template, a legitimate provider-side dependency can surface as an unexpected AccessDenied.

Failure Signature

The stack reaches a valid resource operation and fails on an AWS API action that appears to belong to a different service than the resource being created or associated.

Two observed examples illustrate the class:

an Application Load Balancer creation path required ec2:GetSecurityGroupsForVpc;
a Web ACL association path required a load-balancer-side permission in addition to WAF permissions.

The first denial looked surprising because the resource was an Elastic Load Balancing resource and the missing EC2 action was not a Describe* call.

The second looked surprising because the operation was conceptually a WAF association, yet the integration crossed the WAF and load-balancer permission boundary.

What We Expected

The execution role was intended to permit only the API surface required to create, update, and remove the stack's resources.

The mistaken assumption was that the required API surface could be inferred directly from the service namespace of each CloudFormation resource type.

Root Cause

The policy model was based on template service ownership, while CloudFormation's actual resource operation depended on provider-side API behavior.

For the load-balancer case, the provider called the current EC2 API GetSecurityGroupsForVpc. A narrowly scoped correction can grant that action without granting broad EC2 access. For example:

{
  "Effect": "Allow",
  "Action": "ec2:GetSecurityGroupsForVpc",
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "aws:RequestedRegion": "<AWS_REGION>"
    }
  }
}

Resource: "*" does not mean the entire role should be broad. Some AWS APIs do not offer useful resource-level ARN scoping for the request. Least privilege then comes from granting the smallest action set and applying supported condition boundaries.

The Web ACL example also demonstrates why historical incident evidence must be rechecked before reuse. The observed denial involved elasticloadbalancing:SetWebACL. Current AWS WAF documentation now distinguishes that older Application Load Balancer setting from a newer association model that uses load-balancer-side CreateWebACLAssociation and DeleteWebACLAssociation permissions along with WAF permissions.

That change does not weaken the lesson. It strengthens it: provider and integration API paths can evolve, so the exact current denied action and current service documentation are better inputs than a permanent hard-coded permission list.

Why the Obvious Fix Was Wrong

A common response to an unexpected provider-side denial is to attach a broad AWS managed policy for the adjacent service until the deployment succeeds.

That solves uncertainty by expanding privilege. It also discards the most useful evidence in the failure: AWS already told you the exact action that was denied.

For the observed EC2 lookup, broad EC2 read or write access was unnecessary. For the WAF/load-balancer association, broad load-balancer access was unnecessary. The correction could remain focused on the required integration action.

Minimal Correction

Use the denial as a dependency-discovery signal:

Capture the exact denied action and resource, if one is reported.
Identify the CloudFormation resource or association operation active at the time.
Check the current AWS documentation for that provider/integration path.
Determine whether the missing API supports resource-level scoping.
Add only the required action and the strongest supported resource or condition boundary.
Review a fresh change set before protected execution.
Add the newly discovered dependency to a static policy-contract test.

Do not preserve an action forever merely because one historical provider path used it. Revalidate integration permissions when AWS changes the underlying association model.

Verification

The correction is validated when the same resource operation succeeds under the narrowed execution role without requiring a broader managed policy.

For a provider-side dependency, verification should cover both the forward operation and any corresponding update or removal path that the stack can exercise. The policy-contract test should then assert the narrow dependency so a later refactor does not silently remove it.

What Should Have Caught This Earlier

A useful preflight is a resource-provider dependency ledger for protected infrastructure changes.

For each resource family, record adjacent-service actions discovered through current documentation or validated deployment evidence. Static IAM tests can then check that the execution role includes the required narrow actions before a long-running CloudFormation operation begins.

The ledger should be treated as versioned knowledge rather than a permanent truth table. When an AWS integration changes, update the contract instead of accumulating obsolete permissions.

Reusable Rule

When a least-privilege CloudFormation deployment fails, do not ask only, "What service is this resource from?" Ask, "What APIs does the current resource operation call to create, associate, inspect, update, and remove it?"

That question usually leads to a smaller and safer correction than attaching another broad policy.