ECR Image Scan Gate in GitLab CI Passes Vulnerable Images

Originally published on kuryzhev.cloud

---

A merge request goes green, the image is deployed, and a day later someone opens the ECR console and finds a critical CVE on that exact tag. The pipeline had an ECR image scan gate, and it reported success. This runbook covers why a gate like that passes when it should not, and how to rebuild it so it fails closed.

Symptoms

These are the typical signs of a gate that fails open. You may see one or several.

• The gate job is green, but aws ecr describe-image-scan-findings run by hand shows CRITICAL or HIGH findings for the same image.
• The gate passes on fresh pushes and only "works" on re-runs, when the scan has had time to finish.
• The job log shows an empty or None severity count, and the script treats that as zero.
• Intermittent ScanNotFoundException or LimitExceededException errors appear, and someone added || true to quiet them.
• The scanned tag (latest, or a branch name) points at a different image by the time the deploy job runs.

Watch out for: a green job with no printed counts. If the log never shows a number, the gate did not check anything.

Root cause

ECR scans are asynchronous. A push triggers a scan (if scan-on-push is enabled), but the results arrive later. If the gate queries findings immediately, the response either errors (typically ScanNotFoundException) or returns a status such as IN_PROGRESS with no findingSeverityCounts at all.

The severity counts map also omits severities with zero findings. A naive query for .CRITICAL therefore returns null (or None with --output text) in two very different situations: the scan is clean, or the scan has not finished. A script that treats "empty" as "zero" cannot tell these apart, so it passes both.

Three secondary causes make this worse:

• Swallowed errors. || true or a missing set -e turns an API failure into a pass.
• Mutable tags. The gate scans repo:latest, but the deploy job pulls whatever that tag points at later.
• Scan type differences. Basic scanning (on push or manual) and enhanced scanning (Amazon Inspector, continuous) report different statuses and have different manual-scan behavior. The applicable limits and status values depend on how scanning is configured for your registry, so check the ECR image scanning documentation for your setup.

Fixing the ECR image scan gate means removing every path where "no data" becomes "no findings".

Fix #1: Wait for a terminal scan status, then fail closed

Poll the scan status by image digest, and exit non-zero on anything that is not a completed scan. The job below uses only the AWS CLI's built-in --query, so it needs no jq in the image. Only "scan not found yet" is treated as retryable. Any other API error, such as AccessDenied, fails the job immediately.

ecr-scan-gate:
  stage: verify
  image:
    name: amazon/aws-cli:latest   # pin a specific version tag in real pipelines
    entrypoint: [""]              # the image's default entrypoint is "aws", which breaks script lines
  script:
    - set -eu
    - DIGEST=$(cat image-digest.txt)
    - |
      STATUS="NOT_STARTED"
      for i in $(seq 1 30); do
        if OUT=$(aws ecr describe-image-scan-findings \
              --repository-name "$ECR_REPO" \
              --image-id imageDigest="$DIGEST" \
              --query 'imageScanStatus.status' --output text 2>&1); then
          STATUS="$OUT"
        else
          case "$OUT" in
            *ScanNotFoundException*) STATUS="NOT_STARTED" ;;    # scan not registered yet: retry
            *) echo "$OUT"; echo "API error, failing closed"; exit 1 ;;
          esac
        fi
        echo "attempt $i: scan status = $STATUS"
        case "$STATUS" in
          COMPLETE|ACTIVE) break ;;                              # ACTIVE appears with enhanced scanning
          FAILED|UNSUPPORTED_IMAGE|SCAN_ELIGIBILITY_EXPIRED|FINDINGS_UNAVAILABLE)
            echo "scan unusable: $STATUS"; exit 1 ;;
        esac
        sleep 10
      done
      case "$STATUS" in COMPLETE|ACTIVE) ;; *) echo "scan never completed"; exit 1 ;; esac

The loop ends in a failure unless a terminal good status was seen. Timing out is a failure, not a pass. Verify the exact status values your registry returns against the AWS documentation before relying on the case list.

Fix #2: Gate on explicit severity counts with a default

Once the scan is complete, a missing severity key genuinely means zero findings. That default is safe only after Fix #1 has confirmed the status. Append this to the same script, and make the threshold a variable so teams can tune it. Anything other than a number or None, including an empty result from a failed API call, stops the job.

    - |
      count() {
        # prints a number; maps the CLI's "None" (key absent) to 0 -- safe ONLY after status is COMPLETE/ACTIVE
        V=$(aws ecr describe-image-scan-findings \
              --repository-name "$ECR_REPO" \
              --image-id imageDigest="$DIGEST" \
              --query "imageScanFindings.findingSeverityCounts.$1" --output text) || return 1
        case "$V" in
          None) echo 0 ;;
          ''|*[!0-9]*) echo "unexpected count for $1: '$V'" >&2; return 1 ;;
          *) echo "$V" ;;
        esac
      }
      CRIT=$(count CRITICAL) || { echo "could not read CRITICAL count"; exit 1; }
      HIGH=$(count HIGH)     || { echo "could not read HIGH count"; exit 1; }
      echo "CRITICAL=$CRIT HIGH=$HIGH (max allowed HIGH=${MAX_HIGH:-0})"
      [ "$CRIT" -eq 0 ] || { echo "blocked: critical findings"; exit 1; }
      [ "$HIGH" -le "${MAX_HIGH:-0}" ] || { echo "blocked: too many high findings"; exit 1; }

Gotcha: the log must print the counts. When a reviewer later asks why a build passed, "CRITICAL=0 HIGH=0" in the job output is the evidence. Also note that a gate on severity counts alone cannot express an exception for a known, accepted CVE. Handle that with a reviewed allowlist file rather than by loosening the threshold.

Fix #3: Pin to the digest and handle scan triggers deliberately

Scan the digest that was pushed, and hand that same digest to the deploy job. This closes the mutable-tag gap. Capture the digest at push time and publish it as an artifact. The job image must provide both Docker (for example via a Docker-in-Docker service) and the AWS CLI.

build-push:
  stage: build
  script:
    - set -eu
    - aws ecr get-login-password --region "$AWS_REGION" | docker login --username AWS --password-stdin "${ECR_URI%%/*}"
    - docker build -t "$ECR_URI:$CI_COMMIT_SHA" .
    - docker push "$ECR_URI:$CI_COMMIT_SHA"
    - |
      aws ecr describe-images --repository-name "$ECR_REPO" \
        --image-ids imageTag="$CI_COMMIT_SHA" \
        --query 'imageDetails[0].imageDigest' --output text > image-digest.txt
    - echo "IMAGE_DIGEST=$(cat image-digest.txt)" > build.env
  artifacts:
    paths: [image-digest.txt]
    reports:
      dotenv: build.env     # deploy jobs read $IMAGE_DIGEST and deploy by digest, never by tag

Next, decide who triggers the scan. With scan-on-push enabled on the repository, the gate only reads results. If you rely on a manual aws ecr start-image-scan call instead, basic scanning limits how often one image can be rescanned, so a retried job can hit LimitExceededException. Treat that specific error as "scan already requested, keep polling", and fail on every other error. With enhanced scanning, manual scans are not the mechanism at all, so check the AWS docs for your configuration.

Finally, scope IAM per job. The gate job needs only ecr:DescribeImageScanFindings on that repository. The build job needs ecr:GetAuthorizationToken (which applies to all resources), the push actions (ecr:BatchCheckLayerAvailability, ecr:InitiateLayerUpload, ecr:UploadLayerPart, ecr:CompleteLayerUpload, ecr:PutImage), and ecr:DescribeImages to read the digest. Obtain these roles through GitLab's ID token authentication rather than stored access keys. A missing permission otherwise surfaces as an AccessDenied that an old || true would hide.

Prevention

Make the failure mode part of the pipeline design, not a one-time patch. Use this checklist when you add or review the gate.

[ ] Scan-on-push is enabled on the repository (or enhanced scanning is configured at registry level)
[ ] Gate queries by imageDigest, not by mutable tag
[ ] Gate waits for a terminal status; a timeout exits non-zero
[ ] FAILED / UNSUPPORTED_IMAGE / SCAN_ELIGIBILITY_EXPIRED / FINDINGS_UNAVAILABLE exit non-zero
[ ] Only ScanNotFoundException is retried; other API errors exit non-zero
[ ] Missing severity key is read as 0 ONLY after a completed status
[ ] Non-numeric or empty counts exit non-zero
[ ] No "|| true" and "set -eu" is active in the script
[ ] Counts are printed in the job log
[ ] Deploy job consumes the same digest via dotenv artifact
[ ] IAM roles are least-privilege per job and obtained via OIDC, not static keys
[ ] A known-vulnerable test image proves the gate actually fails

The last item matters most. Push an old base image with known critical CVEs to a throwaway repository and confirm the job goes red. A gate that has never failed has never been shown to work.

Also plan for drift. A clean image today can have new findings next week, and a one-time gate at build time will not see them. Enhanced scanning re-evaluates images continuously. You can combine it with a scheduled GitLab pipeline that re-runs the digest check against what is currently deployed. For more pipeline and AWS runbooks, browse kuryzhev.cloud.

Related

• Step Functions Orchestration for Lambda and S3: Which Pattern?
• CI/CD Quality Gate Mistakes: Approvals and Rollback Pitfalls
• DataSync EventBridge Automation Checklist for S3 Transfers