DEV Community: When It Runs

When cert-manager's Ready Condition Runs Past Expiry Until a Reconcile

When It Runs — Sat, 24 Oct 2026 00:21:22 +0000

On cert-manager v1.21.2, a renewal-blocked Certificate kept Ready=True ("Certificate is up to date and has not expired") past notAfter through our 30-minute window, until a restart reconciled it.

When It Runs — Run Report #RR10 · Testing what infrastructure actually does.

Provider · Component: cert-manager · the Certificate Ready condition (readiness controller) after status.notAfter has passed, when renewal is blocked

Versions tested: measured — cert-manager v1.21.2 (the latest release as of 2026-09-11; v1.21.3, released on 2026-10-23, was not measured; single version). reported — v1.18.2 (issue #7895; a report-layer detail, not measured here). source-read — the v1.21.2 tag.

Config profile: a single-node kind cluster (node kindest/node:v1.33.4, digest-pinned) on a dedicated test environment; a SelfSigned Issuer → a Certificate (duration: 1h, renewBefore: 55m, RSA-2048); renewal blocked by deleting the Issuer after issuance (a block that generates no further events); 30-second polling; an observation window of ≥30 minutes past notAfter; then a controller restart.

Verified on: 2026-10-23 (desk sources re-checked; PR #8529 open and unmerged, and v1.21.3 the latest release) · lab measured 2026-09-28 on v1.21.2.

Affects: cert-manager v1.21.2 Certificates whose renewal is blocked and whose state produces no further change events on the Certificate, its Secret, or its CertificateRequest (measured here: the Issuer was deleted; reported elsewhere: e.g. an ACME DNS01 failure): after status.notAfter passes, Ready=True and the message "Certificate is up to date and has not expired" can remain until the next reconcile (an event, the informer resync, or a restart). The length of that window is only what we measured, within this condition.

TL;DR: On cert-manager v1.21.2, a renewal-blocked 1-hour Certificate stayed READY=True with "Certificate is up to date and has not expired" for the whole 30-minute window we watched after the certificate in its Secret had actually expired, and changed to Ready=False ("Certificate expired on …") only after the controller was restarted, about 34 minutes past expiry — while the API defines Ready to mean, among other things, "a certificate that has not expired."

Full report: https://whenitruns.substack.com/p/when-cert-managers-ready-condition

Reproduction repo: https://github.com/whenitruns/when-cert-managers-ready-condition-runs-past-expiry-until-a-reconcile

When an Argo CD App-of-Apps Wave Runs Past a Child With a Missing Resource

When It Runs — Sat, 17 Oct 2026 19:16:10 +0000

With the documented argoproj.io_Application health check, Argo CD v3.5.3 started wave 2 while a child missing a resource read Healthy; on v3.3.14 that child read Missing and the gate held.

When It Runs — Run Report #RR09 · Testing what infrastructure actually does.

Provider · Component: Argo CD · the sync-wave gate of an app-of-apps parent — what the parent waits for before it starts the next wave of child Applications, when it uses the Application health check from Argo CD's documentation

Versions tested: measured — Argo CD v3.5.3 (the latest release as of 2026-09-26; v3.5.4 and v3.5.5, released on 2026-10-06 and 2026-10-12, were not measured) and v3.3.14 (the latest 3.3 release as of 2026-09-26; v3.3.15 and v3.3.16, released on 2026-10-06 and 2026-10-12, were not measured), with identical inputs; both are supported releases. source-read — controller/health.go at both tags. The behavior change came in v3.4.0 (pull request #25962).

Config profile: a single-node kind cluster (node kindest/node:v1.33.4, digest-pinned; a Kubernetes version both releases list as tested), a fresh cluster per release, Argo CD installed from the release's official install.yaml; in argocd-cm, the documented argoproj.io_Application health check plus a health check registered for the kind that fails to apply; a local Git server holding one commit; a parent Application and three child Applications in sync waves 0, 1 and 2 (automated sync); the wave-1 child applies a Job (inner wave 0) and a custom resource whose CRD is not installed (inner wave 1, SkipDryRunOnMissingResource=true); 10-second polling for 10 minutes.

Verified on: 2026-10-13 (desk sources re-checked; v3.5.5 was the latest release and v3.3.16 the latest 3.3 release) · lab measured 2026-09-27 on v3.5.3 and v3.3.14.

Affects: Argo CD 3.4 and later (measured: v3.5.3) app-of-apps setups that gate sync waves on child Applications with the documented argoproj.io_Application health check: a child Application with a resource that failed to apply — of a kind that has a health check, with its dry run skipped — can report Healthy, and the parent can start the next wave. With the same inputs, v3.3.14 reports that child as Missing and holds the gate.

TL;DR: On Argo CD v3.5.3, an app-of-apps parent using the documented Application health check began creating its wave-2 child 6 seconds after the parent was created, while the wave-1 child reported Healthy with its custom resource (whose CRD is not installed) still unapplied; that resource failed to apply 36 milliseconds later, and the child stayed Healthy and OutOfSync for the rest of the 10-minute window. With the same inputs, v3.3.14 reported that child as Missing and held the gate for the whole window.

Full report: https://whenitruns.substack.com/p/when-an-argo-cd-app-of-apps-wave

Reproduction repo: https://github.com/whenitruns/when-an-argo-cd-app-of-apps-wave-runs-past-a-child-with-a-missing-resource

When a cert-manager Keystore Runs Behind a Password Change

When It Runs — Sat, 17 Oct 2026 15:10:51 +0000

cert-manager v1.21.2's API reference: "The keystore file will be updated immediately." After a password change, the keystores opened only with the old password throughout our 30-minute window.

When It Runs — Run Report #RR08 · Testing what infrastructure actually does.

Provider · Component: cert-manager · the keystore outputs of a Certificate (spec.keystores.jks / spec.keystores.pkcs12 with create: true) — what the keystore entries in the target Secret hold after the keystore password is changed

Versions tested: measured — cert-manager v1.21.2 (current supported release; latest as of 2026-09-11; single version). source-read — the v1.21.2 tag. quoted, not measured — the same API comment at the v1.10.0 and v1.11.0 tags.

Config profile: a single-node kind cluster (node kindest/node:v1.33.4, digest-pinned) on a dedicated test environment; a SelfSigned Issuer → Certificates with keystores.jks and keystores.pkcs12 (profile: Modern2023), both create: true; the keystore password supplied through passwordSecretRef (two Certificates) or the inline password field (one Certificate); the password changed in place (the referenced Secret updated, the inline field patched); 30-second polling; an observation window of ≥30 minutes; then a manual re-issuance with cmctl renew.

Verified on: 2026-10-13 (desk sources re-checked; cert-manager v1.21.2 was the latest release) · lab measured 2026-09-26 on v1.21.2.

Affects: cert-manager v1.21.2 Certificates with keystores.jks or keystores.pkcs12 set to create: true: after the keystore password is changed — in the Secret referenced by passwordSecretRef, or in the inline password field — the keystore entries in the target Secret stay encrypted with the previous password until the certificate is next issued (observed for the 30-minute window we watched; beyond it, from the source), while the Certificate stays Ready=True.

TL;DR: On cert-manager v1.21.2, after the keystore password was changed in the Secret referenced by passwordSecretRef (or in the inline password field), keystore.jks and keystore.p12 in the target Secret still opened only with the old password for the whole 30-minute window we watched, while the Certificate stayed Ready; re-issuing the certificate switched them to the new password — yet the API reference says the keystore file "will be updated immediately."

Full report: https://whenitruns.substack.com/p/when-a-cert-manager-keystore-runs

Reproduction repo: https://github.com/whenitruns/when-a-cert-manager-keystore-runs-behind-a-password-change

When Keycloak's Event Store Runs on Defaults Without `enabled_event_types`

When It Runs — Wed, 14 Oct 2026 21:54:39 +0000

Keycloak 26.7.4 + keycloak provider v5.9.0, enabled_event_types omitted ("Omitting this field enables all event types"): the event store kept LOGIN, not REFRESH_TOKEN; the log listener got both.

When It Runs — Run Report #RR07 · Testing what infrastructure actually does.

Provider · Component: Keycloak · realm user-event storage (the event store) — what gets saved when user events are turned on through the keycloak organization's IaC provider (keycloak_realm_events) with enabled_event_types left out

Versions tested: measured — Keycloak 26.7.4 with the keycloak provider v5.9.0 (keycloak/terraform-provider-keycloak; both the latest release as of 2026-09-26; Keycloak 26.7.5, 26.8.0 and 26.8.1, released on 2026-09-30, 2026-10-01 and 2026-10-13, and provider v5.10.0, released on 2026-10-05, were not measured; a single version pair), applied with OpenTofu 1.12.6. source-read — the Keycloak 26.7.4 and provider v5.9.0 tags.

Config profile: the official image quay.io/keycloak/keycloak:26.7.4 (digest-pinned) in start-dev mode on a dedicated test VM, with the event listener's log category at debug; through OpenTofu, two realms with keycloak_realm_events (events_enabled = true, events_listeners = ["jboss-logging"]) — enabled_event_types left out in rr07, set to four types in the comparison realm rr07-explicit; in each realm a public client with direct access grants and one user; a successful and a failed password grant, then a successful and a failed refresh-token grant.

Verified on: 2026-10-13 (desk sources re-checked; Keycloak 26.8.1 and provider v5.10.0 were the latest releases) · lab measured 2026-09-27 on Keycloak 26.7.4 with provider v5.9.0.

Affects: Keycloak 26.7.4 realms whose user-event saving is turned on through the keycloak provider v5.9.0 keycloak_realm_events resource with enabled_event_types left out: the event store keeps only Keycloak's default saved types, so REFRESH_TOKEN and REFRESH_TOKEN_ERROR are not saved (checked 5 and 120 seconds after the requests). Event listeners, such as the jboss-logging log listener, still receive those events.

TL;DR: With Keycloak 26.7.4 and the keycloak provider v5.9.0, turning on realm events without enabled_event_types, which the provider documents as "Omitting this field enables all event types," saved LOGIN and LOGIN_ERROR to the event store but not REFRESH_TOKEN or REFRESH_TOKEN_ERROR, which reached only the log listener.

1. The Claim

The claim here is the keycloak organization's own IaC provider documentation, set against what Keycloak's event store actually keeps.

What the provider documents. The keycloak provider (keycloak/terraform-provider-keycloak, used from Terraform or OpenTofu) documents the enabled_event_types field of its keycloak_realm_events resource as follows (docs/resources/realm_events.md at the v5.9.0 tag, L45; checked 2026-09-26):

"enabled_event_types - (Optional) The event types that will be saved to the database. Omitting this field enables all event types. Defaults to [] or all event types."

The example on the same page carries the comment "When omitted or left empty, keycloak will enable all event types" (L26), and describes events_enabled as: "When true, events from enabled_event_types are saved to the database, making them available through the admin console. Defaults to false." (L43).

How the default is commonly read. A third-party Keycloak auditing guide (skycloak.io; published 2026-06-15, updated 2026-09-07; checked 2026-09-26) describes the same setting in the admin console the same way: "The default is all types; for a lean DB footprint, select only the types you actively query." The same guide later suggests alerting on "Spikes in REFRESH_TOKEN_ERROR or CODE_TO_TOKEN_ERROR".

What Keycloak documents. Keycloak's server administration guide presents the saved event types as a list you add to — "Click *Add saved types to see other events you can save."* (docs/documentation/server_admin/topics/events/login.adoc L20, 26.7.4) — and the EventType Javadoc describes isSaveByDefault() as: "Determines whether this event is stored when the admin has not set a specific set of event types to save." So Keycloak does say that an empty setting means a default subset. The gap this report looks at is between that behavior and the provider's "Omitting this field enables all event types."

Why it would behave that way (source layer). In Keycloak 26.7.4, EventBuilder.sendNow (server-spi-private/src/main/java/org/keycloak/events/EventBuilder.java L261–277) stores an event only if:

if (eventTypes.isEmpty() && event.getType().isSaveByDefault() || eventTypes.contains(event.getType().name())) {

— that is, when the realm's list of saved types is empty, only types flagged "save by default" go to the event store. Then, outside that condition, every configured event listener receives the event (L270–276). In EventType.java, LOGIN(0, true) (L29) and LOGIN_ERROR (L30) are flagged true, while REFRESH_TOKEN(5, false) (L42) and REFRESH_TOKEN_ERROR (L43) are flagged false. When the list is empty, GET …/events/config answers with the default-saved types filled in rather than an empty list (RealmAdminResource.java L845–850). On the provider side (v5.9.0), omitting enabled_event_types sends an empty list (provider/resource_keycloak_realm_events.go L60; the API field enabledEventTypes has no omitempty, keycloak/realm_events.go L11), and the field is read back into state only when it is set in the configuration (L94) — which suggests tofu plan would report no difference, something we checked in the lab (§3).

What this report tests. When realm events are turned on through keycloak_realm_events with enabled_event_types left out — which the provider documents as "Omitting this field enables all event types" — does Keycloak save all event types, or only its default saved types, leaving REFRESH_TOKEN (success and error) and similar events out of the event store? The event store is what the admin console's Events tab, the admin REST events API, and the event database table show. Event listeners are a separate path: per the source above they receive every event regardless of the saved types, and we observed the jboss-logging listener separately (§3).

2. The Setup

Environment: a dedicated test VM with Docker; Keycloak from the official image quay.io/keycloak/keycloak:26.7.4, pinned by digest (sha256:82a77884f3af238beab1e7afd63b5f530e1b5c0590bd7aa60b40a40463e29b2c; the linux/amd64 image in it, sha256:3d911baa186f352563854039b95f21a7e2c01c76b527fdc64f24a0885b927bdf, is what ran), started with start-dev and the bootstrap-admin variables from Keycloak's container guide. The log level is INFO,org.keycloak.events:debug: org.keycloak.events is the category of the jboss-logging event listener, which logs successful events at debug by default, so this makes its success lines visible; it does not change what the event store keeps. curl 8.5.0 and jq 1.7 for the REST calls.
Version: Keycloak 26.7.4 · the keycloak provider v5.9.0 (release asset terraform-provider-keycloak_5.9.0_linux_amd64.zip, sha256 matching the release's SHA256SUMS) · OpenTofu 1.12.6 (release asset sha256 matching its SHA256SUMS). A single version pair.
Configuration: one OpenTofu configuration (§4) creating two realms. In rr07, keycloak_realm_events sets events_enabled = true and events_listeners = ["jboss-logging"] and leaves enabled_event_types out. In rr07-explicit, it is the same except enabled_event_types = ["LOGIN", "LOGIN_ERROR", "REFRESH_TOKEN", "REFRESH_TOKEN_ERROR"]. events_listeners is set explicitly in both, as in the provider's own example. Each realm has a public client rr07-app with direct access grants and a user alice with a dummy password. In each realm, in the same order: a successful password grant (LOGIN), a password grant with a wrong password (LOGIN_ERROR), a successful refresh-token grant (REFRESH_TOKEN), and a refresh-token grant with an altered token (REFRESH_TOKEN_ERROR). The event store is queried 5 seconds and 120 seconds after the last request.

3. Expected vs. Observed — with `enabled_event_types` left out, which events does the event store keep?

Expected (from the provider documentation): "Omitting this field enables all event types" — so in rr07 all four event types would be saved, the same as in rr07-explicit.

Observed (Keycloak 26.7.4 with provider v5.9.0 via OpenTofu 1.12.6, measured 2026-09-27; all times UTC; raw in our lab record):

Apply, then plan: tofu apply created the eight resources — "Apply complete! Resources: 8 added, 0 changed, 0 destroyed." — and tofu plan -detailed-exitcode right after it exited 0 with "No changes. Your infrastructure matches the configuration." Run again after the requests, it again exited 0.
The saved types the server reports (GET …/events/config): for rr07, eventsEnabled: true, eventsListeners: ["jboss-logging"], and an enabledEventTypes list of 103 types that includes LOGIN and LOGIN_ERROR but not REFRESH_TOKEN or REFRESH_TOKEN_ERROR — the list the server returns when the realm's own list is empty (§1); for rr07-explicit, the four listed types.
The requests (23:03:37–23:03:39): in both realms, the password grant returned HTTP 200; the wrong password, 400 invalid_grant ("Invalid user credentials"); the refresh, 200; the altered refresh token, 400 invalid_grant ("Invalid refresh token").
The event store (GET /admin/realms/<realm>/events, at +5 s and +120 s): the same at both times — rr07 held two events, one LOGIN and one LOGIN_ERROR, and no REFRESH_TOKEN or REFRESH_TOKEN_ERROR; rr07-explicit held one of each of the four.
The listener (the jboss-logging lines in the container log): in both realms, one line for each of the four types — LOGIN and REFRESH_TOKEN at DEBUG, LOGIN_ERROR and REFRESH_TOKEN_ERROR at WARN (the latter with error="invalid_token" and reason="Invalid refresh token").

Raw evidence: the OpenTofu output, the events/config responses, each request with its HTTP status and response, the event-store responses at both query times, and the container log, all kept in our lab record.

4. Reproduction

The minimal reproduction (raw output is kept in our lab record; the configuration and the commands below are in the reproduction repo linked below). All times UTC.

# 0. Start Keycloak 26.7.4 (official image, digest-pinned) in dev mode, with the event listener's
#    log category at debug so its success lines are visible, and wait until it has started.
#      docker run -d --name rr07-kc -p 8080:8080 -e KC_BOOTSTRAP_ADMIN_USERNAME=admin -e KC_BOOTSTRAP_ADMIN_PASSWORD=change_me quay.io/keycloak/keycloak:26.7.4@sha256:82a77884f3af238beab1e7afd63b5f530e1b5c0590bd7aa60b40a40463e29b2c start-dev --log-level="INFO,org.keycloak.events:debug"
#      for i in $(seq 1 60); do docker logs rr07-kc 2>&1 | grep -E 'Keycloak 26\.7\.4 on JVM .* started' && break; sleep 3; done   # the startup line, at most 3 minutes

# 1. Apply the OpenTofu configuration below (main.tf; OpenTofu 1.12.6, provider keycloak/keycloak 5.9.0),
#    then check for drift right away (exit code 0 = no changes, 2 = changes).
#      tofu init
#      tofu apply -auto-approve
#      tofu plan -detailed-exitcode

# 2. Read the saved types the server reports for each realm (admin token from the master realm).
#      TOKEN=$(curl -s -d grant_type=password -d client_id=admin-cli -d username=admin -d password=change_me http://localhost:8080/realms/master/protocol/openid-connect/token | jq -r .access_token)
#      curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8080/admin/realms/rr07/events/config | jq .
#      curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8080/admin/realms/rr07-explicit/events/config | jq .

# 3. Generate the four events in each realm (run once with REALM=rr07, then with REALM=rr07-explicit).
#      T=http://localhost:8080/realms/$REALM/protocol/openid-connect/token
#      RT=$(curl -s -d grant_type=password -d client_id=rr07-app -d username=alice -d password=alice-pass-1 $T | jq -r .refresh_token)   # LOGIN
#      curl -s -d grant_type=password -d client_id=rr07-app -d username=alice -d password=wrong-password $T                                  # LOGIN_ERROR
#      curl -s -d grant_type=refresh_token -d client_id=rr07-app -d refresh_token="$RT" $T                                                   # REFRESH_TOKEN
#      curl -s -d grant_type=refresh_token -d client_id=rr07-app -d refresh_token="${RT}x" $T                                                # REFRESH_TOKEN_ERROR (altered token)

# 4. Query each realm's event store 5 seconds and 120 seconds after the last request (fresh admin token each time).
#      TOKEN=$(curl -s -d grant_type=password -d client_id=admin-cli -d username=admin -d password=change_me http://localhost:8080/realms/master/protocol/openid-connect/token | jq -r .access_token)
#      for REALM in rr07 rr07-explicit; do echo "== $REALM"; curl -s -H "Authorization: Bearer $TOKEN" "http://localhost:8080/admin/realms/$REALM/events?max=100" | jq -r '.[].type' | sort | uniq -c; done

# 5. Look at what the jboss-logging listener wrote for the same requests.
#      docker logs --timestamps rr07-kc 2>&1 | grep 'org.keycloak.events'

# 6. Tear down (the test container only).
#      docker rm -f rr07-kc

The OpenTofu configuration for step 1 (main.tf), exactly as we applied it:

terraform {
  required_providers {
    keycloak = {
      source  = "keycloak/keycloak"
      version = "5.9.0"
    }
  }
}

provider "keycloak" {
  client_id = "admin-cli"
  username  = "admin"
  password  = "change_me"
  url       = "http://localhost:8080"
}

# Realm rr07: user events on, enabled_event_types left out.
resource "keycloak_realm" "rr07" {
  realm   = "rr07"
  enabled = true
}

resource "keycloak_realm_events" "rr07" {
  realm_id         = keycloak_realm.rr07.id
  events_enabled   = true
  events_listeners = ["jboss-logging"]
}

# Realm rr07-explicit: the same, with the four types listed.
resource "keycloak_realm" "rr07_explicit" {
  realm   = "rr07-explicit"
  enabled = true
}

resource "keycloak_realm_events" "rr07_explicit" {
  realm_id            = keycloak_realm.rr07_explicit.id
  events_enabled      = true
  events_listeners    = ["jboss-logging"]
  enabled_event_types = ["LOGIN", "LOGIN_ERROR", "REFRESH_TOKEN", "REFRESH_TOKEN_ERROR"]
}

# In each realm: a public client with direct access grants, and one user.
locals {
  realm_ids = {
    "rr07"          = keycloak_realm.rr07.id
    "rr07-explicit" = keycloak_realm.rr07_explicit.id
  }
}

resource "keycloak_openid_client" "app" {
  for_each                     = local.realm_ids
  realm_id                     = each.value
  client_id                    = "rr07-app"
  access_type                  = "PUBLIC"
  standard_flow_enabled        = false
  direct_access_grants_enabled = true
}

resource "keycloak_user" "alice" {
  for_each       = local.realm_ids
  realm_id       = each.value
  username       = "alice"
  email          = "alice@example.com"
  email_verified = true
  first_name     = "Alice"
  last_name      = "Example"

  initial_password {
    value     = "alice-pass-1"
    temporary = false
  }
}

Outputs for each step land in §3 (above) and our lab record.

A recorded run of this reproduction: https://youtu.be/yFowKYN7kcw

Reproduction repo: https://github.com/whenitruns/when-keycloaks-event-store-runs-on-defaults-without-enabled-event-types

5. Impact

An audit that relies on the event store — the admin console's Events tab, the admin REST events API, or the event database table — has no record of refresh-token activity, successful or failed, in a realm whose events were turned on through this provider with enabled_event_types left out — in our run, neither event type was stored. The belief this quietly breaks is "the provider says omitting the list saves all event types, so refresh-token failures would be there." An investigator who finds no REFRESH_TOKEN_ERROR entries can read that as "no refresh failures happened." The log-listener path is a different matter: in our run, the jboss-logging listener wrote those events to the server log. This is about the completeness of an audit configuration, not a way around authentication: refresh tokens are validated as usual; the events simply are not stored.

6. What We Couldn't Conclude

Other listeners and log settings. We observed the jboss-logging listener with its log category at debug. Other listeners, and other log levels, were not measured.
The plan result beyond our run. The tofu plan result in §3 is what we observed in this run; we make no broader claim about when OpenTofu or Terraform would or would not report a difference.
Realms that already have a saved-types list. In the source, the provider sends an explicit empty list when the field is left out, and Keycloak replaces a realm's saved types with any list it receives that is not null. We measured only newly created realms and did not measure what happens when this configuration is applied to a realm that already has a custom list.
Other tools using this provider. We applied the provider with OpenTofu only.
Other versions. Only Keycloak 26.7.4 with provider v5.9.0; we make no claim about other versions. Keycloak 26.7.5, 26.8.0 and 26.8.1 (released on 2026-09-30, 2026-10-01 and 2026-10-13) and provider v5.10.0 (released on 2026-10-05) were not measured.
Which other types are left out by default. We name only the two we measured. To see which types your realm saves, read GET /admin/realms/<realm>/events/config or the saved-types list in the admin console.
Why the provider documents "all event types." We don't know, and we don't infer it.
Admin events. Admin events are a separate setting and are not covered here.

7. Conclusion

With Keycloak 26.7.4 and the keycloak provider v5.9.0, in the setup recorded above, turning on realm events with enabled_event_types left out kept LOGIN and LOGIN_ERROR in the event store but not REFRESH_TOKEN or REFRESH_TOKEN_ERROR, while listing the four types in the comparison realm saved all four. The provider documents the omitted field as "Omitting this field enables all event types"; Keycloak applies its default saved types when the list is empty, and its admin guide presents the saved types as a list you add to. The point of this report is the gap between those two descriptions, recorded as of the Verified-on date and within this scope only.

When a Renewed cert-manager CA Runs on the Same Key but a New Key Identifier

When It Runs — Wed, 14 Oct 2026 13:14:31 +0000

On cert-manager v1.21.2, a v1.18.6-issued rotationPolicy: Never CA renewed with the same key got a new SKI (SHA-1 → truncated SHA-256); OpenSSL accepted a pre-renewal leaf only with the old CA too.

When It Runs — Run Report #RR06 · Testing what infrastructure actually does.

Provider · Component: cert-manager · Subject Key Identifier generation on the CA Certificate renewal path (the controller's Go crypto/x509 CreateCertificate, with rotationPolicy: Never)

Versions tested: measured — cert-manager v1.18.6 issuing (the latest 1.18 patch; vendor-listed EOL 2026-03-10) → in-place upgrade through v1.19.6 → v1.20.4 (each minor's latest patch, following the vendor upgrade guidance; no behavioral assertions on the transit versions) → v1.21.2 renewing (current supported release; latest as of 2026-09-11). source-read — the controller cmd/controller/go.mod go line: v1.18.6 go 1.24.0 · v1.19.0 go 1.25.0 (the first affected version; the 1.19 line is EOL 2026-07-08) · v1.21.2 go 1.26.0.

Config profile: a single-node kind cluster (node kindest/node:v1.33.4, digest-pinned) on a dedicated test environment; a SelfSigned Issuer → a CA Certificate (isCA: true, rotationPolicy: Never, RSA-2048) → a CA Issuer → a leaf Certificate; the CA renewed in place with the same key.

Verified on: 2026-10-13 (desk sources re-checked; the go.mod lines, PR #8480 open and unmerged, and v1.21.2 the current supported release) · lab measured 2026-09-24 (v1.18.6 → v1.21.2).

Affects: cert-manager CA Certificates that were issued with rotationPolicy: Never on ≤1.18 and are then renewed with the same key on ≥1.19. The controller does not set the certificate's Subject Key Identifier itself, and from v1.19.0 the controller is built with go 1.25.0, whose crypto/x509 fills a missing SKI with a truncated SHA-256 (earlier: SHA-1). A leaf issued before the renewal therefore carries an Authority Key Identifier derived from the old SHA-1 CA SKI, which no longer matches the renewed CA's SHA-256 SKI. Whether this breaks path building/verification is client-dependent — see §5 and §6; measured: with OpenSSL 3.0.13's openssl verify, the pre-renewal leaf failed against a trust store holding only the renewed CA and verified against the old and renewed CA bundled together.

TL;DR: On cert-manager v1.21.2, renewing a v1.18.6-issued rotationPolicy: Never CA with the same key left the public key unchanged but changed the CA's Subject Key Identifier from SHA-1 to a truncated SHA-256, so a pre-renewal leaf's Authority Key Identifier no longer matched the renewed CA — and openssl verify (OpenSSL 3.0.13) rejected that leaf against the renewed CA alone ("unable to get local issuer certificate") while accepting it against the old and renewed CA bundled.

1. The Claim

This report starts from a promise operators lean on, a change recorded only one layer down, and a report that connects the two.

What the docs promise about rotationPolicy: Never. cert-manager's Certificate usage documentation, describing the rotationPolicy field, states (checked 2026-09-14) that with this setting "cert-manager reuses the existing private key on each issuance", and:

"With rotationPolicy: Never, a private key is only generated if one does not already exist in the target Secret resource (using the tls.key key). All further issuances will re-use this private key."

Never was the default until cert-manager 1.18: the 1.18 release notes announce "We have changed the default value of Certificate.Spec.PrivateKey.RotationPolicy from Never to Always" and advise operators to "Explicitly set rotationPolicy: Never on your Certificate resources" to keep the old behavior (checked 2026-09-14). So a long-lived CA issued on an older release, and kept on the old behavior, re-uses one private key across renewals. The surface an operator reads promises private-key continuity. It does not say anything about the continuity of the certificate's identifiers.

Where the identifier change is actually recorded. The Subject Key Identifier is generated by the Go standard library, and its generation changed. The Go 1.25 release notes state (checked 2026-09-14):

"CreateCertificate now uses truncated SHA-256 to populate the SubjectKeyId if it is missing. The GODEBUG setting x509sha256skid=0 reverts to SHA-1."

Whether a given build generates SKIs the new way (truncated SHA-256) or the old way (SHA-1) is decided by the go line of the main module, per Go's godebug documentation (checked 2026-09-14): a GODEBUG setting's default "is derived from … the defaults for the Go toolchain used to build the program, amended to match the Go version listed in go.mod, and then overridden by explicit //go:debug lines in the program." cert-manager's controller module declares go 1.25.0 from v1.19.0 onward (v1.18.6 declares go 1.24.0; v1.21.2 declares go 1.26.0 — read from the tagged go.mod), its packet-of-interest source does not set SubjectKeyId itself, and there is no //go:debug x509sha256skid=… or GODEBUG override in the module. A cert-manager ≥1.19 build therefore generates SHA-256 CA SKIs by default, and a ≤1.18 build generated SHA-1 ones. cert-manager's own release notes for 1.18 through 1.21 and its 1.17→1.18 and 1.18→1.19 upgrade guides do not mention this change (checked 2026-09-14).

The report this test is built on. In issue #8289 (checked 2026-09-14) a user described issuing a rotationPolicy: Never CA Certificate on v1.10.0, then — after upgrading to v1.19.1 — renewing it and finding that the private key was unchanged but the CA's SKI was different. The reporter traced the cause to the Go 1.25 change and closed the issue themselves as "not a bug." Two later comments recorded operational impact (for example, disruption to Galera clusters) — a report layer this report does not itself reproduce.

What this report tests. Not whether the docs are wrong or whether Go's default is wrong — both are documented where they live. The gap is that a rotationPolicy: Never CA renewed across the go 1.24 → 1.25 boundary keeps its key but changes its SKI, so a leaf signed before that renewal carries an Authority Key Identifier that points at an SKI the renewed CA no longer has. §3 measures the identifiers before and after the renewal; whether any given client then fails to build a path is a separate, client-dependent question handled in §5 and §6.

2. The Setup

Environment: a single-node kind cluster on a dedicated Linux test VM (GNU coreutils), node image kindest/node:v1.33.4 (digest-pinned). k8s v1.33 is inside the vendor-listed supported range of every cert-manager version exercised here (1.18: k8s 1.29–1.33; 1.19: 1.31–1.35; 1.20: 1.32–1.35; 1.21: 1.33–1.36), so one node version covers the whole issue → upgrade → renew sequence; kubectl within ±1 minor of the node. The SKI mechanism is independent of the Kubernetes version.
Version: cert-manager v1.18.6 to issue (installed from the official Helm chart cert-manager-v1.18.6.tgz from charts.jetstack.io with one non-default value, crds.enabled=true, so the chart installs the CRDs (chart default false); the in-place upgrades below use each minor's official chart with the same value; image digests and chart hashes recorded in our lab record). The 1.18 line is EOL (vendor-listed 2026-03-10); v1.18.6 is its latest patch. The controller is then upgraded in place to v1.19.6 → v1.20.4 → v1.21.2 — one minor at a time, each the minor's latest patch, following the vendor guidance "We recommend that you upgrade cert-manager one minor version at a time, always choosing the latest patch version for the minor version." No behavior is asserted on the transit versions; the renewal is measured on v1.21.2 (the current supported release). The first affected version, v1.19.0 (go 1.25.0), is read from source only.
Configuration: a SelfSigned Issuer; a CA Certificate with isCA: true, rotationPolicy: Never, RSA-2048; a CA Issuer referencing that CA's Secret; and a leaf Certificate issued by the CA Issuer before the upgrade. After upgrading to v1.21.2 the CA Certificate is renewed in place, which under rotationPolicy: Never keeps the same private key.

3. Expected vs. Observed — does a same-key CA renewal keep the CA's Subject Key Identifier stable?

Expected (what the rotationPolicy: Never promise, read by itself, suggests): renewing the CA with the same key keeps the CA's identity stable, so a leaf issued before the renewal still chains to the renewed CA.

Observed (measured 2026-09-24; raw in our lab record). The measurements taken, before the renewal (on v1.18.6) and after (on v1.21.2):

CA public-key fingerprint before vs. after the renewal: unchanged — SPKI SHA-256 84b323b6927d5dfe25b242dce19f4287ed28c1f2b456d3f4f603588eb2bf1a57 both times. The renewal kept the key; only the certificate was re-issued (serial 25034c5688dd96891327b113cfc02eb4 → 19b489e55575e31270275311b74a2454832fb881).
CA Subject Key Identifier before (v1.18.6): 42b062a23924df0a7492534af9e3d64925be0106 (20 bytes) — equal to the SHA-1 of the CA's public key (the subjectPublicKey bit string), which we computed from the certificate itself.
CA Subject Key Identifier after the renewal (v1.21.2): 9de807cf05452aca93f19f6f1ca515f1ceba4809 (20 bytes) — equal to the first 20 bytes of the SHA-256 of that same public key (the truncated SHA-256 form), and different from the value above. The in-place upgrades to v1.19.6, v1.20.4 and v1.21.2 did not re-issue the CA or the leaf on their own (both certificates were byte-identical after each step); the renewal was triggered on v1.21.2 with cmctl renew (cmctl v2.6.1).
the pre-renewal leaf's Authority Key Identifier vs. the renewed CA's Subject Key Identifier: the leaf issued on v1.18.6 carries AKI 42b062a23924df0a7492534af9e3d64925be0106 — the old CA SKI — which does not match the renewed CA's SKI 9de807cf05452aca93f19f6f1ca515f1ceba4809. The leaf itself carries no SKI.
client verification of the pre-renewal leaf against each measured trust store (the client-verification step): one client, OpenSSL 3.0.13 (openssl verify -show_chain -CAfile <store> <leaf>). (a) Renewed CA alone: error 20 at 0 depth lookup: unable to get local issuer certificate, exit code 2. (b) Old CA + renewed CA bundled: OK (chain depth 1 CN = rr06-ca), exit code 0. No optional client (GnuTLS, Java, Go) was run in this round; the trust-store file hashes are in our lab record.

Raw evidence: the CA certificate before and after the renewal and the pre-renewal leaf, the openssl x509 output of their Subject Key Identifier and Authority Key Identifier extensions, and the public-key fingerprints (hex) are kept in our lab record.

4. Reproduction

The minimal reproduction (raw output is kept in our lab record; the manifest and the commands below are in the reproduction repo linked below):

# 0. Bring up a kind cluster (node k8s v1.33.4) on the test VM and install cert-manager v1.18.6
#    from the official Helm chart, with the CRDs (chart hash + image digests recorded; rollout
#    confirmed before any measurement):
#      kind create cluster --name rr06 --image kindest/node:v1.33.4@sha256:25a6018e48dfcaee478f4a59af81157a437f15e6e140bf103f85a2e7cd0cbbf2
#      helm install cert-manager https://charts.jetstack.io/charts/cert-manager-v1.18.6.tgz \
#        -n cert-manager --create-namespace --set crds.enabled=true --wait
#      for d in cert-manager cert-manager-webhook cert-manager-cainjector; do kubectl -n cert-manager rollout status "deploy/$d" --timeout=300s; done

# 1. Create a SelfSigned Issuer, then a CA Certificate that re-uses one key across issuances
#    (Issuer: spec.selfSigned: {}; Certificate: spec.isCA: true, spec.privateKey.rotationPolicy: Never,
#    algorithm RSA size 2048, issuerRef -> the SelfSigned Issuer, secretName -> the CA secret).
#    Then a CA Issuer referencing that CA secret, and a leaf Certificate issued by the CA Issuer.
#      kubectl apply -f step1-manifest.yaml   # the manifest below, saved to this file
#      kubectl -n rr06 wait --for=condition=Ready certificate/rr06-ca --timeout=120s
#      kubectl -n rr06 wait --for=condition=Ready issuer/rr06-ca-issuer --timeout=120s
#      kubectl -n rr06 wait --for=condition=Ready certificate/leaf1 --timeout=120s

# 2. RECORD #1 (still on v1.18.6): the CA's Subject Key Identifier (SHA-1, 20 bytes) and public key,
#    the leaf's Authority Key Identifier (== the CA SKI above), and the CA public-key fingerprint (SPKI SHA-256).
#      kubectl -n rr06 get secret rr06-ca-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > ca-1_v1.18.6.pem
#      kubectl -n rr06 get secret leaf1-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > leaf1-1_v1.18.6.pem
#      openssl x509 -in ca-1_v1.18.6.pem -noout -subject -issuer -serial -ext subjectKeyIdentifier,authorityKeyIdentifier
#      openssl x509 -in ca-1_v1.18.6.pem -noout -pubkey | openssl pkey -pubin -outform DER | sha256sum   # CA public-key fingerprint (SPKI SHA-256)
#      openssl x509 -in ca-1_v1.18.6.pem -noout -pubkey | openssl rsa -pubin -RSAPublicKey_out -outform DER 2>/dev/null | sha1sum   # SHA-1 of the public key (the subjectPublicKey bit string)
#      openssl x509 -in ca-1_v1.18.6.pem -noout -pubkey | openssl rsa -pubin -RSAPublicKey_out -outform DER 2>/dev/null | sha256sum | cut -c1-40   # the first 20 bytes of its SHA-256
#      openssl x509 -in leaf1-1_v1.18.6.pem -noout -subject -issuer -serial -ext subjectKeyIdentifier,authorityKeyIdentifier

# 3. Upgrade cert-manager IN PLACE, one minor at a time, each the latest patch, per the vendor guidance:
#    v1.18.6 -> v1.19.6 -> v1.20.4 -> v1.21.2
#      helm upgrade cert-manager https://charts.jetstack.io/charts/cert-manager-v1.19.6.tgz \
#        -n cert-manager --set crds.enabled=true --wait
#      helm upgrade cert-manager https://charts.jetstack.io/charts/cert-manager-v1.20.4.tgz \
#        -n cert-manager --set crds.enabled=true --wait
#      helm upgrade cert-manager https://charts.jetstack.io/charts/cert-manager-v1.21.2.tgz \
#        -n cert-manager --set crds.enabled=true --wait
#    (confirm rollout + version at each step; no renewal, no assertions on the transit versions.)

# 4. On v1.21.2, RENEW the CA Certificate with the same key (rotationPolicy: Never keeps the key):
#    trigger an in-place renewal of the CA Certificate and wait for it to complete: the revision has
#    changed and Ready is True (at most 120 one-second reads).
#      rev0=$(kubectl -n rr06 get certificate rr06-ca -o jsonpath='{.status.revision}'); echo "CA revision before: $rev0"
#      cmctl renew -n rr06 rr06-ca
#      for i in $(seq 1 120); do rev=$(kubectl -n rr06 get certificate rr06-ca -o jsonpath='{.status.revision}'); rdy=$(kubectl -n rr06 get certificate rr06-ca -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'); [ "$rev" != "$rev0" ] && [ "$rdy" = True ] && break; sleep 1; done
#      echo "CA revision after: $rev (Ready=$rdy)"

# 5. RECORD #2 (on v1.21.2, after the renewal): the CA's Subject Key Identifier now, compared to RECORD #1;
#    the CA public-key fingerprint, compared to RECORD #1 (expected unchanged); and the pre-renewal leaf's
#    Authority Key Identifier, compared to the renewed CA's Subject Key Identifier.
#      kubectl -n rr06 get secret rr06-ca-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > ca-2_v1.21.2_renewed.pem
#      openssl x509 -in ca-2_v1.21.2_renewed.pem -noout -subject -issuer -serial -ext subjectKeyIdentifier,authorityKeyIdentifier
#      openssl x509 -in ca-2_v1.21.2_renewed.pem -noout -pubkey | openssl pkey -pubin -outform DER | sha256sum   # CA public-key fingerprint (SPKI SHA-256)
#      openssl x509 -in ca-2_v1.21.2_renewed.pem -noout -pubkey | openssl rsa -pubin -RSAPublicKey_out -outform DER 2>/dev/null | sha1sum   # SHA-1 of the public key (the subjectPublicKey bit string)
#      openssl x509 -in ca-2_v1.21.2_renewed.pem -noout -pubkey | openssl rsa -pubin -RSAPublicKey_out -outform DER 2>/dev/null | sha256sum | cut -c1-40   # the first 20 bytes of its SHA-256

# 6. CLIENT VERIFICATION of the pre-renewal leaf, in two trust stores:
#    (a) the renewed CA alone;   (b) the old CA + the renewed CA, bundled.
#    Required client (path that reads AKI/SKI): openssl verify. OpenSSL's issuer test, in addition to
#    matching the issuer DN, compares the sub-fields of the subject's authorityKeyIdentifier (keyid,
#    serial, issuer) against the issuer's subjectKeyIdentifier, serial number, and issuer when present
#    (X509_check_issued(3)). Optional, recorded without a path classification: GnuTLS certtool --verify,
#    Java keytool/PKIX, Go crypto/x509 Verify. Client names, versions, and trust-store file hashes: our lab record.
#      cp ca-2_v1.21.2_renewed.pem store-a_renewed-CA-only.pem
#      cat ca-1_v1.18.6.pem ca-2_v1.21.2_renewed.pem > store-b_old-CA+renewed-CA.pem
#      openssl verify -show_chain -CAfile store-a_renewed-CA-only.pem leaf1-1_v1.18.6.pem   # (a)
#      openssl verify -show_chain -CAfile store-b_old-CA+renewed-CA.pem leaf1-1_v1.18.6.pem   # (b)

# 7. Tear down (the test cluster only).
#      kind delete cluster --name rr06

The manifest for step 1 — the SelfSigned Issuer, the CA Certificate, the CA Issuer, and the leaf Certificate, in their own namespace — exactly as we applied it:

apiVersion: v1
kind: Namespace
metadata:
  name: rr06
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: rr06
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: rr06-ca
  namespace: rr06
spec:
  isCA: true
  commonName: rr06-ca
  secretName: rr06-ca-tls
  privateKey:
    algorithm: RSA
    size: 2048
    rotationPolicy: Never
  issuerRef:
    name: selfsigned
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: rr06-ca-issuer
  namespace: rr06
spec:
  ca:
    secretName: rr06-ca-tls
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: leaf1
  namespace: rr06
spec:
  commonName: leaf1.rr06.example.test
  dnsNames:
    - leaf1.rr06.example.test
  secretName: leaf1-tls
  privateKey:
    algorithm: RSA
    size: 2048
  issuerRef:
    name: rr06-ca-issuer
    kind: Issuer

Outputs for each RECORD step land in §3 (above) and our lab record.

A recorded run of this reproduction: https://youtu.be/GMwCRX-3BhU

Reproduction repo: https://github.com/whenitruns/when-a-renewed-cert-manager-ca-runs-on-the-same-key-but-a-new-key-identifier

5. Impact

A client that builds a chain by matching a leaf's Authority Key Identifier to a CA's Subject Key Identifier can, against a trust store that holds only the renewed CA, fail to build a path for a leaf that was issued before the renewal — even though the CA's key never changed and the leaf has not expired (the exact clients and trust stores are measured in the client-verification step; in this run, openssl verify failed with the renewed CA alone and succeeded with the old and renewed CA bundled). The belief this quietly breaks is "the key didn't change, so certificates already issued by this CA keep working." Renewing a rotationPolicy: Never CA in place is the kind of routine action taken for exactly that reason. This is an availability and interoperability property, not a security weakness: nothing here lets a certificate be forged or a check be bypassed. Reported production incidents (e.g. Galera clusters) sit in the report layer of #8289 and are not reproduced here.

6. What We Couldn't Conclude

Client behavior is not exhaustive. We report only the clients and trust-store configurations we measured (the client-verification step). Whether a given library, language runtime, or middlebox builds the path by DN, by key identifier, or by trying alternates is a per-client question; "TLS verification breaks" is not a general claim of this report. The reporter's own note in #8289 — that "openssl verify does succeed" for the bundled certificate while some applications do not — is consistent with that client-dependence.
The private key. This report's claim is about the identifier, not the key. If a renewal ever changed the public-key fingerprint, that would be key rotation — a different mechanism, outside this report — and we record the before/after fingerprint precisely so that case is visible rather than assumed. The #8289 reporter confirmed the key was unchanged.
"Before the renewal," not "before the upgrade." The affected leaf is one issued before the CA is renewed. We do not measure every issue-time/upgrade-time ordering.
The reported production impact (Galera and similar) is a report-layer account by other users, not our reproduction; we do not attribute those incidents.
Out of measured scope: other trust-store shapes, other key algorithms, and intermediate-CA topologies are not covered. The transit versions (v1.19.6, v1.20.4) and the first affected version (v1.19.0) are not measured for renewal behavior — v1.19.0 is read from go.mod only; the renewal is measured on v1.21.2.
Clients that tolerate the mismatch. A client whose path logic tolerates the AKI/SKI mismatch could still verify the pre-renewal leaf, and that outcome would belong to this finding too: the mechanism (key unchanged, SKI SHA-1 → SHA-256, AKI/SKI link broken) holds whether or not a given client tolerates it. In this run, openssl verify did not tolerate it — it failed against the renewed CA alone (§3, §5).

7. Conclusion

On the versions and in the configuration recorded above, renewing a rotationPolicy: Never CA Certificate across cert-manager's go 1.24 → 1.25 build boundary kept the CA's private key while changing its Subject Key Identifier from SHA-1 to a truncated SHA-256, leaving a pre-renewal leaf's Authority Key Identifier pointed at an identifier the renewed CA no longer carries — and openssl verify then rejected that leaf against the renewed CA alone while accepting it with the old CA bundled in; the measured identifiers and the client-verification results are recorded as of the Verified-on date and within this scope only. The change is documented where it originates — the Go 1.25 release notes and the godebug reference — and cert-manager does not set the identifier itself; the point of this report is that the surface an operator reads for rotationPolicy: Never, which promises key re-use, is not where the identifier change is written down.

When a Kyverno Wildcard Guardrail Runs Past a Post-Policy CRD Until a Restart

When It Runs — Sat, 26 Sep 2026 09:48:11 +0000

`kinds: ["/"] + operations: [DELETE] on Kyverno v1.19.1 (crdWatcher` on): "no policies matched admission request" for a post-policy CRD's custom resource until the admission controller restarted — a pre-policy CRD's resource was denied before and after.

When It Runs — Run Report #RR05 · Testing what infrastructure actually does.

Provider · Component: Kyverno · ClusterPolicy wildcard matching (kinds: ["*/*"]) and the admission controller's policy cache

Versions tested: Kyverno v1.19.1 (current release as of the desk check, published 2026-09-10; re-confirmed as current at lab start on 2026-09-13), installed from the official Helm chart kyverno/kyverno 3.9.1 (repo https://kyverno.github.io/kyverno/); admission-controller image reg.kyverno.io/kyverno/kyverno:v1.19.1 at digest sha256:b31d8511ae5fd6010e2a01ea72ebae08eb82fa51d91af14a1d9fe989949b4edb. The documentation strings and issue/PR states were checked against the live docs and GitHub on 2026-09-12 and re-confirmed on 2026-09-13.

Config profile: A single-node kind cluster running the Kyverno v1.19.1 admission controller with crdWatcher enabled (Helm value admissionController.crdWatcher=true; the live container args included --crdWatcher=true, alongside --resyncPeriod=15m and --v=4); a wildcard ClusterPolicy; and a benign CRD (plus one of its custom resources) installed after the policy. Node Kubernetes version v1.34.0.

Verified on: 2026-09-24 (desk sources re-checked; v1.19.1 is still the current release) · lab measured 2026-09-13 on v1.19.1.

Affects: Kyverno ClusterPolicies that rely on a kinds: ["*/*"] wildcard deny rule (operations: [DELETE]) to cover CRDs installed after the policy is applied (v1.19.1, crdWatcher enabled; measured: the deletion was allowed — the guardrail did not match — until the admission controller was restarted).

TL;DR: On Kyverno v1.19.1 (Kubernetes v1.34.0, crdWatcher enabled), a kinds: ["*/*"] wildcard deny ClusterPolicy did not deny deletion of a labeled custom resource whose CRD was installed after the policy — the deletion was allowed at every poll across a ~32-minute window (past the 15-minute informer resync) and became denied only after the admission controller was restarted, while a pre-policy CRD's custom resource was denied throughout (control).

Full report: https://whenitruns.substack.com/p/when-a-kyverno-wildcard-guardrail

Reproduction repo: https://github.com/whenitruns/when-a-kyverno-wildcard-guardrail-runs-past-a-post-policy-crd-until-a-restart

When OPA's `semver.is_valid` Runs on What SemVer Rejects

When It Runs — Thu, 24 Sep 2026 05:26:00 +0000

On OPA v1.20.2, semver.is_valid returns true for leading-zero versions (01.2.3, 1.02.3) and empty pre-release/build sections (1.2.3-, 1.2.3+), and semver.compare compares 1.02.3 and 1.2.3 as equal.

When It Runs — Run Report #RR04 · Testing what infrastructure actually does.

Provider · Component: OPA (Open Policy Agent) · SemVer built-ins — semver.is_valid and semver.compare

Versions tested: OPA v1.20.2 (official opa_linux_amd64_static release; measured on our lab — see §2). The documentation strings and the built-in's source were checked against the live docs and the v1.20.2 source tag (2026-09-12).

Config profile: A single official opa binary; expressions evaluated with opa eval --format pretty, in both the default (non-strict) mode and --strict-builtin-errors (see §2 and §4).

Verified on: 2026-09-25 (re-measured on v1.21.0, the current release — see the update below) · lab measured 2026-09-12 on v1.20.2; desk sources re-checked 2026-09-22.

Affects: OPA policies that use semver.is_valid or semver.compare as a version trust gate — for example "reject the supplied version unless it is valid SemVer," or "allow when declared ≥ minimum" (measured on v1.20.2; on v1.21.0 these inputs are rejected — see the update below).

TL;DR: OPA's docs describe semver.is_valid as validating "that the input is a valid SemVer string", but on v1.20.2 it returns true for strings the SemVer spec rejects — leading-zero numbers (01.2.3, 1.02.3) and empty pre-release/build sections (1.2.3-, 1.2.3+) — and semver.compare compares 1.02.3 and 1.2.3 as equal rather than erroring the way it does for other malformed input like 1.2; OPA v1.21.0, released 2026-09-24, no longer shows either behavior in our re-run.

Full report: https://whenitruns.substack.com/p/when-opas-semveris_valid-runs-on

Reproduction repo: https://github.com/whenitruns/when-opas-semver-is-valid-runs-on-what-semver-rejects

When OPA's Bundle Loader Runs Past a `.manifest` Typo

When It Runs — Tue, 22 Sep 2026 22:53:25 +0000

In bundle mode, a one-character .manifest key typo (rego_verison for rego_version) draws no diagnostic that names the key; the failure can surface as rego_parse_error pointing at your .rego — OPA v1.20.1

When It Runs — Run Report #RR03 · Testing what infrastructure actually does.

Provider · Component: OPA (Open Policy Agent) · bundle .manifest loading — unknown top-level keys and rego_version

Versions tested: OPA v1.20.1 (official opa_linux_amd64_static release, Build 2026-08-28; measured on our lab VM — see §2). Documentation, published schema, and source facts checked against the live docs and the v1.20.1 source tag (2026-09-02).

Config profile: Single official opa binary; minimal two-file bundles (.manifest + one policy.rego, full contents in §2); one minimal config.yaml used only for the contrast check in §4.

Verified on: 2026-09-22 (desk sources re-checked; v1.20.2 is the current release) · lab measured 2026-09-11 on v1.20.1.

Affects: OPA bundles whose .manifest carries a mistyped or unknown top-level key — for example rego_verison instead of rego_version (measured on v1.20.1).

TL;DR: OPA's own docs state that the bundle loader "has always ignored unknown top-level keys" in .manifest, so when a bundle is loaded in bundle mode (opa build -b, or at runtime) a mistyped rego_version produces no diagnostic that names the key (measured on OPA v1.20.1) — the failure can surface instead as rego_parse_error: `if` keyword is required before rule body pointing at your .rego source file, while a mistyped key in the sibling config.yaml has drawn an explicit unknown configuration option warning since v1.19.0.

Full report: https://whenitruns.substack.com/p/when-opas-bundle-loader-runs-past

Reproduction repo: https://github.com/whenitruns/when-opas-bundle-loader-runs-past-a-manifest-typo

When the `Contact K8S API Server From Container` Rule Runs Silent

When It Runs — Wed, 16 Sep 2026 11:51:01 +0000

The shipped k8s_api_server FQDN condition (fd.sip.name="kubernetes.default.svc.cluster.local") fired on neither Falco 0.43.1 nor 0.40.0 in our lab — and the falco.org example still shows pre-v0.19.0 fd.sip="1.2.3.4"

When It Runs — Run Report #RR02 · Testing what infrastructure actually does.

Provider · Component: Falco · the k8s_api_server macro and the Contact K8S API Server From Container rule (published docs vs. shipped default)

Versions tested: Docs — live on falco.org (checked 2026-08-18). Shipped rule defaults — 0.18.0, 0.19.0, and current falcosecurity/rules main. Runtime firing — our lab (2026-08-18, one kind cluster): Falco 0.43.1 and 0.40.0, both observed not firing on a benign in-cluster API-server contact.

Config profile: Default shipped rules, no override — for the desk layers, the published documentation and versioned shipped rule files; for the lab, each image's bundled falco_rules.yaml as shipped (plus one added observation-only rule that changes no detection logic; full text in §2).

Verified on: 2026-09-14 (desk sources re-checked: the falco.org example, the shipped rule conditions, and #3834; Falco 0.44.1 was then the current release) · lab measured 2026-08-18 on Falco 0.43.1 and 0.40.0.

Affects: Operators who set the k8s_api_server macro by following the currently published example, and environments where the shipped DNS-associated condition never matches — as in a community report (#3834) and in our own lab.

TL;DR: Falco's published k8s_api_server override example still shows the placeholder 1.2.3.4:8080 that the shipped rules replaced with a DNS-associated FQDN condition in v0.19.0 — and in our lab that current condition fired on neither Falco 0.43.1 nor 0.40.0 for a benign in-cluster API-server contact, because the field it matches on (fd.sip.name) stayed empty.

Full report: https://whenitruns.substack.com/p/when-the-contact-k8s-api-server-from

Reproduction repo: https://github.com/whenitruns/when-the-contact-k8s-api-server-from-container-rule-runs-silent

When Falco Runs Out of Metadata

When It Runs — Mon, 14 Sep 2026 12:27:36 +0000

user.uid, user.loginuid, and fd.name under absent metadata on Falco 0.44.1 — plus the reported 4294967295 and /<NA> forms (0.31.0–0.40.0)

When It Runs — Run Report #RR01 · Testing what infrastructure actually does.

Provider · Component: Falco · alert output field rendering when metadata is absent (user.uid, user.loginuid, user.name, fd.name)

Versions tested: Falco 0.44.1 (our lab, 2026-08-18 — modern_ebpf, standard Ubuntu 24.04 kernel). Publicly reported across 0.31.0 (2022) through 0.40.0 (2025) — quoted as reported, not re-measured.

Config profile: Default falco.yaml as shipped in the official 0.44.1 image, with json_output=true and one custom output rule (full text in §2).

Verified on: 2026-08-18 (desk sources re-checked; lab run on 0.44.1 the same day) — re-verification against the then-current release happens at publication.

Affects: Anyone consuming Falco alert fields as values — dashboards, filters, and correlation that treat an absent field as a real one.

TL;DR: On Falco 0.44.1 we observed absent metadata rendered in value-like forms — user.loginuid as -1 (the one documented substitution), fd.name as <NA>/null, and one alert carrying a real uid, -1, and null side by side — while the reported forms user.uid = 4294967295 and the path-shaped /<NA> (#1921, #2126, #3246) did not appear in our scenarios and stand on those reports.

Full report: https://whenitruns.substack.com/p/when-falco-runs-out-of-metadata

Reproduction repo: https://github.com/whenitruns/when-falco-runs-out-of-metadata

DEV Community: When It Runs

When cert-manager's Ready Condition Runs Past Expiry Until a Reconcile

When an Argo CD App-of-Apps Wave Runs Past a Child With a Missing Resource

When a cert-manager Keystore Runs Behind a Password Change

When Keycloak's Event Store Runs on Defaults Without `enabled_event_types`

1. The Claim

2. The Setup

3. Expected vs. Observed — with enabled_event_types left out, which events does the event store keep?

4. Reproduction

5. Impact

6. What We Couldn't Conclude

7. Conclusion

When a Renewed cert-manager CA Runs on the Same Key but a New Key Identifier

1. The Claim

2. The Setup

3. Expected vs. Observed — does a same-key CA renewal keep the CA's Subject Key Identifier stable?

4. Reproduction

5. Impact

6. What We Couldn't Conclude

7. Conclusion

When a Kyverno Wildcard Guardrail Runs Past a Post-Policy CRD Until a Restart

When OPA's `semver.is_valid` Runs on What SemVer Rejects

When OPA's Bundle Loader Runs Past a `.manifest` Typo

When the `Contact K8S API Server From Container` Rule Runs Silent

When Falco Runs Out of Metadata

3. Expected vs. Observed — with `enabled_event_types` left out, which events does the event store keep?