DEV Community: NTCTech

Temporal Guardrails Move the Control Into the Event Log

NTCTech — Wed, 14 Oct 2026 12:15:32 +0000

Temporal guardrails change what an agent policy decides: not only whether this request is allowed, but whether it is allowed given everything the agent has already done. AWS's open-source Dogwood Local Engine documents its event handling and recovery model in enough operational detail to reason about the idea concretely. The language it enforces was introduced in the earlier Dogwood runtime verification post. The part of temporal guardrails worth an architect's attention is the boundary around it, because a stateful verdict is only as trustworthy as the history it was computed from.

This post is scoped to the open-source Local Engine and the language as AWS described it. It makes no claim about the managed AgentCore Policy integration.

Stateless Policy Decides on the Request. Temporal Guardrails Decide on the History.

Dogwood extends Cedar, and its temporal guardrails are built on that base. Any valid Cedar policy is a valid Dogwood policy, evaluation is deny-by-default, and a forbid overrides a permit. What Dogwood adds is a when temporal { ... } clause with operators such as formerly, count_within, count_distinct_within, sum_within and bind. With those, a policy can express prerequisites and ordering, rate limits, distinct-value limits and running totals.

That changes the shape of the decision. A stateless policy is a function of the request and the policy text. A temporal clause makes the verdict a function of the request, the policy text, and an event history the engine accumulated before the request arrived. In AI infrastructure architecture terms, the policy decision point stops being a pure function over a rule set. Once a rule can say "no more than five transfers in an hour" or "only after an approval event," the decision point also becomes a place where state lives, and state has an author, a lifecycle and a failure mode.

The control pattern is not new in kind. A token-budget limit at a gateway, as in the authority model in Nutanix Put MCP Behind a Gateway, already refuses a call because of what already happened. What temporal clauses do is make that "because of what already happened" expressible in the policy language itself rather than in gateway-specific counters.

Where the History Comes From

The Local Engine is a library embedded in a harness. The harness intercepts each tool call, submits a request event to the engine, runs the tool only if the verdict is allow, and then submits a response event. The engine issues verdicts only for request events; response events record outcomes. On arrival, the engine linearizes events under a lock, timestamps them, and appends them to a durable log that is synced to disk before evaluation. It keeps in-memory summaries per temporal clause, takes periodic snapshots, and rebuilds state by loading a snapshot and replaying later events. Policy updates are logged in the same sequence as events, and recovery yields either the complete pre-update or the complete post-update policy set.

AWS is explicit about who is responsible for what. The harness must provide event integrity. The event interface must be reachable only from the harness, and harness and engine state must not be manipulable by other processes or by the effects of tool calls. The library does not perform operating-system isolation.

Responsibility	Where the post places it	Detail
Ordering and timestamping	Engine (documented)	Events linearized under a lock and timestamped on arrival
Persistence and recovery	Engine (documented)	Durable log synced before evaluation; snapshot plus replay
Evaluation	Engine (documented)	Verdicts for request events only
Event accuracy and submitter identity	Harness (documented)	Event interface reachable only from the harness
Isolation of harness and engine state	Harness (documented)	Not performed by the library
Denied action does not run	Harness (documented)	Tool executes only on an allow verdict
Authentication and key management	Not stated	The post does not describe it
Behavior on engine or log failure	Not stated	Fail-open versus fail-closed is not described

Read that table the way you would read a shared-responsibility matrix. The two "not stated" rows are not criticisms; they are the questions your integration has to answer for itself.

Architectural guidance (not AWS's): treat the harness-to-engine event feed as an authenticated channel with one trusted submission path, because the engine will believe whatever it is told. A literal single-writer implementation is one way to get there; AWS's stated requirement is narrower, that the event interface be reachable only from the harness. An agent that can influence what the harness reports, or write to the same state through a tool side effect, is not being governed by the guardrail; it is authoring it. This is the same identity-isolation concern we raised in the agentic control plane problem and in the MCP security architecture analysis of the Agentic Authority Boundary: the component enforcing the rule must not share a trust domain with the thing the rule constrains. The runtime-control stage of the Governance & Runtime Control learning path frames where that boundary sits.

The Blind Window

Here is the mechanism that defines the whole problem with temporal guardrails. AWS documents that a new or modified temporal clause counts only events that arrive after the update. Its history starts at the update, not at the beginning of the log.

That is a reasonable design, and for temporal guardrails it is also a boundary you have to reason about. Whenever a clause is added or changed, there is a period in which it is evaluating a truncated view of the past. The agent's earlier behavior exists in the log, but the clause cannot see it.

The two inferences below are mine, drawn from the documented behavior. AWS does not state them.

ILLUSTRATION — CONSTRUCTED FOR THIS POST, NOT AN AWS EXAMPLE:

Permissive error — An agent submits 40 transfers between 13:00 and 14:00. At 14:00 an operator adds a clause: forbid a transfer when five have already occurred in the trailing hour. Because the clause's history begins at 14:00, it counts none of those 40 at 14:01. Depending on whether the current request counts toward its own verdict, which the post does not state, it permits roughly the next five. A rule written to cap the burst is blind to the burst that motivated it.
Restrictive error — At 09:00 an approval event is logged. At 14:00 an operator adds a clause: forbid release unless an approval occurred in the last 24 hours. The clause never saw the 09:00 approval, so it denies every release until a fresh approval arrives. Inference one: for a rolling window, the blind window is bounded by the clause's own window length, assuming the documented window semantics and the events' timestamps behave as described. Pre-update events stay outside the new clause's history until they would have aged out of the applicable window anyway. A one-hour clause stops being incomplete about an hour after the update; a 24-hour clause about a day after. Longer windows are not only costlier to evaluate, they stay unrepresentative for longer. This is a statement about when the clause matches the intended policy, not a guarantee about any specific verdict.

Inference two: the direction of the error depends on how the rule is written. A "forbid when the count is reached" rule errs permissive during the window. A "forbid unless a prior event is present" rule errs restrictive. Neither error direction can be assumed acceptable without an explicit transition procedure, and the same update process produces both.

This is where the evidence question sits. In AI Systems Need Evidence, Not Just Observability, component 02 of the AI Evidence Artifact Layer is the policy state snapshot: the constraint that was active when execution occurred, immutable and verifiable without access to the current policy configuration. The claim here is narrow. A snapshot of the policy text alone is not sufficient to reconstruct a temporal verdict. To explain why a particular request was allowed or denied, you also need the clause's start point and the event history inside its window. Nothing prevents a snapshot from carrying those. A snapshot combined with the event log and the clause-start boundary provides the essential inputs for reconstructing the decision, provided the exact policy version, the event ordering and timestamps, and the evaluation context are also preserved. But a record that stores only "policy version 14 was active" will not tell an auditor why the same request was allowed at 14:01 and denied at 15:01.

Windows Change Meaning When You Shorten Them

The August post is direct that evaluation of temporal guardrails is stateful and that cost can depend on event-log length. The Local Engine post puts a number on it, with a context that matters: each benchmark point is the median of 200 evaluations on one server-class host, using simulated sessions generating 360 events per hour. At a twelve-hour session length, the 24-hour window took about six milliseconds per decision, while the 15-minute window stayed flat. Those figures describe that one host under those conditions. They are not a production-latency commitment, and they say nothing about your harness, your event volume, your storage or your concurrency.

The same post notes that shortening a window is safe only if the shorter window still meets the requirement. That sentence deserves more weight than it gets. Shrinking a 24-hour clause to 15 minutes to recover latency on that single host is a change in what the control means, not a tuning step. A cumulative-spend ceiling over a day and a burst limit over a quarter hour are different controls that happen to share an operator.

Architectural guidance (not AWS's): record each window together with the risk it was sized against. "Twenty-four hours because the daily settlement cycle is the exposure" survives a performance review. "Twenty-four hours" does not, and it will be the first number someone proposes to cut when the latency graph looks bad. The blind window compounds this when the change modifies the clause: a window-length edit is a clause modification, so it resets that clause's history boundary as described above.

What to Test in Your Own Temporal Guardrails Integration

Diagnostic: "When a temporal clause changes, which events does it count, who wrote them, and what would you read to prove it?"

These four checks for temporal guardrails are architectural guidance. They are not an AWS procedure, and they do not assume Dogwood supports any particular remediation such as back-filling history into a new clause. If it does not, the mitigation is operational: schedule clause changes, and compensate during the window. Run them in a controlled test harness, not against a production log device, and capture evidence for each: the attempted action, the observed result, and the event-log state afterward.

01 — FEED INTEGRITY

Attempt to submit a forged event, and to alter engine or log state, from inside a tool call. The test passes only if the harness is the sole path in, tool side effects cannot reach the state, and the log is verifiably unchanged afterward. A forged event that merely fails to execute does not prove the log stayed intact.

02 — CLAUSE START POINT

Add and modify a clause against a log with known prior events. Confirm the clause ignores them, then confirm your change procedure records the start boundary and the compensating control used during the window.

03 — WINDOW SIZING

For every clause, write down the exposure the window length answers. Measure latency on your own harness and storage rather than importing the single-host benchmark figures.

04 — FAILURE MATRIX

The post does not state fail-open or fail-closed behavior, and the states differ. Exercise each in a test harness: engine unavailable before a verdict; log persistence failure before evaluation completes; failure after a durable append but before a verdict returns; failure after an allow but during tool execution. Define the expected response per state before an incident does.

Where This Stops

This analysis covers the open-source Local Engine and the language as the August post describes it. It does not extend to the AgentCore Policy integration, whose deployment topology the sources do not state, so nothing here should be read as a finding about the managed service.

It is also not a claim that Dogwood is unsafe. The blind window follows from a documented, defensible design choice, and the harness responsibilities are documented rather than hidden. Two limits from AWS's own description matter for how much weight your tests carry: temporal conditions do not currently support Cedar's automated reasoning tools, so you cannot lean on that analysis for them, and liveness properties and absolute wall-clock windows are listed as future work.

Architect's Verdict

Temporal guardrails are a genuine improvement in what an agent policy can express. Rate limits, prerequisites and running totals belong in the policy layer, not in ad hoc gateway counters.

Once temporal guardrails put the control in the log, three questions replace the old one. Who is allowed to write to it, which part of the past can a given clause actually see, and what proves the answer after the fact. AWS answers the first by assigning event integrity to the harness, and answers the second implicitly by starting each clause's history at its update. The third is left to you.

In operation, that makes a clause change a controlled transition: a named start boundary, a compensating control for the window, and a record an auditor can use. Window length belongs to the control, not to performance tuning.

A guardrail that remembers is only as sound as what it was allowed to remember, and from when.

Additional Resources

AI Infrastructure Architecture — the pillar hub for AI infrastructure design decisions, including where runtime governance sits relative to compute, data and orchestration.
Governance & Runtime Control — the learning-path stage covering enforcement points, authority boundaries and runtime control for agentic systems.
MCP, Tool Use, and the New Attack Surface Nobody Is Mapping — the Agentic Authority Boundary and the identity-isolation controls that govern what a harness must keep separate from tool context.
Agentic AI Has a Control Plane Problem — Because It Became the Control Plane — why the component enforcing policy cannot share a trust domain with the agent it constrains.
AI Systems Need Evidence, Not Just Observability — the AI Evidence Artifact Layer and why a policy-state record has to be verifiable independently of current configuration.
Nutanix Put MCP Behind a Gateway. The Real Problem Is Authority. — a gateway-level control that refuses calls based on prior consumption, the counter-based precursor to temporal clauses.
Introducing the Dogwood Local Engine: Temporal Governance for Agent Actions — AWS Open Source Blog; the primary source for the harness model, event log, recovery and benchmark described here.
Introducing Dogwood: Runtime Verification for AI Agents — AWS Open Source Blog; the language description, operators and stated limits.

Originally published at rack2cloud.com

GPU Passthrough Works. Can You Still Move the VM?

NTCTech — Tue, 13 Oct 2026 12:15:01 +0000

GPU passthrough gives a VM direct access to a physical accelerator, constraining which lifecycle operations the hypervisor can perform. Which operations remain available depends on the destination platform, its implementation of that access mode, and the documented conditions for the release. A VMware exit plan that validates a GPU workload by booting it on the destination and comparing throughput to the source has tested the device. It has not tested the VM.

The two tests answer different questions. A workload that boots, loads its driver and matches the source host's numbers has shown that the device works. Whether that same VM can be live-migrated for host maintenance, snapshotted before a change, or restarted automatically after a host failure depends on how the device was attached and on what the destination documents for that attachment. Those are separate capabilities. A sourced statement about one says nothing about the others, and this post treats them that way throughout.

The decision here is narrow. The device-access mode constrains the lifecycle operations available to the VM, so it has to be evaluated per workload, before the destination is committed and before a cutover date is fixed. This is not a hardware compatibility checklist, and it does not claim to cover every hypervisor. Where the documentation is silent, or this review did not reach it, the text says so instead of filling the gap.

The device works. Which VM operations survive is a separate question.

Booting Is Not a Lifecycle Test

In virtualization architecture, a VM's lifecycle is the set of operations the platform performs on it without the workload's involvement. It moves for maintenance, it is captured before a change, and it comes back after a host fails. Migration validation checks that the workload runs. The lifecycle is what the operations team assumes the platform will keep doing for years afterward. Four operations are worth keeping apart.

Live migration — moving a running VM to another host with no interruption, normally for planned maintenance or rebalancing.
Snapshots and suspend/resume — separate capabilities, grouped here because they are often evaluated together during change and migration planning. Snapshot support does not establish suspend/resume support, or vice versa.
HA restart — starting the VM on another host after an unplanned host failure. The VM restarts. It does not migrate.
Placement and drain — where the scheduler may start the VM, and what happens to it when a host is drained for maintenance.

The distinction between live migration and HA restart matters most. A platform that restarts a VM on another node after a failure has provided recovery. It has not provided live migration, and a planned drain that restarts the VM is an interruption the maintenance window has to absorb. A runbook that says "migrate the VMs off the host before patching" assumes live migration. A recovery plan that says "the cluster will restart it" assumes HA restart. A VM can have one without the other, and the device-access mode is one of the things that decides which. GPU passthrough is where that dependency shows up first, because the device is attached directly to the guest.

Diagnostic: "Which of these four operations does this workload's maintenance, backup and recovery plan actually assume, and does its device-access mode provide each one on the destination?"

Direct Assignment, Mediated Access, and the Four Operations

Two device-access modes matter for GPU workloads: GPU passthrough and mediated access. The rest of this post keeps them separate. In direct assignment, which is what GPU passthrough means, the guest is given the physical device. Platforms name this PCI passthrough, DirectPath I/O or Discrete Device Assignment. In mediated or partitioned access, a vendor driver or the hypervisor presents a slice of the device to the VM, and some vendor-supported combinations document live migration under stated conditions. Whether a given combination does is a vendor-matrix question, and this review did not answer it. Treating the two modes as interchangeable is the error this post exists to prevent.

Other device types, such as NICs and SR-IOV functions, raise the same question. They are out of scope here and were not surveyed, and nothing below should be read as covering them.

The table records what this review established for each destination and mode. A sourced cell names its source in the section that follows. A cell marked "Not established" means the review did not find or verify a primary source for it. It is not a statement that the operation is unsupported.

What the documentation establishes, and where it stops.

Destination and mode	Live migration	Snapshots / suspend-resume	HA restart	Placement and drain
vSphere, DirectPath I/O	Blocked (Broadcom)	Blocked (Broadcom)	Not established	Not established
Hyper-V, DDA (Windows Server 2025)	Not currently supported (Microsoft)	Not established	Restarts where pool GPU resources exist (Microsoft)	Drain restarts the VM on a node with pool resources (Microsoft)
Proxmox VE, PCI passthrough	Blocked (Proxmox guide)	Not established	Not established	Not established
Hyper-V, GPU partitioning	Not established	Not established	Not established	Not established
Proxmox VE, mediated devices	Not established	Not established	Not established	Not established
Nutanix AHV, any GPU mode	Not established	Not established	Not established	Not established
Vendor vGPU on any hypervisor	Not established	Not established	Not established	Not established

What the Documentation Establishes About GPU Passthrough

Three destination-and-mode combinations have primary documentation behind them. Each is stated at the scope its source supports.

vSphere DirectPath I/O

Broadcom's knowledge base article Virtual Machine vMotion is not supported on PCI vSphere Direct Path I/O devices at the VM level is direct. When a PCI DirectPath I/O device is assigned to a VM, the VM cannot be live migrated, and snapshots and suspend/resume are blocked. The documented remedy is to shut the VM down and remove the device, after which vMotion works again. The article does not state which vSphere versions it applies to, so the exact build in the source environment is something to check, not assume. It also says nothing about HA restart. That cell stays open, because a restriction on live migration is not evidence about what happens after a host failure. For a VMware exit, this is the source-side baseline for VMs using PCI DirectPath I/O under the conditions the KB describes: live migration, snapshots and suspend/resume are unavailable. Do not generalize it to other GPU-access modes, or assume it applies to every vSphere build without checking the source environment.

Hyper-V Discrete Device Assignment

Microsoft's guidance for GPUs in clustered VMs is explicit about the split. DDA is Hyper-V's form of GPU passthrough. Live migration of VMs using GPUs provided by DDA isn't currently supported, but the cluster can restart those VMs and place them where GPU resources are available after a failure. Draining a server restarts the VM on another node that has sufficient resources in the GPU pool. The guidance applies to Windows Server 2025 and Azure Local 2311.2 and later, and DDA with GPU pools in clustered VMs requires Windows Server 2025 Datacenter edition.

For planned maintenance the consequence is concrete: the drain is a restart, so the maintenance window has to treat it as an interruption. Microsoft points to GPU partitioning as the alternative to investigate when live migration is required. The support conditions for that mode are covered separately and were not verified in this review, so no live-migration capability is claimed for the partitioning row.

Proxmox VE

The Proxmox VE administration guide states it plainly: PCI or USB devices that are passed through currently block live-migration. For GPU passthrough on Proxmox, that sentence is the sourced finding. The copy retrieved for this review was labeled version 9.2.13 and dated 24 September 2026, so confirm the version your destination runs. Proxmox also supports cluster-level resource mappings, which its PCI passthrough documentation says exist so mapped devices can be properly used with HA. That is a statement of design intent. It does not establish that a VM with a mapped GPU restarts cleanly after a node failure, and snapshot behavior for passthrough VMs was not established either.

This is also where Proxmox's operating assumptions diverge from vSphere's. Proxmox Isn't Replacing VMware. It's Replacing Assumptions. treats vMotion, snapshot semantics and HA clustering as VMware implementations of infrastructure primitives rather than primitives themselves. A passthrough VM is a case where the destination's implementation differs from the source's for a reason unrelated to the hypervisor's general design. The reason is the device-access mode.

⚠ Do Not Generalize: These findings apply to direct assignment. A restriction on GPU passthrough is not a restriction on GPUs, and a destination that blocks live migration for a passed-through device says nothing about its mediated or partitioned modes.

Where the Evidence Stops

Four items remain open for this review. None of them should be read as a negative finding.

Nutanix AHV. No primary AHV source was retrieved for this review, for either GPU passthrough or vGPU. The rolling-maintenance model described in Upgrade Physics: Designing for Rolling Maintenance Without Stopping Production relies on evacuating workloads before a host is maintained. Whether an AHV GPU workload can participate in that evacuation under its actual access mode is not established here. That determines whether it can follow the normal maintenance procedure or needs a separately validated one.

Proxmox mediated devices. Proxmox documents mediated devices, including vGPU, as devices created when the VM starts and removed when it stops. That describes lifecycle at the device level. The documentation read for this review does not say whether a VM with a mediated device can live-migrate, so no claim is made either way.

Vendor vGPU matrices. Any vGPU migration claim depends on the GPU vendor's current compatibility matrix and on the destination vendor's own documentation. Neither was retrieved for this review, so the post makes no vGPU migration claim for any destination.

Hyper-V GPU partitioning. Microsoft names it as the option when live migration is required. The supporting conditions were not verified, so the table leaves the row open.

The Migration-Gating Decision Model

The model has four steps, applied per workload, in order. It gates the cutover commitment. It is not a post-cutover validation.

01 — REQUIRED OPERATIONS

For each GPU workload, state which of the four operations its maintenance, backup and recovery plans require. Take this from the runbooks as written, not from what the platform is assumed to provide.

02 — ACTUAL ACCESS MODE

Record whether each VM uses GPU passthrough, a mediated mode or a partitioned mode in production today, from the hypervisor configuration and not from the procurement record.

03 — DESTINATION EVIDENCE

For that mode and each required operation, find the destination vendor's documentation for the exact release. Mark each cell sourced or not established. A cell that is not established is not a pass.

04 — DISPOSITION

Pass. Change the access mode. Substitute an operational procedure, such as a planned downtime window or restart-based recovery. Or leave it unresolved, which holds the workload out of the cutover commitment.

The fourth step is the one that changes plans. A workload whose required operation is not established on the preferred destination does not pass by default. It is held out of the cutover commitment until the evidence is in, and the plan carries a date for getting it. The gate sits ahead of the destination commitment for the same reason. If a required operation fails on the preferred destination for the workload's current mode, the answer might be a different mode, a different destination for that workload, or an operational substitute. All three are cheaper to choose before the target is fixed.

Four steps per workload, applied before the destination is committed.

This is a migration-planning gate, in the same family as the other pre-cutover work in the VMware Migration Strategy stage of the Virtualization Architecture Path. It belongs in the plan alongside that work.

What This Changes Before Target Selection

Two neighboring problems should not be confused with this one. The Hypervisor Is Not the Migration Target — The Operating Model Is covers governance, authority and policy that fail to transfer to a new platform. That gap exists whatever the workloads are doing. Device-access mode is a property of the individual workload that constrains what the platform can do for it, with governance fully in place. And Your VMware Exit Plan Assumed a Tool You Didn't Control covers an exit plan failing on access to the migration tooling itself. The shared lesson is that an exit plan's assumptions include things outside the destination's feature list. The device-access mode is one of them, and it lives in the workload.

The GPU workloads in an estate can be listed. The useful output of the model is a per-workload disposition: which move with the main wave, which move later behind the evidence, which move in a different mode, and which move with a planned interruption written into the cutover plan. That list exists before any destination is chosen, and it should shape the shortlist.

Architect's Verdict

GPU passthrough that works on the destination has passed a performance test. It has not passed a lifecycle test, and different things decide the two. The device-access mode constrains which lifecycle operations the VM can keep, and the destination's documentation for that mode and release determines whether each one is available.

The real problem is that migration validation measures the device and skips the operations. Live migration, snapshots, HA restart and drain are four separate capabilities, and sourced evidence about one is not evidence about the others. Where the evidence is missing, the honest status is not established. It is not "unsupported" and it is not "assumed."

Choose the access mode against the lifecycle the workload requires, before the destination is committed. Hold any workload whose required operation is not established out of the cutover commitment until it is. A VM that boots proves the device. It does not prove the lifecycle.

Originally published at rack2cloud.com

The Root KSK Rollover Is Done. Did Your Resolver's Trust Anchor Change?

NTCTech — Mon, 12 Oct 2026 12:09:39 +0000

The root KSK rollover swapped the key at the top of the DNSSEC chain, and what matters now is whether each of your validating resolvers swapped with it. IANA's Trust Anchors and Rollovers page lists KSK-2024 (key tag 38696) as signing since 2026-10-11, with KSK-2017 (key tag 20326) no longer signing the root zone.

IANA's announcement of 11 October states that the root zone began using a new KSK on that date, key tag 38696, replacing the previous key. That is IANA's statement of the event, not a query of my own.

That is the scheduled state. Whether it is your state is a separate fact, and for most estates nobody has recorded it. Verisign's July data, as APNIC reports it, put roughly 90% of reporting resolvers at KSK-2024, while APNIC's own early-October measurement found about 70% of RFC 8509-capable test results reporting it trusted, with several ISPs under 10%. Those are different populations measured different ways, and neither describes your resolvers. This post is a method for producing that description: three kinds of evidence per resolver, kept on one worksheet.

How a Root KSK Rollover Breaks Validation

A validating resolver does not discover trust; it is configured with it. The trust anchor is the one input DNSSEC cannot supply for itself. Everything beneath it is derived: the KSK signs the root DNSKEY RRset, that RRset publishes the zone-signing key, the ZSK signs the DS records for the TLDs, and each TLD vouches for the zones below it. A resolver validates an answer by walking that chain upward until it reaches a key it was told to trust.

KSK-2024 has been published in the root DNSKEY RRset since January 2025. What the root KSK rollover changed on 11 October is the signer. The RRset is now signed by KSK-2024, and IANA lists KSK-2017 as no longer signing. A resolver whose trust anchors contain only KSK-2017 now receives a DNSKEY RRset signed by a key it was never told to trust. It cannot complete the chain, treats the data as bogus, and returns SERVFAIL. The algorithm did not change, so this is trust-anchor state and nothing else.

The failure is not limited to signed domains. To answer for an unsigned name, a validating resolver must still prove that no DS record exists at the parent, and that proof rests on the same chain. The expected symptom is broad SERVFAIL for that resolver's clients, signed and unsigned names alike.

Timing is where operators get misled. A resolver caches the DNSKEY RRset and its signature, and answers validated against the old signature stay usable until that cached data expires. The symptom therefore reaches different resolvers at different moments and can trail the switch-over. The size of the window is not a constant to quote. It is in the root's response, as the TTL on the DNSKEY RRset and the validity period of its RRSIG.

I am not reporting an outage. This is what a validating resolver without KSK-2024 is expected to do. Whether any resolver you run is in that state is what the rest of this post establishes.

Where the Trust Anchor Doesn't Update Itself

RFC 5011 is why most resolvers needed no operator during the root KSK rollover. A resolver that supports it watches the root DNSKEY RRset, sees a new KSK appear, holds it for at least 30 days, and then adds it to its trust anchors without human action. KSK-2024 has had far longer than that, so a resolver running continuously, with state it can write and persist, has trusted it since early 2025.

The exposure is every resolver that doesn't meet those conditions. That is an inventory problem, not a DNS problem.

Resolvers that don't self-update:

Pinned anchors — a static anchor file or configuration statement carrying only KSK-2017, never touched because it never had to be
Restored and cloned images — a resolver recovered from a snapshot returns with the trust state it had when the snapshot was taken, and one that was off or restored before it finished adopting KSK-2024 can no longer validate the RRset it would need to learn the key
Appliances and embedded resolvers — firmware that ships a fixed anchor, where the update path is a firmware release
Stateless containers — a resolver that can run RFC 5011 but discards its managed state on restart returns to whatever the image shipped, every time
Forwarding chains — a forwarder that trusts the AD bit from an upstream validator inherits that upstream's anchor state, so a failure at a resolver you don't run still lands in your tickets

Resolvers built from images and modules inherit whatever trust state the image carried. That makes this as much a modern infrastructure and IaC problem as a DNS one: the stale anchor is a baked artifact, and drift detection rarely tracks it.

⚠ Common mistake: Reading the anchor file as the resolver's state. A file on disk records what the resolver was configured to load. It does not prove what the running process loaded, and a file edited after the last restart proves nothing at all.

The scheduled revocation of KSK-2017 on 11 January 2027, and its removal on 22 March 2027, are the second half of the exposure. Both are IANA schedule dates, not yet events. A resolver following RFC 5011 learns of the revocation by watching the root. A pinned anchor never does and keeps trusting a retired key. That is revocation scoped at the trust root, and it carries the property the revocation unit argument describes: it reaches only the verifiers that are still watching.

Three Results, Not One: Configured, Runtime, Behavioral

Marking the root KSK rollover done because the date passed is the closure error the architecture of premature closure names. This is not another row in that table: the change is external, you ran no remediation, and the question is whether your resolvers absorbed it.

Record three results per resolver. Each answers a different question and none substitutes for the others.

Result	Question	Evidence	What it can't prove
Configured	Is KSK-2024 (tag 38696) in the trust-anchor source?	The anchor file or configuration for your software: BIND `bind.keys` (`initial-ds 38696`) or a `trust-anchors` statement, Unbound's `auto-trust-anchor-file`, Knot Resolver's `root.keys`	That the running process loaded it
Runtime	Has the running resolver loaded it?	BIND `rndc managed-keys status`; PowerDNS Recursor `rec_control get-tas`; where no command lists anchors (Unbound 1.19 has none), compare process start or reload time to the anchor file's modification time	Anything about validation behavior
Behavioral	Does the resolver validate using it?	The RFC 8509 sentinel test; a known-signed name returned with the `ad` flag set	Which anchor produced the result, or whether a forwarder answered instead

The RFC 8509 sentinel test is the behavioral check, and it is a three-query test. A client queries a deliberately bogus name, then two names built from the key tag, root-key-sentinel-is-ta-38696 and root-key-sentinel-not-ta-38696, as the leftmost label under a DNSSEC-signed test zone. A sentinel-aware validating resolver that trusts key 38696 answers the is-ta name and returns SERVFAIL for the not-ta name. One that does not trust it does the reverse. The bogus name must SERVFAIL; if it answers, something in the path is not validating. Some combinations, such as SERVFAIL on the bogus name and an answer on not-ta, are indeterminate when several resolvers sit behind one address. RFC 8509 §4.3 treats those as unreliable. Record them as indeterminate, not as a pass or a fail.

Three cautions apply. Sentinel support is not universal, so a non-answer can mean unsupported rather than untrusted. A forwarder in front changes which resolver you tested. And a pass is evidence for the path that carried the query, not for every resolver you run.

Diagnostic: "Which key does this resolver trust right now, and what did it do to prove it?"

Telling a Trust-Anchor Failure From Ordinary SERVFAIL

A root trust-anchor failure looks like every other SERVFAIL until you change one variable. Run the failing query against the suspect resolver with checking disabled (dig +cd). If it answers with +cd and fails without it, validation is failing, not reachability. Then query an unsigned name. Broad failure across signed and unsigned names with +cd succeeding points to the top of the chain, since an expired signature or DS mismatch normally fails one zone. Neither test proves a root trust-anchor failure on its own. The stronger diagnosis comes from the sentinel result, the direct anchor evidence from the table above, and a comparison against a second validating resolver known to hold KSK-2024. Read the Extended DNS Error (RFC 8914) where your resolver returns one.

Layered triage, such as the five phases in It's Not DNS (It's MTU), localizes a failure by path. A root-anchor failure isn't a location in that stack: every network-layer test can pass while the resolver still returns SERVFAIL, so the order of checks above is a trust test, not another triage ladder. The same separation holds for failover. A test that measures when traffic actually moves, the question DNS failover testing works through, is a caching question. This one is a trust question, and a resolver can pass the first while failing the second.

The Resolver Evidence Worksheet

The root KSK rollover leaves one row per validating endpoint, forwarders included. The columns are the record you will want to hand someone later:

Field	What goes in it
Endpoint	Resolver or forwarder address, and what depends on it
Software and version	Needed to interpret the commands and sentinel behavior
Validation location	This host, or an upstream
Anchor management	RFC 5011 managed, static file, firmware-embedded, image-baked
Configured / Runtime / Behavioral	Three separate results, each with its evidence and query path
Timestamp and owner	When it was checked and who is accountable for the row

Score each row PASS only when all three results are recorded. Anything unchecked is a FAIL until evidenced. Re-run the worksheet at the KSK-2017 revocation and removal dates, because those are the points at which pinned anchors diverge from managed ones.

Download the Root KSK Rollover Resolver Evidence Worksheet: spreadsheet or PDF — the per-resolver record used in this post, with a command reference by software.

Architect's Verdict

A root KSK rollover is scheduled, announced and documented, which is why it fails quietly. Nobody opens an incident for an event that went to plan, and the plan was never what needed to succeed. The root signed with the new key. Whether your resolvers trust it is a separate fact, held in places that are not treated as configuration.

The real problem is where resolver trust state lives. It sits in images, firmware, a state directory that may not persist, and a forwarder chain owned by someone else. None of those appear in an inventory under the name trust anchor, so the question of which key a resolver trusts has no owner until the day it matters.

A rollover is finished when each resolver can show which key it trusts and a test that proves it. Everything before that is a calendar entry.

Additional Resources

Modern Infrastructure & IaC Architecture — the pillar hub for image- and module-built estates, where stale resolver trust state gets baked in and goes untracked.
Rotating Credentials Is Not Revoking Them. The Revocation Unit Decides Whether You Can. — trust-root revocation reaches only the verifiers still watching, the property the scheduled KSK-2017 revocation shares.
The Architecture of Premature Closure — the signal-fired, state-unverified pattern; this post differs because the change is external and no remediation of yours ran.
It's Not DNS (It's MTU): Debugging Kubernetes Ingress — layered path triage that localizes failures by network location, the contrast case for a trust-state failure.
Why Your DNS Failover Didn't Actually Fail Over — traffic-movement timing across caching layers, a different question from resolver trust.
IANA: Trust Anchors and Rollovers and IANA's 11 October announcement — the signing-since date for KSK-2024, the announcement that the root zone began using it, and the scheduled KSK-2017 revocation and removal dates.

Originally published at rack2cloud.com

DKM ACL Hardening Is Scheduled. The Evidence It Worked Is Not.

NTCTech — Sun, 11 Oct 2026 14:41:09 +0000

DKM ACL hardening for AD FS moves from audit to enforcement with the October 2026 update. Microsoft's page, KB5121391, first published July 14, 2026, says that on Windows Server 2016 and later the update "will automatically remediate insecure ACLs" by October 13, 2026 unless the registry value says otherwise. Windows Server 2012 and 2012 R2 are different: enforcement auto-remediation does not apply, and the work is manual.

That is a schedule. It is not evidence. This note is about what proves the ACL ended up in the expected state, who can read that proof, and how long it exists.

What the DKM ACL Hardening Schedule Actually Promises

AD FS relies on the Distributed Key Manager container to store the symmetric keys that protect the private keys of its token-signing and token-encryption certificates. Microsoft's stated risk: if the container's ACL is overly permissive, an attacker with read access to the DKM key material can decrypt the token-signing private keys. The page frames this as an elevation of privilege, CVE-2026-56155. It does not report exploitation, and neither does this note. The control sits in the governance layer of modern infrastructure architecture: state declared by a vendor, held by something else.

The mechanics, as the page documents them:

Audit, from the July 14 update — Detection runs one minute after the AD FS service starts, then every 24 hours. No changes are made to the ACL. A mismatch logs event 1132.
Opt-in remediation — Set RemediateDkmAcl (DWORD) to 1 under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ADFS on any one AD FS server in the farm. Remediation runs at the next detection cycle, or on a service restart.
Enforcement, from the October 2026 update — Remediation "runs by default without requiring the registry key," equivalent to RemediateDkmAcl = 1. Setting the value to 0 opts out, and the page is blunt that opting out "leaves your DKM container vulnerable."

The page also says further hardening will follow in future updates. Microsoft published the dates in July, so there is no confirmation lag to manage; Patch Visibility Debt covers the opposite problem, where confirmation lags the event. An announced date does not establish that the October update is deployed everywhere, or what the ACL looks like afterward. That is what the announcement leaves open.

The Closure Signal Is an Update. The State Is One AD Object.

Analysis, mine and not Microsoft's: in most change records the closure signal for this item will read "October update installed." The Architecture of Premature Closure names that shape: a closure signal fires on execution, not on the verified state it was meant to represent. DKM ACL hardening is one more instance of it, not a new pattern.

What makes the instance specific is where the state lives. The ACL is not on an AD FS server. Every event the page defines carries a single "Container DN," and the page tells you to "Ensure the AD FS service account has access to the DKM group container in Active Directory." AD FS servers run the detection and one of them performs the change, but the thing changed is a permission on an Active Directory object.

In many organizations that splits ownership: the AD team owns the container's ACL and the AD FS team owns the service and the event log. Your actual owners have to be identified, not assumed. Remediation is an AD FS service action that rewrites a permission the AD team may believe is theirs, and the evidence of it lands in a log the AD team may never read. Who is accountable for the declared state of that ACL is the question configuration drift ownership addresses in general; DKM supplies a specific container to ask it about, and the governance and drift stage of the Modern Infrastructure learning path maps where that question sits.

What Each Event Proves, and What It Cannot

All five DKM ACL hardening events are written to the AD FS/Admin log on the AD FS server that logged them. What each one establishes is narrower than the label suggests.

Event	Proves	Cannot prove
1132 (Warning)	At check time the ACL did not match the expected secure state. Carries the container DN, an inheritance flag, and the current ACL as SDDL.	Who changed it, or that anyone reviewed it. In audit mode, nothing was changed.
1133 (Information)	At check time the ACL matched the expected state. Carries only the container DN.	That remediation ever ran, or that the previous ACL was preserved.
1134 (Error)	The detection task failed, for example on LDAP connectivity.	Anything about the ACL, in either direction.
1135 (Information)	Remediation succeeded and the previous ACL is recorded as SDDL.	That the SDDL still exists tomorrow.
1136 (Error)	Remediation was attempted and failed. Carries the current SDDL and the exception. The service retries within 24 hours.	That the failure is resolved. The page points to manual hardening with ADSI Edit or PowerShell if it persists.

The row that matters operationally is 1135. It records the previous ACL in SDDL as part of the success event, so its retention is the operational risk: Microsoft's own note says that event logs can roll over due to size limits and that an overwritten event means the previous SDDL "will be permanently lost." The page's recommendation is to save the message immediately on the AD FS server, using Get-WinEvent -FilterHashtable @{LogName='AD FS/Admin'; Id=1135} -MaxEvents 1 and writing $event.Message to a file. Microsoft frames this as a recommended precaution, not a requirement. Treat it as the change evidence.

A caution on the page itself. As of October 10, its sample commands carry transcription defects: the property name in the 2012 steps is cut off after "CertificateSharingContaine," the save command's output string is missing its closing quote, and the restore block mixes straight and curly quotes. This note describes the sequence; it is not a runbook. Copy from Microsoft's page, lint, and test in a lab before running anything.

One inference, labeled as such: the 1135 text notes that auto-remediation will not run again while the ACL matches the baseline. If the ACL drifts later and remediation runs again, a second 1135 would record the drifted ACL as "previous." Keep the first one.

The Server 2012 and 2012 R2 Path

For these platforms DKM ACL hardening is a manual sequence, per the page, run on any one AD FS server in the farm before the registry value is enabled: identify the DKM container DN, identify the AD FS service account, and grant that account explicit Allow ACEs for WriteOwner and WriteDacl on the container. Then set RemediateDkmAcl to 1, creating the ADFS key first if it is absent. Skip the grant and the page says remediation attempts will fail, which would surface as event 1136, retried every 24 hours. Microsoft's KB5099445, the July 14 Windows Server 2012 ESU monthly rollup, carries the same DKM item: detection plus an opt-in remediation mechanism. The same page gives October 13, 2026 as the final date for Windows Server 2012 ESUs. That date is stated for Server 2012 on that page, and this note does not extend it to 2012 R2. Analysis: for a 2012 farm, the manual path and the end of the ESU stream share a date.

For Server 2012 and 2012 R2, the October enforcement mode does not automatically enable remediation. Administrators must follow the platform-specific procedure: grant the required permissions to the AD FS service account, then explicitly opt in to remediation. The page does not describe how a farm containing both generations behaves, and this note makes no claim about it. Analysis: that is the case where "update installed" is most reliably true and the ACL is most likely unchanged. These are ESU platforms; Compatibility Is Not The Same Thing As Support is the argument that "still receives updates" and "still has an owner" are separate facts.

What the Hardened ACL Removes

After DKM ACL hardening completes, per the page, only these principals have access to the container:

Principal	Access rights
Domain Admins	Generic All (Full Control)
Enterprise Admins	Generic All (Full Control)
System	Generic All (Full Control)
AD FS Service Account	Read, Write, Create Child, Write Owner, Delete Tree

Inheritance is disabled and all inherited ACEs are discarded. Any other explicit Allow ACE not in that list is removed.

The page does not say who might be on the "other" side of that line. Analysis: any non-standard principal on the container, whether older service accounts, delegated groups or third-party tooling, loses access without the page naming it, and the inventory of who they were is the SDDL in event 1132 or in the previous-ACL field of 1135. If neither was captured, nobody can answer the question afterward.

A second inference, labeled the same way: the page describes tightening access going forward. It makes no statement about key material that was readable before the change, and this note makes none either. Whether that question applies to your farm is a review for you to run; Rotating Credentials Is Not Revoking Them covers why rotating and revoking are different operations.

Before DKM ACL Hardening Enforces

01 — Name the owners

Name the owner of the DKM container ACL and the owner of the AD FS/Admin log. If they are different teams, put both on the change.

02 — Read 1132, 1133 and 1134

Check every AD FS server for 1132, 1133 and 1134 now. If 1132 is present, export its text: it carries the current ACL, which is your baseline.

03 — Remediate or opt out, deliberately

Decide deliberately between remediating now and opting out. Microsoft frames opt-out as for a specific compatibility requirement with a plan to harden manually.

04 — Server 2012 and 2012 R2

On 2012 and 2012 R2, grant the ACEs, set the value, and expect 1135. Watch for 1136.

05 — Preserve the 1135 text

When 1135 appears, save the full message off the server immediately and forward the AD FS/Admin log somewhere a size limit cannot erase.

06 — Record the evidence

Record the 1135 text as the change evidence. Treat 1133 as a status, because it carries only the DN.

Architect's Verdict

A hardening deadline is a closure signal. DKM ACL hardening shows what sits behind one: state in an Active Directory object, evidence in a rolling event log, and ownership split between the team that holds the container and the team that runs the service.

The control is not the weak part. Microsoft's sequence is detailed, phased and announced months ahead. The weak part is the habit of recording "update installed" as "ACL hardened," and the risk that the previous ACL survives only in evidence nobody preserved outside a rolling event log.

The update closes the ticket. Only the event text closes the question.

Originally published at rack2cloud.com

The Vendor Retention Window Wasn't a Recovery Control

NTCTech — Sat, 10 Oct 2026 12:19:08 +0000

A vendor retention window is a lifecycle schedule, not a recovery control, and the Microsoft 365 deletion reported on September 28 is a clean example of the distance between the two. If the only copy of a tenant's data sits inside the engine that enforces the schedule, a failure in that engine can become a failure of the recovery plan.

What Microsoft has confirmed is narrow. A spokesperson told The Register that an error resulted in remaining data, if any, being deleted before the retention period ended. The affected content was tied to expired Microsoft 365 Business Premium subscriptions held by nonprofit customers. Microsoft emailed one customer that it had investigated recovery options and that the data cannot be recovered. That is the confirmed record, and this post stays inside it.

What Microsoft Has and Hasn't Said

The table separates Microsoft's attributed statements from what the public record leaves open. Nothing in the right column is a claim about Microsoft. It is the list of things a reader cannot currently know.

Stated by Microsoft (via The Register, 2026-09-28)	Not disclosed in the reporting
An error resulted in remaining data, if any, being deleted before the retention period ended	The cause of the error
The affected content was tied to expired Microsoft 365 Business Premium subscriptions	The number of affected organizations
Recovery options were investigated and the data cannot be recovered (email to one customer)	The deletion date, or how early the deletion ran
Microsoft is unable to list what was deleted, or whether any data was deleted	The retention period that applied to the affected tenants
The offer to customers is a free session to set up a new environment	The reason recovery is not possible

Counts circulating in secondary coverage rest on a single unconfirmed support-staff account and are not used here.

One row deserves more weight than its length suggests. The party that deleted the data is also the party best placed to say what it held, and it says it cannot. For a customer without an independent copy or inventory, the vendor's inability to identify what was deleted leaves no reliable vendor-side account of the loss. Whatever the customer retained independently becomes critical to reconstructing what existed and what may be missing.

The unknowns matter because they decide what a customer can reconstruct. Without a deletion date, a scope or an inventory from the vendor, an affected team cannot size its own loss from the vendor side. The burden of knowing what existed shifts to whatever records the customer kept independently, and a team that kept none is left asking the custodian a question the custodian has said it cannot answer.

A Vendor Retention Window Is a Schedule, Not a Control

Microsoft's published lifecycle for Microsoft 365 business subscriptions describes three stages: Expired, Disabled, Deleted. In the standard case, the headline durations are 30 days and 90 days for the first two, with different timings for enterprise and volume agreements, and CSP subscriptions follow a separate Partner Center reference. That is general product guidance. It is not evidence of the contractual or operational window that applied to the affected tenants, and nothing here assigns it to them.

The schedule is legitimate, documented product behavior, and nothing in the record says it routinely fails. It is also a statement about what the vendor's engine will do. It says nothing about what a customer can do if the engine does something else. The same documentation recommends backing up before deletion, and states that a deleted subscription cannot be reactivated and that buying a new subscription does not restore the data. Microsoft's own lifecycle documentation treats the lifecycle as the thing you back up around, not as the backup.

A vendor retention window answers one question: how long does the custodian intend to hold the data. An independent copy answers a different one: if the custodian's hold fails for any reason, is there a copy it cannot touch. The first is a property of the vendor's lifecycle engine. The second is a property of your data protection architecture.

Microsoft's documentation also distinguishes expiration from explicit deletion. An explicit deletion can bypass the Expired and Disabled stages, with SharePoint Online and OneDrive data deleted immediately. That is a different lifecycle path, not an explanation of the nonprofit incident. The architectural point is narrower: documented lifecycle stages describe expected vendor behavior, and they do not provide an independent recovery copy. A vendor retention window describes the vendor's process, and your recovery requirement has to be met by something you control.

Whether the affected customers held an independent copy is not in the public record, and the argument does not depend on it. A hold enforced by the same engine that can delete is one failure domain, not two.

What This Is Not

This is not an argument that SaaS vendors delete data when relationships end, and it is not a claim about how Microsoft's lifecycle normally behaves. Two neighboring problems are different decisions.

The Data Still Exists. You Just Can't Reach It. covers data you still hold that the identity, network and tooling around it make unreachable. Here the question is earlier: whether a copy exists at all outside the custodian's deletion boundary. The Retention Conflict Backup Architecture Wasn't Built to Arbitrate covers two legitimate obligations competing over one record. Nothing competes in this incident. One schedule is reported to have run before its own window closed.

The Check

Applied to any consequential SaaS tenant, the test for whether a vendor retention window is doing any recovery work has three parts. A yes on the first does not carry the other two.

Diagnostic: "Independence: does a copy exist outside the SaaS vendor's lifecycle and deletion boundary?"

Diagnostic: "Custody: who controls that copy, its credentials, and its retention policy?"

Diagnostic: "Recovery evidence: when was a representative restore last completed and verified?"

Independence. A second tenant at the same vendor under the same admin plane is not outside the boundary. Neither is an export folder inside the tenant it protects. The copy has to survive the vendor deleting, expiring or disabling the original, and the test is whether you can name where it lives, whatever the vendor retention window says.

Custody. Who holds the credentials, and who sets the retention? If access to the copy depends on the identity provider or admin account of the tenant it protects, the two share a fate. If the copy's retention is governed by another vendor default that you cannot independently verify or control, you may have recreated the same dependency at another layer. Establish who controls retention, what deletion events can affect the copy, and what evidence demonstrates that the copy survives them.

Recovery evidence. A copy outside the engine establishes a separate preservation boundary. It does not establish that the copy is complete, usable or restorable. The only evidence is a representative restore, with a date, a scope and a record of what failed. The Governance & Recovery Assurance stage covers how that evidence is kept honest.

A representative restore means more than opening one file. Sample each workload type the tenant holds, such as mailboxes, document libraries and the permissions that govern them. Restore into a clean location the original tenant does not control, verify contents against a known reference, and time it. If the restore needed the original tenant to be healthy, the copy was never independent.

Run the check before the next lifecycle event, not after it. A subscription lapse, a license reassignment, a program change or an offboarding is the event that starts the vendor's clock.

Architect's Verdict

A vendor retention window is a statement about the vendor's intent and the vendor's engine. It is not a recovery control, and the Microsoft 365 report does not change that so much as make it visible.

The underlying problem is a single failure domain. When the custodian, the deletion schedule and the only copy sit inside the same failure domain, the customer has no independent copy to fall back on if the lifecycle process fails. In this case, Microsoft also said it could not identify what may have been deleted. The three-part check is the answer: a copy outside the boundary, controlled by you, with a restore you have completed and verified.

The window is the vendor's schedule. The restore is yours.

Originally published at rack2cloud.com

BYOC Isn't a Deployment Decision. It's a Control-Plane Delegation Nobody Scoped.

NTCTech — Fri, 09 Oct 2026 12:07:55 +0000

BYOC delegation is the part of a bring-your-own-cloud decision that the deployment review never scopes. The review asks where the data plane runs, hears "your account," and files the question as answered. The question it skips is who receives authority over that account when you install the thing. In the offerings reviewed here, those two answers are not the same answer.

This post reads the BYOC documentation for Redpanda, Databricks and Aiven, and treats WarpStream as a vendor claim at the far end of the range. The set is narrow: four data and analytics platforms, and the pages I read are weighted toward AWS. Every empirical statement below is tied to a page I read. "Not stated" means the pages I read do not state it, not that the vendor lacks it.

The Question the Review Asks, and the One It Skips

In cloud architecture strategy, a BYOC review that stops at residency asks one question: where does the data plane run? The documentation answers it the same way each time. Redpanda states that you deploy the data plane in your own VPC and that customer data never leaves it. Databricks operates and manages clusters through a cross-account IAM role in the customer's AWS account. Aiven creates a VPC in the customer's account.

The residency answer looks like the authority answer. It is not.

The authority question is BYOC delegation: who receives permission over my environment when I install this. In Redpanda, Databricks and Aiven, the deployment process establishes that authority through IAM: a bootstrap step, cross-account role, or template the buyer applies. The residency review never reads it, because nothing in the residency question points there.

What "Your Account" Means in BYOC

Start with what the documentation establishes.

In standard Redpanda BYOC, you bootstrap a VM in your VPC with rpk cloud byoc apply. That VM launches the agent. Redpanda then assigns the IAM policies and creates dedicated roles per workload, which it describes as fine-grained and least privilege. The agent pulls cluster specifications from the control plane, and its IAM permissions let it call the cloud provider API to create and manage cluster resources, including VPC peering networks. The control plane handles provisioning, Kubernetes version upgrades and infrastructure maintenance.

In Aiven's AWS custom cloud, you generate a Terraform template in the Aiven console, apply it in your account, and hand the resulting Role ARN back to Aiven. Aiven assumes that role to run operations such as creating the VMs for service nodes, and the documentation describes the grant as permission to create resources and manage them onward. The VPC is one Aiven creates in your account, within a CIDR range you choose.

Then the constraint, stated precisely. Redpanda documents that it does not support customer access to or modification of internal data plane resources, and gives its reason: Redpanda manages configuration changes internally to support a 99.99% SLA for BYOC clusters. That is a specific, documented limit on what the customer can do inside the account. It is not a general claim that the customer has no control, and this post does not make one.

"Runs in your account" therefore establishes where the resources live. It does not establish who can create, change or remove them. Who owns a control plane once the architecture splits is the question the Control Plane Architecture stage of the learning path takes up.

Same Location, Different Authority

BYOC delegation varies even when location does not, and Redpanda provides the cleanest control: one vendor, one data plane location, two variants.

In standard BYOC, Redpanda manages the networking lifecycle and assigns the IAM policies. In BYOVPC, you supply the VPC and the IAM role the agent assumes. You create the networking and security resources in advance, and Redpanda states that the agent does not create or change them. The agent still deploys and operates the cluster, using the role you supply with only the permissions you grant. Redpanda says BYOVPC requires fewer permissions than standard BYOC.

Databricks shows this is not a Redpanda naming quirk. The cross-account role for a Databricks-managed VPC carries permissions to create and delete the VPC, subnets, route tables, NAT and internet gateways and VPC endpoints, plus launch and terminate instances. For a customer-managed VPC the documented set is smaller: the VPC, subnet, route and NAT creation and deletion permissions do not appear in that table, and the documentation states the set can be scoped down further with custom resource restrictions.

Dimension	Vendor-managed BYOC	Customer-managed variant
Data-plane location	Customer account (Redpanda, Databricks)	Customer account (Redpanda, Databricks)
IAM authority	Redpanda assigns the policies and creates the roles	Redpanda BYOVPC: customer supplies the role, and the agent is limited to what it grants
Delegated permissions	Databricks: full VPC and network lifecycle set	Databricks: smaller set, scopable further
Network resources	Redpanda manages the networking lifecycle	Customer creates them in advance (Redpanda) or provisions the VPC (Databricks)
Operational burden	Not stated in the pages read	Not stated in the pages read

The location row is identical in both columns. The authority rows are not. Same location is not same authority.

The Decision Is Made at Deployment, and Reduced Authority Can Be a Product Boundary

Two further facts from the same pages change how to read the table. Redpanda states that existing clusters cannot be converted to BYOVPC and that a BYOVPC cluster cannot change VPCs after creation. BYOVPC is an add-on that requires Premium support. On the Databricks side, the customer-managed VPC variant requires the Premium plan or above.

This is not a pricing argument. The architectural point is that BYOC delegation is a design decision, not an implementation detail. In these two offerings, narrower vendor authority is a variant you select and a tier you qualify for, not a setting you adjust later. For Redpanda the variant is fixed at creation. The Databricks page I read does not state whether a workspace can move between VPC types.

The far end of the range is a vendor claim, and it should be read as one. In a June 2024 blog post, WarpStream says its agents need only access to an object storage bucket and an outbound connection to its control plane, that no IAM roles or permissive security policies are required, and that the customer is responsible for running the stateless compute. I have not independently verified any of that, and it does not appear as a row in the inventory below. It shows what the variable looks like when a vendor chooses to remove the grant instead of reducing it.

The BYOC Delegation Inventory

Diagnostic: "What can this vendor's control plane actually do inside my account, and what do the vendor's own documents say about it?"

A review of BYOC delegation should produce an inventory, not a residency statement. Here is the inventory for the offerings reviewed, with four columns. The fourth is validation, not a control: it records what the documentation says happens to the deployed data plane if the vendor control plane is unreachable, and it is evidence-dependent. Where a page I read does not state something, the cell says NS.

Offering (variant)	Install-time grant	Resource scope	Customer-side constraint	Control-plane-loss behavior (validation, not a control)
Redpanda BYOC (standard)	Customer bootstraps a VM in their VPC that launches the agent. Redpanda then assigns the IAM policies and creates dedicated per-workload roles. The agent uses them to call the cloud provider API.	Cluster resources, Kubernetes version upgrades and infrastructure maintenance, VPC peering, and the networking lifecycle.	None stated for scoping the grant. Customers cannot access or modify internal data plane resources.	Stated: clusters remain available if the connection to the control plane is lost. Management during the outage: NS.
Redpanda BYOVPC (add-on, Premium support)	Customer supplies the VPC and the IAM role (instance profile) the agent assumes. The agent operates with only the permissions the customer grants.	Customer pre-creates networking and security resources, and the agent does not create or change them. It still deploys and operates the cluster infrastructure.	Customer grants the permissions. The example module restricts the IAM it grants to tagged resources and the cluster definition references a permissions boundary policy. Provisioning checks that the roles meet a minimum permission set, so there is a floor as well as a ceiling.	NS on this page. The page states the cluster needs outbound access to send telemetry to the control plane.
Databricks (Databricks-managed VPC)	Cross-account IAM role in the customer's AWS account. The documented purposes include creating and deleting the VPC, subnets, route tables, NAT and internet gateways and VPC endpoints, plus launching and terminating instances.	EC2 and VPC resources for the workspace.	Organization-level SCPs that deny AssumeRole or EC2/VPC access can break the role setup, so customer-side policy can constrain it.	NS.
Databricks (customer-managed VPC, Premium or above)	Same role mechanism, smaller permission set. The VPC, subnet, route and NAT create and delete permissions do not appear in this table. Instance launch and terminate, volumes, fleets and launch templates remain.	Narrower, because the customer provisions the VPC.	The documentation states the set can be scoped down further with custom resource restrictions.	NS.
Aiven (AWS custom cloud)	Customer applies a generated Terraform template that yields a Role ARN. Aiven assumes that role to run operations such as creating node VMs. The documentation describes it as permission to create resources and manage them onward.	A VPC Aiven creates in the customer account (customer-chosen CIDR, /16 to /24), node VMs, and an S3 bucket if object storage is enabled. The customer selects which Aiven projects or units may use the cloud.	The template can be viewed, downloaded and modified before it is applied. How the role's permissions can be scoped: NS.	NS.

Three readings follow, and none of them goes beyond the cells.

First, the density of NS. The control-plane-loss column is populated for one of the three vendors, and even there it states availability, not whether management continues. The constraint column is documented for Redpanda BYOVPC and Databricks, and partial for Aiven.

Second, a pointer on Aiven. The IAM permissions JSON on the Aiven page belongs to the user who deploys the template, not to the role Aiven assumes. The Aiven documentation I reviewed does not state how that role's permissions can be scoped. It does show the template can be viewed before you apply it, which is the place to start reading the grant.

Third, an inversion. In standard Redpanda BYOC, the documented constraint binds the customer, not the vendor: the customer cannot access or modify internal data plane resources in the account the customer owns.

Unknown is a result. If a vendor's documentation establishes where the workload runs but does not establish what happens when its control plane disappears, the architecture review should record that gap, not infer an answer. Same location plus documented BYOC delegation is still not an understood architecture. It is an architecture with a populated inventory and a list of questions to put to the vendor.

What This Post Does Not Argue

Four adjacent arguments already have homes, and this post cites them instead of repeating them.

Your Cloud Isn't Compromised. Your Vendor Is. Now What? covers why third-party authority persists after the trust behind it fails.
That SaaS integrations accumulate authority over time belongs to The SaaS Control Plane Problem. The BYOC delegation examined here is the authority granted at deployment, not what builds up afterward.
The delegation object and its schema are covered by Enterprise Architecture Has Identity Governance. It Doesn't Have Delegation Governance. The inventory above is a BYOC-specific fill-in, not a competing model.
How a delegation is revoked belongs to Rotating Credentials Is Not Revoking Them. This post does not teach the mechanism, and the inventory has no removal-path column. Two scope limits apply. The empirical claims cover the BYOC offerings reviewed here and no others. And the inventory has no column for whether a vendor can change the grant over time, because none of the pages I read state it. That row is under investigation, not claimed.

Architect's Verdict

BYOC is a control-plane delegation decision disguised as a deployment-location decision. The residency answer is real and the documentation supports it. It is also the less informative of the two answers a review needs.

The underlying problem is that the install step establishes the BYOC delegation while the review records the location. In the offerings reviewed here, the vendor-created roles, the resources a customer may not touch, and the narrower variants gated behind a tier are all visible in public documentation. They go unscoped because the review has no artifact that asks for them.

The BYOC delegation inventory is that artifact, and its empty cells are part of its output. Where the documentation is silent, you have found a question for the vendor, not a reason to assume the answer you were hoping for.

Residency tells you where the data sits. Only the inventory tells you who can act on it.

Additional Resources

Cloud Architecture Strategy — the pillar hub covering placement, ownership and exit decisions across cloud architecture.
Control Plane Architecture — the learning path stage on who owns the control plane once responsibility splits across vendor and customer.
Your Cloud Isn't Compromised. Your Vendor Is. Now What? — what happens to standing third-party authority once the trust behind it fails.
The SaaS Control Plane Problem — how authority accumulates through SaaS integrations, the layer after the install-time grant covered here.
Enterprise Architecture Has Identity Governance. It Doesn't Have Delegation Governance. — the delegation object this post's inventory fills in for BYOC.
Redpanda BYOC Architecture — vendor documentation for the standard BYOC control plane and agent model.

Originally published at rack2cloud.com

KMS Circular Dependency: The Cluster Came Back. The Keys Didn't.

NTCTech — Thu, 08 Oct 2026 12:27:57 +0000

A KMS circular dependency turns an online cluster into an unreadable one: the data needs a key, and the service that holds the key lives on the data. Hosts can be running, membership can be intact, and the storage can still be unable to complete the unlock path.

What "Available" Hides

Availability and readability are different properties. In virtualization architecture, "available" usually describes the coordination layer: hosts joined, quorum held, services running. "Readable" describes something narrower: the cluster can obtain the key material it needs to decrypt the data it is holding. A cluster can have the first and lack the second.

The two-node edge deployment argument separates availability from capacity the same way: quorum says the cluster is up and says nothing about what that state costs. The gap here opens on a different axis. An encrypted cluster can be up and unable to read its own disks.

The scope is narrow, and it sits at the edge of an earlier comparison. Availability vs Authority scopes its Nutanix outcomes to clusters that don't depend on a key manager outside the cluster. This post covers the ones that do. The claim is that storage survivability is configuration-dependent, and that one function, unlock, shows it with vendor primary documentation behind it.

Five Storage Functions, One Populated Row

Between power-on and a usable datastore, a storage platform performs more than one function, and each function has its own dependency path. Five are worth separating.

Function	Dependency question	Status in this post
Unlock	Does the key service stay reachable when the storage is cold?	Worked below
Mount	What must be present before the unlocked volume presents to hosts?	Under investigation, not claimed
Discover	What must resolve before hosts and VMs find the datastore?	Under investigation, not claimed
Replicate	What must be reachable for a replication link to authenticate and move data?	Under investigation, not claimed
Validate	What must be reachable before a restored state can be checked as usable?	Under investigation, not claimed

Only the first row carries evidence here: unlock, where the KMS circular dependency lives. The other four are the same question asked of different functions. None is shown to fail the same way, and nothing below implies it.

This sits beside the Storage Survivability Boundary, Framework #109: the point where a storage architecture can no longer absorb component failure, rebuild activity, or locality loss without breaking its performance or availability assumptions. A cluster can sit well inside that boundary and still be unreadable, because the unlock dependency isn't part of the arithmetic the boundary models.

The KMS Circular Dependency: Unlock and Decrypt, Worked

The evidence standard for this section has three rungs: the dependency a vendor documents, the consequence it documents, and the mechanism it documents. Where the vendor stops, this post stops.

vSAN: The Circular Case Is Documented

Broadcom's KB 453198 (vSAN 8.x and 9.x) names the KMS circular dependency directly. It occurs when the KMS virtual machines sit on the same encrypted vSAN datastore they are responsible for unlocking. While the cluster is online and the keys are cached in host memory, Broadcom notes, the configuration appears functional. A full power outage or a host reboot clears that cache. At startup the ESXi hosts cannot mount the encrypted disk groups without a network connection to the KMS. The KMS virtual machines cannot power on, because the datastore underneath them is locked. Each side waits on the other.

The guidance for a KMS circular dependency follows from the mechanism: never locate the KMS on the datastore it unlocks. A new KMS installation that holds none of the original keys will not decrypt existing data, so recovery means restoring the original KMS or its keys, with restoring the KMS to non-encrypted storage as the preferred path. Without external backups of the KMS appliances or exported keys, the data is permanently inaccessible.

⚠ Do not reboot your way out: Broadcom's guidance is explicit: do not reboot hosts while key access is unavailable. Keys can remain cached in memory after a KMS outage, and a reboot destroys that cache and removes the remaining ability to access the data. Do not recreate disk groups or disable encryption to bypass the lock. Recreating disk groups can destroy the remaining possibility of recovering the data.

Two variants appear in Broadcom's vSAN 8.0 administration guide, which documents the baseline the same way: a rebooting host does not mount its disk groups until it receives the key-encryption key from the KMS. From vSAN 7.0 Update 3, key persistence lets hosts retain their keys across a reboot, in a TPM where one is present, so encryption operations can continue while the key server is unavailable. Native Key Provider, supported from vSAN 7.0 Update 2, needs no external KMS: vCenter Server generates the key material and pushes it to the hosts, which then generate the data encryption keys. KB 453198 describes the cached-key case and does not mention either variant. This post does not claim how they interact with the KMS circular dependency.

Nutanix: The Warning Is Documented, the Cycle Isn't

The AOS Security 7.6 Security Guide is the governing source for the Nutanix claims here. The relevant language is unchanged from AOS Security 6.8. The guide separates two encryption modes, and the evidence differs by mode.

For self-encrypting drives using an external KMS, the key service is outside the cluster and the Controller VM uses KMIP to upload and retrieve the drive keys. One KMS is required, several are recommended so the KMS is not a single point of failure, and the guide advises clustering them (p. 149). After a full power-off and power-on with protection enabled, the Controller VM retrieves the drive keys from the KMS to unlock the drives. If it cannot get the correct keys, it cannot access the data (p. 149).

Software-only encryption does not inherit that sentence. The guide gives it an access-condition list instead. The system accesses the external KMS when a cluster starts, when a key is regenerated, when the KMS type is switched, and when the NCC heartbeat checks that the KMS is alive. For software-only encryption specifically, it also accesses the KMS when a service starts or restarts and when AOS is upgraded (pp. 149–150).

For an external KMS in both modes, the guide carries an explicit caution: do not host a key management server VM on the encrypted cluster that is using it, because a problem with that VM could result in complete data loss (pp. 149, 164). That is a topology prohibition and a stated consequence. The guide does not describe the startup sequence that produces the loss.

Native KMS removes the external KMS dependency by design, within conditions. Native KMS (local) is software-only and needs at least three nodes. One- and two-node clusters are not supported for it (pp. 147, 160). Its keys must be backed up (p. 147). Native KMS (remote), meant for one- and two-node remote-office clusters, is available only when the cluster is registered to Prism Central (p. 160), and the guide notes that the root key is retained on Prism Central if the cluster is destroyed while still registered (p. 167). That documents a dependency on Prism Central registration. It does not document how or when keys are retrieved, and this post does not claim it.

What Each Vendor Documents, and Where It Stops

Both vendors warn against placing the key-management function on the encrypted storage it is responsible for unlocking. The documentation differs in how much of the failure it exposes.

Rung	Broadcom (vSAN 8.x, 9.x)	Nutanix (AOS Security 7.6)
Dependency documented	Hosts need the KMS to mount encrypted disk groups at startup	The CVM retrieves keys from an external KMS; access conditions listed by mode
Consequence documented	Locked datastore; a new KMS without the original keys cannot decrypt; no external backups means permanent inaccessibility	SED: the CVM cannot access data without the correct keys. Hosted KMS VM: complete data loss warning
Mechanism documented	Yes: hosts cannot mount without the KMS, and the KMS VMs cannot power on from the locked datastore	Not for a hosted KMS VM, and not for Native KMS (remote) retrieval

No ranking follows. A vendor that documents less of a failure has not been shown to fail differently, and a vendor that documents the whole cycle has not been shown to be more exposed to it.

The Dependencies Recovery Plans Forget names four dependency objects that recovery plans treat as assumptions: identity, DNS, certificates, and network. A key manager is not on that list, and the circular case is the version where the dependency is not only forgotten. It cannot start until the storage it unlocks is already readable. The System Recovered. Your Recovery Boundary Didn't. argues that recovery boundaries get drawn around what the organization owns instead of what the service depends on, which is a question about dependencies left outside the boundary. This one can sit inside the boundary and still fail, because of ordering.

Configuration Preconditions

Survivability here depends on a configuration state: where the key service runs relative to the storage it unlocks. Eight configurations follow, graded by what the evidence supports. The grade labels the evidence class. It does not mean recommended. One row establishes the external placement condition; another creates the KMS circular dependency outright. The remaining rows set the conditions around it.

Platform	Configuration	Cold-start / access dependency	Evidence / Scope
vSAN	External KMS, off the encrypted datastore	Hosts need a network connection to the KMS to mount encrypted disk groups at startup	PRIMARY — KB 453198 — vSAN 8.x, 9.x
vSAN	KMS VM on the encrypted datastore it unlocks	Creates the KMS circular dependency: unlock requires the KMS the datastore hosts	PRIMARY — KB 453198 — vSAN 8.x, 9.x
vSAN	Host key persistence with a TPM	Hosts retain keys across a reboot and encryption operations continue while the key server is unavailable (vSAN 7.0 U3+); whether it eliminates or changes the circular failure path is not established here	PRIMARY — vSAN 8.0 ADMIN GUIDE
vSAN	Native Key Provider	No external KMS: vCenter Server generates the key material and pushes it to the hosts (vSAN 7.0 U2+); whether it eliminates or changes the circular failure path is not established here	PRIMARY — vSAN 8.0 ADMIN GUIDE
Nutanix	External KMS, SED	After a full power-off and power-on, the CVM retrieves drive keys from the KMS; without the correct keys it cannot access the data	PRIMARY — SED — 7.6
Nutanix	External KMS, software-only	Guide lists KMS access at cluster start, key regeneration, KMS type switch and NCC heartbeat, plus service start/restart and AOS upgrade	PRIMARY — SOFTWARE — 7.6
Nutanix	Native KMS, local	Software-only; minimum 3 nodes, 1- and 2-node clusters not supported; keys must be backed up	PRIMARY — 3-NODE — 7.6
Nutanix	Native KMS, remote	Available only when the cluster is registered to Prism Central; retrieval mechanism not documented here	PRIMARY — PRISM CENTRAL — 7.6

AOS 7.5+ also documents Prism Central-managed external KMIP for software-only encryption. After association, the cluster communicates directly with the KMS and Prism Central is not in the cluster-to-KMS data path (AOS Security 7.6, pp. 103–104). This is distinct from Native KMS (remote) and is not included as a separate configuration row here.

Diagnostic: "If every host in this cluster powered off at once, which service has to answer before the first datastore mounts, and where does that service run?"

What This Post Does Not Claim

The SED power-cycle sentence is not generalized to software-only encryption. Software-only is covered by its own access-condition list.
The Native KMS (remote) retrieval mechanism is not established. Prism Central registration is the documented dependency.
Cloud KMS, which the 7.6 guide documents for Azure Key Vault only (p. 108), is not treated as equivalent to a KMS VM hosted on the cluster, and is not worked here.
AOS 7.5+ Prism Central-managed external KMIP is distinct from Native KMS (remote).
Whether a KMS circular dependency forms in a software-only Nutanix cluster is not claimed. The guide states the prohibition and the consequence only.
vCenter-offline behavior on vSAN is not claimed.
Whether TPM key persistence or Native Key Provider eliminates or changes the circular failure path is not claimed.
Mount, discover, replicate, and validate are not claimed.
No platform is ranked, and nothing here extends beyond the documented versions and configurations: vSAN 8.x and 9.x per Broadcom KB 453198, and Nutanix AOS Security 7.6 per the Nutanix Security Guide. The relevant Nutanix language is unchanged from the 6.8 guide.

Recovery Readiness Assessment — test your own recovery path against this failure mode.

Architect's Verdict

Online and readable are different states, and encrypted storage is where confusing them costs the most. Both vendors' primary documentation says the same thing about the key service: it does not belong on the storage it unlocks.

The failure is not encryption by itself. It is a dependency nobody drew. In the configuration where a KMS circular dependency forms, the cluster that most needs the key service is the one that cannot start it. Broadcom documents that wait step by step. Nutanix documents the prohibition and the consequence and leaves the sequence unstated. A vendor that documents less has not been shown to fail differently.

Configuration decides the outcome. Same platform, same encryption, one placement decision apart, and one cluster restarts while the other waits on itself. The review question is not whether encryption is enabled. It is where the key service runs when everything is off, who can restore it, and from what.

A cluster that has come back has not recovered until it can read what it came back with.

Additional Resources

Virtualization Architecture — the pillar hub for the hypervisor, control-plane, and storage platform decisions this unlock dependency sits inside
Nutanix vs VMware: Availability vs Authority in the Post-Broadcom Datacenter (2026) — the earlier comparison whose Nutanix outcomes assume no key manager outside the cluster
Two Nodes Can Preserve Availability. They Can't Preserve N+1 Capacity. — the same availability-versus-capacity separation, applied to failure capacity instead of readability
VMware vSAN 8.0 Administration Guide: data-at-rest encryption — the administration guide's encryption section: restart behavior, key persistence from vSAN 7.0 Update 3, and Native Key Provider
Recovering Encrypted vSAN Datastores from KMS Circular Dependencies (Broadcom KB 453198) — the primary source for the vSAN circular mechanism and recovery constraints
Nutanix AOS Security 7.6 Security Guide — the primary source for the Nutanix external-KMS, Native KMS, and access-condition claims; page cites in this post follow the PDF edition

Originally published at rack2cloud.com

Kubernetes Swap Turns an OOMKill Into a Latency Problem

NTCTech — Wed, 07 Oct 2026 12:05:31 +0000

Kubernetes swap changes what memory pressure does to a pod: a workload that would have been OOMKilled can remain alive and run slowly instead. That sounds like a pure improvement until you look at what decides which pods get that treatment. Under LimitedSwap, the kubelet and container runtime use the container's memory request when determining its swap allowance. The scheduler uses the same field for placement and does not account for swap.

This post covers one configuration and stays inside it: cgroup v2 nodes running LimitedSwap, and the pods that mode allows to use swap. It extends the model in Kubernetes Requests vs Limits rather than replacing it. That model still describes the default node. This is what changes when you turn swap on.

One number, two jobs: the memory request places the pod and sizes its swap allowance.

Kubernetes Swap Is Off by Default, and Opt-In Per Node

On a default Kubernetes node, workloads cannot use swap. The kubelet refuses to start on a node with swap enabled unless you set failSwapOn: false, and even then it tells the container runtime to allocate zero swap to workloads. Letting pods use it is a separate, explicit choice: set memorySwap.swapBehavior to LimitedSwap in the kubelet configuration, as described in the Kubernetes swap documentation. Different nodes can use different behaviors, which makes this a node pool decision, and each node reports its swap capacity in its status, so you can see which pools have swap provisioned. It sits in the platform layer of modern infrastructure architecture and is felt by every team whose pods land there.

The mechanics matter for what follows. The kubelet does not enforce swap itself. It directs the container runtime to apply the setting, and on cgroup v2 that setting is memory.swap.max on the container's cgroup. On cgroup v1, Kubernetes workloads are not allowed to use swap at all.

Scope of this post:

Covered: cgroup v2 nodes running LimitedSwap, non-high-priority Burstable pods, the request-derived swap allowance, and the shift from OOMKill to latency.
Not covered: eviction interaction, swap-aware scheduling, benchmark or density numbers, and which release moved the feature to which stage.
Basis: behavior as documented on kubernetes.io and in KEP-2400. Nothing here was measured.

The Memory Request Becomes the Swap Entitlement

The requests-vs-limits model treats the memory request as a scheduling signal. The scheduler adds it to a node's accounting ledger and places the pod. Under LimitedSwap the same number gets a second job. The amount of swap a container may use is proportional to its memory request, the node's physical memory, and the total swap available to pods on that node. Raise the request and the allowance grows with it. Lower it and the allowance shrinks.

Consumer	What it does with the memory request
kube-scheduler	Places the pod on a node with enough unreserved memory in its accounting. Does not account for swap.
kubelet and container runtime	Derive the container's swap allowance from the request and apply it to the container's cgroup.

Per the Kubernetes documentation, the scheduler does not currently account for swap usage, and the docs name that as a factor that heightens noisy-neighbor risk. A node can be packed on requests alone while its pods compete for the same swap device. Kubernetes swap gives the same request a second runtime consequence: where a pod lands is still one decision, but the request also helps determine how much swap the container may use when memory runs short.

An OOMKill Is Loud. Swap Is Quiet.

On a default node, a container that hits its memory limit is killed by the OOM killer, the pod reports OOMKilled, and the restart counter moves. It is an event, and most monitoring is built around events.

An OOMKill is an event. Swap pressure can leave a pod running with no event attached.

Swap changes the ending. The Kubernetes documentation says swap can help prevent pods from being terminated during memory pressure spikes. The same page says that having swap available reduces predictability, that swapping data back into memory can be slower by many orders of magnitude, and that swap changes a system's behavior under memory pressure. Both statements describe one mechanism. The pod that survives may now be waiting on a much slower storage path.

How slow depends on the storage behind the swap. The docs state that performance on a node with swap depends on the underlying physical storage, and is significantly worse in an IOPS-constrained environment, such as a cloud VM with I/O throttling, than on SSD or NVMe. The pod is the same either way. The outcome depends on a device it has no say in.

Swap also changes who pays. The docs warn that enabling it increases noisy-neighbor risk, because pods that use their RAM frequently can cause other pods to swap. Kubernetes swap can turn a pod-level memory-pressure event into contention for a shared node storage resource.

⚠ A silent failure mode: A pod that would have been OOMKilled may now be alive and slow. If your alerting keys on OOMKilled status and restart counts, swap removes the signal without removing the problem.

The signals move to places most dashboards do not look. The kubelet exposes container_swap_usage_bytes and container_swap_limit_bytes on its resource metrics endpoint, and kubectl top pod --show-swap reports per-pod swap use. None of that appears in pod status the way an OOMKill does.

Who Gets Swap, and How Much

Eligibility for Kubernetes swap is narrower than "pods on a swap-enabled node." Under LimitedSwap, only non-high-priority pods in the Burstable QoS class may use swap, and BestEffort and Guaranteed pods are prohibited, according to the Kubernetes 1.32 swap post. KEP-2400 scopes the feature to Burstable pods.

Pod class	Swap under LimitedSwap
Guaranteed	None
BestEffort	None
Burstable, high priority	None
Burstable, other	Proportional to the memory request

Eligibility under LimitedSwap is narrower than pods on a swap-enabled node.

The documentation decides what counts as high priority, so check it for the version you run. There is also a built-in opt-out: a container in a Burstable pod whose memory request equals its memory limit gets no swap. Request sizing is now a swap decision as well as a scheduling decision.

That creates a tension the default requests-vs-limits model does not capture. Sizing requests from steady-state usage is sound for placement. Under LimitedSwap it also means a container with a modest request gets a modest allowance, however spiky its peaks. Anything that sets the memory request is now also setting the potential swap allowance, whether that request comes from a chart default or an autoscaler's recommendation covered in VPA vs HPA.

Before You Enable It: One Question and Four Checks

Diagnostic: "For this workload, is a slowdown on a device I am not monitoring better than a restart I would have seen?"

If the answer is yes for a class of workload, swap is a reasonable tool for it. If the answer is no, setting the request equal to the limit opts the container out.

Before enabling LimitedSwap:

Confirm the node pool runs cgroup v2. On cgroup v1, workloads cannot use swap.
Decide which workloads may page. The request sets the allowance, high-priority pods are excluded, and request equal to limit opts a container out.
Move alerting beyond OOMKilled. Watch container swap usage against its limit, and per-pod swap in kubectl top.
Check what backs the swap. The docs tie performance to the storage underneath, single out IOPS-constrained environments as the worst case, and strongly encourage encrypting the swap space.

Nothing here argues against swap. Kubernetes swap is a reasonable tool when the trade is chosen deliberately. What it argues against is enabling it as a node setting and leaving requests and alerts exactly where they were. The scheduler will keep placing pods using requests, while the kubelet and container runtime apply the swap allowances derived from those requests.

Architect's Verdict

Kubernetes swap does not soften a memory limit. It changes the failure path available to a memory-pressured pod, and the memory request helps determine how much swap the container may use.

The real problem is that two long-standing habits stop lining up. Teams commonly size requests for the scheduler and alert on the OOMKill. Under LimitedSwap the request also helps set a swap allowance that the scheduler does not account for, and the kill can become a slowdown that no restart counter records. Neither change is a bug. Both are documented. The risk is enabling swap as a node setting without revisiting what requests mean and what alerts watch.

An OOMKill announces itself. Swap fails quietly, and only the teams that chose to measure it will notice.

Additional Resources

Modern Infrastructure & IaC Architecture — the pillar hub for platform-layer decisions like node configuration and runtime behavior
Kubernetes Requests vs Limits — the scheduler-and-kernel model this post extends, including QoS classes and default OOM behavior
VPA vs HPA: Why Most Teams Choose the Wrong Autoscaler — how autoscalers change requests, which now also change swap allowances
Kubernetes Documentation: Swap memory management — official reference for NoSwap, LimitedSwap, metrics and the documented risks
Kubernetes 1.32: Fresh Swap Features for Linux Users — eligibility rules, the request-proportional allowance and the request-equals-limit opt-out

Originally published at rack2cloud.com

A Completeness Gate Is Only as Complete as Its Required Set. Don't Let the Model Write It.

NTCTech — Mon, 05 Oct 2026 12:13:51 +0000

A completeness gate has two different problems: determining what must exist, and determining whether it exists. We keep treating them as one.

One common design hands both to the same model. A generated change plan, policy bundle or control mapping goes in, the model is asked whether anything required is missing, and it answers against its own idea of what "required" means. If it reports nothing missing, the gate passes. Nothing alerts, because nothing failed. The check ran correctly against a list that was already short.

Every Completeness Gate Solves Two Problems

The first problem is enumeration: which elements must be present? The second is comparison: is each of them present in this artifact? Any gate that checks a deployment plan against a control catalogue, or a generated policy against the clauses a standard requires, does both, whether or not the design separates them. Wherever model-generated plans, policies or change requests pass through a gate like this, it belongs to AI infrastructure architecture as much as to the pipeline that hosts it.

Comparison is a membership question, and it can only report on elements the required set contains. Enumeration decides the ceiling. A flawless comparison against an incomplete set returns a clean result for an incomplete artifact.

Job	Question	How it fails
Enumerate the required set	What must exist?	Required elements are missing from the set, so the gate under-rejects. Requirements the specification never states are added, so it over-rejects.
Compare artifact to set	Is each required element present?	A required element is in the set and absent from the artifact, and the comparison does not report it.

The two enumeration failures behave differently under review. An invented requirement is written down, so a reviewer can challenge it. A missing requirement is an absence, and finding it is the same problem the author failed to solve. The research below argues this asymmetry, and it is why a model-written list can survive inspection and still be short.

A check that reports success while the objective went unmet already has a name on this site, False Completion (#82), and the cost of continuing to verify automated output is the subject of the Automation Validation Tax (#172). This post is narrower than either: it is about how the set a gate checks against gets built. How an approval step decays into a rubber stamp is covered in GhostApproval.

What the Evidence Shows, and Where It Stops

Two pieces of research bear on how a completeness gate should be built, and neither is about infrastructure.

AbsenceBench gives a model an original document and an edited copy and asks which elements were removed. On its GitHub pull-request portion, 14 models scored between 17.3 and 40.0 percent F1 across their configurations, with the best at 40.0, even though the authors note that a simple program solves the task in linear time. The models are mid-2025 generation, the omissions are literal deleted lines, and the paper reports no error bars.

Judging Is Not Enumerating measures models as authors of the sets that other systems check against: test suites, answer keys, rubrics. It tests four separate reference constructions, and in each the models judged whether a candidate belongs far better than they authored the set itself. The size of the gap depends on the construction. On the paper's algorithmic construction it was about 0.29 to 0.34 F1 and did not close across a 24-fold range of model sizes. On executable code, models that judged at F1 0.74 to 0.90 wrote test suites that admitted only 19 to 42 percent of the correct solutions. In a separate test, planted over-inclusions were detected six to seven times more often than planted omissions. A control then located the deficit: asked to emit the predicate rather than list its members, the same models reached an F1 of about 0.99. In that control the rule was already in the prompt, so the model was restating a rule, not deriving one.

The authors scope the finding to one-shot authoring without test-time reasoning. Enabling reasoning on two closed frontier models closed or nearly closed the gap on the simplest, rule-based construction. It did not reliably repair test-suite authoring, where the expected behavior has to be derived from prose. The paper's one production observation is a K-12 assessment pipeline of 43,227 scored items, where errors in model-authored answer keys were omissions rather than inclusions by about ten to one, as graded by a commercial evaluator.

⚠ What this evidence does not show: It does not show that current models cannot do this: reasoning narrowed or closed the gap where a rule could be executed. It does not test semantic absence, because AbsenceBench removes literal lines. And it contains no infrastructure incident. Applying it to plans, policies and change requests is an architectural extrapolation from benchmark and research results.

Restate the Rule. Enumerate in Code.

The completeness gate architecture that follows separates the two jobs and gives each to the component that does it reliably. The model may translate a rule that has already been specified into an executable predicate. Code then runs that predicate against an authoritative, machine-readable source to enumerate the required set. A deterministic diff compares the artifact to that set, and the gate reports the missing elements by name.

Step	Who does it	Why
Specify the rule	A person or a standard	The rule exists before the gate does
Translate it to a predicate	The model, with the result reviewed	Restating a given rule is the task the research control shows models doing well
Enumerate the required set	Code, from the source	The result is reproducible and adds no probabilistic step
Compare artifact to set	A deterministic diff	Set difference is a mechanical operation
Report	The gate	Each missing element is named

Specify, translate, enumerate, diff: the model touches only the second step. It participates in expressing the rule. It does not become the database of what must exist.

When the predicate and the source are correct, deterministic enumeration gives a reproducible result without introducing another probabilistic judgment step. That is the whole argument, and it is about cost and verifiability, not about what models are capable of. A model can still propose predicates, and a reasoning model may enumerate acceptably in simple cases. The design should not make model enumeration the authoritative source of the required set when the set can be derived deterministically from an authoritative machine-readable source. Infrastructure already uses versions of this separation: policy-as-code evaluates machine-readable Terraform plans against explicit rules rather than asking a model to invent the required checks at evaluation time.

Four design properties keep the split intact in any completeness gate.

01 — Source-derivable set

The required set must be computable from something that exists outside the model: an inventory, a schema, a control catalogue, a plan diff. If no such source exists, there is nothing for code to enumerate, and the gate cannot claim completeness.

02 — Reviewable predicate

The model's output is a predicate a person can read, test and version. A wrong predicate is visible text, while an omission from a model-written list is not. That visibility is this post's inference from the research finding that over-inclusions are easier to catch than omissions.

03 — Deterministic enumeration

Enumeration runs as code against the source and produces the same set on every run. Where a reasoning model enumerates instead, treat its output as a proposal to diff against the code-derived set, not as the set itself.

04 — Over-rejection control

A predicate that encodes a requirement the specification never stated blocks valid work. Trace every predicate to a clause in the specification, and review rejections as seriously as passes.

Where This Stops Working

A completeness gate built this way works where the required set can be derived from a machine-readable source. Plenty of requirements cannot be derived that way. "The runbook should cover failure of the identity provider" and "the change should consider downstream consumers" have no computable predicate, and nothing here says how to gate them. Nor does the evidence: AbsenceBench removes literal lines, and the research controls use rules a model was given. For requirements stated only in prose, a model-derived set may be the only practical option. In that case the gate is no longer proving completeness against a deterministic required set. It is performing a model-mediated review, its omissions stay hard to see, and its output should say so, so that a pass is read as weaker evidence.

Two neighboring questions are out of scope here. Who is entitled to reject a gate's conclusion is a separate authority question, covered for recovery claims in Recovery Evidence Boundary and for AI platform evidence in AI evidence verification; it does not depend on how the required set was built. And a probe reports on what it probes, which is the limit What Kubernetes Health Checks Actually Guarantee covers for readiness and liveness checks.

Test the Gate, Not Just the Artifact

A completeness gate is itself a system that can be wrong, so test it with artifacts whose contents you already know. Run a known-complete artifact through and confirm it passes. Run one with a deliberately removed element and confirm the gate names it. If the gate cannot find a planted omission, its clean results carry no information. A fixed answer set defined before the run is the same move AI workload RPO makes for restore tests.

The research points the same direction. Its one mitigation that worked for authored code test suites was discarding any suite that rejected a known-correct probe, which cut false rejection from 58 to 92 percent down to 5 percent or less, at the cost of keeping only 5 to 39 percent of suites. A planted-omission test is the mirror image, and it is cheap to run on every change to the predicate or the source.

Architect's Verdict

A completeness gate can only report on what its required set contains. If a model wrote that set in one pass, the gate inherits the model's omissions and returns clean results for incomplete artifacts.

The mistake is treating enumeration and comparison as one job. The research shows that judging membership and enumerating requirements are different tasks, and that model performance can diverge sharply between them. Where a rule exists, let the model translate the rule and let code do the listing. Reasoning narrows the gap in simple cases and does not reliably close it elsewhere, so the design should not depend on it.

Where the required set can be derived from an authoritative source, derive it. Where it cannot, say that the gate checked a model-derived list. Either way, a gate that has never found a planted omission has not shown it can find a real one.

A completeness gate is only as complete as the set it checks against.

Originally published at rack2cloud.com

The Ransomware Takedown Closed the Actor's Infrastructure. It Didn't Close Your Exposure.

NTCTech — Sat, 03 Oct 2026 12:07:15 +0000

A ransomware takedown changes what an attacker can do next; it does not tell a victim what was taken, who holds copies, or whether its exposure is closed.

On 30 September 2026, law enforcement took control of the KillSec leak site and secured at least 110 terabytes of stolen data against further unauthorised access, according to Europol. Eurojust coordinated authorities from nine countries and reports three arrests, eight house searches, and five seized servers the group used to store victim data. It describes a group responsible for almost 1,000 attacks. That is a real result. It is also a result about the actor.

This note is about the other side of that sentence: what a victim can establish afterward, and what no press release can establish for it.

What a Ransomware Takedown Changes, and What It Doesn't

A ransomware takedown that seizes a leak site and the servers behind it does real work. The actor can no longer publish through that site, at least 110 terabytes of stolen data has been secured, and investigators have evidence to analyse. Those are changes in the actor's capability and in the investigators' position. None of them touches the victim's state: what was taken, how much, where copies sit, who has seen them. Those facts were set by what happened before the seizure, and the seizure does not reach back to them.

Recovery closure already has a model, the incident recovery process, and this note does not revisit it. That model asks whether the restored environment can be trusted again. It does not ask what left the environment first, and that question arrives with neither the restore nor the seizure notice. A victim can hold a clean closure record for the environment and an open, unevidenced question about the data, in the same incident.

In data protection architecture, the unit of concern is the data and where it went, not the actor's current operating status. Treat a ransomware takedown as an intelligence event on your side of the incident. It is new information about the actor. It is not a state change in your exposure record. Treating it as one substitutes the actor's status for evidence about your own data.

The Non-Payer Path Was Already Open

Eurojust's account of the group's method is specific. KillSec got into organisations through poorly secured access points, particularly those linked to cloud storage, and copied the data to infrastructure it controlled. It then threatened to publish. Victims were sent samples as proof of possession. If a victim did not pay, the files were made available for free download. In some cases the group received substantial payments.

Read that as a sequence and the seizure sits at the end of it. For organisations that refused, release was the group's standard practice before 30 September. The seizure removes the leak site. It does not recall a file that was already downloadable, and neither the Eurojust release nor Europol's headline says anything about copies that left the group's infrastructure. Whether a particular victim's files were released, or downloaded by anyone, is not something either source states.

One artifact from that sequence does sit with the victim: the proof-of-possession samples. If your incident record holds them, they are direct evidence that the actor held specific files, which makes them the firmest anchor for scope you will have. They bound the scope from below, not from above. A sample shows that something was taken. It does not show that nothing else was.

Whether to pay is a separate decision with its own constraints, covered in the payment-ban dependency post. The point here is narrower. Whatever a payer was promised came from the actor, and whatever a refuser faced was the actor's publication practice. Neither is evidence the victim holds.

The Victim List Is Still Being Written

Eurojust says the next phase is examining the seized devices and data and tracing the financial proceeds, and that this evidence may help identify other victims, attacks and people involved. That sentence cuts two ways. The investigators' victim list is not complete. And the evidence that completes it is the same corpus a victim would want answers from.

An organisation can therefore be a KillSec victim today without knowing it, and learn so later from an authority rather than from its own telemetry. The Eurojust release does not describe how victims get told. Plan for a notification that arrives after your own incident record was closed, from outside your own timeline. The size figure is a floor on what investigators secured, not a measurement of what any individual victim lost.

The entry point matters here. Eurojust describes access through poorly secured access points, particularly those linked to cloud storage. If that describes your environment, the records that show what was read or copied may sit with the storage platform rather than on your network, and how long they are retained is a setting chosen long before the incident. By the time an authority makes contact after a ransomware takedown, the evidence that would have answered the question may already have aged out.

What a Victim Can and Cannot Verify After a Ransomware Takedown

The question after a takedown is practical: what can this organisation put evidence behind? Two short lists.

Can establish:

External communications — which systems communicated externally during the intrusion window, from flow or egress records held outside anything the intruder could write to
Reachable data stores — which repositories held data the compromised access could reach, from your own data-store inventory checked against that access scope
Identification as a victim — whether law enforcement or the actor has identified your organisation as a victim
Defined scope — whether the evidence you hold supports a defined exfiltration scope, or only fails to rule one out

Cannot establish:

Downloads — who else downloaded copies
Affiliate copies — whether affiliates retained copies of the data they helped take
Redistribution — whether files released before the seizure were redistributed
Seizure completeness — whether the data secured in the seizure is all of what was taken

The fourth item on the first list carries the most weight, and it is the easiest to get wrong. Evidence that supports a defined scope is positive: records showing what left and where it went. Records that merely fail to show a transfer rule nothing out unless you know the logging would have captured it. A gap in the logs is not a finding of no exfiltration.

The first list depends on evidence collected before and during the incident, and on whether that evidence can be trusted. Records written by infrastructure an intruder could reach are weak proof of anything; the evidence-layer trust boundary covers why a copy of evidence counts as independent only if the attacker cannot exercise the same authority over the original and the copy. The second list cannot be built from anything the victim holds. It sits with the actor, the affiliates, and whoever downloaded. Some of it may narrow if investigators later share what the seized systems show. Nothing in the Eurojust release says they will.

Architect's Verdict

The ransomware takedown closed the actor's infrastructure. It did not close the victim's exposure, and it is not evidence about it. Two different things came out in the same press release.

What a takedown leaves behind is a verification problem. Exposure closure requires exfiltration-scope evidence, not attacker status. A victim holding that evidence can establish a scope. The rest of the question, who holds copies, was outside what any victim could verify before the seizure and remains outside it after.

Do not read an actor's defeat as your own closure. The actor's status is public record. Your exposure is whatever your evidence can account for.

Originally published at rack2cloud.com

Rotating Credentials Is Not Revoking Them. The Revocation Unit Decides Whether You Can.

NTCTech — Fri, 02 Oct 2026 12:03:12 +0000

The revocation unit decides whether credential rotation can ever be atomic. It is the scope one revoke action removes: a token, an identity, or a trust root. Most teams treat rotation and revocation as one operation. They aren't. Rotation issues new secrets. Revocation removes the authority of everything that could have used the old ones, and it works only if the architecture can name that set and remove it in a single step.

The Server Was Fixed. Persistent Access Wasn't. covered what survives when remediation never reaches the layer where persistence lives. This post covers a layer remediation did reach: the credential layer itself, where a rotation can complete on schedule and still leave a surviving credential able to read whatever replaced the old one. The decision question is different. It is not which layers recovery has to validate, but what unit revocation acts on.

The Revocation Unit Is an Architecture Decision

Before any incident, the architecture has already answered a question, usually by default: when you revoke, what exactly are you revoking?

Revocation unit	One revoke action removes	What may remain outside the unit
Token	A single bearer credential	Other tokens the same identity can mint, and any secret that token could read
Identity	Every credential issued to one principal	Static secrets that principal already read and copied elsewhere
Trust root	Everything that chains to one issuer or signing key	Any credential that was never issued through that root

That answer belongs in data protection architecture, not in the incident runbook. A runbook can only sequence actions the architecture permits, and the revocation unit fixes what any single action is capable of removing before anyone is in the room.

A token-level unit is precise and narrow. An identity-level unit is wider but stops at the principal's boundary. A trust-root unit can be the broadest, if the affected authority chains through that root, and it only reaches what was issued through it. None of the three is wrong. The decision is whether the unit you chose can enclose the set you will need to remove.

Reachability: What the Revocation Unit Has to Enclose

That set is the reachability graph: every credential that can read, write, or mint any secret in the affected set, followed transitively. A token that can read a vault path holding a signing key that can mint further tokens is in the graph, and so is everything those tokens can reach in turn.

Revocation is complete only if the unit encloses the full closure of that graph. Anything left outside is a survivor, and a survivor with read access to the store holding the replacement secrets can simply collect them.

Reachability is a graph property, not a list. It grows with every integration, every shared service account, every pipeline granted a convenient scope, and it is recorded nowhere unless the architecture is built to record it.

The usual response is to rotate everything. That does not escape the problem, it relocates it. The mechanism doing the rotating holds credentials of its own, and those are in the graph too. A rotation job that can write to every store can also be read through, impersonated, or replayed, so the revocation unit has to enclose the rotator as well as the secrets it rotates.

Atomicity: Why Rotation Opens a Window

Atomic revocation means that at one instant, no credential in the closure remains valid except those deliberately reissued. If the revocation unit is smaller than the closure, revocation stops being an operation and becomes a sequence: kill this token, rotate that secret, then the next. A sequence has a window.

A generic sequence shows the shape. A pipeline token can read a shared store. Responders revoke that token, then rotate the database password, then the signing key. Another token issued to the same service account was never in the revocation unit, so it is still valid. It reads the new database password the moment it is written, and the signing key after it. Every step in the sequence succeeded, and the closure still leaked.

Inside that window, the failure mode is replacement-secret exposure. Rotation writes fresh, valid, high-value material into a store that surviving credentials can still read. The rotation does not just fail to lock the attacker out. It hands them the new keys. The defenders see a completed rotation and a dead old secret, while the party holding a surviving credential holds a valid replacement.

Atomic at the unit is not instantaneous at every verifier. Cached validations, token lifetimes and propagation delay leave a residual window even under a trust-root revocation. The design property is not zero. It is a window that is bounded and known in advance, rather than one discovered afterward from logs.

Verification: You Can Only Prove Closure Over a Set You Can Name

Revocation is usually treated as an action: run it, confirm it ran. The stronger claim is about proof. If you cannot enumerate the affected authority set, you cannot prove that revocation closed it. You can show that the revocation action executed. You cannot show that no reachable authority survived.

Enumerability is a prerequisite for proving closure, not for every revocation. A trust-root revocation can invalidate an entire authority domain without listing each credential inside it. But the proof then moves up a level: you have to establish that everything with reach to the replacement secrets chains through that root. That is still a claim about a set, and it has to be checkable.

A closure test has three parts. Establish the set of principals that could read, write or mint any secret in scope, either by listing them or by showing they all chain through one revoked root. For each, show it is invalid or unreachable after the replacement was written. Keep that evidence somewhere the rotated system cannot rewrite. The question of who has standing to confirm that evidence is the governance problem Your Recovery Program Has a Recovery Evidence Boundary addresses. Here the problem is mechanical: whether the set can be established at all.

Restore Testing Is Becoming An Adversarial Discipline already assumes the recovery team's own credentials can end up in the attacker's hands. Closure verification extends that posture to replacement secrets: assume a survivor exists, and go and find it. Architectures built to survive adversarial compromise need credential closure as an explicit completion criterion rather than an assumption.

What the Trivy Incident Documents

Aqua Security's advisory for the Trivy supply chain compromise records that after the March 1 initial disclosure, credential rotation was performed without being atomic: not every credential was revoked at the same moment. It adds that an attacker could have used a still-valid token to exfiltrate newly rotated secrets during a rotation window that lasted a few days. The wording is conditional, and this post keeps it conditional.

That makes it a documented demonstration of the mechanism, not a reconstruction of the attack. The advisory does not establish how many credentials survived or what was taken, and this post does not claim to.

One more detail from the same advisory is worth keeping. Exposure is scoped per component and per consumption path. Whether a workflow was affected depended on which component it pulled, how it pinned it, and when it ran, and a workflow that pinned one action safely could still pull a compromised one through it. That is reachability appearing in a dependency graph rather than as one clean window.

Design Properties That Decide Atomicity

Four properties, all fixed at design time, determine whether an architecture can revoke atomically. None depends on any particular incident.

Secret-store topology

Where replacement secrets are written, and who can read that location at write time. One shared store readable by an entire pipeline means one surviving credential reads every replacement. Scoped paths per consumer shrink the closure.

Issuer design and credential lifetime

Static, long-lived credentials have no built-in expiry, so they are revoked or they persist. Short-lived credentials issued from a central root let trust-root revocation collapse the unit onto the issuer: stop issuing and the survivors age out within a lifetime you chose in advance. That is the bounded window described above. For the case where authority keeps working after the trust behind it has failed, see also Your Cloud Isn't Compromised. Your Vendor Is. Now What?. The question here is narrower: whether the unit can reach that authority at all.

Cross-secret reachability

Some secrets mint others: publishing credentials, signing keys, role-assumption paths. Rotating a leaf secret without the secret that can mint it leaves a way to recreate what you just removed. The closure has to include the minters.

Verification requirements

The architecture has to make the affected set answerable on demand: a record of which principals can reach which secrets, issuance records, and a way to query validity. If the answer has to be reconstructed from logs after the fact, the architecture cannot prove closure. It can only show that rotation executed.

Architect's Verdict

A revocation event is complete when no surviving credential can reach the replacement secrets. It is not complete when the rotation finishes.

The real problem is that revocation gets treated as an action to execute rather than a property the architecture either supports or does not. The revocation unit, the reachability graph and the enumerability of the affected set are all fixed long before an incident. During the incident, you discover what you built.

If you cannot enumerate the affected authority set, you cannot prove that revocation closed it. You can show that the action executed. You cannot show that no reachable authority survived.

Additional Resources

Data Protection Architecture — the pillar page for recovery, resiliency and backup architecture; this post sits in its credential-and-authority layer
The Server Was Fixed. Persistent Access Wasn't. — what survives when remediation never reaches the layer persistence lives in; this post extends that argument inside the credential layer
Restore Testing Is Becoming An Adversarial Discipline — the adversarial posture toward the recovery team's own credentials that closure verification extends to replacement secrets
Your Recovery Program Has a Recovery Evidence Boundary — the governance question of who has standing to confirm recovery claims; this post covers the mechanical precondition, a provable set
Your Cloud Isn't Compromised. Your Vendor Is. Now What? — see also: authority that keeps working after the trust behind it has failed
Trivy ecosystem supply chain compromise (GHSA-69fq-xp46-6x23) — Aqua Security's advisory; primary source for the non-atomic rotation language used in the evidence section
CVE-2026-33634 — NVD record carrying the same advisory text

Originally published at rack2cloud.com

The Gateway Consolidated Your Controls. NetScaler Compromise Response Has to Unconsolidate Them.

NTCTech — Thu, 01 Oct 2026 12:23:56 +0000

NetScaler compromise response is difficult for the same reason the appliance is valuable: it concentrates authentication, remote access, certificates and administrative trust in one node, so every step of response has to unwind that concentration.

Citrix disclosed eight NetScaler ADC and Gateway vulnerabilities on September 27, 2026, two of them already exploited. The guidance that followed from Citrix, CISA and Google's Mandiant and Threat Intelligence Group reads less like a patch procedure than a list of interruptions, and the order depends on what you find on the appliance.

Every control the gateway concentrated is a connection that compromise response has to cut.

What Was Disclosed, and What Wasn't

Citrix published bulletin CTX697096 on September 27, 2026, covering CVE-2026-88771 through CVE-2026-88778. The bulletin says exploitation of CVE-2026-88771 and CVE-2026-88772 has been observed on unmitigated deployments. The first is an unauthenticated command-execution flaw that affects every deployment in its default configuration. The second is a memory-overflow flaw that requires DTLS, which is on by default for VPN virtual servers. The bulletin covers customer-managed deployments; Citrix upgrades its own cloud-managed services.

CISA added both to its Known Exploited Vulnerabilities catalog the same day, with a September 30 federal remediation date and a requirement for forensic triage under BOD 26-04. Its alert also carries a caution that matters later in this note: preserve forensic evidence before applying updates, because updates can cost forensic visibility.

The bulletin says nothing about how long exploitation had been running, how widely, or by whom. The public source for that is Google's Mandiant and Threat Intelligence Group, whose September 29 report describes a campaign against CVE-2026-88772 that was "ongoing since at least early September," with likely impact on government, financial services, technology, education, and legal and professional services organizations in North America and Europe. For CVE-2026-88771 the report relies on the vendor's disclosure rather than its own analysis. It does not attribute the activity, and neither does this note. How long an exposure runs before an operator can see it is a cost in its own right, covered in patch visibility debt, and it is why the missing start date matters for NetScaler compromise response.

Concentration Is the Point of the Appliance

A NetScaler in the path exists so that authentication brokering, TLS termination, remote access and policy enforcement happen in one controlled place instead of inside every application behind it. Deciding where to concentrate enforcement is a cloud architecture strategy decision, and it is also a decision about where failure will concentrate. Google's report notes that these devices face the internet, sit outside endpoint detection, and often store or process credentials that reach deeper into the network. Because TLS terminates on the appliance, upstream network devices generally cannot see the request paths or headers inside it.

None of that is an argument against gateways. The concentration is the design working. The argument is about what happens afterward, because NetScaler compromise response inherits the concentration. Whatever the appliance holds, response has to invalidate. Whatever it fronts, response has to be ready to interrupt. The question for an architect is not whether the appliance is important, which was settled when it was placed in the path, but whether anyone has written down what it costs to take it out of service, and who may decide to.

NetScaler Compromise Response Is a Sequence of Interruptions

Three documents describe what to do, and they start from different states. CISA writes for the gap between disclosure and update. Google's guidance writes for an estate that may or may not be compromised. Citrix's standing article on suspected compromise writes for an appliance you no longer trust. Each step in NetScaler compromise response has a service cost, and several steps cost the same service.

Each step in the response sequence has a service cost; isolation and rotation reach every user of the node.

Evidence comes first, if you can afford it. Citrix's article starts with preservation: snapshot a virtual appliance, record system time and NTP settings before isolation, keep remote syslog and management-console logs, and collect a support bundle, while noting that the vendor does not support forensic investigation. Its next step is a core file of the packet engine, and the article states that the system performs a warm restart during that process and drops SSH sessions. Google asks for a memory-inclusive snapshot of a virtual appliance before any reboot where operationally possible, and several of its detections depend on logs the appliance does not forward by default. That matters because the actor Google describes scrubbed its traces from appliance files and, per Google's hunting guidance, from web access logs, which is the boundary in your logs are not evidence if the infrastructure that produces them can rewrite the record.

Then isolation, which for a gateway means removing remote access. Google says isolating a compromised or suspected appliance can cause significant business disruption where the appliance provides remote access, and that broad internet isolation or strict allow-listing disrupts remote workforces. It recommends a targeted, phased approach that prioritizes the fixed build, with isolation weighed against risk tolerance and operational requirements. When the appliance fronts services owned by several teams, who may make that call is an ownership question before it is a technical one, the pattern in an ownership problem, not a technology problem.

In a high-availability pair, Google adds a step that turns the redundancy mechanism into the thing being contained: halt configuration synchronization and assess each node independently, so a compromised node cannot replicate attacker-modified configuration to its standby. Google presents this as a preventive step, not a reported occurrence. The architectural reading is mine. The feature that keeps the pair available is also a propagation path, so containment means running the pair as two unvalidated nodes until each is cleared. Delay is exposure. Action is lost redundancy.

The mechanism that keeps the pair available is also the path containment has to close.

Then the fix, which is not one action either. Citrix's article has virtual appliances replaced rather than repaired, firmware upgraded before any configuration is restored, and the restored backup verified to predate the compromise. That last condition needs a date. Google's early-September floor covers one of the two exploited flaws, and the vendor bulletin gives no date for either. Citrix's article also says that where law enforcement involvement is possible, counsel should weigh in before rebuild, because evidence preservation can outrank restoring service. And a fixed build closes the flaw without evicting an actor who persisted through it. Google describes web-server configuration changes, an elevated-privilege setting on the shell and an appliance reboot used to make access survive, which is the argument in the server was fixed and persistent access wasn't.

What Concentration Makes You Rotate

The rotation guidance in NetScaler compromise response is where the concentration becomes visible. Google and Citrix both list what has to change, and the lists overlap almost entirely. They differ on timing: Citrix revokes before the rebuild and rotates restored secrets afterward, while Google places rotation after patching.

What the appliance concentrates	What the guidance has you change	Who feels it
Remote-access sessions, including administrative, Gateway, VPN and ICA/HDX	Revoke and terminate the sessions	Every user connected through the node
User accounts authenticated through Gateway or AAA virtual servers	Change credentials on their respective systems (Citrix)	Every user population the node authenticated
Appliance administrator, local and SSH credentials, plus key encryption keys (Citrix)	Rotate	Administrators and the automation that uses them
Certificates and private keys	Revoke and replace	Every service that presents the node's certificate
LDAP, RADIUS, TACACS, SNMP and API integration credentials	Change on the systems they authenticate to	Identity, network and monitoring teams at once
Connected infrastructure such as StoreFront, delivery controllers and session hosts	Review for lateral movement	Teams that never owned the appliance

The first two columns come from the two documents. The third is my analysis.

Read down the first column and you have the appliance's real dependency map: the directory, the authentication path, the certificate chain, the management plane and the user population, all in one node. Nobody drew it that way when the appliance was deployed, and it was there the whole time.

Questions to Run Against Your Estate

Run these against your estate before NetScaler compromise response is something you are doing rather than planning:

Pre-incident questions:

What terminates on each NetScaler, and which teams own the services behind it?
Who may authorize isolating a Gateway, and is that authority written down?
Is the list of secrets, certificates and integration credentials on each appliance maintained, or would you be assembling it during response?
Does the runbook say when to halt synchronization in a high-availability pair, and what redundancy you give up while it is halted?
Which configuration backup would you trust, and how would you show when it was taken relative to the compromise?

Architect's Verdict

A gateway consolidates controls on purpose, and compromise response has to unconsolidate them on demand. NetScaler compromise response is not a patch. It is an ordered series of interruptions to everything the appliance was trusted to hold together.

Every incident response interrupts service, so that alone is not the argument. The difference is scope. Each decision in this sequence reaches every service that shared the node, and the public guidance enumerates them: sessions, administrative access, certificates, directory and authentication integrations, management integrations, and the infrastructure connected behind it.

The rotation inventory is the architecture diagram. It shows what the appliance concentrated, in the order a responder has to undo it, and it is more accurate than any design document written when the appliance was deployed.

Gateways earn their place in the path. The cost of that placement arrives as a list, on the day you trust the node least.

Additional Resources

Cloud Architecture Strategy — pillar hub for the architecture decisions behind where enforcement, and failure, concentrate
Your Logs Are Not Evidence If The Infrastructure That Produces Them Can Rewrite The Record — why appliance-side logs cannot carry the investigation alone
The Server Was Fixed. Persistent Access Wasn't. — why a fixed build does not end a compromise
SharePoint Vulnerabilities And The Cost Of Patch Visibility Debt — the cost of the gap between exposure and operator visibility
Multi-Cloud Coherence Is an Ownership Problem, Not a Technology Problem — why cross-team decisions stall on ownership before technology
Defending Against Active Exploitation of Citrix NetScaler ADC and Gateway Appliances — Mandiant and Google Threat Intelligence Group campaign analysis and containment guidance, September 29, 2026
Steps to Take if NetScaler ADC is Suspected to be Compromised (CTX694799) — Citrix's own compromise-response sequence
NetScaler ADC and Gateway Security Bulletin for CVE-2026-88771 through CVE-2026-88778 (CTX697096) — the vendor bulletin, published September 27, 2026
Critical Zero-Day Vulnerabilities Exploited in Citrix NetScaler ADC, Gateway (CISA) — CISA alert carrying the preserve-evidence-before-updating caution

Originally published at rack2cloud.com