DEV Community

Cover image for Uumi: A Governed Agent Fleet for Enterprise Credential Rotation
Ukpaa Chigozie
Ukpaa Chigozie

Posted on

Uumi: A Governed Agent Fleet for Enterprise Credential Rotation

For Google's All Things Agentic Hackathon, we built Uumi: a governed agent fleet for enterprise credential rotation.

Uumi's agents drive a rotation from exposure to revocation. They check declared workload bindings, choose the strategy, build versioned browser playbooks, and evaluate recovery branches fixed during planning. They never receive the credential value.

How does that even work?

We split reasoning from authority. Google Agent Development Kit (ADK) agents reason over redacted, run-bound facts and return typed decisions. Deterministic services execute credential mutations, validate each request, verify the replacement, and commit each state transition. Google Cloud Workflows keeps the operation moving across hours or days.

This article explains the design, then follows one credential rotation from exposure to verified completion.

Contents

TL;DR

Uumi coordinates credential rotation across the provider, Secret Manager, and every declared consumer. An exposure event or authenticated operator request creates a fenced run. That run creates the replacement, stores it, deploys it at zero traffic, verifies it through real consumer identities, and rolls it out under observation. When policy protects revocation, the operator must authorize the exact action before it runs. Completion requires provider-side revocation proof, retirement of the old secret version, and committed evidence.

Four separately deployed ADK agents specialize in inventory assessment, planning, browser playbooks, and console coordination. Workflows sequences their work across the run. Firestore holds revisions, leases, fencing tokens, plans, approvals, attempts, and state transitions. The MCP broker validates every brokered tool call, while the guarded browser worker validates proposed console actions. The verifier performs the independent checks that allow a stage to advance.

Gemini never receives plaintext credential values. API-created values move directly from the provider connector into Secret Manager. Browser-disclosed values pass through Secure Capture while model visibility and recording are paused.

Retries begin with reconciliation. Uumi compares the provider and Secret Manager with pre-mutation baselines, removes an orphan only when it can attribute it to the timed-out attempt, and then retries. Recovery is constrained to a branch fixed during planning. Compensation remains a separate terminal outcome.

We built Uumi with Google ADK and Gemini on Vertex AI. Workflows carries the long-running operation, Cloud Run hosts the services, Firestore stores state, Pub/Sub and Eventarc deliver committed events, and Secret Manager stores credential versions. Section 2 covers the full Google Cloud design.

1. What enterprise credential rotation means

Enterprise credential rotation is a live migration between two credential generations.

One credential can have a provider-side identity, several Secret Manager versions, multiple runtime consumers, different service accounts, active deployment revisions, a linked exposure incident, and an audit history. Each dependency must move to the replacement before the predecessor can be retired.

Creating a new credential is only the first mutation. A complete rotation proves five conditions:

  1. The replacement exists at the provider and in one exact, enabled Secret Manager version.
  2. Every declared consumer can access that version under its own runtime identity.
  3. The target generation is deployed, verified, rolled out, and healthy under real traffic.
  4. Provider-side revocation is proved before the old secret version is disabled. API-managed providers must reject authentication with the predecessor; browser-managed providers must pass the published revoke checkpoint.
  5. The final evidence and audit snapshot are durable, any linked incident is resolved, and the credential lock is released.

The order protects availability and containment at the same time. Early revocation can break a workload. Late revocation leaves the exposed credential usable. Uumi keeps both generations visible, binds every stage to the target generation, and advances only on evidence.

Credential rotation from trigger to durable proof

Figure 1A. One enterprise credential rotation, from trigger to committed completion.

Each credential can have only one active rotation. A credential lock prevents agents, operators, and duplicate incident deliveries from creating competing target generations.

2. How we built Uumi on Google Cloud

Uumi starts from either a signed exposure event or an authenticated operator request. Both create the same locked rotation run.

Manual and incident-triggered run admission

Figure 1B. Manual and incident inputs converge on one run-creation contract.

Run creation and dispatch are separate steps. Firestore commits the run, first step, credential lock, idempotency record, and run.created outbox event together. The publisher can retry delivery until Pub/Sub accepts that committed event. Eventarc then starts Workflows.

Durable dispatch through the transactional outbox

Figure 1C. A committed outbox event starts Workflows through Pub/Sub.

At each stage, Workflows sends the current revision and fence to the coordinator. The coordinator asks an agent for a typed decision, the broker for a tool call, or the verifier for a pinned probe. The broker or browser worker performs the mutation. The verifier checks the result. Firestore commits the next state after the coordinator applies the stage gate.

Governed stage execution across managed targets

Figure 1D. Workflows sequences each stage across the coordinator, agent fleet, MCP broker, and verifier.

Stage evidence follows its own durable path. Evidence objects carry stable identifiers, content digests, and object generations. Redacted audit events append to a per-organization hash chain before a separate outbox delivers them to a locked regional Cloud Logging bucket.

Evidence storage and audit delivery

Figure 1E. Stage evidence and audit events follow separate durable paths.

The Google Cloud implementation is divided by responsibility:

Responsibility Uumi implementation
Agent reasoning Four ADK agent applications using Gemini 3.7 Flash through Vertex AI
Browser reasoning Gemini 3.7 Flash for bounded Computer Use
Agent operations Agent Runtime, Agent Registry, Agent Gateway, Agent Identity, Memory Bank, and Model Armor
Long-running sequencing Google Cloud Workflows
Durable state Firestore transactions, immutable bindings, leases, fencing tokens, approvals, plans, and attempts
Governed mutation MCP broker for typed connectors; guarded Compute Engine worker for browser actions
Messaging Pub/Sub, Eventarc, and transactional outboxes
Service runtime Python services on Cloud Run
Isolated browser runtime One-run Compute Engine workers with no public IP, Shielded VM, CMEK, and default-deny Secure Web Proxy egress
Credential storage Secret Manager with immutable version references
Operator access Firebase authentication and IAP-protected browser view and takeover
Evidence and telemetry Cloud Logging, Cloud Trace, Cloud Monitoring, and redaction-safe OpenTelemetry
Persisted-data perimeter Regional policy, CMEK, and VPC Service Controls where supported

The long-lived Python services run on Cloud Run: the web gateway, API, incident ingestion, outbox publisher, MCP broker, coordinator, IAP browser gateway, notification worker, and audit publisher. The client is served through Firebase Hosting. Each browser job runs on a dedicated Compute Engine VM.

Public ingress is limited to the authenticated web app, signed webhook ingestion, and the IAP-protected browser gateway. The API, broker, coordinator, and internal workers remain private. Each workload has its own service identity and minimum role set.

The policies that govern a run are immutable:

  • ControlVersion fixes allowed tools, protected tools, triggers, checks, observation limits, and recovery policy.
  • PlaybookVersion fixes browser objectives, selectors, checkpoints, declared fields, and Secure Capture metadata.
  • ProbeVersion fixes a verifier target and success condition.
  • RotationPlan fixes the strategy, scopes, consumers, rollout milestones, observation duration, and recovery identities.
  • ProtectedAction fixes the exact mutation an approval can authorize.

Agents reason from these objects. Execution uses the published versions bound to the run.

3. How one credential rotation works

Uumi represents a rotation as twelve durable stages:

Trigger -> Preflight -> Plan -> Create -> Store -> Deploy -> Verify -> Rollout -> Observe -> Approval -> Revoke -> Complete

Each stage produces evidence before the next state commits.

Admit one run

Incident ingestion authenticates and normalizes the event. A stable event ID prevents duplicate deliveries from opening another incident. The incident service writes the dedupe record and incident in one transaction, correlates the event with credential inventory, and loads the credential's published control version.

Automatic rotation starts when correlation confidence meets the required threshold and trigger policy allows it. Other incidents remain recorded for review.

Incident-triggered run creation

Figure 2A. A verified incident creates one deduplicated, credential-locked rotation run.

Manual admission uses the same state contract. Firebase authenticates the operator. The web gateway verifies the identity, mints the service token for the private API, and forwards the idempotency key. The API creates the run against the credential's published control version.

Manual run admission

Figure 2B. A manual request creates the same locked run through the authenticated web path.

The outbox path wakes Workflows. Workflows reloads the run and asks the API to start it. The start transaction acquires the lease, increments the fencing token, and moves the run to RUNNING.

Revision numbers reject stale writes. The lease names the current worker. The fence invalidates requests from every previous worker or resume cycle. Every stage must match all three.

Shared dispatch and lease acquisition

Figure 2C. Durable dispatch starts the run under a new lease and fencing token.

Fix the facts and bind the plan

Preflight reads the run, credential lock, inventory snapshot, immutable ControlVersion, and published probe versions. The Inventory Agent receives a redacted task containing credential metadata and the declared consumer bindings. It checks consumer coverage and flags declared consumer IDs that lack a matching stored binding.

The Preflight proof binds the current generation to the run. Plan can then begin. The Planner Agent returns a typed rotation proposal. Recovery branches already exist in the published ControlVersion. The coordinator materializes those branches as RecoveryPlan records, creates the rotation plan once, computes its digest, and binds that exact plan to the run.

Preflight validation and plan binding

Figure 3. Preflight binds current facts before Plan commits the rotation and recovery plans.

The plan selects a strategy implemented by deterministic code: multi-consumer, dual-slot, parallel, or immediate. The published control version bounds the observation window and recovery choices. The canonical state machine fixes the stages; the published control fixes allowed and protected tools, checks, and recovery branches. Published playbooks fix browser actions.

Create, store, and prove access

Create is where the run takes one of two execution paths. An API-managed provider receives typed calls through the MCP broker and a governed connector. A dashboard-only provider runs a published PlaybookVersion inside an isolated browser session. The Console Operator Agent loads one immutable step at a time, Gemini Computer Use proposes an action, and the browser worker validates and executes it through a separate signed request. Secure Capture moves any one-time value directly into Secret Manager while model visibility and recording are paused.

Both paths converge at Store. Each binds the replacement to one provider identity, one exact Secret Manager version, and durable evidence before deployment begins. Their provider-side proof differs at revocation: the API path proves that the old credential is rejected; the browser path proves the published revoke checkpoint.

Figures 4A–4C follow the API-managed path. Section 5 covers the plaintext boundary for both paths.

On the API-managed path, Create asks the MCP broker to execute provider.createCredential. Before calling the provider connector, the broker reloads the run and validates the live lease, fence, capability, credential, provider connection, and target resource.

Provider credential creation under broker admission

Figure 4A. The broker validates the current run and target before creating the replacement.

The provider returns its identifier and a one-time value. The broker transfers that transient SecretValue directly to the secret connector. The connector writes one immutable Secret Manager version, computes a SHA-256 fingerprint, and returns metadata. The broker clears the transient value before returning.

Direct secret transfer and transient-value clearance

Figure 4B. The replacement moves directly into Secret Manager before transient memory is cleared.

Store reads metadata for that exact version. It requires an enabled version and rejects connector results containing a plaintext field.

Exact secret-version validation

Figure 4C. Store confirms that the replacement version is enabled without retrieving plaintext.

Uumi then resolves each declared consumer's Cloud Run service account and exact secret binding.

Consumer identity and secret-binding discovery

Figure 4D. Declared consumer bindings resolve to service accounts and secret references.

The broker mints a short-lived token for each consumer identity and accesses the exact replacement version. It validates a non-empty result, clears the transient secret, clears the token, and returns only the access result.

Per-consumer exact-version access validation

Figure 4E. Each consumer proves access to the replacement version under its own identity.

Deploy and verify

Deploy creates a candidate Cloud Run revision whose SecretKeyRef points to the exact replacement version. A generation label binds the revision to this rotation. The active revision keeps all traffic; the candidate starts at zero. Uumi records both the candidate and rollback revision before traffic moves.

Zero-traffic candidate deployment

Figure 5A. The replacement is deployed at zero traffic while the rollback revision remains live.

For an API-managed provider, Verify checks four independent facts:

  1. The provider reports that the replacement exists.
  2. Secret Manager reports that the exact version is enabled.
  3. Authentication with the replacement succeeds.
  4. Cloud Run reports a ready candidate carrying the expected generation label.

The verifier clears its transient value and stores a VerificationReport. The coordinator receives evidence identifiers.

Provider, secret-version and candidate-runtime verification

Figure 5B. Verification spans the provider, exact secret version, replacement authentication, and candidate runtime.

Browser-managed runs apply the checks available to that provider type: Secret Manager state, candidate runtime readiness, and optional telemetry. The immutable browser checkpoint supplies the provider-side evidence.

Roll out and observe

Rollout advances through the traffic milestones fixed in the plan. At each milestone, the broker shifts every bound deployment, waits for the declared split to converge, and returns the result. The verifier runs the milestone's pinned probes. Traffic advances after every deployment and required probe passes.

Progressive traffic rollout

Figure 6A. Traffic advances through verified milestones before the replacement becomes active.

Observe evaluates the target generation against the published control version. It verifies readiness, checks target-generation health and error thresholds, looks for continued old-generation use, and runs the pinned HTTP probe when configured.

Generation-scoped observation probes

Figure 6B. Observe evaluates the generation and probes pinned by the control version.

Workflows renews the lease for the observation duration plus a fixed safety margin, waits for the bound window, and executes Observe. A passing report commits the Observe proof and advances the run to Approval. A failing report moves the run to Cleanup with its evidence attached.

Observation window and next-state commit

Figure 6C. The observation window ends in a fenced transition to Approval or Cleanup.

Approve and revoke

When the published controls require revocation approval, the coordinator evaluates protected steps one at a time. For the first step without a matching approval, it builds a request containing the ProtectedAction and pauses. The approval service persists that action with one Approval and one-time capability. A later protected step can produce another pause.

Per-action approval request and pause

Figure 7A. One protected step produces one approval request and one pause.

After the operator decision and capability consumption, the resume transaction checks the paused run and expected revision, acquires a new lease, increments the fence to F+1, returns the run to RUNNING, and writes run.resumed. Approval validation happens when the Approval stage runs again.

Fenced resume transaction

Figure 7B. Resume acquires a new lease and fence before Approval runs again.

The resume event follows the same outbox path used at run creation. Workflows reloads the new revision and fence.

Durable resume dispatch under the renewed fence

Figure 7C. The resume event restarts Workflows with the renewed fence.

On re-entry, the Approval stage reloads the approval and protected action. It confirms that the approval is valid and unexpired, that its one-time capability has been consumed, and that the action still matches the plan, evidence, generation, and current fence. The broker independently validates the approved decision, consumed capability, run, tool, and action digest before executing the call. When approval is disabled by policy, the stage records approval-not-required and advances.

Per-action approval validation and stage completion

Figure 7D. The Approval stage validates each protected action before Revoke.

For an API-managed provider, the first revocation action removes the old provider credential. The verifier then proves that the replacement still authenticates, the old provider identifier is absent, and authentication with the old credential is rejected while its Secret Manager version remains enabled. The next action disables that exact old version. Policy can protect either action with the approval flow above. Uumi then marks the old generation REVOKED and the linked incident CONTAINED.

API revocation through old-credential rejection and containment

Figure 7E. API revocation proves replacement success and old-credential rejection before containment.

Providers without a suitable credential API use an isolated browser. The Console Operator Agent loads the published revoke step and checkpoint. For a protected step, the browser worker receives the consumed approval capability. It gives Gemini Computer Use a safe frame and bounded objective, validates the proposed action, executes it, and returns declared outputs.

Browser-managed revocation execution

Figure 7F. Browser-managed revocation executes one approved immutable step.

The browser path proves the immutable revoke step and expected checkpoint, validates the available Secret Manager and runtime evidence, then disables the old secret version.

Browser-managed verification, secret disablement and containment

Figure 7G. Browser checkpoint evidence precedes old-version disablement and containment.

Complete validates the active replacement, revoked predecessor, required reports, audit chain, and linked incident. The final Firestore transaction records COMPLETED and releases the credential lock. Section 7 shows the evidence and transaction path behind that commit.

4. Agent fleet and authority

Uumi deploys four agents with separate skills and separate identities:

Agent Responsibility Authority boundary
Inventory Agent Check declared consumer coverage and identify stale bindings Read-only, redacted task context
Planner Agent Select the rotation strategy and evaluate one recovery branch fixed during Plan Published controls and recovery branches only
Playbook Builder Agent Build versioned browser procedures from sanitized walkthrough evidence No credential values
Console Operator Agent Load immutable console steps and return ready, drift, or pause decisions No direct browser-action execution

The Playbook Builder runs before a browser-managed rotation. An operator can supply written instructions, upload an MP4, WebM, or MOV walkthrough, or reference an existing video in Cloud Storage. Uumi sanitizes written instructions immediately. For video, Video Intelligence extracts speech, on-screen text, and shot boundaries from an immutable Cloud Storage object generation.

Secret-like values are redacted before the Playbook Builder Agent receives the walkthrough analysis. The agent converts that evidence into a typed PlaybookDraft containing allowed domains, ordered steps, objectives, selectors, checkpoints, declared outputs, and Secure Capture fields. The API validates the contract, and the operator reviews the generated procedure before saving it as a draft. Publishing requires PLAYBOOK_APPROVE authority. Only a published, immutable PlaybookVersion attached to the provider connection can enter a rotation.

Each registration ties one agent kind and skill to an Agent Runtime deployment, Registry entry, Gateway, and managed identity. Routing succeeds only when exactly one ready registration matches.

Skill-bound routing across the agent fleet

Figure 8A. Skill-bound routing resolves one active deployment for each agent task.

Agents return decisions. The MCP broker executes mutations after reloading the run, requested connection, credential, consumer bindings, and controls from Firestore. It validates the capability claims against the requested tool and target. Approval-backed requests also require the live approval and exact protected-action binding.

Tool authority and broker-side validation

Figure 8B. The broker reloads and validates authority before every governed connector call.

Browser work adds two gates. browser.operate binds a proposal request to the session, rotation run, signed capability, nonce, and immutable playbook step.

browser.operate authority and playbook validation

Figure 8C. browser.operate admits a proposal for one bound session, run, and playbook step.

browser.execute uses a separate signed request and nonce. It consumes the exact pending proposal and detects whether the step is protected.

browser.execute authority and proposal consumption

Figure 8D. browser.execute consumes one pending proposal and detects protected steps.

For a protected step, the approval must be approved, consumed, unexpired, and bound to the same action, run, tool, and digest.

Protected browser approval validation

Figure 8E. Protected execution requires a live approval with the exact action binding.

The browser protocol is therefore two-phase. Gemini proposes an action from a safe frame and bounded objective. The worker validates the proposal. A separate signed request executes the accepted action.

Browser action proposal and guarded execution

Figure 8F. Browser actions pass proposal validation before separate signed execution.

The worker then reads the page again, validates the expected checkpoint, and extracts only the fields declared by the immutable step.

Completion checkpoint and declared-output extraction

Figure 8G. A completion response is accepted only after checkpoint and declared-output validation.

Registry and routing select the deployment. Agent Identity gives it a dedicated principal. Gateway and Model Armor inspect traffic. Agent Runtime carries run-bound context, while Memory Bank stores approved facts with provenance and expiry. Firestore remains the source of truth for execution state and approvals.

5. How we ensure the credential never enters the model

Gemini never receives a plaintext credential.

Plaintext exists only inside the broker's deterministic provider/secret path, the verifier, or Secure Capture, and only long enough to store or test it. It never appears in prompts, responses, Agent Runtime sessions, Memory Bank, browser replay, audit payloads, or stage evidence.

API-managed providers

The agent returns a typed plan. The coordinator sends a typed request. The provider connector returns the provider identifier and transient SecretValue to the broker. The broker transfers the value directly into Secret Manager. The coordinator receives the provider identifier, immutable secret-version reference, and fingerprint.

Plaintext isolation during API credential creation

Figure 9A. API-created plaintext travels from the provider connector to Secret Manager without entering agent context.

Browser-managed providers

A provider console may display a new credential once. Before reading that field, the worker arms Secure Capture. The session enters CAPTURING, pauses model visibility, and pauses recording. Secure Capture reads the declared field and provider identifier, writes the value into Secret Manager, computes the fingerprint, masks the field, verifies that the secret is absent from visible body text and input/textarea values, clears the clipboard, and zeroes mutable byte buffers.

The worker receives the provider ID, Secret Manager reference, fingerprint, masked-value digest, and capture timestamp. It then completes capture on the session. Gemini receives neither the credential nor a frame containing it.

Secure Capture transfer and sanitization

Figure 9B. Secure Capture pauses the model and recording while moving browser-disclosed plaintext into Secret Manager.

After cleanup, model visibility and recording resume for an automated session. Both remain paused while human takeover is active. The operator can inspect the live browser without turning the model or replay stream back on.

Session state after Secure Capture

Figure 9C. After cleanup, automated sessions resume model visibility; takeover sessions keep it paused.

Secret isolation is enforced by the data path. The plaintext has no route into model context.

6. What makes it agentic if revocation requires approval?

Revocation is irreversible, and policy can mark it as protected. When approval is required, the human answers one narrow question: may this exact action run now?

The agentic work spans the rest of the operation. The fleet:

  • checks declared consumers against stored inventory;
  • selects a rotation strategy and observation duration;
  • constructs provider-specific browser procedures;
  • returns ready, drift, or pause decisions for immutable console steps; and
  • evaluates the one recovery branch fixed during planning when execution fails.

Those decisions happen asynchronously across a long-running rotation. Agents can adapt within the published control version, immutable plan, current generation, and available recovery branches.

By the time Uumi asks for approval, the replacement is already created, stored, deployed, verified, rolled out, and observed. Figures 7A–7D show the gate: the operator decides each revocation action, approval supplies an expiring one-time capability, resume creates a new fence, and the coordinator and broker validate the action again before execution.

The human decides the protected action. Uumi coordinates every other stage autonomously within policy, and execution remains bounded to the current run, plan, generation, and capability.

7. How we manage retries, fencing, reconciliation, and compensation

Distributed systems fail between request and response. Credential creation is especially dangerous: the provider may commit the new credential while Uumi receives a timeout. An immediate retry can leave two live credentials and no reliable record of which one belongs to the run.

Every broker mutation gets a durable ToolAttempt containing its run, request, lease, reconciliation state, and any checkpoint or result. Before replay or reclaim, the broker reloads the current run and target context, then verifies the fence, capability, connection, and resource binding.

Reject stale work

The revision proves that a worker read the current run state. The lease grants ownership for a bounded time. The fencing token invalidates every request admitted before a newer owner or resume cycle. Coordinator state transitions carry the revision and fence. Broker requests carry the fence and are checked against the live run and lease. Protected actions and recovery operations are fence-bound.

A worker can finish after its lease expires. It still cannot commit through a stale fence. That check prevents a delayed process from overwriting work already owned by its successor.

Reconcile before retry

A replay reads the attempt first. A finished attempt returns its saved result. A checkpointed attempt finishes from the checkpoint. A live lease rejects another worker. An expired lease can be reclaimed.

ToolAttempt replay and expired-lease reclaim

Figure 10A. Replay returns durable progress or reclaims only an expired attempt.

Reclaim starts reconciliation. The broker checks the provider outcome, lists Secret Manager versions, and compares both systems with the baselines captured before the original call.

Reconciliation order after an expired mutation attempt

Figure 10B. Reconciliation compares provider and Secret Manager state with pre-mutation baselines.

Provider cleanup follows a strict attribution rule. Zero new candidates permits retry. One exact candidate is revoked before retry. Multiple candidates remain held for intervention.

Provider candidate cleanup

Figure 10C. Provider cleanup proceeds only when one orphan is exactly attributable.

Secret Manager uses the same rule independently. Zero new enabled versions permits retry. One exact candidate is disabled. Multiple candidates remain unresolved.

Secret-version candidate cleanup

Figure 10D. Secret-version cleanup applies the same single-candidate rule.

After reconciliation, the broker can retry creation. It stores the returned value, clears the transient buffer, checkpoints the result, and then finishes the attempt.

Safe retry and direct secret transfer

Figure 10E. Retry stores one replacement, checkpoints the result, and then finishes the attempt.

Durable attempt state, before-and-after comparison, and attributable cleanup provide safety even when the provider offers no idempotency key.

Fix recovery before failure

Deploy, Verify, Rollout, and Observe can fail after changing runtime state. Uumi therefore materializes their recovery branches during Plan, before any of those mutations begin.

The recovery contract requires a new lease and fence, the bound plan digest, the failed-stage binding, and an eligible branch selected during Plan.

Recovery admission and branch eligibility

Figure 10F. Recovery can select only a branch bound during Plan.

Each RecoveryPlan fixes its actions and records whether any action requires protection. A RecoveryResult carries the resulting evidence. The recovery-outcome transaction validates the active recovery and current fence before it can change run state.

Retry mode is valid only for a retryable prior failure and returns the run to RUNNING. A completed rollback, rollforward, cleanup, or escalation outcome produces the terminal COMPENSATED status, clears the lease, and releases the credential lock.

Fenced recovery outcome commit

Figure 10G. Recovery commits retry or compensation as a fenced outcome.

Compensation has its own terminal status. Operators can distinguish a completed rotation from an authorized rollback or containment action.

Successful completion uses the same evidence, fencing, and transactional guarantees.

Commit evidence and release the lock

Every successful stage stores evidence first. The evidence store returns an ID, digest, and object generation. The coordinator evaluates the stage's GatePolicy and sends the proof to Workflows. The API repeats the control-version and gate checks inside the fenced Firestore transaction that advances the run.

Stage proof becomes durable run state

Figure 11A. A stage advances only after its evidence is stored and revalidated inside the state transaction.

Audit events are redacted before persistence. One Firestore transaction reads the organization's chain head, assigns the next sequence, computes the event hash, writes the event and delivery outbox, and updates the chain head.

Hash-chain append and delivery enqueue

Figure 11B. Redacted audit events append to the organization hash chain in one transaction.

The audit worker claims outbox records in organization order and writes them to Cloud Logging with the event hash as insertId. A retryable failure releases the lease and schedules another attempt. Exhausted attempts enter the dead-letter path and remain visible.

Ordered audit delivery and retry handling

Figure 11C. Audit delivery preserves per-organization order across retries and dead-lettering.

Completion requires an active target generation, a revoked old generation, and a valid audit chain. The coordinator stores the audit snapshot used by the final proof.

Completion preconditions and audit evidence

Figure 11D. Completion requires an active replacement, a revoked predecessor, and a valid audit chain.

The coordinator terminates any remaining browser runtime, validates the latest target-generation Verify report and old-generation Revoke report, and moves a linked incident from CONTAINED to RESOLVED.

Runtime termination, report validation and incident resolution

Figure 11E. Finalization terminates the browser runtime, validates reports, and resolves the linked incident.

The final transaction validates the completion gate, applies COMPLETED with the lease cleared, writes the run, final RunStep, and run.completed outbox record, deletes the credential lock, and commits. The API returns the completed run after that commit.

Final completion commit and lock release

Figure 11F. Run completion, the final outbox event, and credential-lock release commit atomically.

Where to go from here

Uumi's core rotation path is now in place. Next, we are expanding coverage: more provider connectors, runtime types, inventory adapters, probe libraries, and policy packs, all under the same credential, authority, evidence, and recovery contracts.

We also want operators to read the proof faster through clearer incident timelines, richer verifier reports, stronger conformance tests, and production-scale fault injection.

Uumi can take a credential from exposure to verified completion while keeping plaintext out of model context and restricting every mutation to explicit authority. A completed run means the replacement is active, every declared consumer has been verified, the predecessor generation is revoked, its old secret version is disabled, and the evidence is durable.

Top comments (0)