TL;DR: A setup job should mint a credential restricted to one customer-usage writer, place it in a secret store without printing it, authenticate with the stored value, and fail unless the returned subject, tenant, and scopes exactly match the request. For a healthtech meter, that identity check matters more than a green “secret written” response: the invoice pipeline must be able to show which workload was allowed to append usage for which customer account, while the credential itself stays out of logs and build artifacts.
Treat bootstrap as a small security transaction with an explicit commit point. Provisioning is not complete when a key exists. It is complete only after the key has crossed into the secret store, the setup process has discarded its plaintext copy, and an authenticated probe proves that the stored value represents the intended machine identity. This framing also forces failure handling into the design: before the commit point, any partial key is untrusted and must be removed; after it, rotation becomes an ordinary lifecycle operation with a known owner and version.
How should a setup script provision and verify a scoped API key?
Imagine the incident review, without inventing an incident: a healthtech platform receives one usage event for customer clinic-042, the event later contributes to a metered invoice, and the audit trail says only that “CI” wrote it. The dashboard is green. The useful questions are less comfortable: which subject authenticated, which tenant boundary applied, which scopes were effective, and did the same credential have permission to read patient data or alter invoices?
If those answers cannot be recovered without exposing the key, the bootstrap design has already failed its primary decision axis: auditability of access. The invariant is straightforward: a credential used to append billable usage must resolve to one stable workload subject, one expected account boundary, and the smallest effective scope set. An environment name such as production is context, not identity.
No identity, no commit.
I would page on repeated identity-verification failures, unexpected subjects, or broader scopes. I would not page merely because a rotation occurred. A rotation is expected state change; an accepted credential that cannot prove its owner is the event that threatens attribution.
Model the handoff before choosing an API
There is no universal API-key provisioning protocol, so keep vendor routes outside the safety logic. The core needs three adapters: a provisioner that returns a secret once, a store that accepts bytes under a versioned name, and an identity endpoint that reports the authenticated principal. That separation lets a team replace an account API or secret backend without weakening the checks that decide whether bootstrap succeeded.
For the example meter, request only an append permission for a single tenant. Do not grant a generic admin scope because setup is convenient. Do not reuse the deployment credential either; deployment and usage submission produce different audit questions and deserve different subjects.
| Boundary | Required evidence | Reject when |
|---|---|---|
| Provisioning | requested subject, tenant, and scope set | the request is ambiguous or overbroad |
| Secret write | versioned secret reference, not plaintext | the destination is mutable without version evidence |
| Authentication | returned subject, tenant, and effective scopes | any field differs from the request |
| Audit | request ID and secret version | the record contains the key or its hash |
Do not log a key fingerprint as a consolation prize. A stable fingerprint becomes another correlator, is unnecessary for the decision above, and can encourage teams to treat possession evidence as identity evidence. Log the non-secret subject, tenant, scope set, secret version, and request ID instead. OWASP's secrets guidance recommends centralized storage, least privilege, rotation, expiration where possible, and care to prevent secrets from reaching logs.
That is enough evidence.
Make the verification path executable
The following program is deliberately adapter-driven. Its in-memory implementations make the example runnable with go run, while production adapters would call an account control plane and a secret manager over authenticated, encrypted connections. The safety function does not know their URLs.
package main
import (
"context"
"crypto/rand"
"encoding/base64"
"errors"
"fmt"
"slices"
)
type Request struct {
Subject string
Tenant string
Scopes []string
}
type Identity struct {
Subject string
Tenant string
Scopes []string
}
type Provisioner interface {
Create(ctx context.Context, request Request) ([]byte, error)
Revoke(ctx context.Context, credential []byte) error
}
type SecretStore interface {
Put(ctx context.Context, name string, value []byte) (version string, err error)
Get(ctx context.Context, name, version string) ([]byte, error)
Delete(ctx context.Context, name, version string) error
}
type Authenticator interface {
WhoAmI(ctx context.Context, credential []byte) (Identity, error)
}
func bootstrap(ctx context.Context, request Request, name string,
p Provisioner, store SecretStore, auth Authenticator) (string, error) {
credential, err := p.Create(ctx, request)
if err != nil {
return "", fmt.Errorf("provision credential: %w", err)
}
committed := false
defer func() {
if !committed {
_ = p.Revoke(ctx, credential)
}
clear(credential)
}()
version, err := store.Put(ctx, name, credential)
if err != nil {
return "", fmt.Errorf("store credential: %w", err)
}
stored, err := store.Get(ctx, name, version)
if err != nil {
_ = store.Delete(ctx, name, version)
return "", fmt.Errorf("read stored version: %w", err)
}
defer clear(stored)
identity, err := auth.WhoAmI(ctx, stored)
if err != nil {
_ = store.Delete(ctx, name, version)
return "", fmt.Errorf("authenticate stored credential: %w", err)
}
wantScopes := append([]string(nil), request.Scopes...)
gotScopes := append([]string(nil), identity.Scopes...)
slices.Sort(wantScopes)
slices.Sort(gotScopes)
if identity.Subject != request.Subject || identity.Tenant != request.Tenant ||
!slices.Equal(gotScopes, wantScopes) {
_ = store.Delete(ctx, name, version)
return "", errors.New("verified identity does not match requested boundary")
}
committed = true
return version, nil
}
type memoryBackend struct {
value []byte
request Request
version string
}
func (m *memoryBackend) Create(_ context.Context, request Request) ([]byte, error) {
raw := make([]byte, 32)
if _, err := rand.Read(raw); err != nil {
return nil, err
}
m.request = request
return []byte(base64.RawURLEncoding.EncodeToString(raw)), nil
}
func (m *memoryBackend) Revoke(_ context.Context, _ []byte) error { return nil }
func (m *memoryBackend) Put(_ context.Context, _ string, value []byte) (string, error) {
m.value = append([]byte(nil), value...)
m.version = "1"
return m.version, nil
}
func (m *memoryBackend) Get(_ context.Context, _, version string) ([]byte, error) {
if version != m.version {
return nil, errors.New("unknown version")
}
return append([]byte(nil), m.value...), nil
}
func (m *memoryBackend) Delete(_ context.Context, _, _ string) error {
clear(m.value)
m.value = nil
return nil
}
func (m *memoryBackend) WhoAmI(_ context.Context, credential []byte) (Identity, error) {
if !slices.Equal(credential, m.value) {
return Identity{}, errors.New("unauthorized")
}
return Identity(m.request), nil
}
func main() {
backend := &memoryBackend{}
request := Request{
Subject: "ci-usage-writer",
Tenant: "clinic-042",
Scopes: []string{"usage:append"},
}
version, err := bootstrap(context.Background(), request,
"healthtech/clinic-042/usage-writer", backend, backend, backend)
if err != nil {
panic(err)
}
fmt.Printf("bootstrap verified: subject=%s tenant=%s secret_version=%s\n",
request.Subject, request.Tenant, version)
}
The compensating path deserves scrutiny. If storage or verification fails, the program deletes the untrusted stored version and attempts revocation. Production code should use a short, separate cleanup context so cancellation of the main setup context does not prevent cleanup; it should also emit a non-secret request ID for reconciliation. The example keeps one context to expose the transaction shape without pretending that generic adapters share retry semantics.
This sample requires Go 1.21 because it uses clear and the slices package.
One subtle limit remains: Go's clear reduces accidental retention in these byte slices, but it cannot promise that no copy exists elsewhere in a runtime, transport, or backend. The stronger controls are architectural: never place the returned value in command arguments, environment dumps, formatted errors, test snapshots, or shell tracing; restrict the setup runner; and make the secret short-lived enough that rotation is routine.
Test the identity mismatch, not only the happy path
A useful test changes the returned identity after the secret write. Make the fake authenticator return clinic-024, add usage:read, or replace ci-usage-writer with a human subject. Each mutation must make bootstrap fail and trigger deletion plus revocation. Also test a store that acknowledges Put but cannot read the exact version back; “latest” is a race-prone reference when two setup jobs overlap.
Test logs too. Feed a recognizable canary credential through every error path, capture process output, and fail the test if the canary appears. This is where a bland success metric stops helping. The check should distinguish these states: provisioning never happened, provisioned but not stored, stored but not authenticated, identity mismatch, cleanup incomplete, and verified.
Retries require idempotency at the adapter boundary. If the provisioning service accepts an idempotency token, derive a new opaque token for one bootstrap attempt and persist its non-secret request ID. If it does not, reconcile by requested subject and tenant before creating another credential, then revoke superseded credentials after a verified replacement exists. Never infer success from an earlier process exit code alone.
Know when a static key is the wrong primitive
This pattern applies when the target service accepts an API key and the CI system must bootstrap it. It does not make a long-lived bearer secret ideal. When both ends support workload federation or another short-lived, audience-bound mechanism, prefer that design: the pipeline exchanges an existing workload identity for temporary access instead of storing a reusable credential. OAuth 2.0 defines bearer-token handling, and RFC 8707 defines resource indicators that can restrict where an access token is intended to be used; neither specification turns an arbitrary API key into a federated identity.
The decision is operational. Static bootstrap is defensible when its scope is narrow, its owner is machine-specific, rotation and revocation are tested, reads from the secret store are auditable, and the authenticated service can report effective identity. The limitation of this pattern is the reusable bearer secret: it is a poor fit when the service cannot expose identity, when all keys inherit account-wide permissions, or when a CI runner cannot keep secret output out of logs. The trade-off is extra setup and rotation work in exchange for compatibility with services that do not accept workload federation; that reason disappears as soon as both sides can exchange a workload identity for short-lived, audience-bound access.
Prefer federation then.
Before enabling the usage writer, run one synthetic event against a non-billable test tenant, verify the authenticated subject in the access record, and confirm that a forbidden operation is actually denied. Then alert on identity drift and cleanup failures, not on dashboard color. The final artifact is evidence of a bounded principal, not evidence that a string was copied successfully.
Top comments (0)