Set a per-tenant spend ceiling before issuing a scoped credential, and make refusal an explicit, observable outcome when that ceiling is reached. In a healthtech monorepo, this is safer than tying one long-lived key to a developer or allowing every project to spend from an unbounded pool.
Short answer: model each credential as a project-and-tenant grant, store its owner and expiry separately from the secret, rotate by overlap rather than replacement, and reject new work with a typed response when the tenant budget is exhausted. That gives the platform team a bounded blast radius without making a staff change an emergency.
The important distinction is traffic that is refused on purpose versus traffic that disappears because a key was revoked accidentally. Your dashboards, runbooks, and client behavior need to tell those cases apart.
The failure signal: ownership changes become traffic incidents
A monorepo makes shared code easy to copy and shared credentials easy to misunderstand. A developer creates a key for a project, another team imports it into a worker, and six months later nobody can say which tenant is paying for a request or who is allowed to revoke it. In healthtech, that ambiguity is more than an accounting nuisance: an emergency rotation can interrupt claims processing or device ingestion while an old credential remains usable in a forgotten job.
I plan for two budgets at once. The first is the tenant's spend ceiling. The second is the amount of refused traffic the product can tolerate before an SLO is missed. A hard ceiling that returns a clear 429 or 403 may protect the invoice while breaking a clinical workflow; a soft ceiling that keeps accepting work may preserve availability while violating a contract. The decision belongs in the service-level objective, not in a hidden retry loop.
The signal I want is a tuple, not a single counter: tenant, project, credential ID, owner group, decision (allowed or refused), reason, and policy version. Keep the secret out of it. A credential ID can be logged and traced; the credential value cannot.
One short rule: no owner, no key.
How should a healthtech monorepo group API credentials by project and tenant?
Use a small record that separates identity, authorization, and material. The project is the deployment boundary; the tenant is the data boundary; the owner group is the accountability boundary. A developer can request a key, but the key belongs to the project and tenant, with an owner group that survives that developer's departure.
A practical record has these fields:
| Field | Purpose | Rotation behavior |
|---|---|---|
| credential_id | Stable audit reference | Never reused |
| project_id | Monorepo service boundary | Revoked with the project grant |
| tenant_id | Data and billing boundary | Checked on every authorization decision |
| owner_group | Human accountability | Updated without changing the secret |
| key_hash | Verification material | Replaced during overlap rotation |
| expires_at | Maximum lifetime | Enforced even if nobody remembers cleanup |
| spend_policy_id | Ceiling and refusal rule | Versioned for audit |
The API should issue a secret once, return the identifier thereafter, and make revocation idempotent. Do not use the developer's email as the primary owner: aliases change, contractors leave, and a person is not a durable service boundary. A group or service account can own the grant while the audit log records who requested each change.
In Go, the command path can remain boring. That is a feature.
package credentials
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"time"
)
type Grant struct {
ID string
ProjectID string
TenantID string
OwnerGroup string
KeyHash string
SpendPolicyID string
ExpiresAt time.Time
RevokedAt *time.Time
}
type Store interface {
Create(ctx context.Context, grant Grant) (secret string, err error)
Revoke(ctx context.Context, id, actor string) error
}
func HashSecret(secret string) string {
sum := sha256.Sum256([]byte(secret))
return hex.EncodeToString(sum[:])
}
func Revoke(ctx context.Context, store Store, id, actor string) error {
if id == "" || actor == "" {
return fmt.Errorf("credential id and actor are required")
}
return store.Revoke(ctx, id, actor)
}
The storage interface hides whether the backing system is self-hosted or managed. What matters is that the authorization check can read the project, tenant, expiry, and current policy in one consistent transaction. A cache may reduce latency, but it must have a bounded staleness window that fits the revocation SLO.
Rotation without a refused-traffic surprise
Rotation is a two-key interval, not a delete-and-recreate event. Create the replacement, distribute it to the workload, observe successful authentication, then revoke the previous key. Keep the overlap shorter than the maximum token or job lifetime. If a worker can run for 20 minutes, a five-minute overlap is fiction.
The rollout sequence should be auditable:
- Authorize the owner group to rotate the project-and-tenant grant.
- Mint the replacement and record its activation time.
- Deliver it through the workload's secret channel, never through a pull request.
- Watch authentication failures and refused-traffic rates by credential ID.
- Revoke the old identifier after the observed overlap.
I once assumed a single revoke call was the whole operation; the hard part was the queue. A batch consumer had already leased work under the old key, so revoking immediately caused retries to pile up and turned a security action into a latency incident. The correction was to make the consumer checkpoint before rotation and to cap its retry window at 90 seconds. Your mileage may vary, but the lease duration is a number you can measure rather than guess.
Refusal needs a stable contract. Return 403 when the credential is invalid or outside its project and tenant scope; return 429 when the spend policy deliberately refuses new work; include a machine-readable reason such as tenant_budget_exceeded; and do not ask clients to retry a permanent 403. A 429 can be retried only when the response includes a bounded retry policy and the operation is idempotent.
Spend ceiling versus refused traffic: what should the SLO measure?
A budget policy is incomplete until it names the failure it prefers. Track at least four time series: accepted requests, refused requests by reason, authentication failures, and estimated spend by tenant. Add a fifth series for rotation age so an old credential cannot remain quietly valid. Alert on the ratio of refused requests and on a sudden rise in authentication failures after a rotation event.
The decision table belongs in the runbook:
| Situation | Default action | Why |
|---|---|---|
| Ceiling reached, non-critical batch | Refuse with 429 | Protects spend; work can be rescheduled |
| Ceiling reached, clinical-path request | Allow only if an approved emergency budget exists | Preserves the SLO with an explicit cost boundary |
| Unknown project or tenant scope | Refuse with 403 | Prevents cross-tenant access |
| Credential expired | Refuse with 403 and page only on unexpected volume | Expiry is a policy result, not an outage |
| Rotation failure before activation | Keep old key until the rollback checkpoint | Avoids an avoidable traffic cliff |
The catch is that a hard ceiling is not suitable when the business has no safe degraded mode. In that case, keep the ceiling as an alert and require an on-call approval path for a narrowly scoped emergency allowance. Stick with a strict refusal when replaying or delaying the operation is safer than spending beyond the contract.
Do not hide refusal behind generic 500 responses. That turns a controlled policy decision into an incident, causes clients to retry aggressively, and makes the spend ceiling less effective precisely when it is needed.
Verification, rollback, and the limits of this pattern
Before rollout, run a matrix that covers project mismatch, tenant mismatch, expired keys, repeated revoke calls, concurrent rotation, and a budget crossing between authorization and execution. Test the audit trail as an artifact: an auditor should be able to reconstruct who issued, used, rotated, and revoked a credential without seeing its value.
For rollback, preserve the old key only until the checkpoint that proves the replacement is active, then revoke it. If the replacement cannot be delivered, cancel the rotation and leave the old key subject to its original expiry. Never restore a revoked secret from a log or database backup.
This design does not solve every identity problem. It is not suitable when a single request legitimately spans many tenants and cannot carry a trustworthy tenant context; use a separate delegated authorization model there. It also does not replace workload identity, network controls, or data-layer authorization. The key is one boundary in a layered system.
The platform decision is therefore modest: group credentials by durable project and tenant ownership, rotate with overlap, and make spend refusal visible. That combination keeps a developer departure from becoming a credential fire drill while preserving a measurable choice between cost and traffic.
Top comments (1)
The distinction you draw between refused-on-purpose traffic and traffic that vanished because a key was revoked is the part most credential rotation runbooks skip. We got bitten by exactly your queue case: a batch consumer had leased work under the old key, revocation turned a security action into a latency incident, and the retries looked like an attack in the dashboard.
Checkpoint-before-rotate plus a capped retry window is the right shape. Do you store the ceiling as a property of the credential or of the tenant? We tie it to the credential and it makes overlap rotation awkward when both keys spend from the same pool.