Node.js Monthly Usage Statements — 2-Phase PDF Email Credential Rotation

A page fires at 00:14 UTC: statement_delivery_failures has crossed its threshold, 37 customer-support accounts have no delivery receipt, and every failed job names the same credential version. The responder needs to know which billing period was frozen, which artifact was rendered, which key version authorized each send, and whether retrying can create a second email.

TL;DR: generate each monthly usage statement from an immutable, tenant-scoped snapshot; render the PDF once; and send it through an idempotent delivery record that stores the credential version, attempt history, and provider receipt. Rotate the production email API key in two phases: make the new version available before switching new sends, retain the old version while in-flight work drains, then revoke it only after evidence shows no eligible job still depends on it. The Node.js scheduler may initiate this workflow, but it must not own time, rendering, email, and secret state in one callback.

This is an auditability problem before it is a scheduling problem. A cron expression can start work. It cannot prove that account support-1042 received exactly the PDF derived from its closed billing window, or explain why the send crossed a credential rotation boundary.

What should the on-call see first?

The useful page is an exception set, not a generic "billing failed" count. For each affected account, it should expose a non-secret statement ID, billing-window identifier, artifact digest, delivery state, attempt number, credential version label, and the last classified error. Never emit the API key itself. OWASP's secrets-management guidance calls for lifecycle controls, auditing, rotation, and careful handling of secrets in logs; the operational record should therefore identify a version without containing secret material.

One row might say that stmt_support_1042_2026_08 rendered successfully, attempt 2 used mail-key-v18, and no provider receipt was recorded. That is enough to decide whether an idempotent retry is appropriate. It is also narrow enough that a support engineer can investigate an account without gaining access to a production credential.

The alert should link to the failed cohort and the rotation event. It should not invite a blind replay of the whole month.

Work backwards from that page. The earlier signal was not “37 emails failed”; by then customer impact had accumulated. It was a growing age of the oldest ready delivery while the completion rate fell below the arrival rate, split by credential version. A single global success percentage would hide a small tenant cohort, while an alert per customer would turn ordinary variance into on-call noise. Page on sustained risk to the delivery SLO; retain tenant-level dimensions for diagnosis.

How should Node.js generate and email each monthly usage statement PDF?

The monthly boundary should create a durable statement identity from the customer account and billing period. An atomic operation freezes the usage inputs and records the statement. Rendering consumes that record and stores a content digest alongside the PDF location. Delivery refers to the statement and artifact; it does not recalculate usage during a retry.

That distinction matters during key rotation. If a send fails after the provider accepted it but before the worker records the receipt, regenerating the PDF and sending again changes two variables at once. An idempotency key derived from the stable delivery ID gives the downstream system a chance to recognize a replay, while the local state machine preserves uncertainty instead of calling it success or failure prematurely.

A compact Go model makes the boundaries visible even when the scheduler itself runs in Node.js:

package billing

import "time"

type DeliveryState string

const (
    Ready DeliveryState = "ready"
    Sending DeliveryState = "sending"
    Delivered DeliveryState = "delivered"
    Uncertain DeliveryState = "uncertain"
)

type Delivery struct {
    ID string
    AccountID string
    StatementID string
    ArtifactSHA256 string
    State DeliveryState
    Attempt int
    CredentialVersion string
    ProviderReceipt string
    LeaseUntil time.Time
}

Uncertain is deliberate. A timeout does not prove rejection. Reconcile that state through a receipt lookup or controlled retry rather than collapsing every ambiguous network result into Ready.

Ambiguity survives.

How can a key rotate while workers are sending?

Use a two-phase handoff with explicit evidence. First, publish a new secret version to the workload identity allowed to send statements, verify that workers can resolve it, and leave the old version valid. Second, atomically change the active-version pointer used when workers claim new deliveries. Existing attempts keep the version they already recorded; newly claimed work uses the new one.

Do not revoke on a timer alone. The retirement check must establish that no active lease, retry, or unresolved attempt names the old version, and that the observation window covers the maximum worker lease plus the permitted retry delay. Those values are local policy, so they belong in configuration and capacity planning rather than copied sample numbers.

The claim operation should bind work to a credential version without exposing the credential to the database:

func Claim(d *Delivery, activeVersion string, now time.Time, lease time.Duration) error {
    if d.State != Ready || d.LeaseUntil.After(now) {
        return ErrNotClaimable
    }

    d.State = Sending
    d.Attempt++
    d.CredentialVersion = activeVersion
    d.LeaseUntil = now.Add(lease)
    return nil
}

The worker resolves activeVersion through its authorized secret channel only after the claim commits. That ordering creates an audit trail: the database says which version the attempt intended to use, the secrets audit says which workload accessed that version, and the delivery record later captures the receipt. Access should be least-privilege and attributable to a workload identity, not a shared operator account.

Rollback is symmetrical. If the new version cannot complete a canary cohort, stop new claims, point future claims back to the still-valid old version, and investigate. Already completed deliveries remain completed. Already claimed deliveries preserve their recorded version. Overlap is capacity: without it, rollback becomes another emergency rotation.

Instrument the transition, then set the page

Add metrics at state transitions rather than around the scheduler callback. Count claims, completions, uncertain outcomes, and permanent rejections by credential version and non-secret error class. Measure ready-queue age and end-to-end delivery latency. Record rotation events in an append-only audit stream with actor, workload, old version label, new version label, and timestamp; keep secret values out of labels, traces, and exception text.

The SLO should describe the customer-visible result, such as eligible monthly statements delivered within the defined billing window, with exclusions written before an incident. Scheduler execution is a dependency indicator, not the result. A healthy cron invocation can enqueue nothing, duplicate a month, or feed a stalled renderer.

Capacity planning starts with the first-day burst. If 120,000 accounts become eligible and the delivery window is 6 hours, the system must sustain at least 5.56 completed statements per second before retries, canaries, rendering variance, and downstream throttling are added. Those figures are an example, not a benchmark. Size each stage independently, cap concurrency at the email boundary, and reserve headroom to drain retries without starving new work.

A rotation warning can fire when old-version claims continue after the active pointer changes. A page should wait for evidence that the delivery objective is threatened: queue age consuming its budget, sustained completion deficit, or uncertain outcomes accumulating faster than reconciliation can clear them. The exact threshold must come from the delivery window, observed variance, and on-call response time.

Buy or build the control plane?

The hard choice is not a feature checklist. It is where audit evidence, failure recovery, and on-call ownership live.

Boundary	Managed service	Self-hosted component	Decision test
Secret lifecycle	Less maintenance; audit retention and identity semantics follow the service contract	More control; the team owns availability, upgrades, and audit durability	Can one attempt be tied to one workload, one version, and one rotation event?
Queue and schedule	Operational load shifts outward; redrive behavior may constrain recovery	Retry policy stays controlled; paging and capacity stay too	Can the monthly burst drain inside the SLO during a slowdown?
PDF rendering	Isolation and patching may be delegated; data boundaries widen	Fonts and runtime are controlled; sandboxing and patching remain local	Can the artifact be reproduced and its digest retained without leaking tenant data?
Email delivery	Receipts may be available; lock-in appears in event semantics	Protocol control increases; reputation and abuse handling become team duties	Can an ambiguous send be reconciled without duplicate mail?

Reject any design, managed or self-hosted, that cannot export the evidence needed to reconstruct one delivery. A bespoke secrets system also needs unusually strong justification: encryption is a small slice of lifecycle management, while access control, rotation, revocation, backup, and audit availability remain on the platform team's pager.

Cost belongs in the decision, but invoice price is not the dominant term. Model operator time, retained audit volume, peak capacity, incident recovery, and migration effort. Lock-in is tolerable only when the boundary is explicit and receipts, statement metadata, and credential-version history can be retained in a portable record.

This design has real limitations. A two-phase rotation is unsuitable for a low-volume internal report where delayed manual delivery is acceptable and no external email credential exists; its leases, reconciliation state, and audit retention impose operational work. At the other extreme, a team that cannot staff secret-store availability, revocation testing, queue recovery, and mail-abuse handling should choose managed boundaries for those duties, provided the service contract exposes the version-level evidence the SLO requires. The trade-off is less direct control over event semantics and retention, not an automatic loss of reliability.

No boundary is free.

The threshold can become its own incident

An alert at the first failed send will page on transient noise. An alert only after the delivery window expires arrives too late. Both thresholds are easy to explain and poor to operate.

Start from the error budget and the time needed to diagnose, roll back a credential pointer, and drain queued work. Then test the alert with a staged rotation: delay a canary cohort, inject an ambiguous response, and verify that the old version cannot be revoked while a lease still names it. The exercise should also prove that an operator lacking secret-read permission can reconstruct the attempt from audit data.

False positives have a measurable cost. They train responders to distrust the page, encourage broad replay actions, and spend the attention needed for real customer impact. Review alerts that did not require action, but do not silence the underlying metric; move noisy conditions to a ticket or dashboard, and reserve paging for a credible threat to the statement-delivery SLO.

The retirement gate is plain: the new version is serving new claims, old-version in-flight and uncertain counts are zero for the required observation window, audit events are durable, and rollback evidence has been captured. Only then revoke the old key.

Further reading

• OWASP Secrets Management Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html