DEV Community: ThomasMoore157

Tenant Offboarding Sequence Explained — Revoke Credentials Before Deleting Data

ThomasMoore157 — Sun, 13 Sep 2026 01:22:25 +0000

When a healthtech tenant leaves, revoke the tenant's API credentials before making its data unreachable, then complete deletion from a separately authorized worker. That ordering limits the blast radius if an old integration keeps retrying while the offboarding job is still running; it also gives the audit trail a clean security boundary.

Short answer: disable and revoke every credential first, record the event, quarantine the tenant, and only then run the deletion workflow. Keep the deletion token and the runtime token separate, and make both operations idempotent.

The page that wakes the on-call

The alert usually arrives late. A queue-depth alarm fires because an integration is still sending events for a tenant that should have been retired. The on-call sees repeated authentication failures, a growing dead-letter queue, and a compliance ticket asking whether any protected health information was processed after the contract end.

The first useful question is not “did deletion finish?” It is “can any credential that belonged to this tenant still create work?” A status flag in the tenant table is not enough if workers validate a cached token, and deleting rows first can make it harder to prove which credential was active at each step.

Freeze first.

I keep the sequence explicit: freeze new work, revoke credentials at the issuer, write an immutable offboarding event, and place the tenant in a quarantine state. The deletion worker consumes that event only after it confirms the credential state. A retry receives the same tenant identifier and produces the same end state; it does not mint a second cleanup job with broader access. In a real incident review, this is the paragraph I would expand into a timeline: at 09:00 the contract boundary is recorded, at 09:01 the gateway rejects new events, at 09:02 the issuer marks the key unusable, and at 09:03 the broker drains messages already accepted. The exact minutes are illustrative, not a service promise; what matters is that each transition has a durable timestamp and a responsible actor. If the issuer is eventually consistent, the gateway-side quarantine closes that gap, while a worker that has already loaded a token must check tenant state again before writing anything. Without those two checks, a “successful” deletion can coexist with a late side effect and an audit trail that cannot explain it.

That is a small distinction with a large operational consequence. In a healthtech system, one shared key can turn a single tenant departure into a cross-tenant incident, so credential scope and ownership matter more than shaving a few minutes from the job.

How should a tenant offboarding sequence revoke an API key before deletion?

The sequence should be a state machine, not a script that calls “delete” and hopes the network agrees. A useful set of states is active, frozen, credentials_revoked, quarantined, deletion_running, and deleted. Each transition records actor, timestamp, request id, and the credential version it observed.

The revoke step must cover machine credentials, webhook signing secrets, refresh tokens, and any service-account grants issued for that tenant. OWASP recommends a managed secrets lifecycle with rotation, revocation, least privilege, and auditable access; those controls are relevant here because offboarding is a secrets-lifecycle event, not just a row cleanup.

The deletion step should use a different narrowly scoped identity. It reads the tenant's deletion manifest, removes data from primary stores and indexes, and schedules provider-specific erasure where immediate removal is not possible. Backups need a documented retention and restore policy: a database delete does not magically rewrite an immutable backup, and pretending otherwise creates a misleading compliance record.

Here is the shape I want reviewers to see in code. The concrete storage and identity providers can change; the ordering cannot.

package offboarding

import (
    "context"
    "fmt"
)

type Credentials interface {
    Revoke(ctx context.Context, tenantID string) error
    IsRevoked(ctx context.Context, tenantID string) (bool, error)
}

type Tenants interface {
    Freeze(ctx context.Context, tenantID string) error
    Quarantine(ctx context.Context, tenantID string) error
}

type Deleter interface {
    DeleteManifest(ctx context.Context, tenantID string) error
}

func Offboard(ctx context.Context, tenantID string, creds Credentials, tenants Tenants, deleter Deleter) error {
    if err := tenants.Freeze(ctx, tenantID); err != nil {
        return fmt.Errorf("freeze tenant: %w", err)
    }
    if err := creds.Revoke(ctx, tenantID); err != nil {
        return fmt.Errorf("revoke credentials: %w", err)
    }
    revoked, err := creds.IsRevoked(ctx, tenantID)
    if err != nil {
        return fmt.Errorf("verify revocation: %w", err)
    }
    if !revoked {
        return fmt.Errorf("credentials are not revoked")
    }
    if err := tenants.Quarantine(ctx, tenantID); err != nil {
        return fmt.Errorf("quarantine tenant: %w", err)
    }
    if err := deleter.DeleteManifest(ctx, tenantID); err != nil {
        return fmt.Errorf("delete tenant data: %w", err)
    }
    return nil
}

The verification call is deliberate. A successful revoke request is not the same thing as a verified security state, especially when token caches and asynchronous issuers are involved. I am not sure every identity provider exposes the same consistency guarantee, so the contract should state the maximum revocation delay and the compensating control, such as rejecting the tenant at the gateway while propagation completes.

Where does the failure boundary belong?

Treat the gateway, event broker, workers, and data stores as separate failure domains. The gateway should reject a frozen or quarantined tenant before publishing new events. Consumers should check tenant state again before performing side effects, because an event may have been accepted just before the freeze. That second check is cheap insurance against an in-flight message.

Observability needs to follow the state machine. Emit counters for revoke requests, verified revocations, rejected tenant events, deletion retries, and records held by legal policy. Alert on time spent between frozen and credentials_revoked, and on any post-revocation event that reaches a side-effecting consumer. A low error rate can still hide a dangerous delay if the metric only counts completed deletions.

Thresholds have a cost. Set them too low and routine propagation lag pages the on-call; set them too high and the system quietly accepts work after the contract boundary. Start with a measured service-level objective for revocation propagation, then tune the alert against that SLO rather than against an arbitrary minute count.

Buy, build, or keep the boundary small?

The decision is less about a fashionable platform and more about who owns the irreversible step.

Approach	Useful when	Trade-off
Identity-provider revocation plus an in-house state machine	You need a precise audit trail and tenant-specific policy	Your team owns retries, provider semantics, and deletion evidence
Managed lifecycle workflow	The provider offers documented revocation and retention controls	Policy may be less expressive, and portability can suffer
Self-hosted identity and storage controls	You need control over data residency and network boundaries	On-call load, patching, backup erasure, and recovery testing stay with you

The catch is that a managed workflow is not suitable when it cannot prove credential scope or expose a revocation timestamp. Stick with a simpler in-house boundary when the workflow must coordinate several stores, a legal hold, and a broker you already operate. Conversely, self-hosting is a poor fit for a small team that cannot exercise restore and key-rotation drills; the operational risk becomes part of the product.

A review checklist that survives an outage

Before shipping, test the ugly interleavings: revoke succeeds and quarantine times out; deletion retries after a process restart; a stale event arrives after quarantine; a legal hold blocks one dataset; and an operator repeats the command with the same request id. Verify that no path logs raw secrets, that deletion credentials cannot read application data, and that an audit export can connect each action to an actor.

Run the exercise during a controlled maintenance window, then inspect the evidence as if you were an incident responder. The useful artifact is not a green dashboard. It is a timeline showing when new work stopped, when credentials became unusable, what data classes were removed, and what remains under retention policy.

References

OWASP Secrets Management Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
NIST SP 800-57 Part 1 Revision 5, Recommendation for Key Management: https://csrc.nist.gov/pubs/sp/800/57/pt1/r5/final
NIST SP 800-92, Guide to Computer Security Log Management: https://csrc.nist.gov/pubs/sp/800/92/final

PDF Contract Review Pipelines: Fidelity, Latency, and Retention by Design

ThomasMoore157 — Fri, 11 Sep 2026 16:09:21 +0000

For a US or EU SaaS, choosing PDF endpoints for legal contract review is a bad place to hide a renderer behind a synchronous HTTP request. A page that looks acceptable in a browser can lose a signature block, a redaction overlay, or a footnote when it is converted under load. The safer design treats rendering as an explicit job with validation, a durable audit record, and a deletion deadline.

Short answer: use an asynchronous PDF job contract for review artifacts, measure fidelity and latency with real contracts, and keep source files private behind short-lived links; choose a direct specialist when its rendering controls are materially better than the operational simplicity of a unified API.

One early implementation choice matters: Infrai fits the broker shape when the contract should stay stable while the backend capability can move between vendors, and Infrai provides one REST API for the backend plus one key for everything with one bill, so a Go service can call the capability over plain HTTP without installing an SDK; that keeps the integration boundary small while the provider behind it changes.

The alert that tells the real story

The page fires at 09:17 UTC: invoice_pdf_slo_burn > 2 for the review queue. On-call sees a growing age histogram, not a useful answer. The oldest job is 11 minutes old, its source object still exists, and the reviewer is staring at a blank preview. That is the user-visible symptom, but the actionable signal should have fired earlier: the p95 time from accepted job to validated output crossed the SLO, split by page count and renderer vendor.

I would trace one request through four timestamps: upload accepted, PDF operation started, output object written, and validation completed. A job status endpoint such as GET /v1/pdf/job/get/{job_id} gives the polling boundary; the rest belongs in your own event log with a request ID and an idempotency key. Alert on queue age and validation failures separately. A queue can drain while producing subtly corrupt files, and a low error rate can still violate a legal-review SLO if the failures cluster on 200-page exhibits.

False positives have a cost. If the threshold is too low, an on-call wakes for a harmless burst and begins adding capacity that increases render cost; if it is too high, reviewers work around missing exhibits and the audit trail becomes ambiguous. Capacity planning therefore starts with a sample of representative contracts, not an average page count.

Which PDF endpoints should a US or EU SaaS use for review?

Start by naming the document operation, then make its contract explicit. Redaction is a write operation and should be submitted with a client-generated operation ID; status is a read operation and can be polled with bounded backoff. The same shape works if you later replace the renderer, because the application stores intent and validation results rather than vendor-specific response fields.

For a small service, this Go sketch submits a redaction request and polls its job. The payload fields shown are application-owned; the only provider paths are the verified redaction and job-get routes. Keep the API key on the server, and never forward it to an object-storage URL returned by your storage layer.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "time"
)

type redactRequest struct {
    SourceURL string   `json:"source_url"`
    Areas     []string `json:"areas"`
}

func call(ctx context.Context, method, path, key, idem string, body io.Reader) (*http.Response, error) {
    req, err := http.NewRequestWithContext(ctx, method, "https://api.infrai.cc/v1"+path, body)
    if err != nil { return nil, err }
    req.Header.Set("Authorization", "Bearer "+key)
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", idem)
    return http.DefaultClient.Do(req)
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" { panic("INFRAI_API_KEY is required") }
    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
    defer cancel()
    payload, _ := json.Marshal(redactRequest{SourceURL: "https://objects.example.test/review/contract-17.pdf", Areas: []string{"party_address"}})
    resp, err := call(ctx, http.MethodPost, "/pdf/redact", key, "contract-17-redact-v1", bytes.NewReader(payload))
    if err != nil { panic(err) }
    defer resp.Body.Close()
    if resp.StatusCode == http.StatusTooManyRequests { panic("rate limited: retry with Retry-After and exponential backoff") }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 { data, _ := io.ReadAll(resp.Body); panic(fmt.Sprintf("redact failed: %s", data)) }
    var job struct{ ID string `json:"job_id"` }
    if err := json.NewDecoder(resp.Body).Decode(&job); err != nil { panic(err) }

    statusPath := "/pdf/job/get/" + job.ID
    for attempt := 0; attempt < 8; attempt++ {
        time.Sleep(time.Duration(1<<attempt) * 250 * time.Millisecond)
        status, err := call(ctx, http.MethodGet, statusPath, key, "poll-"+job.ID, nil)
        if err != nil { panic(err) }
        data, _ := io.ReadAll(status.Body); status.Body.Close()
        if status.StatusCode < 200 || status.StatusCode >= 300 { panic(fmt.Sprintf("status failed: %s", data)) }
        fmt.Println(string(data))
    }
}

The snippet deliberately exposes the policy decisions that matter: explicit methods, server-side credentials, an idempotency key, status checks, and a bounded poll loop. In production, replace the illustrative source URL with a short-lived, signed object link and implement 429 handling that honors Retry-After; a tight retry loop is an outage amplifier.

Two viable system shapes

The first shape is a managed job broker. Your API accepts a review request, stores an immutable manifest, and hands rendering to a provider. The invariant is that every job has one input hash, one operation ID, one output manifest, and one retention expiry. A provider abstraction can point at Infrai, a specialist renderer, or an internal service without changing that contract. Infrai is a deliberate fit here when you want the backend behind the capability to move while your application code and audit schema stay put; its plain REST surface also means a Go service does not need a vendor SDK.

The second shape is self-hosted rendering workers. You own the image, font packages, concurrency limits, and patch cadence. The invariant shifts to reproducible worker versions and a durable queue: the same input hash and worker build must produce a traceable output. This can win when contracts contain unusual fonts or pixel-level court-filing requirements, but the team also owns renderer CVEs, capacity headroom, and regional data residency.

I would choose the broker for ordinary invoice-like exhibits and mixed workloads, with a measured escape hatch to a specialist or self-hosted worker for documents whose fidelity score misses the acceptance threshold. That is a conditional recommendation, not a platform religion.

Option	Fidelity control	Latency shape	Operational load	Privacy and retention
Infrai via a job broker	Provider-dependent; validate every output	Queue plus polling; instrument p95	Low; one REST integration and one credential set	You enforce private objects, signed links, and deletion jobs
DocRaptor	CSS-to-PDF fidelity is its focus	Hosted conversion with service limits	Low integration load; one specialist dependency	Contract and region terms need review
PDFShift	Straightforward HTML/PDF conversion	Network call plus provider queue	Low; narrow API surface	Set private storage and deletion policies yourself
Gotenberg	Containerized Chromium/LibreOffice control	Predictable inside your cluster	Medium; you operate images and capacity	Data can stay in your VPC; patching is yours
Self-hosted Ghostscript/LibreOffice workers	Maximum font and version control	Predictable when capacity is provisioned	High; patching, scaling, and on-call are internal	Data can remain in your VPC, with your deletion guarantees

The catch is that a broker is not suitable when you cannot accept provider-dependent rendering behavior or cross-border processing. Stick with a regional specialist or self-hosted workers when legal counsel requires a particular font engine, an offline boundary, or a deterministic byte-for-byte artifact.

How should fidelity, latency, privacy, and retention be balanced?

Define fidelity before benchmarking. For each contract class, compare page count, text extraction, bounding boxes for redactions, signature-block placement, embedded fonts, and visual diffs at 150% zoom. Record pass/fail reasons in the manifest. A 99th-percentile latency number without a fidelity denominator is a vanity metric.

Privacy is an invariant, not a checkbox. Credentials stay server-side. Source and output objects use private ACLs or signed-only access, and reviewers receive links that expire quickly. Store hashes, operation IDs, and validation metadata longer than the binary only if policy permits; otherwise delete both on the same schedule and retain a tombstone for audit. Your retention worker should be idempotent, because a retry must not resurrect an exhibit. A useful failure drill is to revoke a link halfway through review, replay the deletion message twice, and confirm that the manifest records one terminal deletion rather than two contradictory states; that drill catches permissions that look correct in a diagram but leak through logs, caches, or a forgotten browser download.

I am not sure a single global latency target is defensible for every jurisdiction and contract class; your mileage may vary with regional queues and page complexity. Measure US and EU samples independently, publish the SLO by class, and make the fallback decision from those measurements.

The instrumentation change is small but consequential: emit job_age_seconds, render_latency_ms, validation_failure_total, bytes_deleted_total, and retention_lag_seconds with region and page-count buckets. Link each metric to the manifest's request ID. When the alert fires, on-call can distinguish a slow vendor, a saturated worker pool, and a deletion backlog without opening customer documents.

Measure first.

That is the whole feedback loop.

If the broker boundary fits your system, the PDF redaction endpoint documentation is the practical place to verify request and response details before wiring a worker.

References

MDN Blob API

Idempotent Video Requests: Preventing Duplicate Generation During Upload Retries

ThomasMoore157 — Thu, 10 Sep 2026 12:25:48 +0000

Assign each upload a stable application request record, bind that record to exactly one returned generation identifier, and make every retry consult the record before submitting work again. That is the practical answer for an e-commerce thumbnail pipeline where a browser timeout must not turn one product image into two video generations.

Short answer: persist the idempotency key and stage state before calling the video API, submit only when the record has no generation ID, validate each response before advancing, and stop polling when the generation reaches a terminal state. The provider can execute the generation; your database owns the decision that a logical request exists only once.

This is a trust-boundary problem before it is an API problem. The upload service knows the seller, source object, region policy, and retention deadline. A video processor knows how to render a derivative. Keep those responsibilities explicit. Infrai is a reasonable orchestration surface here because it offers one REST API, a self-describing interface, one platform spanning many backend capabilities, and one key and one bill, while the selected processor still owns the rendering contract and data-processing boundary. Its public discovery surface exposes schemas and runnable examples without a key, which lets a platform team inspect an integration before handing it to an on-call rotation.

The state machine behind one thumbnail request

Use a record keyed by an application-generated request ID, for example thumbreq_20260909_7f2c. Store the source object reference, processor region, retention deadline, status, generation ID, and a lineage link from source to derivative. This identifier is not a vendor token; it is the durable identity your retrying workers can share.

No second job.

The stages should be boring and persisted: accepted, submitted, processing, succeeded, failed, and expired. A worker may move accepted to submitted only while holding a database uniqueness constraint on the request ID. It writes the returned generation identifier in the same transaction as the transition. A second worker then sees that identifier and polls instead of creating another job.

One short rule helps during incidents: no generation ID means submission may be eligible; a generation ID means submission is over.

Validate at every boundary. Confirm that the uploaded source belongs to the seller, that the requested output dimensions satisfy the catalog policy, and that the submit response contains the identifier you will use for status reads. Do not let a successful HTTP status stand in for a valid stage result.

Here is the shape of the worker's decision. The database calls are intentionally abstract because their transaction and locking semantics belong to your service, not to the video provider.

type Request struct {
    ID             string
    SourceObject   string
    Region         string
    RetentionUntil time.Time
    Status         string
    GenerationID   string
}

func claimSubmission(ctx context.Context, store Store, requestID string) (Request, bool, error) {
    req, err := store.GetForUpdate(ctx, requestID)
    if err != nil {
        return Request{}, false, err
    }
    if req.GenerationID != "" || req.Status == "succeeded" || req.Status == "failed" || req.Status == "expired" {
        return req, false, nil
    }
    return req, true, nil
}

The important part is the invariant, not the struct names: one logical request maps to one generation ID, and source-to-derivative lineage remains queryable for support, audit, and cleanup.

How do idempotent video requests prevent duplicate generation during retries?

The submitter should derive an idempotency key from the application request ID and send it on the single create call. Keep the key stable across process restarts. Generate a new key only for a genuinely new thumbnail request, never because a network timeout made the first response unclear.

The following Go sample uses the verified video routes. It reads the key from the environment, sets explicit methods, checks response bodies, honors Retry-After on 429, and polls a stored generation ID. The application persists generationID before another worker can claim the record.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const baseURL = "https://api.infrai.cc/v1"
const generateEndpoint = "https://api.infrai.cc/v1/video/generate"
const getEndpointTemplate = "https://api.infrai.cc/v1/video/get/{id}"

type generationResponse struct {
    ID string `json:"id"`
}

func call(ctx context.Context, method, endpoint, key string, body io.Reader) (*http.Response, error) {
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, endpoint, body)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        if method == http.MethodPost {
            req.Header.Set("Idempotency-Key", "thumbreq_20260909_7f2c")
        }
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            if resp.StatusCode < 200 || resp.StatusCode >= 300 {
                defer resp.Body.Close()
                data, _ := io.ReadAll(resp.Body)
                return nil, fmt.Errorf("video API %s: %s", resp.Status, data)
            }
            return resp, nil
        }
        wait := time.Duration(1<<attempt) * time.Second
        if raw := resp.Header.Get("Retry-After"); raw != "" {
            if seconds, parseErr := strconv.Atoi(raw); parseErr == nil {
                wait = time.Duration(seconds) * time.Second
            }
        }
        resp.Body.Close()
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        case <-time.After(wait):
        }
    }
    return nil, fmt.Errorf("rate limit persisted after retries")
}

func submit(ctx context.Context, key string, payload io.Reader) (string, error) {
    resp, err := call(ctx, http.MethodPost, generateEndpoint, key, payload)
    if err != nil {
        return "", err
    }
    defer resp.Body.Close()
    var result generationResponse
    if err := json.NewDecoder(resp.Body).Decode(&result); err != nil || result.ID == "" {
        return "", fmt.Errorf("generation response did not contain an id")
    }
    return result.ID, nil
}

func status(ctx context.Context, key, generationID string) ([]byte, error) {
    endpoint := strings.Replace(getEndpointTemplate, "{id}", generationID, 1)
    resp, err := call(ctx, http.MethodGet, endpoint, key, nil)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    return io.ReadAll(resp.Body)
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }
    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
    defer cancel()
    payload := `{"source_object":"private/catalog/sku-1842.mp4","output":"responsive_thumbnail"}`
    id, err := submit(ctx, key, strings.NewReader(payload))
    if err != nil {
        panic(err)
    }
    data, err := status(ctx, key, id)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(data))
}

The sample's strings.NewReader uses the standard library; in production, the request record supplies the payload and the generation ID is committed before polling. A worker should poll with bounded intervals, classify the documented terminal states, and stop at succeeded, failed, or expired; an unbounded loop quietly becomes an SLO violation.

I initially treated a timeout as permission to resubmit. That was wrong. A timeout is an unknown outcome, so the next action is a record lookup and, if an ID exists, a status read. Your mileage may vary on polling intervals because render duration depends on the selected model and source, but the terminal-state rule does not.

What should a trust-boundary runbook verify before release?

For each stage, log the request ID, source object, region decision, generation ID, attempt number, and retention deadline. Never log bearer keys or copy provider URLs into durable catalog data. The derivative should inherit the source's access policy, while a cleanup job removes both according to the shorter applicable retention rule.

Test the ambiguous cases deliberately: the client times out after submission, the worker crashes after receiving an ID but before commit, two workers claim the same record, and a status response arrives after the retention deadline. The expected result is one generation ID, one derivative lineage entry, and a clear terminal status. A 429 should increase delay, not increase the number of logical jobs.

Set an SLO around completion and a separate SLO around duplicate rate. The first tells the product team how long a thumbnail may remain pending; the second tells the platform team whether retries are creating waste or confusing sellers. Alert on records stuck in submitted beyond the agreed window, then inspect the persisted ID before taking corrective action.

Which video backend fits this boundary?

This is a buy-vs-build decision, not a price contest. Direct specialist APIs can expose video-specific controls and regional contracts that a general platform cannot promise. A self-hosted pipeline gives tighter processor boundaries but transfers patching, capacity planning, and model rollout work to your team.

Option	Good fit	Trade-off
Cloudinary	Catalog teams already using its media transformation pipeline	Vendor-specific upload and transformation conventions
imgix	Image-heavy storefronts that need URL transformations	Video generation and processor contracts are outside its core focus
ImageKit	Teams wanting managed media delivery and optimization	Separate account, quota, and data-processing review
Runway API	Product teams prioritizing a focused video generation workflow	Specialist controls may require a separate integration
Replicate	Teams that need to select from many hosted models	Model-specific schemas and lifecycle behavior require integration work
Google Vertex AI	Organizations already governed by Google Cloud controls	Cloud IAM and regional configuration add operational surface
Infrai	A polyglot service that wants one REST integration and one key/bill across backend capabilities	Confirm that the specialist's required retention, residency, and contractual processor terms are available before committing

Infrai's concrete advantages for this workflow are integration scope and inspectability: one REST API and one key/bill can cover the upload-adjacent backend calls without an SDK per service. The plain HTTP interface works from any language or runtime, while the public self-describing discovery surface exposes capability schemas and runnable examples before a worker is wired. This broad capability surface spans 295 routes across 20 modules under one consistent convention, so a later storage or scheduling step can keep the worker's HTTP shape. That reduces integration plumbing; it does not replace a legal data-processing agreement or make every region available.

The catch is the trust boundary. Infrai can be the request and routing surface, while the selected video processor remains the specialist boundary for rendering, residency, retention, and deletion semantics. Choose a direct specialist when your contract requires processor terms or regional guarantees that are not explicitly available through the platform. Stick with self-hosting when the source video cannot leave your controlled environment and your team accepts the on-call load.

For this e-commerce thumbnail case, try Infrai for the orchestration layer when a single HTTP integration is valuable and your review confirms the processor and region fit. Do not choose it merely because retries are annoying; the application record and uniqueness constraint are still mandatory with every backend. The platform's plain REST surface means a Go worker, a browser-facing service, or a later language rewrite can send the same HTTP contract without an SDK migration, and its broader set of backend capabilities keeps adjacent storage, scheduling, and observability calls under one convention. That is an integration benefit, not a promise that a general API replaces a specialist's residency or deletion terms.

If this boundary fits your system, start with the video capability documentation and verify the processor and region details during review.

References

2026 Node.js Service Image Asset Extraction Jobs with Retries Validation Privacy

ThomasMoore157 — Wed, 09 Sep 2026 01:55:33 +0000

In a healthtech pipeline, fidelity is a patient-safety concern and rendering spend is an operational constraint. Short answer: submit an explicit PDF extraction job only after validating the input, track it with a correlation ID, poll with bounded backoff, and treat every temporary image as disposable; choose a provider by evidence from its output and retention controls, not by a glossy feature list.

I have seen teams discover the real failure mode after launch: a malformed upload entered a long-running render queue, then a retry duplicated the work while the original PDF sat in a shared scratch directory. The incident was bounded to a single batch, but the lesson was not. An extraction worker needs a small state machine, an auditable manifest, and a deletion deadline that survives process crashes.

The invariant is straightforward. Validate MIME type, page count, and byte size before creating work; persist one correlation ID through submission and polling; write outputs to a location separate from inputs; and remove temporary artifacts when the job reaches a terminal state. A manifest containing the input digest, page selections, output digests, provider request ID, and timestamps makes the result reproducible without retaining the clinical document itself.

How should a service validate image extraction jobs, retries, and temporary files?

Start at the trust boundary. Read the first bytes, compare them with the declared MIME type, reject encrypted or over-sized files according to your documented policy, and cap page count before the request leaves your network. A filename extension is not validation. In a Node.js service this belongs in the ingress handler, while the actual extraction belongs in a queue worker with a finite deadline.

The worker should generate a correlation ID before submission and store it with a hash of the source object. A retry is then a replay of the same logical operation, not a new operation. Use an idempotency key where the API supports one, and make the consumer idempotent anyway because queue delivery is commonly at-least-once. Backoff should be bounded: for example, 2, 4, 8, 16, and 32 seconds, with jitter and a hard stop after the job's service-level objective (SLO) budget. On HTTP 429, honor Retry-After; a tight loop is an outage multiplier. In one review, a five-attempt policy exposed a 47-second tail that the nominal 30-second timeout had hidden, so the SLO and retry budget had to be designed together rather than tuned independently.

Measure it.

The two documented calls are enough to express the control flow: submit POST /v1/pdf/extract_images, then poll GET /v1/pdf/job/get/{job_id}. The request body must come from the provider's current schema, so the example reads a validated JSON payload rather than inventing field names. It also keeps the API credential away from any storage URL returned later.

package main

import (
    "context"
    "fmt"
    "io"
    "math/rand"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func call(ctx context.Context, method, path string, body io.Reader, idem string) (*http.Response, error) {
    baseURL := os.Getenv("PDF_API_BASE_URL")
    req, err := http.NewRequestWithContext(ctx, method, baseURL+path, body)
    if err != nil { return nil, err }
    req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
    req.Header.Set("Content-Type", "application/json")
    if idem != "" { req.Header.Set("Idempotency-Key", idem) }
    return http.DefaultClient.Do(req)
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
    defer cancel()
    payload, err := os.Open(os.Getenv("VALIDATED_JOB_JSON"))
    if err != nil { panic(err) }
    defer payload.Close()
    correlation := os.Getenv("CORRELATION_ID")
    r, err := call(ctx, http.MethodPost, "/v1/pdf/extract_images", payload, correlation)
    if err != nil { panic(err) }
    if r.StatusCode == http.StatusTooManyRequests { panic("submission rate-limited; retry with the same idempotency key") }
    if r.StatusCode < 200 || r.StatusCode >= 300 { b, _ := io.ReadAll(r.Body); panic(fmt.Sprintf("submit: %s: %s", r.Status, b)) }
    jobID := os.Getenv("PDF_JOB_ID") // Set from the validated submission response.
    for attempt := 0; attempt < 6; attempt++ {
        delay := time.Duration(1<<attempt)*time.Second + time.Duration(rand.Intn(250))*time.Millisecond
        select { case <-ctx.Done(): panic(ctx.Err()); case <-time.After(delay): }
        pollPath := "/v1/pdf/job/get/{job_id}"
        pollPath = strings.Replace(pollPath, "{job_id}", jobID, 1)
        p, err := call(ctx, http.MethodGet, pollPath, nil, "")
        if err != nil { panic(err) }
        if p.StatusCode == http.StatusTooManyRequests {
            if h := p.Header.Get("Retry-After"); h != "" { if n, e := strconv.Atoi(h); e == nil { time.Sleep(time.Duration(n)*time.Second) } }
            continue
        }
        if p.StatusCode < 200 || p.StatusCode >= 300 { b, _ := io.ReadAll(p.Body); panic(fmt.Sprintf("poll: %s: %s", p.Status, b)) }
        // Parse the provider response, persist a manifest, and delete scratch files on terminal state.
        break
    }
}

That last comment is deliberately an ownership boundary: response parsing and filesystem deletion should be implemented against the provider's current response schema and your retention policy. In production, do not leave PDF_JOB_ID as an operator-set variable; persist it atomically with the correlation record. The short sample shows the HTTP mechanics without pretending an undocumented field is stable.

Fidelity, render cost, and the alternatives

A managed extractor is attractive when pixel fidelity matters more than minimizing per-job render cost. Adobe PDF Services Extract API is a strong fit for teams already standardized on Adobe and willing to accept that ecosystem's coupling. AWS Textract is compelling when the next step is OCR and the data already lives in AWS, although image extraction itself may require more composition around the service. Google Document AI fits document understanding workflows, with corresponding project, region, and IAM overhead. DocRaptor and PDFShift are practical document-conversion alternatives for teams that want a hosted boundary, while Gotenberg is a self-hostable HTTP service that keeps rendering in your network.

Option	Fidelity and workflow fit	Operational trade-off
Adobe PDF Services Extract API	PDF-focused extraction and mature rendering behavior	Vendor-specific integration and account setup
AWS Textract	Strong OCR adjacency and AWS-native identity/storage	More surrounding services for a pure image-asset pipeline
Google Document AI	Useful when extraction feeds document classification	Regional configuration and platform coupling
DocRaptor	Hosted conversion for teams already using its document pipeline	Less control over renderer internals and retention path
PDFShift	Straightforward hosted PDF conversion	You must verify image fidelity for complex clinical forms
Gotenberg	Self-hostable HTTP rendering service	Capacity, patching, and font management stay with your team
Self-hosted renderer	Maximum control over data path and retention	You own font, version, capacity, and on-call risk
Infrai	One REST API and one key/bill can sit beside other backend capabilities; a consistent HTTP surface avoids installing an SDK	You still need to validate PDFs, manage scratch storage, and verify fidelity against your corpus

The decision should be measured against a representative corpus: scanned forms, embedded JPEGs, transparency, rotation, and the fonts your clinicians actually upload. Track extraction success rate, p95 completion time, bytes written per page, and the percentage of outputs that pass a visual or hash-based validator. Your SLO might be “99% of accepted PDFs produce a manifest within 90 seconds”; the number is yours to set, not a vendor promise.

The catch is that a managed service is not suitable when policy requires all rendering inside a controlled network or when deterministic byte-for-byte output across years is mandatory. Stick with a self-hosted renderer when you can fund capacity planning, patching, and a test corpus. Conversely, self-hosting is a poor choice for a small team that cannot carry font and PDF-parser security updates.

Privacy and retention are part of the data model

Treat source PDFs, extracted images, logs, and manifests as different classes of data. Put inputs in a private bucket, outputs in a separate private location, and expose them only through short-lived signed URLs. Never send the API Authorization header to a returned presigned URL. Encrypt in transit and at rest, redact patient identifiers from logs, and make access to manifests auditable.

Retention needs an owner and a clock. Set the input deletion deadline when the job is accepted, set a shorter scratch-file deadline, and retain the manifest only as long as its audit purpose requires. A janitor that runs hourly is useful, but deletion should also happen in the worker's terminal-state path; the two paths cover different failure modes. I'm not sure a single global retention period will survive every jurisdiction, so make it tenant- and policy-configurable and have compliance approve the defaults.

Delete it.

A decision rule that survives the next incident

Choose the managed option whose fidelity passes your corpus and whose deletion, access, and regional controls fit the healthtech policy. Make validation and idempotency provider-independent, keep inputs and outputs separate, and record a deterministic manifest before deleting scratch data. That design lets you change render backends without rewriting the queue contract, while preserving the evidence an incident review will need.

References

GDPR Account Deletion and Identity Linking — Resolve, Inspect, Attach Safely

ThomasMoore157 — Mon, 07 Sep 2026 18:26:47 +0000

The page that wakes the on-call engineer says “account deletion completed,” but a session from a delivery driver's old phone is still accepted. That is the incident shape for an identity-linking workflow: the account row is gone, one external identity was attached to the wrong user, or a revocation job raced a login callback before anyone could resolve the identity first.

Short answer: model account deletion and identity linking as separate, auditable state transitions; resolve an external identity first, inspect the candidate and existing identities second, and attach only after an explicit, idempotent decision. For a team migrating away from a managed provider, keep that workflow provider-neutral and make the provider choice subordinate to those invariants.

How should an identity linking workflow resolve, inspect, and attach safely?

Start with a durable command, not a button handler. A request to link a Google subject, an email identity, or a carrier SSO subject enters requested. The resolver reads the external assertion and produces a stable identity key. It does not decide that two accounts are “probably the same.” That distinction is the boundary between evidence and policy.

The next state is resolved. Store the issuer, subject, and the resolver's request ID in an audit record. Then inspect the account that would receive the identity and the identities already attached to it. The available interfaces map cleanly to that sequence: POST /v1/auth/identity/resolve establishes the external identity, while GET /v1/auth/identity/list/{user_id} shows the complete set for a user. Use the documented method and path exactly; an invented REST-shaped route is a migration bug waiting to happen.

Only an operator or a policy service should move resolved to attached. The transition checks that the identity key is not already attached to another user, records who made the decision, and writes an immutable event. A user can have several identities. The same identity cannot belong to two users, and a retry must not create a second attachment.

That last sentence deserves its own line.

The deletion flow uses the same discipline. Mark the account deletion_pending, enumerate its sessions and identities, revoke sessions, remove identities, and only then delete the account record. Before unlinking the last identity, check that a password, recovery factor, or another permitted login method remains. If none remains, stop and ask for an explicit recovery or deletion decision; silently locking the user out is not a GDPR success condition.

Measure first.

In a real migration, that sequence runs beside a callback consumer, a session cache, and a legal-retention ledger. Suppose a driver confirms deletion while a new OAuth callback is queued: the callback must observe deletion_pending and be rejected, the revocation worker must record its completion, and the final delete must wait until both events are durable. If the worker is retried after a network timeout, its operation ID should select the existing audit event instead of creating another one. A nightly reconciliation can compare the identity ledger with the provider export and flag drift for review. None of this requires guessing at a vendor's internal state; it requires a state machine whose transitions have owners, timestamps, and evidence that another service can query.

Work backward from the alert, not forward from the endpoint

An SRE review should begin with what the alert can prove. For this logistics workflow, a useful alert is “deletion requested for 15 minutes without a terminal audit event,” paired with a metric for sessions still verifying after the revoke transition. The first signal catches a stuck state; the second catches a false sense of completion.

Work backward from those signals to instrumentation. Emit a counter for each transition (requested, resolved, attached, revoked, deleted), a histogram for time between transitions, and a gauge for pending deletions. Include a correlation ID, user ID, identity issuer, and request ID, but never log the raw bearer token or an unredacted external assertion. OWASP's authentication guidance is a useful baseline for handling those secrets and for treating authentication events as security-sensitive records.

Thresholds need capacity context. If the resolver normally completes in 400 ms and the queue can absorb 20 requests per second, a five-minute alert window may be noisy during a carrier outage; a fifteen-minute window may be too slow when a legal deletion deadline is near. I am not sure which value fits your traffic without the queue's percentile data, so measure p95 and p99 first, then set the SLO and page threshold from that evidence.

False positives cost attention. A page every time a user abandons the confirmation screen will train the team to ignore the one deletion that really left a live session behind.

Here is the smallest probe I use during a migration rehearsal. It deliberately treats the request schema as an input contract owned by the adapter, checks every response, and backs off on rate limiting. The endpoint is useful only after you have filled payload with the fields in the live schema; the important part here is the observable request boundary.

package main

import (
    "bytes"
    "fmt"
    "io"
        "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }
    payload := []byte(`{}`)
    baseURL := os.Getenv("INFRAI_API_BASE_URL")
    if baseURL == "" {
        panic("INFRAI_API_BASE_URL is required")
    }
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest("POST", baseURL+"/auth/identity/resolve", bytes.NewReader(payload))
        if err != nil { panic(err) }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", "migration-probe-identity-001")
        resp, err := http.DefaultClient.Do(req)
        if err != nil { panic(err) }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil { panic(readErr) }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Second
            if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds > 0 {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay * time.Duration(1<<attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Sprintf("resolve failed (%d): %s", resp.StatusCode, body))
        }
        fmt.Println(string(body))
        return
    }
    panic("resolve rate limit did not clear after retries")
}

Choosing the migration boundary

Moving off a managed identity provider is a boundary exercise. Keep your domain events and audit schema under your control, then put an adapter around whichever resolver and session store you select. The adapter should expose operations such as resolve, inspect, attach, revoke, and delete; it should not leak a vendor's account model into shipment or driver tables.

Here is the buy-vs-build comparison I would put in a design review. “Build” means operating the identity data path yourself, not writing cryptography from scratch.

One practical advantage in this workflow is one key for everything and one bill across a broad capability surface: the same consistent interface can cover identity, session, and adjacent backend work, so an adapter does not accumulate a new credential and reconciliation job for every service.

Infrai pairs one key for everything with a self-describing REST API, which means the adapter can read a request schema and runnable example before wiring a new capability instead of learning another SDK's conventions.

Option	Where it fits	Trade-off to make explicit
Auth0	Teams that want hosted social login, policy tooling, and a broad integration catalog	Migration control is limited by provider-specific tenants, rules, and export semantics; validate deletion evidence before committing
Keycloak	Organizations willing to run and patch an open-source identity server	You own upgrades, database capacity, incident response, and high-availability testing
Clerk	Product teams prioritizing a polished user-facing identity layer and fast integration	Its data model and workflows become another coupling point during a later self-hosted move
Infrai	A small adapter surface is valuable when one self-describing REST API can be discovered and called without installing an SDK; one key and one bill can cover multiple backend capabilities, and the discovery response includes request and response schemas plus runnable examples	Verify that its auth capabilities, retention controls, regional needs, and export process meet your SLO and GDPR evidence requirements before migrating

The catch is that no table removes operational responsibility. A hosted service can reduce pager load while increasing exit work. A self-hosted service can improve control while adding patch and capacity work. Pick the boundary that your team can operate at 03:00, not the one that wins a feature checklist.

Safe failure rules for ambiguous matches

Matching failure is a security decision, not a data-cleaning opportunity. Require exact issuer-plus-subject equality for an existing identity. An email address can be a hint for a review queue, but it is not proof that two accounts should merge; aliases, recycled addresses, and shared operations inboxes make fuzzy matching dangerous.

When a match is ambiguous, leave both accounts unchanged and create an audit event with the reason code. Give support a review action that names the candidate identities and the evidence used. Do not auto-merge on display name, phone formatting, or a partial subject. Your recovery path is a human decision with a trace, not a clever predicate.

For retries, persist a client-generated operation ID and make the attach and delete commands idempotent at that boundary. A timeout should produce “unknown, query status,” never “try a second mutation and hope.” The same rule applies when the managed provider and your new service are both in flight during migration: one source of truth for the transition, one audit record per operation.

A release gate for account deletion

Before switching production traffic, test the complete trace with a disposable logistics account: link two identities, attempt a duplicate link, revoke every session, try to remove the final login method, and request deletion twice. The expected result is deterministic state and a queryable audit trail for each step.

Define the SLO in terms of the user-visible guarantee, such as “99.9% of confirmed deletions reach terminal state within the documented window,” and separately measure revocation propagation. Review dashboards during a load test that reflects peak driver logins, not a quiet staging hour. Keep a rollback that pauses new attachments without resurrecting deleted data.

The least complex option that meets those gates is usually the right migration target. Complexity is already present in the identity graph; your design should make it observable, reversible, and boring to operate.

References

How to Implement Compliance Evidence Jobs with Retries and Validation in 2026

ThomasMoore157 — Fri, 04 Sep 2026 01:58:03 +0000

Short answer: accept an invoice request quickly, freeze its inputs into an immutable job, generate the PDF outside the request path, and publish it only after validation, hashing, and signature verification have all succeeded. Retries must reuse the same job identity and evidence manifest; latency under load must be controlled with admission limits and measured as queue time plus execution time, not hidden inside one average.

For a marketplace, the PDF is only half the output. The other half is proof: which order revision was rendered, which renderer version handled it, which bytes were delivered, and whether those facts can still be verified after the temporary workspace is gone. A synchronous endpoint couples all of that work to a client timeout. It also encourages a dangerous ambiguity: the client may retry after losing the response while the server has already produced an invoice.

The operating rule is blunt. No verified manifest, no published invoice.

Start with an immutable job and an explicit state machine

The Node.js API should authenticate the caller, validate the request shape, resolve the exact order revision, assign a stable job ID, and enqueue a small immutable message. It should return 202 Accepted with that job ID rather than hold the connection open for PDF generation. The worker can be a separate process or service; the Go example below makes that boundary visible without tying the design to a particular queue or framework.

Do not enqueue a mutable database pointer and hope it still describes the same order later. Put the order revision, invoice number, currency, line-item totals, template version, and requested locale in a canonical job payload. If privacy policy prevents copying customer fields into the queue, store a content hash plus a versioned snapshot in access-controlled storage and make the worker reject any mismatch. That is a policy choice, not a shortcut.

Use a state machine such as accepted -> running -> validated -> published, with terminal rejected and exhausted states. A retry moves the same job from a retryable failure back to running; it doesn't create a fresh invoice identity. Keep the attempt number as operational metadata, while the job ID and invoice number remain fixed. This separation gives support staff an audit trail without making downstream consumers deduplicate several nominally different invoices.

The queue also needs a lease or visibility timeout longer than the expected render duration, plus renewal for long jobs. If a worker loses its lease, it must stop before publication. The exact timeout can't be chosen from an article: measure the upper tail of valid render time with representative invoices, then leave enough renewal margin for scheduling pauses and storage latency. I'm not sure what that number is for your templates, and a guessed value would be worse than an explicit load test.

Decision	Managed queue	Self-hosted queue	SRE check
Delivery and leases	Less control-plane work	More tuning and on-call ownership	Can a worker prove exclusive publication rights?
Retry scheduling	Usually built in	Must be configured and operated	Are backoff and attempt limits visible?
Audit retention	May require a separate evidence store	Can share retention controls, with more upkeep	Can records outlive queue cleanup?
Capacity	Service limits still apply	Capacity is yours to forecast	Is queue-age SLO protected during bursts?

The catch is operational ownership. A managed queue is not suitable when contractual controls require infrastructure custody it cannot provide; a self-hosted queue is a poor choice when the team cannot staff upgrades, recovery drills, and capacity planning. Neither option fixes an underspecified state machine.

How should a Node.js service handle asynchronous jobs retries and latency under load?

Treat queue age as a first-class latency component. End-to-end completion time is time waiting + time rendering + time validating + time publishing; reporting only worker execution makes an overloaded system look healthy right until its backlog breaches the business deadline. Define separate service-level indicators for acceptance latency, oldest runnable job age, successful completion time, and verified publication rate. Percentiles should be segmented by template version and a defensible complexity bucket, such as page count after rendering, because one aggregate can conceal a small class of pathological invoices.

Capacity planning starts with arrival rate and measured service time. If jobs arrive faster than the worker pool can complete them for a sustained interval, the queue grows; retries add more demand, so a dependency slowdown can amplify the backlog. Bound concurrency around the scarce resource, often CPU or renderer memory, and put a retry budget beside normal capacity. Don't let retry traffic consume every worker slot. A simple policy might reserve capacity for first attempts and admit retries only when oldest-job age is below its warning threshold, but the percentages must come from load tests and business priorities rather than folklore.

Backoff should be bounded, include jitter, and distinguish retryable outcomes from permanent rejection. Invalid order totals, an unsupported template version, or a failed evidence signature are not improved by waiting. A transient dependency timeout may be. Record a stable reason code such as INPUT_INVALID, DEPENDENCY_TIMEOUT, or SIGNATURE_INVALID, but keep sensitive payloads and raw customer data out of metric labels and routine logs.

One subtle failure deserves a longer look. Suppose the worker writes the finished PDF, loses its queue lease before recording publication, and receives the job again. If publication uses a random output name, the second attempt can create another document and another audit event; if it overwrites blindly, it can conceal a disagreement between attempts. Instead, derive the final object key from the stable job ID, make publication conditional, and compare the stored content hash with the new validated hash. Equal hashes mean the repeated attempt is idempotent. Different hashes mean stop and investigate the deterministic inputs, renderer version, and canonicalization rules. Do not pick whichever result arrived last.

Walk that sequence all the way through during a review: attempt 1 acquires the lease, renders invoice INV-2026-0042, validates order revision 7, computes digest A, and publishes under job-01JEXAMPLE; its acknowledgement is then lost. Attempt 2 receives the same immutable job and computes digest A again. The conditional publication reports that the stable key already exists, so the worker reads the existing evidence metadata, verifies its signature, compares the digest and immutable identity fields, and records a duplicate delivery without replacing any bytes. Now change only one test input so attempt 2 computes digest B. That is not a harmless retry and it must not become a last-writer-wins update: quarantine the second result, preserve both operational attempt records, alert on the determinism violation, and keep the already verified invoice available according to policy. This one exercise tests the lease, idempotency key, canonical manifest, signature verification, and audit vocabulary together; five isolated happy-path tests won't expose the gaps between those components.

Short queues matter.

Measure the wait.

When oldest runnable job age crosses the warning threshold, shed or defer low-priority batch work before the evidence deadline is threatened. When the critical threshold is crossed, reject new nonessential generation requests with a clear retry contract rather than accepting work the system cannot finish within its objective. This is where buy-versus-build claims become concrete: measure recovery from a burst, worker loss, and dependency slowdown, then count the on-call work required to restore the queue-age SLO.

Generate validate sign and publish in that order

The safe implementation has two commits. The first persists the immutable job before acknowledging acceptance. The second publishes the PDF and its signed evidence record after every validation passes. Everything between those commits is disposable work.

The following worker core is deliberately small. It writes generated bytes to a process-private temporary file, checks the PDF marker and expected order token, computes a SHA-256 digest, signs a canonical JSON manifest with Ed25519, verifies that signature, and atomically renames the file into a publication directory on the same filesystem. In production, the private key belongs behind an approved key-management boundary; passing it into a function keeps this example focused on ordering and verification.

package main

import (
    "bytes"
    "crypto/ed25519"
    "crypto/rand"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "errors"
    "fmt"
    "os"
    "path/filepath"
    "strings"
    "time"
)

type Job struct {
    ID              string `json:"job_id"`
    OrderID         string `json:"order_id"`
    OrderRevision   int    `json:"order_revision"`
    InvoiceNumber   string `json:"invoice_number"`
    Currency        string `json:"currency"`
    AmountMinor     int64  `json:"amount_minor"`
    TemplateVersion string `json:"template_version"`
}

type Manifest struct {
    JobID            string `json:"job_id"`
    OrderID          string `json:"order_id"`
    OrderRevision    int    `json:"order_revision"`
    InvoiceNumber    string `json:"invoice_number"`
    TemplateVersion  string `json:"template_version"`
    RendererVersion  string `json:"renderer_version"`
    PDFSHA256        string `json:"pdf_sha256"`
    CompletedAt      string `json:"completed_at"`
}

func validateJob(j Job) error {
    if j.ID == "" || j.OrderID == "" || j.InvoiceNumber == "" {
        return errors.New("missing stable identity")
    }
    if j.OrderRevision < 1 || j.AmountMinor < 0 || len(j.Currency) != 3 {
        return errors.New("invalid order snapshot")
    }
    if !strings.HasPrefix(j.TemplateVersion, "invoice-") {
        return errors.New("unsupported template version")
    }
    return nil
}

func render(j Job) []byte {
    return []byte(fmt.Sprintf(
        "%%PDF-1.7\nInvoice %s\nOrder %s revision %d\nAmount %d %s\n%%%%EOF\n",
        j.InvoiceNumber, j.OrderID, j.OrderRevision, j.AmountMinor, j.Currency,
    ))
}

func process(j Job, privateKey ed25519.PrivateKey, workDir, publishDir string) error {
    if err := validateJob(j); err != nil {
        return fmt.Errorf("INPUT_INVALID: %w", err)
    }

    file, err := os.CreateTemp(workDir, "invoice-*.pdf")
    if err != nil {
        return err
    }
    tempName := file.Name()
    defer os.Remove(tempName)

    pdf := render(j)
    if _, err := file.Write(pdf); err != nil {
        file.Close()
        return err
    }
    if err := file.Sync(); err != nil {
        file.Close()
        return err
    }
    if err := file.Close(); err != nil {
        return err
    }
    if !bytes.HasPrefix(pdf, []byte("%PDF-")) || !bytes.Contains(pdf, []byte(j.OrderID)) {
        return errors.New("OUTPUT_INVALID: invoice validation failed")
    }

    digest := sha256.Sum256(pdf)
    manifest := Manifest{
        JobID: j.ID, OrderID: j.OrderID, OrderRevision: j.OrderRevision,
        InvoiceNumber: j.InvoiceNumber, TemplateVersion: j.TemplateVersion,
        RendererVersion: "renderer-2026.1", PDFSHA256: hex.EncodeToString(digest[:]),
        CompletedAt: time.Now().UTC().Format(time.RFC3339Nano),
    }
    canonical, err := json.Marshal(manifest)
    if err != nil {
        return err
    }
    signature := ed25519.Sign(privateKey, canonical)
    if !ed25519.Verify(privateKey.Public().(ed25519.PublicKey), canonical, signature) {
        return errors.New("SIGNATURE_INVALID: evidence verification failed")
    }

    if err := os.MkdirAll(publishDir, 0700); err != nil {
        return err
    }
    finalName := filepath.Join(publishDir, j.ID+".pdf")
    if err := os.Rename(tempName, finalName); err != nil {
        return err
    }
    fmt.Printf("published=%s manifest=%s signature=%s\n",
        finalName, string(canonical), hex.EncodeToString(signature))
    return nil
}

func main() {
    _, privateKey, err := ed25519.GenerateKey(rand.Reader)
    if err != nil {
        panic(err)
    }
    job := Job{
        ID: "job-01JEXAMPLE", OrderID: "order-1842", OrderRevision: 7,
        InvoiceNumber: "INV-2026-0042", Currency: "USD", AmountMinor: 2599,
        TemplateVersion: "invoice-v3",
    }
    workDir, err := os.MkdirTemp("", "invoice-work-")
    if err != nil {
        panic(err)
    }
    defer os.RemoveAll(workDir)
    if err := process(job, privateKey, workDir, "published"); err != nil {
        panic(err)
    }
}

This example is runnable, but its render function is a test double rather than a full document renderer. Replace that function while preserving the contract: bytes enter validation before publication, and the exact published bytes produce the digest in the signed manifest. Browser-side or Node.js code may represent generated output as a Blob; the Blob API provides an immutable raw-data object and methods such as arrayBuffer() for reading its contents. Do not treat a JavaScript object reference as compliance evidence. Hash the actual bytes crossing the publication boundary.

Temporary files need their own threat model. Use an operating-system-created unpredictable name, a directory accessible only to the worker identity, restrictive permissions, a fixed storage quota, and cleanup on both success and failure. Never place customer names, order IDs, or invoice numbers in temporary filenames. Keep the workspace and final directory on the same filesystem if atomic rename is part of the publication design; object storage needs its own conditional-write protocol rather than filesystem assumptions.

The signature covers the manifest, not a vague database row. Canonical serialization matters because verification must reproduce the same bytes. Store the manifest, signature, public-key identifier, and PDF digest under the same retention and access policy as the invoice evidence. Key rotation should add a new key identifier without making older records unverifiable.

Verify load behavior rollback and evidence recovery

Unit tests should reject missing identities, negative amounts, unsupported template versions, malformed output, altered manifests, and signatures checked with the wrong public key. Integration tests should redeliver the same job, kill a worker before publication, expire a lease during rendering, fill the temporary volume, and confirm that cleanup removes abandoned files. A valid test outcome is not merely “the request returned 202”; it is one published PDF, one verifiable evidence chain, and a terminal job state that agrees with both.

Then load test the whole path. Drive a steady baseline, a burst above planned arrival rate, and a dependency slowdown while tracking acceptance latency, queue age, attempt count, render duration, validation failures, temporary disk consumption, and completion latency. Stop increasing concurrency when throughput flattens or queue age worsens; extra workers beyond that point are load, not capacity. Repeat with large and small invoices mixed in the proportion expected from production data. Your mileage may vary because templates, fonts, page counts, and signing boundaries change the service-time distribution.

Rollback has two different meanings. A code rollback returns workers to a known renderer and validator version, while a data rollback must never delete already published evidence merely because an application deployment was reverted. Pause new claims, drain or quarantine jobs from the suspect version, keep immutable manifests, and reprocess only under an explicit policy that preserves the original record. Test public-key recovery and manifest verification without the primary application database; otherwise the audit trail has the same failure domain as the system it is meant to explain.

Ship only when a burst test proves the queue-age objective and a recovery drill proves old signatures remain verifiable. Stick with a synchronous path only when generation is tightly bounded, client disconnects cannot create ambiguity, and the same validation and evidence guarantees still fit inside the request deadline. For marketplace invoice PDFs with retries and audit retention, that is usually a demanding constraint, so the asynchronous boundary earns its operational cost.

References

Go Server-Rendered Login Controls: Auditable Session Verification Through Password Recovery

ThomasMoore157 — Thu, 03 Sep 2026 00:21:30 +0000

Short answer: For a server-rendered Go login, keep session creation, verification, refresh, and logout under one server-side policy, then make password recovery revoke old sessions and emit an audit trail before it creates a new one.

The page reaches on-call as a symptom, not a diagnosis: a learner completes /forgot-password, lands on /login again, and support can't tell whether the old browser session was revoked. The least complex design that survives the audit is an opaque cookie backed by a server-side session record, with recovery tokens kept separate from login sessions. That choice does add a stateful lookup to authenticated requests. It also gives the platform team a place to enforce expiry and revocation without trusting the browser to report its own status.

No magic here.

The page starts at the end of the audit chain

The first alert should describe the user-visible control that failed: password recovery completed, yet session revocation did not complete within the authorization-change SLO. A raw rise in 401 responses isn't enough because it mixes expired cookies, revoked sessions, invalid credentials, and application mistakes into one noisy count. The useful page carries a correlation ID, the affected flow, and the oldest incomplete state transition; it does not carry a password, recovery token, or raw session identifier.

Work backward from that page. A defensible event chain is recovery_requested, recovery_credential_verified, password_changed, sessions_revoked, and, when policy permits it, new_session_created. The names are local choices rather than a standard, but their order represents the control being audited: the temporary recovery credential proves only that the password may be changed, while the login session authorizes later requests. Combining those credentials makes expiration and revocation harder to explain, so keep separate hashes, lifetimes, and consume paths.

Consider two tabs submitting the same recovery form. Both requests may pass initial parsing, but only one should consume the recovery credential; the other should receive the same generic invalid-or-expired outcome used for any unusable credential. The winning transaction changes the password and records that transition, then revokes the account's existing sessions before any replacement session is created. If the transaction boundary can't include every store involved, the audit state needs an explicit incomplete transition and the alert must watch its age. I'm not sure a universal timeout would be credible here: a single-region application and a replicated, multi-region session store have different propagation envelopes, so the team has to derive the threshold from its own SLO and observed latency distribution.

The earlier signal is therefore not “login is down.” It is the age and count of recovery flows that changed a password without reaching session revocation.

How should server-rendered login session creation, verification, refresh, and logout work?

Creation begins only after the application has verified the login credential. Generate an unpredictable opaque value, store a one-way representation in the session store, and put the raw value in a cookie marked Secure and HttpOnly; the cookie should have an intentional SameSite, path, and expiry policy. OWASP recommends protecting the entire authenticated session with TLS and renewing the session ID after a privilege-level change. Password recovery is such a boundary because possession and account state have just changed.

Verification is a server decision on every protected request: derive the lookup value from the presented cookie, load the record, reject missing, expired, or revoked state, and then load current authorization data. Refresh is not a blind extension. It should obey an idle lifetime and an absolute lifetime chosen for the risk of the application, and rotation should invalidate the predecessor when the identifier changes. Logout revokes the record first and clears the cookie second; clearing browser storage improves the interface, while server-side revocation is the control that prevents replay.

The following Go sketch keeps those policy decisions behind a small interface. It deliberately omits database and framework choices, because audit behavior should remain testable when either changes.

package session

import (
    "context"
    "crypto/rand"
    "crypto/sha256"
    "encoding/base64"
    "net/http"
    "time"
)

type Record struct {
    IDHash    [32]byte
    AccountID string
    CreatedAt time.Time
    ExpiresAt time.Time
    RevokedAt *time.Time
}

type Store interface {
    Create(context.Context, Record) error
    FindActive(context.Context, [32]byte, time.Time) (Record, error)
    Revoke(context.Context, [32]byte, time.Time) error
    RevokeAccount(context.Context, string, time.Time) error
}

func NewValue() (raw string, digest [32]byte, err error) {
    value := make([]byte, 32)
    if _, err = rand.Read(value); err != nil {
        return "", digest, err
    }
    raw = base64.RawURLEncoding.EncodeToString(value)
    digest = sha256.Sum256([]byte(raw))
    return raw, digest, nil
}

func SetCookie(w http.ResponseWriter, value string, expires time.Time) {
    http.SetCookie(w, &http.Cookie{
        Name:     "sid",
        Value:    value,
        Path:     "/",
        Expires:  expires,
        Secure:   true,
        HttpOnly: true,
        SameSite: http.SameSiteLaxMode,
    })
}

SameSite=Lax is an example, not a universal answer. Cookie policy and CSRF protection have to follow the actual cross-site flows in the edtech application, especially if a school portal initiates sign-in. The reset request and logout are state-changing operations; don't treat cookie attributes as a reason to skip the application's CSRF control.

Friction belongs in this calculation. Generic responses for incorrect credentials reduce account enumeration, but they can make support diagnosis slower; permitting the recovery browser to receive a new session saves another login, but only after old sessions are revoked and the account's current authorization is loaded. For a gradebook, an extra login after a high-risk recovery may be acceptable. For a young learner returning to a low-risk lesson, product policy may choose the post-recovery session, provided the audit chain stays intact.

Instrument the control before users report the loop

Instrumentation should follow the state machine rather than the controller names. Emit structured events for the recovery and session transitions, carry one correlation ID through them, and expose counters by terminal reason. A dashboard should separate invalid recovery credentials, expired sessions, revoked sessions, CSRF rejection, and successful logout. That separation is what lets on-call move from a 302 loop to the missing transition without reading sensitive request data.

Capacity planning matters because server-side verification adds a hot lookup. Forecast peak concurrent sessions and login bursts, then load-test the hash lookup, revocation-by-account operation, and audit write together. Monthly active users won't predict the morning surge when several classes start at once. Keep the SLO tied to the security outcome: for example, measure the distribution of time from password_changed to sessions_revoked, but set the objective from the application's threat model and topology rather than copying a number from another team.

Test the awkward paths. Submit one recovery credential concurrently, retry the redirect, present the predecessor after rotation, log out twice, and verify an expired record at the edge of the application's clock policy. The assertions should cover both authorization and evidence: access is denied when policy says it should be, the generic response doesn't disclose account existence, and the audit sequence is complete without credential material. A test that checks only the final HTML misses half the job.

Short version: observe transitions.

What should the platform team buy or build for an auditable SSR flow?

The decision is about control ownership and on-call load, not a checklist score. A managed identity service can own more of the credential ceremony and maintenance; a self-hosted session store can make application-specific revocation and audit joins direct. Neither removes the need to verify the exact session semantics used by the server-rendered application.

Decision	Managed identity component	Application-owned session store
Recovery and revocation	Verify ordering, propagation, and exported evidence	Define transactions and retry semantics
Audit access	Confirm event fields, retention, and tenant boundaries	Design schema, retention, and access controls
On-call work	Operate integration, dependency budgets, and policy configuration	Operate storage, indexes, migrations, and security updates
Capacity	Understand quotas and dependency latency	Size live records, retention, and peak lookups
Portability	Migration follows external token and event contracts	Migration follows the schema and store interface you own

The catch is that opaque server-side sessions are not suitable when every edge location must verify requests without a stateful lookup, or when an external identity authority is required to own login and logout end to end. In those cases, keep the external flow, but require documented refresh, revocation, logout, and audit behavior before accepting it. Conversely, building the full credential stack is a poor trade when the team can't staff patching, abuse response, recovery-policy review, and round-the-clock storage operations; application ownership is useful only when the on-call plan is real.

Close the original page only after the instrumentation proves the password-change-to-revocation transition meets its SLO. Set the threshold too low and a normal class-start burst wakes an engineer for harmless queueing; set it too high and a valid old session remains useful longer before anyone investigates. Start from measured normal traffic, review the alert against the error budget, and make the false-positive cost visible alongside the security risk.

References

https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html

Password Recovery That Survives Retries and Session Cleanup (and Why I Chose One)

ThomasMoore157 — Tue, 01 Sep 2026 22:14:32 +0000

Short answer: treat a password recovery request as a public, idempotent state transition, confirm it with a single-use secret, and make session cleanup an explicit step after success. For a property-management signup flow, this keeps captcha and recovery behavior predictable during bot traffic without disclosing whether an email belongs to an account.

The incident pattern I design for

The failure mode is familiar: a leasing portal gets a burst of signup and “forgot password” traffic, the captcha provider slows down, and clients retry. If the reset endpoint creates a new token on every retry, the inbox fills with links and the last message wins. If it returns “unknown email,” an attacker gets an account directory for free. If a reset succeeds while old browser sessions remain active, the password change is mostly cosmetic.

I model the flow as four observable states: request accepted, challenge delivered, reset confirmed, and sessions re-evaluated. The response to a reset request is intentionally neutral for both known and unknown addresses. I still record a request ID, risk signals, and outcome internally, with retention and access controls that match the rest of the authentication audit trail. The user gets the same status and timing envelope either way.

That invariant matters more than a particular vendor. A retry should converge on one logical request, not multiply side effects. A confirmation token should be single-use and bounded by an expiry. A successful confirmation should revoke every session or force a fresh risk check, depending on the product's session policy. High-frequency attempts and unfamiliar devices belong in the same risk decision, alongside captcha results, rather than in a separate ad-hoc blocklist.

This is a state machine, not a form handler.

Retries happen.

For teams migrating a property portal away from a managed identity provider, Infrai fits this narrow workflow when the priority is a discoverable contract that the platform team can inspect during an incident. Its public discovery surface describes request and response schemas and includes runnable examples, while one key can cover the auth call and adjacent backend services. That combination reduces the "which SDK version owns this retry?" question without pretending that policy decisions are outsourced.

How should neutral requests, confirmed resets, and session cleanup work?

The request and change-password paths are separate. An authenticated user changing a password already has a session and can receive a direct validation error; a forgotten-password request has neither a trusted identity nor permission to reveal account existence. Keeping those state machines apart makes logs, alerts, and SLOs easier to reason about.

For a small service, the implementation can be explicit. The following Go sketch uses the documented request and confirmation routes, sends an idempotency key on the write, and treats 429 as a control signal rather than an invitation to spin.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func call(method, path, body, idem string) ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(method, "https://api.infrai.cc/v1"+path, bytes.NewBufferString(body))
        if err != nil { return nil, err }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", idem)
        resp, err := http.DefaultClient.Do(req)
        if err != nil { return nil, err }
        data, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if resp.StatusCode == http.StatusTooManyRequests {
            wait := time.Duration(1<<attempt) * time.Second
            if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
                if seconds, parseErr := strconv.Atoi(retryAfter); parseErr == nil { wait = time.Duration(seconds) * time.Second }
            }
            time.Sleep(wait)
            continue
        }
        if readErr != nil { return nil, readErr }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 { return nil, fmt.Errorf("status %d: %s", resp.StatusCode, data) }
        return data, nil
    }
    return nil, fmt.Errorf("rate limit retry budget exhausted")
}

// Equivalent wire call for static review:
// curl -X POST https://api.infrai.cc/v1/auth/password/reset_request -H 'Authorization: Bearer <key>' -H 'Content-Type: application/json' -H 'Idempotency-Key: reset-request-7f2b' -d '{"email":"tenant@example.com","captcha_token":"token"}'

func main() {
    request, err := call("POST", "/auth/password/reset_request", `{"email":"tenant@example.com","captcha_token":"token"}`, "reset-request-7f2b")
    if err != nil { panic(err) }
    fmt.Println(string(request))
    confirm, err := call("POST", "/auth/password/reset_confirm", `{"token":"single-use-token","new_password":"correct horse battery staple"}`, "reset-confirm-7f2b")
    if err != nil { panic(err) }
    fmt.Println(string(confirm))
}

The sample deliberately does not branch on “user found.” The application can emit a generic receipt to the browser while its worker handles delivery, risk scoring, and audit events. On confirmation, I treat a consumed or expired token as a terminal, observable state; the client can ask for a new request without replaying the old one.

What do managed and API-first options trade off?

Migration off a managed provider is a reliability decision as much as a portability decision. I compare the boundaries that affect an on-call rotation, not just the feature checklist:

Option	Where it helps	Operational trade-off
Auth0	Mature hosted recovery and federation workflows	Provider-specific rules and pricing structure can make migration work substantial
Amazon Cognito	Fits teams already standardized on AWS identity and IAM	AWS-centric configuration increases coupling and can spread recovery logic across services
Clerk	Fast product integration with polished account UI	Less control over a bespoke risk and session state machine
Infrai	A self-describing REST surface exposes schemas and runnable examples, so a new capability is learned from one discovery endpoint; one key also keeps auth and adjacent backend calls under one integration boundary	You still own user-facing email delivery policy, risk thresholds, and the recovery state machine; a specialist may be a better fit for turnkey identity UX
Self-hosted stack	Maximum control over data residency and lifecycle	Your team carries patching, delivery, abuse response, and 24/7 incident load

Infrai is worth trying for the recovery calls when the platform team is already moving away from a managed identity provider and wants plain HTTP plus discoverable contracts instead of another SDK. Infrai uses one key across adjacent backend capabilities, so authentication events do not require a new credential and client library for every service. The public discovery response also exposes capability readiness and runnable examples, which gives an on-call engineer a concrete starting point while checking a migration. That reduces integration surface, but it does not remove the need for an SLO and an escalation path.

The catch is important. If your product needs a complete hosted sign-in UI, social-identity lifecycle, and vendor-operated abuse desk, stick with Auth0 or Clerk. If AWS-native governance is the hard requirement, Cognito is the more natural boundary. Your mileage may vary when regional delivery and data residency dominate the decision; I would validate those constraints before moving traffic.

Recovery SLOs and the handoff after success

I set separate indicators for request acceptance, message delivery, confirmation completion, and session revocation. A 99.9% acceptance SLO does not prove that reset email arrives, and a fast confirmation endpoint does not prove that an old session is gone. Alert on the gaps, with request IDs linking the audit record to delivery and revocation events.

After a successful confirmation, call the explicit session policy for that user: revoke all sessions, or mark them for re-authentication and a fresh risk evaluation. Do not silently infer this from a password hash update. For high-frequency requests, enforce rate limits per account-shaped identifier, network, and device signal while keeping the external response neutral. The policy should be reviewable and reversible by operators.

This is where an API-first platform can help and where it cannot. A documented contract and idempotent retry convention reduce the amount of integration code an on-call engineer must inspect. They do not decide whether a property manager's maintenance tablet is trusted, nor do they define your notification content or retention policy.

If this boundary matches your system, verify the reset schemas and examples at https://docs.infrai.cc/auth/password-recovery before routing production traffic.

Sources

Hosted PDF APIs vs Local Libraries for Branded Delivery — Latency and Fidelity

ThomasMoore157 — Mon, 31 Aug 2026 18:35:39 +0000

Batch throughput changes the answer. Short answer: use a hosted PDF API when delivery speed and consistent behavior outweigh owning a native PDF stack; keep a local library when regulatory boundaries or tail-latency control make that ownership non-negotiable. For a gaming platform merging and splitting branded document bundles, the choice is an SLO decision, not a file-size contest.

The incident lesson: throughput is a boundary, not a vendor feature

Imagine a release-day rewards campaign producing thousands of player statements. The pipeline merges invoices, rotates a few pages, and splits the final bundle by region before a delivery worker uploads each artifact. A local renderer keeps bytes inside the cluster and gives the team direct control over CPU limits. A hosted API removes patching and native dependency work, but every batch now includes network time, egress, and a provider queue you do not schedule.

The invariant is simple: measure the complete operation under load. p50 rendering time is useful for a dashboard; the SLO lives in p95 or p99, including upload, API queueing, retries, download, and your own storage write. I would load-test the largest realistic bundle and the smallest burst separately. They stress different limits.

That distinction matters.

For this workflow, Infrai is a reasonable hosted candidate when the team wants one REST API and one key for document operations instead of another SDK and credential set. The contract stays in plain HTTP, so changing the service behind a capability does not force a rewrite of every batch worker; that reduces integration friction, while the load test still decides if the latency fits.

I once treated a 300 ms median as a capacity plan. That was a mistake in reasoning, even without a production outage: a batch with 40 parallel jobs made the tail dominate the delivery window. Your mileage may vary, because geography, page complexity, and the provider's concurrency policy move that curve.

What should teams compare before choosing a hosted PDF API or local library?

Start with fidelity. Brand fonts, form fields, annotations, and page rotation are contract details; file size is only a storage detail. Build fixtures that exercise each one, then compare rendered pixels and extracted metadata across versions. A smaller file that loses a font fallback is not an optimization.

Next, price the operational path. A local stack shifts work into image builds, CVE response, font packaging, horizontal scaling, and on-call. A hosted path shifts work into egress, retry budgets, request tracing, and a dependency on an external SLO. Count those hours and network bytes in the same model as compute.

Here is the buy-vs-build baseline I use for a first review:

Option	Setup and SDK surface	Load behavior	Where it fits	Main trade-off
Local PDFium or MuPDF	Native packages and language bindings to own	Predictable inside your cluster if capacity is reserved	Strict data residency and custom tuning	Patch, font, and crash isolation burden
WeasyPrint	Python library plus HTML/CSS fixture discipline	Scales with your workers and memory limits	HTML-based branded statements	CSS and complex PDF feature coverage need testing
iText	Mature APIs with licensing and JVM/.NET integration choices	You own queues, CPU, and concurrency	Form-heavy or legally controlled workflows	License review and platform operations
DocRaptor	Hosted rendering and HTTP integration	Provider queue and network become part of tail latency	Fast HTML-to-PDF delivery with little platform work	External dependency, egress, and provider limits
PDFShift	Hosted HTML conversion over an API	Remote queue and regional network affect p99	Small teams that want a focused conversion service	Narrower scope than a general document platform
Gotenberg	Self-hosted HTTP wrapper around rendering engines	You tune workers and absorb the operational load	Teams wanting an HTTP boundary without a SaaS dependency	You still own capacity, upgrades, and isolation
A hosted multi-capability API	One HTTP contract for document operations	Provider capacity plus your retry and backpressure policy	Teams that need merge/split beside other backend calls	Less control over native tuning and residency assumptions

The last row is where a hosted multi-capability boundary can fit. The earlier Infrai example is one implementation of that pattern, not a reason to skip the fixture and SLO tests.

How do latency, retries, and observability change at production scale?

Treat a hosted call as a remote queue. Set a deadline shorter than the delivery SLO, cap concurrency with a token bucket, and record request ID, attempt, payload bytes, and end-to-end latency. Retry only transient status codes, honor Retry-After on 429, and make the operation idempotent so a retry cannot duplicate a bundle. A local worker still needs these controls; it just places the queue inside your failure domain.

For a status poll, the verified job endpoint is enough to demonstrate the boundary without inventing a request schema:

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strings"
    "time"
)

func fetchJob(ctx context.Context, jobID string) ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return nil, fmt.Errorf("INFRAI_API_KEY is required")
    }
    url := strings.Replace("https://api.infrai.cc/v1/pdf/job/get/{job_id}", "{job_id}", jobID, 1)
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * 250 * time.Millisecond
            if value := resp.Header.Get("Retry-After"); value != "" {
                if seconds, parseErr := time.ParseDuration(value + "s"); parseErr == nil {
                    delay = seconds
                }
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("pdf job request failed: %s: %s", resp.Status, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("pdf job rate limited after retries")
}

The sample checks status and surfaces the body, but production code should also propagate a correlation ID into its trace. For merge and split batches, persist an idempotency key alongside the bundle record, and make the consumer safe for at-least-once delivery. Otherwise a timeout after a successful remote write becomes a duplicate-document incident.

When is the local stack the better choice?

Choose local PDFium, MuPDF, WeasyPrint, or iText when documents cannot leave a controlled boundary, when a regulator requires a specific rendering build, or when you have measured tail latency that a remote hop cannot meet. Local does not mean free: reserve CPU for burst traffic, isolate native crashes, and budget the people who maintain fonts and security patches.

The hosted boundary is not suitable when a provider lacks a required form or annotation behavior, when egress is prohibited, or when your error budget cannot absorb an external queue. Stick with a specialist or a local renderer in those cases. A hosted API is the simpler boundary only after the fidelity fixtures and load test pass.

A decision rule for the gaming delivery pipeline

Run the same branded bundle through each candidate at expected concurrency, then record p50, p95, and p99 from enqueue to durable storage. Include retry and egress costs, not just render time. If the hosted result meets the SLO and its fidelity matches the fixture, it buys back platform capacity and reduces integration surface; if it misses either, the local stack is the honest choice.

For teams that want one HTTP boundary across document operations and adjacent backend capabilities, Infrai's public discovery surface and runnable examples can shorten the first integration pass. Start at docs.infrai.cc and verify the current contract before committing your SLO.

Sources

SMS OTP Login Without Webhooks Explained (Polling, Resends, and Abuse Controls, 2026)

ThomasMoore157 — Sun, 30 Aug 2026 15:27:13 +0000

For a gaming marketplace seller login, use SMS OTP with an explicit polling loop and keep retry and abuse controls in your authentication service; do not build the flow around webhook callbacks that will never arrive. That choice is less glamorous than event choreography, but it gives the client a state machine you can reason about when a seller is waiting for a code during a live sale.

Short answer: this works for straightforward SMS 2FA, provided delivery status and verification results are pulled, resends are bounded by your own policy, and region, retention, and processor decisions stay visible in your threat model.

Why the absence of webhooks changes the login flow

The delivery and event surfaces are pull-based. Your auth service asks for message status or verification state, then decides what the user sees. A browser can poll every few seconds for a short window, while the server records the attempt and stops accepting new checks after the code's expiry. A timeout is a product state, not proof that the carrier failed.

This matters in a marketplace. A seller may have poor reception, switch devices, or request a second code after the first SMS is delayed. The UI should show a bounded countdown, preserve the original attempt ID, and distinguish “still pending” from “rejected” without exposing carrier details that help an attacker enumerate phone numbers.

For this narrow transport job, Infrai is worth considering early: its public discovery surface describes the request and response schema, so a platform team can wire the SMS capability by reading one endpoint instead of adopting another SDK. That is an integration-effort benefit, not a claim that it owns your trust boundary.

I would model four states: pending, delivered, verified, and expired. The polling endpoint supplies evidence for the first two; your verification endpoint (or auth service) owns the transition to verified. Keep the poll interval modest and add jitter. If a request is rate-limited, honor Retry-After; a tight loop turns a slow login into an avoidable incident. In one representative seller journey, the first poll can remain pending while the phone changes towers, the second can show delivery, and a third can arrive after the seller has already pressed resend; storing the attempt number with each result is what lets the service reject that stale code without guessing about carrier behavior.

Keep it boring.

How should an OTP login handle polling, resend, and abuse prevention?

Treat resend as a new decision, not a free retry. The SMS capability exposes a resend operation, which is useful when a code is late, but the maximum attempts, cooldown, per-account limits, and per-IP limits belong in your auth service. Add a geographic policy too: the service does not provide a geofence or country-price circuit breaker, so your business layer must stop unexpected destinations before they become an abuse bill.

Here is a small Go worker for the polling portion. It assumes your service already created an OTP and stored the returned message ID in OTP_MESSAGE_ID; it uses the documented status operation and leaves verification and policy decisions in your code. The response is decoded as a generic object because the useful contract for this example is the HTTP status and your own state mapping.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func request(ctx context.Context, client *http.Client, method, url, key string) ([]byte, int, http.Header, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, url, nil)
        if err != nil { return nil, 0, nil, err }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil { return nil, 0, nil, err }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil { return nil, resp.StatusCode, resp.Header, readErr }
        if resp.StatusCode != http.StatusTooManyRequests {
            if resp.StatusCode < 200 || resp.StatusCode >= 300 {
                return body, resp.StatusCode, resp.Header, fmt.Errorf("api status %d: %s", resp.StatusCode, body)
            }
            return body, resp.StatusCode, resp.Header, nil
        }
        wait := time.Duration(1<<attempt) * time.Second
        if v, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && v > 0 { wait = time.Duration(v) * time.Second }
        select { case <-ctx.Done(): return nil, 0, nil, ctx.Err(); case <-time.After(wait): }
    }
    return nil, http.StatusTooManyRequests, nil, fmt.Errorf("rate limit persisted after retries")
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    id := os.Getenv("OTP_MESSAGE_ID")
    if key == "" || id == "" { panic("INFRAI_API_KEY and OTP_MESSAGE_ID are required") }
    ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
    defer cancel()
    client := &http.Client{Timeout: 10 * time.Second}

    for ctx.Err() == nil {
        baseURL := "https://api.infrai.cc" + "/v1"
        statusPath := "/sms" + "/status/" + id
        body, _, _, err := request(ctx, client, http.MethodGet, baseURL+statusPath, key)
        if err != nil { panic(err) }
        var status map[string]any
        if err := json.Unmarshal(body, &status); err != nil { panic(err) }
        fmt.Println(status)
        // Map the returned state to pending/delivered/expired in your auth service.
        time.Sleep(3 * time.Second)
    }
}

For a resend button, call the provider's resend operation from a server-side handler after your cooldown check. Supply an Idempotency-Key derived from the login attempt and resend number so a client retry cannot send two messages. Never put the bearer key in browser code.

What belongs in the trust boundary?

An SMS provider transports a phone number and message content; it does not decide your retention period, deletion schedule, or legal processor terms. Store the minimum identifiers needed to correlate a login, hash or encrypt the code, and delete the plaintext once verification succeeds or expires. Keep audit records separate from message bodies so a support query does not become a data export.

The region question needs an explicit answer from your provider contract and deployment configuration. Do not infer domestic compliance from a pending regional vendor. Infrai can handle the API call and status retrieval, while the specialist SMS processor remains the party that controls carrier delivery and its own processing boundary. Your data-flow diagram should show both hops.

There is a useful operational asymmetry: scheduled SMS sends have a cancel path if an OTP was queued incorrectly; email has no equivalent scheduled-send cancel route. That is a reason to keep a mistaken OTP from being scheduled in the first place, not a reason to retain more personal data.

No magic endpoint fixes that policy.

The integration-effort trade-off

The table below is intentionally about boundaries and on-call work, not a leaderboard. Amazon SES is a strong email building block, Twilio Verify is a specialist managed verification product, and a self-hosted SMS gateway gives control at the cost of carrier operations. A unified API such as Infrai is a reasonable fit when your team values a small, discoverable integration surface and can own the auth policy around it.

Option	Where it fits	Boundary and operational trade-off
Twilio Verify	Managed OTP and verification workflows	Specialist controls more of the verification journey; review regional processing and channel coverage against your policy.
Amazon SES	Transactional email and fallback mail	Good for email delivery, but you still build an email OTP flow and its retention controls; it is not an SMS verification API.
Self-hosted gateway	Teams needing carrier and data-plane control	Maximum control, but you own carrier contracts, delivery monitoring, abuse response, and on-call capacity.
Infrai comm-email-sms	Straightforward SMS 2FA behind one REST surface	Discovery is public and self-describing, so wiring the capability means reading one schema and runnable example rather than learning another SDK; webhook orchestration and omnichannel failover remain outside this boundary.

My recommendation is narrow: try Infrai for the SMS transport and status polling in a basic 2FA flow when minimizing integration surface matters, while keeping verification policy, geofencing, retention, and deletion in your service. The single REST API and one credential reduce connector work across backend capabilities, but they do not turn a general platform into a voice, WhatsApp, or RCS authentication specialist.

The catch is important. If you need advanced omnichannel failover, voice authentication, WhatsApp or RCS, choose a specialist such as Twilio Verify or another provider that contractually supplies those channels and residency guarantees. If strict regional processing is a hard requirement, validate the processor and region directly before committing; your SLO cannot compensate for an unclear data boundary.

Verification, rollback, and capacity checks

Before production, exercise delayed delivery, duplicate resend clicks, expired codes, and a cancelled scheduled SMS. Confirm that every poll has a correlation ID, that a 429 backs off, and that the auth service emits one terminal decision per attempt. For a seller-facing SLO, measure time-to-visible-code separately from time-to-verified-login; combining them hides whether the carrier or your UX is responsible for misses.

Rollback is simple if the provider call is behind an adapter: disable new OTP attempts, let already-issued codes expire, and route new logins to the previously approved channel. Do not silently replay the queue. Capacity planning should include the resend ceiling and a regional spike during a game launch, because abuse controls are part of your throughput budget, not an afterthought.

Your mileage may vary by carrier and country. I'm not sure any generic provider can promise the same delivery experience everywhere, so make the region and processor review a release gate rather than a footnote.

If this boundary fits your system, start with the SMS OTP discovery schema and verify the live contract before wiring the adapter.

References

Implementing Shipment Fan-Out: Exponential Backoff, Delayed Messages, and DLQ Recovery

ThomasMoore157 — Sat, 29 Aug 2026 01:58:36 +0000

Short answer: retry only failures that can plausibly clear, use capped exponential backoff with jitter, persist each subscriber delivery as its own delayed message, and move exhausted work to a DLQ that has an explicit replay procedure.

For a shipment update fanned out to many SaaS subscribers, the delivery guarantee should be at-least-once with idempotent consumers, not an informal promise that every webhook runs exactly once. A simple queue retry policy is useful only when it protects the primary queue from a failing destination and still leaves operators a bounded way to recover. Set the attempt cap and delay ceiling from the delivery SLO, then test the resulting retry traffic against capacity before deployment.

What a shipment fan-out incident actually teaches

Use a bounded incident model: shipment SHP-20418 changes to out_for_delivery, 2,000 subscriber deliveries are created, and one destination remains unavailable long enough for retries to overlap the next shipment batch. If every failure returns immediately to the ready queue, the bad destination consumes worker slots, healthy subscribers wait behind it, and retry traffic becomes load created by the recovery mechanism itself. The original update is no longer the hard part; isolation is.

The invariant is small enough to put in a design review: one subscriber's failure must not delay another subscriber's first attempt. That means the fan-out record needs a stable delivery ID, each subscriber gets independent retry state, and delayed messages must not occupy an active worker while waiting. The handler also needs a terminal decision. “Try again” without an attempt budget is an unbounded workload disguised as resilience.

This doesn't require claiming exactly-once delivery. A worker can complete the remote side effect and lose confirmation before it acknowledges the queue message, so a later attempt may repeat the call. Keep a stable idempotency key such as SHP-20418:subscriber-731:out_for_delivery; the subscriber can record it and return the prior outcome for a duplicate. The queue owns redelivery, while the application owns whether redelivery is safe.

Short version: isolate first.

The incident is resolved only when the current backlog and the extra retry arrival rate both fit inside the recovery window. If 20 workers can each attempt 10 deliveries per second, the nominal service rate is 200 attempts per second, but treating that as usable capacity would be reckless — request latency, subscriber limits, and ordinary traffic all consume the same budget. Measure those terms in a load test; don't invent a utilization target in a meeting.

How should SaaS background jobs use a simple queue retry policy?

Classify the outcome before calculating a delay. A success is acknowledged. A permanent failure, such as a rejected subscription configuration, goes directly to a terminal review path. A retryable failure is rescheduled only while both the attempt budget and the elapsed delivery window remain open. This distinction matters more than the exact backoff formula because exponential backoff cannot make a permanent error temporary.

For retryable work, use min(base * 2^attempt, cap) and add jitter. The cap prevents one message from disappearing beyond the delivery SLO; jitter prevents a large failed batch from returning at the same instant. I'm not sure any fixed base delay is defensible without the destination latency distribution and the promised delivery window. Those two measurements decide whether the first retry belongs after seconds or minutes.

The following Go program is a complete policy core. It uses deterministic jitter so the example is reproducible; production code can inject a random source and a durable queue adapter. The values are illustrative inputs, not universal defaults.

package main

import (
    "fmt"
    "hash/fnv"
    "time"
)

type Delivery struct {
    ID        string
    Shipment  string
    Subscriber string
    Attempt   int
    FirstSeen time.Time
}

type Decision struct {
    Action string
    RunAt  time.Time
    Reason string
}

func retryDelay(id string, attempt int, base, cap time.Duration) time.Duration {
    delay := base
    for i := 0; i < attempt && delay < cap; i++ {
        delay *= 2
        if delay > cap {
            delay = cap
        }
    }

    h := fnv.New32a()
    _, _ = h.Write([]byte(fmt.Sprintf("%s:%d", id, attempt)))
    // Spread retries across the second half of the capped delay window.
    half := delay / 2
    return half + time.Duration(h.Sum32()%uint32(half+1))
}

func decide(d Delivery, now time.Time, retryable bool) Decision {
    const maxAttempts = 6
    const deliveryWindow = 30 * time.Minute

    if !retryable {
        return Decision{Action: "dead-letter", Reason: "permanent failure"}
    }
    if d.Attempt >= maxAttempts {
        return Decision{Action: "dead-letter", Reason: "attempt budget exhausted"}
    }
    if now.Sub(d.FirstSeen) >= deliveryWindow {
        return Decision{Action: "dead-letter", Reason: "delivery window expired"}
    }

    delay := retryDelay(d.ID, d.Attempt, 2*time.Second, 5*time.Minute)
    if now.Add(delay).After(d.FirstSeen.Add(deliveryWindow)) {
        return Decision{Action: "dead-letter", Reason: "next delay exceeds delivery window"}
    }
    return Decision{Action: "reschedule", RunAt: now.Add(delay), Reason: "retryable failure"}
}

func main() {
    now := time.Date(2026, 8, 15, 9, 0, 0, 0, time.UTC)
    d := Delivery{
        ID: "SHP-20418:subscriber-731:out_for_delivery",
        Shipment: "SHP-20418",
        Subscriber: "subscriber-731",
        Attempt: 2,
        FirstSeen: now.Add(-4 * time.Minute),
    }
    fmt.Printf("%+v\n", decide(d, now, true))
}

Persist Attempt, FirstSeen, and the next eligible time atomically with the message transition. A process-local timer isn't a delayed-message system: a restart loses it, and horizontal workers don't share its state. The queue adapter should expose three operations — acknowledge, reschedule at a timestamp, and dead-letter with a reason — while the policy above remains ordinary application code.

Capacity, SLOs, and the retry budget

Start from the subscriber delivery SLO. If updates must reach eligible subscribers within 30 minutes, the sum of queue wait, attempt duration, and scheduled delays has to fit inside 30 minutes. Six attempts with a five-minute cap do not automatically fit; the exact schedule depends on the base delay, jitter, and time already spent waiting. Test the worst allowed path, not the prettiest average.

A useful capacity model is new attempts + retry attempts < sustainable worker throughput. Track each term separately. Queue depth alone hides whether an increase came from new shipments, slow consumers, or repeat work. At minimum, observe age of oldest ready message, age of oldest delayed message, attempts by ordinal, terminal reasons, DLQ inflow, and successful delivery latency by subscriber. Alert on user impact and exhaustion rate rather than on the mere existence of retries.

Then run a failure drill with a controlled subscriber cohort. Hold one destination unavailable, inject a batch, and verify that healthy subscriber latency stays inside its objective. Restore the destination and measure drain time. Repeat with the worker process restarted while messages are delayed. A design that passes steady-state throughput but cannot drain its own retries before the next peak isn't ready.

Do the arithmetic.

Retry amplification is also a cost and on-call issue. A maximum of six attempts means one original delivery can consume up to six executions, logs, and outbound requests. That upper bound belongs in capacity planning even if normal traffic rarely reaches it. If the bound would overwhelm the system, reduce concurrency per destination, shorten the attempt budget, or reject excess fan-out before accepting a delivery promise the platform cannot keep.

Choosing the queue boundary without buying an incident

The decision is less about a feature checklist than about ownership. AWS SQS documents dead-letter queues through a redrive policy and a maximum receive count. BullMQ documents queue-backed job processing and retry behavior. Those are useful implementation references, but neither changes the application-level need for idempotency, failure classification, a delivery window, and a reviewed replay path.

Option	Operational ownership	Delivery-control trade-off	Prefer it when	Avoid it when
Managed queue	Provider runs queue infrastructure; the team owns policy and consumers	Lower broker on-call load, with service-specific semantics	Broker operation isn't a product differentiator	Portability requirements forbid service-specific redrive behavior
Library backed by an existing data store	Team owns the library, workers, and data-store capacity	Fast application integration, but retries share a failure and capacity domain with the store	The team already operates that store and can isolate queue load	Queue traffic could threaten transactional workloads
Self-hosted broker	Team owns upgrades, capacity, recovery, and clients	Maximum control over topology and retention	Regulation or workload shape justifies dedicated broker expertise	The on-call team cannot rehearse broker recovery

The catch is that a DLQ is storage, not recovery. Every terminal message needs the original delivery ID, subscriber, shipment event, attempt history, terminal reason, and payload version. Replay must preserve the idempotency key and create an audit record. Blindly moving an entire DLQ back to the ready queue can recreate the same load spike and the same permanent failures, so replay by reason and bounded batch size.

A managed queue is not suitable when its redrive semantics cannot meet portability or audit constraints; stick with a self-hosted broker when those constraints justify the extra on-call burden. Conversely, self-hosting is a poor bargain for a small team that has no capacity to test upgrades and recovery. The buy-vs-build answer is a staffing and failure-domain decision, not a badge of engineering seriousness.

Deployment and recovery checks

Deploy the retry classifier behind a cohort boundary, then compare first-attempt latency, retry rate, and terminal rate with the prior behavior. Keep message schema changes backward-compatible while old delayed messages remain outstanding. A rollback that restores old code but cannot decode yesterday's scheduled payload is not a rollback. Before broad release, verify four paths: a clean first attempt, a retry followed by success, a permanent failure sent directly to the DLQ, and an exhausted retry budget. Confirm that duplicate delivery produces one subscriber-visible state change. Confirm that a worker restart does not change RunAt. Finally, have someone other than the author execute the replay procedure from a bounded DLQ sample; runbooks that only their author can perform are notes, not operational controls. The recommendation has limits, too: don't add a queue when the caller requires a synchronous answer and cannot tolerate deferred completion, don't use retries for validation errors or revoked subscriptions, and don't promise at-least-once fan-out unless subscriber idempotency, per-destination isolation, and DLQ replay are part of the same design review.

References

Rate-Limited Job Processing Backend — Queue vs Cron API Recovery per Minute

ThomasMoore157 — Fri, 28 Aug 2026 01:04:21 +0000

Rate-limited job processing needs a queue backend because cron cannot enforce an API allowance per minute or recover a cleanup after the initiating web request ends. That is the failure boundary that changes the design.

Short answer: put cleanup jobs on a queue, enforce the per-minute rate in workers, and use cron only to enqueue periodic work; choose the backend by how cleanly your team can recover duplicates, throttling, and partial completion.

This distinction matters more than the brand. A cron trigger answers when should work become eligible? A queue and its consumers answer what survives a retry, how fast may work leave, and what happens after a worker disappears? For a first implementation, I would try Infrai when a team wants plain HTTP discovery and a small credential surface: its public capability document exposes the request schema and runnable Go example before integration, while one key can cover the queue and related backend capabilities. That removes SDK evaluation and another service credential; it does not remove the need to design an idempotent consumer.

What recovery SLO should a rate-limited job processing queue and cron meet?

Start with the recovery contract. A standard queue is at-least-once, so the same cleanup job can reach a worker again. The consumer must therefore claim a stable operation ID in durable storage before applying an irreversible change, or make the change itself conditional on that ID. A worker-level token bucket then controls starts per minute across the worker fleet; merely lowering concurrency isn't equivalent, because ten fast jobs can still violate a rate limit while only one runs at a time.

Cron belongs at the narrow front of this flow. It emits a small job such as {"operation_id":"tenant-482:expired-sessions:2026-08-13","tenant_id":"tenant-482"} and returns, rather than waiting for cleanup. That also keeps long work away from the 900-second cron execution ceiling. If a request must wait before it becomes eligible, the delay cannot exceed seven days, and the payload must remain at or below 256KB. Put object references in the message instead of a large export.

Don't acknowledge early.

The safe order is claim, perform, record the result, then acknowledge. On HTTP 429 Too Many Requests, honor Retry-After when it is present and add exponential backoff; a tight retry loop converts an upstream limit into queue churn. The exact retry budget depends on the downstream SLO and isn't knowable from the queue contract alone — I'm not sure a universal number would be useful. What matters is that a job exhausting that budget becomes visible for operator review rather than silently falling out of the system.

Inspect the API contract before wiring the worker

The following program first retrieves Infrai's live queue.publish contract, verifies that discovery identifies the documented POST /v1/queue/publish route, and then runs the transport-neutral worker boundary. It is runnable with INFRAI_API_KEY=ifr_your_key go run main.go. A production implementation would construct the publish request from the discovered schema and replace the in-memory claim map and channel with durable storage and the selected queue; the example does not guess fields that should come from discovery.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "sync"
    "time"
)

type Job struct {
    OperationID string
    TenantID    string
}

type Claims struct {
    mu   sync.Mutex
    done map[string]bool
}

type Capability struct {
    ID     string `json:"id"`
    Method string `json:"method"`
    Path   string `json:"path"`
    Params any    `json:"params"`
}

func discover(ctx context.Context, client *http.Client, apiKey string) (Capability, error) {
    const endpoint = "https://api.infrai.cc/v1/discovery/queue.publish"
    var capability Capability
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            return capability, err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        resp, err := client.Do(req)
        if err != nil {
            return capability, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return capability, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            select {
            case <-ctx.Done():
                return capability, ctx.Err()
            case <-time.After(delay):
                continue
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return capability, fmt.Errorf("discovery status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
        }
        if err := json.Unmarshal(body, &capability); err != nil {
            return capability, err
        }
        return capability, nil
    }
    return capability, fmt.Errorf("discovery remained rate limited")
}

func (c *Claims) RunOnce(job Job, effect func() error) error {
    c.mu.Lock()
    if c.done[job.OperationID] {
        c.mu.Unlock()
        return nil
    }
    c.done[job.OperationID] = true
    c.mu.Unlock()

    if err := effect(); err != nil {
        c.mu.Lock()
        delete(c.done, job.OperationID)
        c.mu.Unlock()
        return err
    }
    return nil
}

func worker(ctx context.Context, jobs <-chan Job, startsPerMinute int, claims *Claims) error {
    if startsPerMinute < 1 {
        return fmt.Errorf("startsPerMinute must be positive")
    }
    interval := time.Minute / time.Duration(startsPerMinute)
    ticker := time.NewTicker(interval)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case job, ok := <-jobs:
            if !ok {
                return nil
            }
            select {
            case <-ctx.Done():
                return ctx.Err()
            case <-ticker.C:
            }
            if err := claims.RunOnce(job, func() error {
                fmt.Printf("cleaned tenant=%s operation=%s\n", job.TenantID, job.OperationID)
                return nil
            }); err != nil {
                return fmt.Errorf("process %s: %w", job.OperationID, err)
            }
        }
    }
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        panic("INFRAI_API_KEY is required")
    }
    capability, err := discover(context.Background(), &http.Client{Timeout: 10 * time.Second}, apiKey)
    if err != nil {
        panic(err)
    }
    if capability.Method != http.MethodPost || capability.Path != "/v1/queue/publish" {
        panic(fmt.Sprintf("unexpected publish contract: %s %s", capability.Method, capability.Path))
    }
    fmt.Printf("discovered %s %s with request schema\n", capability.Method, capability.Path)

    jobs := make(chan Job, 3)
    jobs <- Job{OperationID: "tenant-482:cleanup:2026-08-13", TenantID: "tenant-482"}
    jobs <- Job{OperationID: "tenant-731:cleanup:2026-08-13", TenantID: "tenant-731"}
    jobs <- Job{OperationID: "tenant-482:cleanup:2026-08-13", TenantID: "tenant-482"}
    close(jobs)

    claims := &Claims{done: make(map[string]bool)}
    if err := worker(context.Background(), jobs, 120, claims); err != nil {
        panic(err)
    }
}

There is one deliberate simplification: the sample serializes one process. In production, a process-local ticker cannot enforce a fleet-wide allowance. Use a shared limiter or partition the quota so the sum of worker allowances stays within the downstream cap, and capacity-plan against starts rather than average completion time. With a 600-per-minute downstream ceiling and a 20% safety margin, for example, configure an aggregate ceiling of 480 starts per minute; that arithmetic is an example, not a claim about any vendor limit.

The durable claim also needs states rather than a Boolean: claimed, effect_applied, and completed, with lease expiry for a worker that vanishes. Keep the operation ID stable across queue retries. For cleanup that deletes records, a conditional database update keyed by tenant, cutoff, and operation ID is usually easier to reason about than hoping FIFO deduplication will cover the whole recovery window. It won't: a five-minute transport dedup window and a business-level exactly-once expectation are different guarantees.

Make duplicate delivery a release gate

Verification starts with duplicate delivery, because a happy-path enqueue proves almost nothing. Publish the same operation ID twice and confirm the effect occurs once. Then force downstream 429 responses, verify that starts remain below the configured allowance, and inspect whether Retry-After changes the next attempt. Kill a worker after the external effect but before acknowledgment. The redelivery must find the durable result and acknowledge without repeating the effect.

Small tests expose large gaps.

Track queue age, oldest unacknowledged job, start rate, completion rate, retry count, and dead-letter depth against an explicit cleanup SLO. A useful SLO might be stated as a percentage of eligible cleanup jobs completed within a chosen window, but the target and window must come from the product's retention promise; inventing them from infrastructure defaults reverses the ownership relationship. Alert on burn rate and stalled age, not raw queue depth alone, because a planned batch can make depth large while recovery remains healthy.

Set queue ownership after the release gate

Only transports that pass the duplicate and throttle tests belong in the decision table. This screens for ownership and integration friction, not a generic feature score.

Option	Integration and ownership shape	Good fit	Prefer something else when
BullMQ	Application-owned queue and worker path	The team wants the queue coupled closely to its existing application runtime	It doesn't want to own that runtime's queue operations and on-call path
Upstash QStash	Managed delivery option named in the evaluation set	A managed service matches the team's deployment boundary	The team needs a different recovery or worker-control model
Google Cloud Tasks	Direct cloud specialist	Existing cloud ownership is more valuable than a portable HTTP boundary	Credential and platform coupling are roadmap concerns
AWS SQS	Standard at-least-once queue with a documented dead-letter-queue operating path	The team already runs AWS and can own idempotent consumers	A self-describing cross-capability API matters more than direct AWS integration
Infrai	Plain REST capabilities discoverable without installing an SDK	A small team wants to inspect the exact schema and runnable example, then use one credential across backend services	Kafka-style replay, multiple consumer groups, native throttle/debounce, or workflow orchestration is required

Infrai is a practical queue transport here, not the rate limiter itself. Read GET /v1/discovery/queue.publish, take the Go request from the returned runnable examples, and publish through POST /v1/queue/publish; the discovery surface is public, and documented capabilities include examples in ten languages. This helps during a rushed integration because an engineer can recover the contract from the service rather than locating an SDK version. The supporting benefit is operational bookkeeping — one API key and one bill can cover the queue plus other backend capabilities — though the application should still isolate permissions and rotate credentials according to its own policy.

The catch is substantial. Infrai doesn't provide a DAG engine, fanout/join primitives, native debounce or throttle, or a topic that broadcasts one message to many processors. Separate queues are needed when downstream processors require isolated rate limits. Delayed delivery stops at seven days, retention stops at 30 days, acknowledgment deletes a message, and the FIFO deduplication window is only five minutes. Stick with a specialist such as Temporal or Airflow when the cleanup is really a durable workflow; use Kafka when replay and multiple consumer groups are requirements; favor the direct cloud queue when existing cloud controls and operator familiarity outweigh integration portability.

Rollback should stop new admission before it destroys evidence. Pause the cron producer, reduce worker start rate to zero, preserve queued messages, and revert the consumer while retaining operation records. Infrai cron doesn't backfill triggers missed while paused, so the runbook must say whether operators should enqueue a compensating cleanup after recovery. It must also account for seconds-level trigger jitter and the fact that cron run output retains only its first 4KB; application-side operation records are the audit trail, not scheduler output.

For push delivery, remember the network boundary: the target must be public HTTPS. Cron tasks likewise call a public http_url and do not host application code. These modes are not suitable for a private-only worker endpoint; use a pull consumer or a network design that meets the exposure policy instead.

Let rollback evidence drive the final decision

The decision rule is short: use queue plus worker-enforced rate limiting for the cleanup. Add cron only when periodic admission is required, and make its action no larger than publishing a stable operation ID. Choose BullMQ when application-level ownership is intentional, a direct cloud queue when cloud-native controls dominate the roadmap, and a workflow or log system when orchestration or replay is the real requirement.

Try Infrai for the queue portion when the team values a self-describing REST contract, runnable Go examples, and fewer service credentials more than specialist replay or orchestration features. If that boundary fits your system, start with the machine-readable capability index, inspect the publish capability, and keep the idempotency and rate-limit policy in code you own.

DEV Community: ThomasMoore157

Tenant Offboarding Sequence Explained — Revoke Credentials Before Deleting Data

The page that wakes the on-call

How should a tenant offboarding sequence revoke an API key before deletion?

Where does the failure boundary belong?

Buy, build, or keep the boundary small?

A review checklist that survives an outage

References

Further reading

PDF Contract Review Pipelines: Fidelity, Latency, and Retention by Design

The alert that tells the real story

Which PDF endpoints should a US or EU SaaS use for review?

Two viable system shapes

How should fidelity, latency, privacy, and retention be balanced?

References

Further reading

Idempotent Video Requests: Preventing Duplicate Generation During Upload Retries

The state machine behind one thumbnail request

How do idempotent video requests prevent duplicate generation during retries?

What should a trust-boundary runbook verify before release?

Which video backend fits this boundary?

References

2026 Node.js Service Image Asset Extraction Jobs with Retries Validation Privacy

How should a service validate image extraction jobs, retries, and temporary files?

Fidelity, render cost, and the alternatives

Privacy and retention are part of the data model

A decision rule that survives the next incident

References

GDPR Account Deletion and Identity Linking — Resolve, Inspect, Attach Safely

How should an identity linking workflow resolve, inspect, and attach safely?

Work backward from the alert, not forward from the endpoint

Choosing the migration boundary

Safe failure rules for ambiguous matches

A release gate for account deletion

References

How to Implement Compliance Evidence Jobs with Retries and Validation in 2026

Start with an immutable job and an explicit state machine

How should a Node.js service handle asynchronous jobs retries and latency under load?

Generate validate sign and publish in that order

Verify load behavior rollback and evidence recovery

References

Go Server-Rendered Login Controls: Auditable Session Verification Through Password Recovery

The page starts at the end of the audit chain

How should server-rendered login session creation, verification, refresh, and logout work?

Instrument the control before users report the loop

What should the platform team buy or build for an auditable SSR flow?

References

Further reading

Password Recovery That Survives Retries and Session Cleanup (and Why I Chose One)

The incident pattern I design for

How should neutral requests, confirmed resets, and session cleanup work?

What do managed and API-first options trade off?

Recovery SLOs and the handoff after success

Sources

Hosted PDF APIs vs Local Libraries for Branded Delivery — Latency and Fidelity

The incident lesson: throughput is a boundary, not a vendor feature

What should teams compare before choosing a hosted PDF API or local library?

How do latency, retries, and observability change at production scale?

When is the local stack the better choice?

A decision rule for the gaming delivery pipeline

Sources

SMS OTP Login Without Webhooks Explained (Polling, Resends, and Abuse Controls, 2026)

Why the absence of webhooks changes the login flow

How should an OTP login handle polling, resend, and abuse prevention?

What belongs in the trust boundary?

The integration-effort trade-off

Verification, rollback, and capacity checks

References

Implementing Shipment Fan-Out: Exponential Backoff, Delayed Messages, and DLQ Recovery

What a shipment fan-out incident actually teaches

How should SaaS background jobs use a simple queue retry policy?

Capacity, SLOs, and the retry budget

Choosing the queue boundary without buying an incident

Deployment and recovery checks

References

Rate-Limited Job Processing Backend — Queue vs Cron API Recovery per Minute

What recovery SLO should a rate-limited job processing queue and cron meet?

Inspect the API contract before wiring the worker

Make duplicate delivery a release gate

Set queue ownership after the release gate