How to Build Invoice Processing — Async Jobs, Retries, Validation, and Secure Files

#invoicing #asyncjobs #reliability

To implement invoice processing in a Node.js service, keep the request small and move asynchronous work to idempotent jobs; retries, validation, and secure temporary files then become explicit controls for latency under load. A pipeline can be correct and still miss its latency target when parsing, remote calls, and file handling stay inside the request that created the invoice.

Short answer: accept a small, validated command, enqueue an idempotent job, and keep temporary files encrypted, bounded, and short-lived; measure queue wait separately from processing time so load-related latency is visible.

The page that arrives after the damage

The on-call page rarely says “invoice latency is high.” It says the API's p95 crossed 800 ms, workers are busy, and a customer has two receipts for one invoice. By then, the original request has disappeared from the trace. A queue-depth chart may look normal because consumers are pulling messages quickly while each message waits on a slow tax or document service.

The first useful split is request_accept_ms, queue_wait_ms, work_ms, and result_persist_ms. A trace should carry an invoice ID and an idempotency key from the HTTP handler through the worker. Alert on queue wait and end-to-end age independently. A low queue depth with old jobs points to a stuck or slow consumer, not a healthy system.

One failed deployment taught me to inspect the payload before inspecting the broker. A 14 MB scanned invoice had been copied three times: request buffer, retry payload, and a temporary conversion file. Memory pressure caused longer garbage-collection pauses, which looked like downstream latency. The fix was a byte limit at ingress and a file-backed stream with explicit cleanup. In the postmortem, the queue dashboard showed consumers pulling messages at a normal rate, so the team initially increased concurrency. That made the host swap harder and extended every request. Reading the trace by phase exposed the real sequence: upload buffering consumed memory, conversion opened a second copy, and a retry serialized the bytes once more. After switching to bounded streaming, recording file size, and rejecting oversized input before enqueue, the worker had a stable memory ceiling. The lesson is specific: queue throughput is not a latency budget, and a payload that fits in a happy-path test can still dominate the system when ten workers process it together.

Measure twice.

That distinction matters under load. A retry that starts before the first attempt has finished is a duplicate delivery, not resilience.

What should an async invoice service validate before work starts?

Validation belongs at two boundaries. The API validates shape and authorization before enqueueing; the worker validates again because queues are durable data and schemas evolve. Reject unknown currency codes, impossible totals, expired upload references, and timestamps outside an agreed clock-skew window. Store a normalized command rather than the original untrusted JSON.

Use an idempotency record keyed by (tenant_id, idempotency_key). Its state can be accepted, processing, succeeded, or failed_permanent. A unique constraint makes the first write win. On a duplicate request, return the existing status and never enqueue a second job.

Here is a compact worker loop. It uses a generic queue interface so the same decisions apply whether the service is written in Node.js, Go, or another language.

package invoice

import (
    "context"
    "errors"
    "time"
)

type Job struct {
    TenantID string
    InvoiceID string
    Attempt int
    Payload []byte
}

type Queue interface {
    Receive(context.Context) (Job, error)
    Ack(context.Context, Job) error
    Retry(context.Context, Job, time.Duration) error
}

func Consume(ctx context.Context, q Queue, process func(context.Context, Job) error) error {
    for {
        job, err := q.Receive(ctx)
        if err != nil {
            return err
        }
        err = process(ctx, job)
        switch {
        case err == nil:
            if ackErr := q.Ack(ctx, job); ackErr != nil {
                return ackErr
            }
        case errors.Is(err, context.Canceled):
            return err
        case job.Attempt >= 5:
            // Persist a permanent failure and acknowledge the message.
            if ackErr := q.Ack(ctx, job); ackErr != nil {
                return ackErr
            }
        default:
            backoff := time.Duration(1<<job.Attempt) * time.Second
            if retryErr := q.Retry(ctx, job, backoff); retryErr != nil {
                return retryErr
            }
        }
    }
}

The retry count is a policy, not a magic number. Classify errors first: malformed input is permanent, a timeout is transient, and an authorization failure needs an operator or customer action. Add jitter to backoff so thousands of invoices do not retry on the same second. The worker must be safe to run twice because a process can die after committing a result but before acknowledging the queue.

How do retries, validation, and temporary files affect latency under load?

Measure the critical path with histograms, not one average. Record payload size, attempt number, queue wait, downstream wait, temporary-file bytes, and cleanup duration. Keep high-cardinality values such as invoice IDs in logs or exemplars, not metric labels. A useful alert combines age and error rate: old work with few errors often means saturation; high errors with young work often means a dependency or schema problem.

Temporary files need an owner and a deadline. Create them in a dedicated directory with restrictive permissions, stream input to a maximum size, and delete them in a defer block after the durable result is committed. Encrypt at rest when the host or volume is not already covered by an equivalent control. Never put a path supplied by a client into a filesystem call; generate the name server-side and keep the original name as metadata only.

import (
    "context"
    "fmt"
    "io"
    "os"
)

func withTempFile(ctx context.Context, dir string, input io.Reader, limit int64, use func(string) error) error {
    f, err := os.CreateTemp(dir, "invoice-*")
    if err != nil {
        return err
    }
    name := f.Name()
    defer os.Remove(name)
    defer f.Close()

    if _, err := io.Copy(f, io.LimitReader(input, limit+1)); err != nil {
        return err
    }
    if info, err := f.Stat(); err != nil || info.Size() > limit {
        return fmt.Errorf("invoice exceeds byte limit")
    }
    if err := f.Close(); err != nil {
        return err
    }
    return use(name)
}

The example assumes the caller has already selected a private directory and an encryption policy; those are deployment decisions, not properties of a queue. For browser uploads, the Blob API describes how bytes are represented, but the server still needs content-type sniffing, size limits, and malware scanning.

Choosing template ownership without hiding operational cost

For server-side signing, template ownership is the decision that changes your blast radius. If your team owns the template, store a versioned immutable artifact and require an explicit migration for every field change. You control rendering, tests, and retention, but you also own legal review and the rendering runtime. If a signing provider owns the template, integration is quicker, while schema drift, provider-specific fields, and export portability become constraints.

Write the choice down in a small decision record. Include who can publish a template, how a signed document is reproduced, which audit events are retained, and what happens when a template is retired. Test a golden invoice for every version; compare the hash of the rendered bytes, not a screenshot.

The catch is operational fit. Provider-owned templates are not suitable when you need offline rendering or strict portability. Keep ownership in your service when a regulated audit requires deterministic reproduction; choose a managed template system when legal authors need a safe editor and your team accepts its lifecycle rules. Either path still needs your own idempotency key and audit event store.

A runbook for the next load test

Start with a fixed invoice corpus: tiny, typical, and near the byte limit. Drive traffic until queue wait reaches the agreed budget, then inspect worker CPU, memory, downstream concurrency, and temporary-directory usage. Kill a worker after it persists a result and verify that a redelivery does not create a second signature. Advance the clock to test expiry and retry jitter.

Keep the runbook close to the alert. It should name the dashboard, the idempotency lookup, the dead-letter procedure, and the cleanup command. I am not sure a single latency SLO fits every tenant; your mileage may vary when large PDFs or cross-region signing are part of the contract. Split SLOs by workload class before raising worker counts.

DEV Community

How to Build Invoice Processing — Async Jobs, Retries, Validation, and Secure Files

The page that arrives after the damage

What should an async invoice service validate before work starts?

How do retries, validation, and temporary files affect latency under load?

Choosing template ownership without hiding operational cost

A runbook for the next load test

References

Further reading

Top comments (0)