FrostY45

Posted on Oct 9

Cost Attribution in Hosted Application Logging for Postgres SaaS API Workers Explained

#observability #sre #logging

Short answer: choose hosted application logging by asking whether one failed customer-support notification can be reconstructed and charged to the right tenant, service, and retention class. For a European Postgres SaaS with API processes, queue workers, and cron jobs, searchable ingestion is only the entry requirement. The consequential trade-off is how much evidence you retain, which dimensions remain safe to index, and whether retry attempts can be joined without turning customer IDs into an unbounded index.

I have been paged for missed jobs and duplicate deliveries. In that moment, a cheap-looking log destination is irrelevant if the API acceptance, worker attempt, provider response, and scheduled reconciliation cannot be connected. The operational rule I took from those incidents is blunt: preserve a small, stable event envelope everywhere, keep high-cardinality values searchable as fields rather than metric labels, and assign every byte to an owner before setting retention.

How should a Postgres SaaS evaluate cheap hosted application logging?

Start from the question the on-call engineer must answer: did the notification fail before enqueueing, while a worker claimed it, at the external delivery boundary, or during a retry? A single error line cannot establish that sequence. Each component should emit an event with the same delivery_id, plus a fresh attempt_id for every try.

For a customer-support system, I would keep tenant_id, channel, component, outcome, and retention_class as structured fields. The message body, recipient address, access token, and raw support conversation do not belong in general application logs. They expand exposure without improving the usual incident query. Redact them at the source, before any network hop.

The database is part of the evidence chain, but its full query text usually is not. Log a stable operation name such as claim_delivery, the result, duration, and a trace or delivery identifier. This lets an engineer distinguish an empty queue from a failed claim without recording customer content or creating a different index key for every SQL statement.

Logs are evidence.

Here is a compact event shape. The same JSON contract can be emitted by Node.js API and worker processes; the Go example makes the validation path explicit.

package deliverylog

import (
    "encoding/json"
    "errors"
    "io"
    "time"
)

type Event struct {
    Timestamp      time.Time `json:"timestamp"`
    Service        string    `json:"service"`
    Component      string    `json:"component"`
    TenantID       string    `json:"tenant_id"`
    DeliveryID     string    `json:"delivery_id"`
    AttemptID      string    `json:"attempt_id,omitempty"`
    Channel        string    `json:"channel"`
    Outcome        string    `json:"outcome"`
    ErrorClass     string    `json:"error_class,omitempty"`
    RetentionClass string    `json:"retention_class"`
}

func WriteEvent(w io.Writer, e Event) error {
    if e.Timestamp.IsZero() || e.Service == "" || e.TenantID == "" ||
        e.DeliveryID == "" || e.Outcome == "" || e.RetentionClass == "" {
        return errors.New("incomplete delivery log event")
    }
    return json.NewEncoder(w).Encode(e)
}

Do not use the presence of a log line as the idempotency mechanism. The worker should enforce idempotency in durable state, then record the decision it made. Logs explain the transition; they do not own it.

Build the incident timeline before comparing plans

A useful evaluation starts with four event classes, not a vendor feature grid. The API records acceptance or rejection. A worker records claim, attempt, and durable outcome. The delivery boundary records a normalized response class rather than a secret-bearing payload. A cron reconciliation records what it inspected and which stale items it safely requeued.

That creates a query path such as delivery_id -> attempts -> outcome. It also exposes gaps. If API logs use a request ID, workers use a queue message ID, and cron uses only a timestamp, search cannot manufacture causality later. Propagate the delivery identifier through the queue payload and make the attempt identifier unique.

I initially care less about dashboards here than a replayable incident worksheet. Select a known synthetic delivery, induce a retryable failure in a test environment, and verify that the search results show acceptance, the first failed attempt, the retry decision, and exactly one terminal state. Then repeat with a deliberately duplicated queue message. The durable idempotency check should reject the duplicate, and the event should say so.

Keep the test bounded. No real recipient data should be necessary.

Attribute storage before tuning retention

Hosted logging cost is driven by more than the advertised ingestion unit. Retained volume, indexed fields, query frequency, regional placement, archive or export, and alert execution can all affect the bill or the operational burden. Pricing changes, so compare candidates with your own measured event distribution rather than a copied monthly estimate.

Measure first.

Use an allocation key that survives organizational changes. tenant_id helps explain customer-driven volume, while service and component identify the engineering owner. retention_class makes policy visible. None should be accepted blindly: tenant identifiers are high cardinality, and placing them in metric labels can create an expensive number of time series. Prometheus instrumentation guidance explicitly warns against labels with unbounded cardinality. Keep aggregate failure metrics on bounded dimensions such as component, channel, and outcome; investigate individual deliveries in logs or traces.

A one-day measurement can be misleading when weekly reconciliation runs or incident retries are bursty. Capture at least one complete scheduling cycle, then calculate volume by owner and class. A simple ledger is enough:

Allocation view	Field	Decision it supports
Customer demand	`tenant_id`	Which accounts generate investigation volume?
Runtime owner	`service` and `component`	Which team can reduce noisy events?
Evidence value	`retention_class`	Which events need longer availability?
Failure shape	`error_class` and `outcome`	Is a bounded aggregate alert possible?

Do not turn that table into a license to index every value. A candidate must demonstrate whether high-cardinality fields are stored, indexed, sampled, or scanned at query time. Those choices change both search latency and cost attribution. Test them with the same event corpus.

I choose a small searchable envelope over indexing every available attribute because incident correlation needs stable identifiers, while unrestricted indexing makes ownership and cardinality harder to control. Being paged for both a missed job and a duplicate delivery changed that priority: the useful record is the one that distinguishes no attempt, a failed attempt, and an idempotently rejected repeat. More fields do not repair a missing transition.

Retention should follow evidentiary value, not logger severity. A routine success may be useful only for a short reconciliation window, while a terminal failure or idempotency rejection may need longer retention for support review. The exact periods depend on contractual, security, and legal requirements; there is no universal number to copy.

Compare operating boundaries, not headline prices

Three deployment patterns are worth separating. A fully managed regional service reduces the system your team operates, but you must verify data location, deletion behavior, export, access control, and failure handling. A managed index with a separate shipper gives more control over buffering and routing, while leaving your team responsible for delivery semantics between those parts. A self-hosted stack offers the widest infrastructure control and also puts upgrades, capacity, backups, and query availability on the same on-call rotation.

The cheapest boundary is the one your team can operate during a logging outage. Ask what happens when the destination is slow or unavailable. Application requests must not wait indefinitely for log export. Workers need bounded buffering, explicit drop accounting, and backpressure behavior that cannot exhaust memory or disk. The service should continue its primary job according to a documented policy, while a metric reports lost or delayed telemetry.

For Europe-region workloads, verify the region of ingestion, indexing, durable storage, backups, and support access separately. A region selector on an endpoint does not by itself answer the whole data-flow question. Obtain the current contractual and architectural evidence from each candidate, because these details can change.

Use a scorecard with pass/fail gates before subjective scoring:

Can an engineer find all events for one synthetic delivery_id across API, worker, and cron components?
Can access be restricted so support and engineering roles see only the fields they need?
Can the system prove deletion and retention behavior for each class?
Does an exporter outage have a bounded effect on application memory, disk, and latency?
Can usage be reported by service, component, tenant, and retention class without exposing message content?
Can raw evidence be exported in a documented format without losing timestamps and identifiers?

Run those checks against production-shaped data. Ten hand-written lines prove almost nothing about cardinality, bursts, multiline handling, or query behavior.

Put failure handling in the write path

A logger call should be small, but the policy around it needs teeth. The following Go wrapper enforces an allowlisted envelope and records export failure through a bounded counter interface. In a Node.js implementation, keep the same contract and lifecycle rules: serialize structured JSON, flush on graceful shutdown with a deadline, and never retry forever inside the request or job path.

package deliverylog

import (
    "context"
    "errors"
    "time"
)

type Exporter interface {
    Export(context.Context, Event) error
}

type Counter interface {
    Inc(component, reason string)
}

type Recorder struct {
    Exporter Exporter
    Dropped  Counter
    Timeout  time.Duration
}

func (r Recorder) Record(parent context.Context, e Event) error {
    if r.Exporter == nil || r.Dropped == nil || r.Timeout <= 0 {
        return errors.New("invalid recorder configuration")
    }

    ctx, cancel := context.WithTimeout(parent, r.Timeout)
    defer cancel()

    if err := r.Exporter.Export(ctx, e); err != nil {
        r.Dropped.Inc(e.Component, "export_failed")
        return err
    }
    return nil
}

The counter labels are deliberately bounded. delivery_id and tenant_id stay out of metrics. The caller decides, by operation, whether an export error should be returned, buffered, or tolerated; that decision must be documented and tested because an API request, a queue worker, and a cron reconciliation have different failure budgets.

Deploy the schema before depending on it. First add the fields, then verify ingestion and redaction in a non-production environment. Next ship queries and alerts that tolerate both old and new events. Only after all producers emit the contract should you make the new field mandatory. This sequence keeps a rolling deployment from creating a false observability incident.

There are conditions where this design is insufficient. Regulated audit records may require an append-only audit system with controls beyond operational logging. Very high-volume event streams may need sampling for diagnostic successes, provided failures and idempotency decisions remain available under the approved policy. Low-volume internal tools may not justify tenant-level allocation at all.

The decision is ready when the team can reproduce a notification failure, identify its owner, state what was dropped, and explain the retention charge without reading customer content. Buy or operate only after that drill passes. Search is a feature; accountable evidence is the system.

DEV Community