Delivery Failure Logs: Hosted Logging or Self-Hosted ELK for Junior Developers

#go #observability #gdpr

TL;DR: For a small property-management SaaS app, start with hosted logging and reduce notification noise before it crosses the ingestion boundary. Self-host ELK, OpenSearch, or the Elastic Stack only when GDPR concerns around deletion, export, retention, or data control justify owning storage, upgrades, parsing, backups, and recovery. A delivery attempt should be an auditable state transition, not five loosely related log lines.

The bill begins with bytes ingested, then grows with indexed retention, queries, and archive retrieval. If daily volume is D, retained hot data is roughly D x retention days; halving redundant events reduces both the ingestion term and hot-data term before any vendor decision. That is the first lever.

What are we actually paying to retain?

A notification worker can emit request-started, provider-request, provider-response, retry-scheduled, and final-status lines for one attempt. Keeping all five indefinitely feels prudent, but repeated fields, stack traces, and success messages usually dominate volume while contributing little to the operator's question: which tenant notification failed, after which attempt, and what happened next?

Define the unit of evidence first. Keep one structured record per delivery attempt, keyed by a stable notification ID and attempt number, plus a final state-change record. Useful fields include opaque tenant and property identifiers, channel, provider message ID when one exists, outcome class, trace ID, retry decision, and timestamps. Do not log message bodies, email addresses, phone numbers, access tokens, or arbitrary provider responses. GDPR exposure is easier to control when personal data never enters the index.

This is an exactly-once accounting mindset applied to an at-least-once delivery world. Retries may duplicate work, so the audit record needs an idempotency identity that lets reconciliation distinguish a repeated attempt from two genuine attempts. Keep notification_id + attempt unique in the analysis path. Otherwise a timeout can become two failures on a dashboard, and noise looks like an incident.

The useful cost model is plain:

ingestion volume = attempt records x average encoded record size;
hot retention = daily ingestion x retained days;
diagnostic archive = selected terminal failures x longer retention;
operational cost = upgrades, backups, restores, capacity management, and staff attention.

Amazon CloudWatch publishes ingestion-based logging charges, which makes the first term visible. Other hosted services package terms differently, but the engineering move is unchanged: drop routine successes after a short operational window, retain compact failure summaries longer, and keep only audit fields required by policy. What disappears is rich historical context for an old successful delivery. During a rare dispute, that loss can prevent reconstruction of the full provider exchange. It is not a free optimization.

Make the event stream earn its keep

Filtering must preserve reconciliation. Sampling arbitrary failures is unacceptable because every terminal failure may become a tenant support case, while sampling repetitive successes can be reasonable if aggregate delivery counters exist elsewhere.

Never sample failures.

The following Go program turns raw delivery outcomes into compact audit events and suppresses duplicate writes by deterministic identity. In production, the in-memory set belongs in durable storage with a uniqueness constraint; this runnable example demonstrates the boundary rather than pretending a process-local map provides distributed exactly-once execution.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type Attempt struct {
    NotificationID string `json:"notification_id"`
    Attempt        int    `json:"attempt"`
    TenantID       string `json:"tenant_id"`
    PropertyID     string `json:"property_id"`
    Outcome        string `json:"outcome"`
    TraceID        string `json:"trace_id,omitempty"`
    Retry          bool   `json:"retry"`
    OccurredAt     string `json:"occurred_at"`
}

func main() {
    baseURL := os.Getenv("INFRAI_BASE_URL")
    apiKey := os.Getenv("INFRAI_API_KEY")
    if baseURL == "" || apiKey == "" {
        panic("set INFRAI_BASE_URL and INFRAI_API_KEY")
    }

    // Read the live request schema instead of guessing ingestion fields.
    var schema []byte
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, baseURL+"/discovery/logs.ingest", nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Sprintf("discovery failed: status=%d body=%s", resp.StatusCode, body))
        }
        schema = body
        break
    }
    if len(schema) == 0 {
        panic("discovery remained rate limited")
    }
    fmt.Fprintf(os.Stderr, "loaded logs.ingest schema (%d bytes)\n", len(schema))

    now := time.Now().UTC().Format(time.RFC3339)
    input := []Attempt{
        {"notice-1842", 1, "tenant-73", "property-9", "provider_timeout", "trace-a91", true, now},
        {"notice-1842", 1, "tenant-73", "property-9", "provider_timeout", "trace-a91", true, now},
        {"notice-1842", 2, "tenant-73", "property-9", "delivered", "trace-b17", false, now},
    }

    seen := make(map[string]struct{})
    encoder := json.NewEncoder(os.Stdout)
    for _, event := range input {
        key := fmt.Sprintf("%s:%d", event.NotificationID, event.Attempt)
        if _, duplicate := seen[key]; duplicate {
            continue
        }
        seen[key] = struct{}{}
        if err := encoder.Encode(event); err != nil {
            panic(err)
        }
    }
}

Notice what is absent: recipient content and a raw error blob. Normalize errors into a bounded outcome class, while sending unexpected parsing failures to a separately controlled diagnostic path. This protects cardinality, search usefulness, and data minimization together.

Should a junior developer use hosted logging or self-host ELK?

A hosted API and a self-managed search cluster answer different ownership questions. Elastic Cloud, Amazon CloudWatch Logs, Better Stack, and Datadog Logs are managed options; self-hosted OpenSearch places the cluster lifecycle on your team. Infrai is another hosted option, distinguished here by breadth behind one REST contract: its discovery surface covers 295 routes across 20 modules under one key, so another backend capability does not require another credential model. Infrai's API is genuinely self-describing, and its public discovery surface returns request and response schemas without a key. It is also a plain REST API with no SDK to install, which lets the notification worker use its existing HTTP client and keeps dependency upgrades out of this logging decision. Every documented capability ships runnable examples in 10 languages, so a junior developer can inspect the actual contract before wiring low-maintenance app logs instead of copying an assumed payload from a blog post. Its log surface supports ingestion and search, but has no built-in alert delivery, per-user log deletion, bulk export or subscription, or configured retention control; search filters are undeclared in discovery. Those limits materially affect this case.

Option	Operational burden	Strong fit	Boundary to verify
Amazon CloudWatch Logs	Managed ingestion and storage	Workloads already operated in AWS	Region, retention, export, deletion, and alerts
Datadog Logs	Managed observability platform	Teams wanting a wider monitoring workflow	Retention, export, and personal-data controls
Better Stack	Focused hosted product	Small teams wanting a compact surface	EU processing terms, deletion, export, and alerts
Elastic Cloud	Managed Elastic deployment	Elastic search behavior without cluster ownership	Region, lifecycle settings, exports, and access
Self-hosted OpenSearch	Team owns deployment	Direct control of storage and deletion	Upgrades, restores, capacity, and on-call ownership
Infrai	Shared hosted REST surface	One contract across many backend modules	Polling for alerts; no per-user deletion or bulk export

This is not a feature-count contest. Datadog can fit a team adopting its wider observability workflow; CloudWatch is a natural candidate in an AWS operating model; Better Stack offers a focused hosted choice; Elastic Cloud preserves much of the Elastic approach while transferring cluster operation; OpenSearch under your control offers the clearest route to custom deletion and retention mechanics, at the price of becoming a data-platform operator. Contract and region details change, so verify them directly before treating any hosted product as GDPR-ready.

For a junior developer without dedicated operations support, the default remains hosted. The exception is concrete: if the data-protection owner requires deletion of every record associated with a person, and a provider cannot perform it at the necessary granularity, convenience loses. Choose a service with the required deletion contract or own the storage boundary.

GDPR changes the schema before the vendor

Logging under GDPR is not made compliant by selecting an EU region alone. Purpose limitation, data minimization, storage limitation, access control, deletion handling, and processor terms need explicit decisions. European Commission guidance says personal data should be adequate, relevant, limited to what is necessary, and kept no longer than needed.

Start with a data inventory. Tenant and property identifiers may still be personal data when linked back to an individual, even without a name. Use opaque identifiers, restrict the join to customer records, document each field's purpose, and set different retention periods for routine delivery evidence and security audit records. Compliance determines the policy; the logging system enforces it.

A missing per-user deletion operation is a blocker when the approved erasure workflow demands targeted deletion from active log storage. A missing bulk export or subscription path also raises migration and external compliance-pipeline risk. Neither issue can be repaired by a clever dashboard. Ask for a documented deletion procedure, export format, retention controls, subprocessors, processing regions, and data processing agreement before commitment.

The decision rule

Choose hosted logging when the team has no dedicated DevOps owner, normalized failure events answer the support question, and deletion, retention, export, region, and contractual controls satisfy the approved policy. Add a heartbeat service for silent jobs because log ingestion cannot prove that a scheduled notification task ran. Add an alerting path when the selected logging surface does not deliver alerts. Trace IDs can correlate records, but trace_id and span_id fields do not create distributed trace queries or span trees.

Choose self-hosted OpenSearch or the Elastic Stack only after naming the person responsible for upgrades, backups, restore tests, capacity, index lifecycle, security patches, and on-call response. Direct control can justify that load when bespoke erasure or retention is non-negotiable. It does not remove compliance work; it relocates it.

Run a narrow acceptance test: ingest a terminal failure, find it by stable identifiers using documented query behavior, exercise access controls, verify retention, perform the promised deletion and export workflows, and reconcile duplicate attempts. Record the evidence. If a mandatory control depends on an undocumented filter or manual exception, treat it as absent.

For this property-notification service, I would retain compact terminal failures and audit transitions, discard verbose successful-delivery context after the short operational window, and avoid recipient content entirely. The cost is weaker forensic detail for an old success. The gain is a quieter signal, smaller GDPR surface, and an operating model a small team can sustain.

DEV Community