DEV Community

ZebedeeHolloway9023
ZebedeeHolloway9023

Posted on

Managed Log Retention Over Self Hosted Indexing for GDPR User Data Deletion

#observability #gdpr #logging

TL;DR: For a nightly logistics pipeline, choose managed log search when rapid incident reconstruction and delegated operations matter most; choose self-hosted indexing only when the team can own deletion, lifecycle, and access controls as production systems. In either case, the decisive GDPR control is earlier: keep names, email addresses, street addresses, authentication tokens, and free-form request bodies out of operational logs, use opaque identifiers, and set a short retention expectation. A search index is not a user-data store.

The architecture decision is therefore conditional but not neutral. I would select managed search for a small EU/US SaaS team reconstructing nightly imports, provided its verified lifecycle controls match the retention policy. I would reject any service that requires per-user erasure but cannot demonstrate it, and I would move to a controlled index only when that capability is an explicit requirement the team is prepared to operate.

Decision record and invariants

The system imports shipment updates once each night. Operators must be able to answer which run started, which stage accepted an item, which operation retried, and whether the final counts reconciled. They do not need to discover who received the shipment. That distinction defines the log schema more usefully than a generic instruction to avoid sensitive data.

Four invariants follow.

  1. Every event carries opaque run_id and operation_id values, never a name, email address, postal address, or authentication token.
  2. A retry retains its operation identifier and increments attempt, so duplicate observations cannot become duplicate business facts.
  3. Reconciliation uses stage transitions, bounded reason codes, and aggregate counts; it does not copy shipment records or carrier payloads into the index.
  4. Audit and business records live outside operational logs, with their own lawful basis, access policy, and retention schedule.

The ingestion boundary is the principal failure boundary. Query-time masking is too late because the original value has already crossed into storage, backups, replicas, and operator-accessible search. Hashing can help in narrowly defined correlation work, but it is not automatic anonymization: a stable hash may remain linkable to a person. An opaque random identifier, backed by a separately controlled mapping only when that mapping is necessary, creates a clearer boundary.

This is an exactly-once mindset applied to evidence rather than delivery. The pipeline may emit the same attempt twice. Reconstruction deduplicates on a stable event key, preserves distinct attempts, and never mistakes repeated telemetry for another committed shipment update.

How should GDPR log retention handle user data deletion?

Suppose one run receives 84,216 update records, reports 317 retry attempts, and closes with a reconciliation difference of 12. These are design inputs for the example, not benchmark results. An investigator needs timestamps, run and stage identifiers, a stable operation identifier, attempt numbers, bounded outcomes, and input/output counts. Twelve is actionable. A consignee's email is not.

Free-form error text is the most likely route around an otherwise disciplined schema. During an investigation, a convenient message starts carrying a serialized request; later it holds nested carrier responses and addresses. Selective erasure then becomes an uncertain search across strings, escaped JSON, and old index generations. Prefer a code such as TRANSIENT_TIMEOUT or SCHEMA_REJECTED, plus an opaque operation reference that leads an authorized workflow to the authoritative business record.

Keep it boring.

GDPR does not assign one universal lifetime to application logs. Article 5 requires data minimization and storage limitation, while Article 17 defines the right to erasure and its exceptions. Engineering should document how long nightly failures ordinarily remain discoverable, retain searchable operational evidence only for that purpose, verify expiry behavior, and test deletion against nested and derived fields. Counsel still has to determine the lawful basis, the actual schedule, and any applicable exception. A logger cannot make that determination.

Managed search or a controlled index

The products below can all receive privacy-conscious events. None makes unrestricted ingestion safe, and the meaningful difference is who owns the lifecycle and what erasure controls can be verified.

Option Control surface relevant to this decision Reconstruction fit Boundary that changes the choice
Datadog Logs Managed indexes expose exclusion filters and index-level retention configuration Strong when a small team values integrated search without operating an index cluster Verify retention and access settings; filtering after sensitive data is emitted does not repair the producer schema
Elastic Stack Ingest pipelines can transform documents, index lifecycle management can expire indices, and Delete by Query can target stored documents Strong when flexible mappings and direct lifecycle ownership justify the operational burden A deletion query needs a reliably indexed subject key and still does not excuse unstructured personal data
Grafana Loki Retention is configured by the operator, while its label model rewards a small set of bounded dimensions Strong when the team already operates a Grafana logging estate and primarily reconstructs by run and stage User identifiers are a poor labeling strategy; retention and deletion remain operator responsibilities
Infrai Public discovery returns full request and response schemas, billing information, and runnable examples; documented capabilities include examples in 10 languages under a plain REST interface Reasonable for a narrow Go worker that benefits from inspecting an ingestion contract without adding an SDK Do not choose it where per-user log deletion, configurable retention or cold storage, bulk export, subscriptions, native alerts, trace-tree queries, symbolication, session replay, or heartbeat monitoring are requirements

For this pipeline, Datadog is the managed default when the team wants the shortest path to searchable operational evidence and its configured retention has been validated. Elastic is the stronger candidate when selective document operations and explicit lifecycle ownership justify staffing the index as a real production dependency. Loki fits an existing Grafana operation particularly well when run_id, stage, and a few bounded dimensions drive reconstruction.

The fourth option has a narrower boundary. Infrai provides one self-describing REST API under one key, callable over plain HTTP from any language or runtime without installing an SDK; public discovery exposes 295 capabilities across 20 modules, with runnable examples in 10 languages. A Go pipeline worker can therefore read the ingestion contract before wiring it, which reduces dependency and contract-discovery friction, while the shared credential simplifies key rotation. It does not erase the privacy limitation: there is no log deletion interface keyed by user, no bulk export or subscription interface, and no clear configuration entrypoint for retention or cold storage. Log-search filter parameters are undeclared as well, so an incident procedure must not depend on assumed filters.

That boundary is decisive. If an erasure request must locate and remove a person's log events, require a documented, tested mechanism before adoption. If the producer schema makes personal identification impossible and short-lived operational search is enough, integration simplicity can legitimately carry more weight.

Critical path in Go

The critical path is the typed event boundary. The following complete program validates a compact pipeline event, rejects accidental identity-bearing keys in diagnostic attributes, derives a deterministic event identifier, and emits newline-delimited JSON for a collector. It deliberately does not call a vendor route because transport details are secondary to the invariant being enforced.

package main

import (
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type PipelineEvent struct {
    EventID      string            `json:"event_id"`
    RunID        string            `json:"run_id"`
    Stage        string            `json:"stage"`
    OperationID  string            `json:"operation_id"`
    Attempt      int               `json:"attempt"`
    ResultCode   string            `json:"result_code"`
    InputCount   int               `json:"input_count"`
    OutputCount  int               `json:"output_count"`
    OccurredAt   time.Time         `json:"occurred_at"`
    Diagnostics  map[string]string `json:"diagnostics,omitempty"`
}

var forbiddenKeys = map[string]struct{}{
    "name": {}, "email": {}, "address": {}, "auth_token": {},
    "request_body": {}, "response_body": {},
}

func (e *PipelineEvent) validate() error {
    if e.RunID == "" || e.Stage == "" || e.OperationID == "" {
        return errors.New("run_id, stage, and operation_id are required")
    }
    if e.Attempt < 1 || e.InputCount < 0 || e.OutputCount < 0 {
        return errors.New("attempt must be positive and counts cannot be negative")
    }
    for key := range e.Diagnostics {
        if _, forbidden := forbiddenKeys[strings.ToLower(key)]; forbidden {
            return fmt.Errorf("diagnostic field %q is forbidden", key)
        }
    }

    identity := fmt.Sprintf("%s:%s:%d:%s", e.RunID, e.OperationID, e.Attempt, e.ResultCode)
    digest := sha256.Sum256([]byte(identity))
    e.EventID = hex.EncodeToString(digest[:16])
    return nil
}

func readIngestContract(client *http.Client, baseURL, apiKey string) error {
    url := strings.TrimRight(baseURL, "/") + "/discovery/logs.ingest"
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, url, nil)
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("discovery failed: status=%d body=%s", resp.StatusCode, strings.TrimSpace(string(body)))
        }
        return nil
    }
    return errors.New("discovery remained rate limited after retries")
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    baseURL := os.Getenv("INFRAI_BASE_URL")
    if apiKey == "" || baseURL == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY and INFRAI_BASE_URL are required")
        os.Exit(1)
    }
    client := &http.Client{Timeout: 10 * time.Second}
    if err := readIngestContract(client, baseURL, apiKey); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }

    event := PipelineEvent{
        RunID:       "run_20261008_0200",
        Stage:       "carrier_normalize",
        OperationID: "op_7f91c2",
        Attempt:     2,
        ResultCode:  "OK",
        InputCount:  1,
        OutputCount: 1,
        OccurredAt:  time.Date(2026, 10, 8, 2, 14, 12, 0, time.UTC),
        Diagnostics: map[string]string{"carrier_code": "ACCEPTED"},
    }

    if err := event.validate(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    if err := json.NewEncoder(os.Stdout).Encode(event); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}
Enter fullscreen mode Exit fullscreen mode

The digest is an idempotency aid, not a privacy technique: its inputs are already opaque operational values. Re-emitting the same terminal observation produces the same event_id; emitting attempt 1 and attempt 2 produces distinct evidence joined by operation_id. Consumers can deduplicate identical observations without collapsing the retry history that explains an incident.

The initial request reads the live ingestion contract from the public discovery surface; its base URL and bearer credential come from environment variables, and it uses an explicit method, a 10-second client timeout, status checking, and bounded retry behavior for HTTP 429. It does not submit an event because the request fields must come from that returned schema rather than assumptions in an article. The denylist is a final tripwire, not the primary schema. Arbitrary keys can still conceal personal data under innocent names, and values can contain it regardless of their keys. Production code should therefore allow a small, reviewed set of diagnostic fields rather than accept an open map; the map appears here only to make the failure boundary executable and visible.

Rejected option and its valid use case

The rejected design stores entire carrier requests and responses, retains them indefinitely, and expects a future deletion job to search by email, address, or customer identifier. It feels attractive during an incident because every detail is nearby. It is poor evidence engineering: serialization changes, identifiers appear in unexpected fields, backups lag, and proving complete erasure becomes harder than explaining why the data was collected.

Self-hosted indexing is not rejected categorically. It is valid when regulation or contract terms require controls the managed candidate cannot demonstrate, when the organization already has operators accountable for lifecycle policy and recovery, and when subject-linked deletion is designed into mappings from the start. In that case, use a dedicated, access-controlled subject reference, test deletion across live and restored indices, and keep the operational event narrow anyway. Owning the index creates control; it does not remove the minimization obligation.

The final decision can be stated as an audit test: an investigator should reconstruct the 02:00 run, its 317 retries, and its difference of 12 from operational identifiers and counts alone. If that works, personal data adds risk without adding necessary evidence. If it does not, repair the event model before buying a more powerful search product.

References

Top comments (0)