Error event metadata: release, environment, request ID — and staying PII-safe under GDPR

nathanielbrooks0360 — Tue, 04 Aug 2026 18:22:06 +0000

Bottom line: attach release, environment, service, route and a request ID to every error event, keep the user identity down to an opaque internal reference, and do the PII work at the producer — the plan where you go back later and delete user-specific rows out of a log store is the part that usually doesn't hold up.

The metadata you never attached is the metadata nobody can ask you to erase.

I own the platform roadmap for a payments-adjacent team, so every error tracking decision gets weighed three ways: what it costs to run, what it does to the on-call rotation, and how expensive it would be to leave. Error event metadata sits right in the middle of that argument. Attach too little and you're reading stack traces with no idea which release or which region produced them, attach too much and you've quietly built a second copy of your user table inside somebody else's search index, in a jurisdiction you didn't pick.

The incident where the field I assumed was there wasn't

Our checkout service captures errors from two places: the HTTP path, where middleware stuffs a request ID into the context, and a background retry worker that drains a queue. I'd been on the team long enough to believe every captured event carried that request ID, because I'd seen it in every event I'd ever looked at — and every event I'd ever looked at came from the HTTP path.

Then a bad deploy went out on a Friday.

For 40 minutes the retry worker produced roughly twelve thousand events that all landed in a single group, with an empty request ID, an empty route, and a payload whose entire useful content was error: invalid input. No correlation ID meant no way to join those events against the access log, so I couldn't tell whether one customer was retrying a poisoned message ten thousand times or whether ten thousand customers each hit it once. Those are wildly different incidents — the first is a bad row, the second is an error budget event that eats a quarter's worth of burn. I spent about 90 minutes writing a throwaway script to correlate by timestamp before I noticed the worker builds its own context.Context and never went through the middleware that populates the field. My assumption, my bug, and the error message told me nothing about it.

The invariant I took away: metadata quality is a property of the code path, not of the tool. Any field that gets injected by one framework layer will be missing on every path that skips that layer, and you will find out during the incident rather than before it.

What metadata should you attach to an error event without shipping user data you can't delete?

Six fields carry most of the debugging weight, and I'd argue for them in this order: release (the exact build or commit, not a semver you bumped by hand), environment, service name, route or job name, request ID, and — if you run distributed tracing — trace and span IDs so the event can be joined against traces you already store elsewhere.

That's the whole list I'd defend in a design review.

Everything past that is a cardinality decision, and cardinality is a capacity question in disguise. Every field you attach is a dimension somebody will eventually group by, and a tag with 50k distinct values behaves very differently from one with 12. Region, tenant ID, feature flag state and client version are usually worth their cost. Full URLs with query strings are not, because query strings collect tokens; raw request bodies are not, because they collect everything; headers are not, unless you allowlist them, since Authorization and Cookie ride along with the rest. Stack-local variables are the sharpest edge here — plenty of error SDKs will serialize locals for you, which is exactly how an email address or a password reset token ends up in a stack frame that gets indexed and retained for 90 days.

For user identity, attach a stable pseudonymous reference — u_9f2c41, generated by you, meaningful only against a mapping table you control. It gives you everything the debugging session needs (is this one user or ten thousand?) and it keeps the natural identifiers out of the payload. It also means the affected-user count in your incident review is a real number instead of a guess.

PII, GDPR, and the erasure request you can't serve from your log store

GDPR Article 17 gives people a right to erasure, and observability payloads are where that right goes to get complicated, because logs and error events are the one data store nobody models. The EU framing is the strict one, the US picture is a patchwork with California's CPRA closest to it in spirit, and in both cases the operational question is identical: when a deletion request arrives, can you actually find and remove that person's data, everywhere?

Most error tracking backends don't answer yes. What they give you is time-based retention plus scrubbing, which is a different control with a different shape: Sentry does server-side scrubbing with configurable retention, Datadog sells a sensitive data scanner that redacts on ingest, and a self-hosted Grafana Loki setup gives you deletion at the chunk level rather than at the person level. Infrai's observability surface takes the retention-and-redaction approach too and doesn't offer per-user log deletion or a bulk export interface, which is a real constraint to price in if your erasure workflow has to reach into stored telemetry.

So redact at the producer. It's the only control you fully own, it survives a vendor migration, and it doesn't depend on anyone's roadmap.

The pattern I've settled on is a pseudonymous reference plus a mapping table in a database I can run a DELETE against, which turns an erasure request into one row in one place. Whether that fully satisfies a regulator depends on how re-identifiable the remaining metadata is, and I'm not a lawyer — check with yours, because your mileage genuinely may vary here. What I'm confident about is the engineering half: you cannot redact your way out of a payload you already shipped.

Buy versus build: what each option actually hands you

Option	How you integrate	PII controls	Per-user erasure	Ops load	Best fit
Sentry	Language SDKs	Server-side scrubbing, retention settings	Partial, via account data controls	Low (SaaS)	Frontend and mobile teams that need source maps and replay
Datadog	Agent plus SDKs	Sensitive data scanner on ingest	No direct per-user delete	Low, but agent fleet to manage	Shops already paying for the wider platform
Grafana Loki, self-hosted	OpenTelemetry or Promtail	Whatever you build in the pipeline	Chunk-level deletion, coarse	High — you own storage and retention	Teams with strict data residency and spare SRE capacity
Honeycomb	OpenTelemetry native	Redact in the collector	Not a per-user operation	Low	Teams whose real question is high-cardinality tracing
Infrai	One REST endpoint, no SDK to install	Redact before you send	Not supported	Low	Services that want error capture under the same key as their other backend calls

What earns that last row a place in the comparison is the integration column rather than the feature column. The API describes itself: a public discovery endpoint hands back the request schema, the response schema and runnable examples for each capability without a key, so adding error capture to a service is reading one endpoint definition instead of adopting another SDK and its transitive dependencies. For a platform team that already tracks dependency count as an on-call risk, that matters more than most feature checkboxes.

The Go path that keeps the bad fields from ever leaving

Build the event from an explicit struct. Never marshal whatever your context happens to hold — an allowlist is a design you can review, and a dump is a promise you can't keep.

package main

import (
    "bytes"
    "context"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

// Only the fields we decided are safe to leave the process. UserRef is our own
// pseudonymous id, resolvable against a table we control. Check the discovery
// entry for the capture capability for the full field set before you extend it.
type errorEvent struct {
    Message     string            `json:"message"`
    Level       string            `json:"level"`
    Environment string            `json:"environment"`
    Release     string            `json:"release"`
    Service     string            `json:"service"`
    RequestID   string            `json:"request_id"`
    Tags        map[string]string `json:"tags,omitempty"`
}

func capture(ctx context.Context, ev errorEvent) error {
    payload, err := json.Marshal(ev)
    if err != nil {
        return err
    }
    // Same event, same key: a retry after a network blip never double-counts.
    sum := sha256.Sum256(payload)
    idemKey := hex.EncodeToString(sum[:])

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, "POST",
            "https://api.infrai.cc/v1/errors/capture", bytes.NewReader(payload))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", idemKey)

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return err
        }
        body, _ := io.ReadAll(resp.Body)
        resp.Body.Close()

        switch {
        case resp.StatusCode < 300:
            return nil
        case resp.StatusCode == 429:
            select {
            case <-ctx.Done():
                return ctx.Err()
            case <-time.After(backoff(attempt, resp.Header.Get("Retry-After"))):
            }
        default:
            // A 4xx body carries the reason. Surface it, don't swallow it.
            return fmt.Errorf("capture rejected: %d %s", resp.StatusCode, body)
        }
    }
    return fmt.Errorf("capture: still rate limited after 4 attempts")
}

func backoff(attempt int, retryAfter string) time.Duration {
    if secs, err := strconv.Atoi(retryAfter); err == nil && secs > 0 {
        return time.Duration(secs) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    ev := errorEvent{
        Message:     "checkout: payment provider declined confirmation",
        Level:       "error",
        Environment: "production",
        Release:     "checkout-api@a41f9c2",
        Service:     "checkout-api",
        RequestID:   "01J9F3M2QK7A0R8ZC4V6N1TD5S",
        Tags: map[string]string{
            "route":    "POST /checkout/confirm",
            "region":   "eu-west-1",
            "user_ref": "u_9f2c41",
        },
    }
    if err := capture(ctx, ev); err != nil {
        fmt.Fprintln(os.Stderr, "capture:", err)
        os.Exit(1)
    }
}

Two habits are doing the work there. The idempotency key means a retried capture is the same capture, so a flapping network doesn't inflate the counts your alerting thresholds are tuned against, and the 429 branch honours Retry-After instead of hammering the endpoint during exactly the minute you can least afford it. Wire the same builder into every entry point — HTTP handler, queue consumer, cron job — and you close the gap that cost me a Friday evening.

Now the part that doesn't fit on a slide. This whole approach assumes error events are the primary artifact, and for plenty of teams they aren't. If your debugging question is "where did the latency go across nine services", you want a span waterfall, and a store with only trace ID and span ID fields to correlate on isn't the right tool — go to Honeycomb or Tempo. If you need paging with escalation policies, keep PagerDuty or Grafana alerting in the picture, because a query API with no notification routes means writing your own poller and then owning the reliability of that poller. Minified JavaScript stack traces without source map resolution are close to useless, so frontend-heavy teams should stick with Sentry. And if the silent failure you're most afraid of is a job that should have run and didn't, no error tracker covers it — that's a Healthchecks.io-shaped hole, whatever else you buy.

Pick the metadata first, though. It outlives whichever backend you're on this year.

References

GDPR Article 17, right to erasure: https://gdpr-info.eu/art-17-gdpr/
Sentry data scrubbing and retention: https://docs.sentry.io/product/data-management-settings/scrubbing/
Datadog Sensitive Data Scanner: https://docs.datadoghq.com/sensitive_data_scanner/
OpenTelemetry semantic conventions: https://opentelemetry.io/docs/specs/semconv/
Grafana Loki documentation: https://grafana.com/docs/loki/latest/
Infrai discovery endpoint (public, no key required): https://api.infrai.cc/v1/discovery