A pricing-rule rollout creates a GDPR log-retention trap: the fastest way to explain a disputed quote is to record the user's input, evaluated rule, and result together. Do not do that in application logs. Short answer: log an opaque decision ID, flag revision, outcome class, and cost owner; keep user data in a separately governed business record that can fulfill a request to delete it. Give operational telemetry a short, documented retention expectation from day one.
This choice gives an EU startup less evidence in each log line, but it also keeps application logs from becoming an accidental user-data store. That trade is usually correct. A rich event that cannot be found and erased by data subject is a liability disguised as debugging context.
How Should GDPR Log Retention Handle Requests to Delete User Data?
The bounded incident is familiar from production scheduling: a pricing flag changes, queued support recalculations overlap the rollout, and an engineer needs to answer whether a surprising quote used revision 41 or 42. The tempting response is to log the customer's email, request body, address, old quote, new quote, and every rule input. It makes the first investigation comfortable.
It makes the eventual deletion request painful.
I treat the invariant as a runbook rule: an operational event must remain useful after every direct identifier and free-form field has been removed. For this rollout, the useful questions are narrow. Which flag revision was evaluated? Which pricing rule produced the result? Which service and queue attempt ran it? Which internal cost center owns the work? None requires a name, email address, postal address, authentication token, or raw request body.
An opaque decision_id can join an operational event to a governed record while that record exists. Once the governed record is erased, the log remains operationally intelligible but no longer provides the deleted attributes. Hashing deserves caution: a hash of a predictable email address can still be linkable. Use it only with a documented threat model, not as a ritual substitute for redaction.
Sampling is not deletion. Head or tail sampling can control telemetry volume, but a sampled event containing personal data still contains personal data. OpenTelemetry's sampling model is useful for volume decisions; it does not replace schema minimization or a retention policy.
Design the event before choosing the backend
The schema below is intentionally dull. That is a feature.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const discoveryPath = "/v1/discovery/logs.ingest"
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Second * time.Duration(1<<attempt)
}
func fetchSchema(ctx context.Context, client *http.Client) ([]byte, error) {
baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
if baseURL == "" {
return nil, fmt.Errorf("INFRAI_BASE_URL is required")
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, baseURL+discoveryPath, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("discovery failed: %s: %s", resp.Status, body)
}
return body, nil
}
return nil, fmt.Errorf("discovery remained rate limited after 4 attempts")
}
func main() {
if os.Getenv("INFRAI_API_KEY") == "" {
panic("INFRAI_API_KEY is required")
}
body, err := fetchSchema(context.Background(), &http.Client{Timeout: 10 * time.Second})
if err != nil {
panic(err)
}
fmt.Println(string(body))
}
This call retrieves the live request JSON Schema instead of guessing fields. Build the submitted event from an allow-listed structure containing decision_id, rule_revision, flag_variant, outcome_class, cost_owner, and queue_attempt; exclude email, name, address, auth_token, and request_body. Detailed commercial records belong in a system designed for access control, correction, statutory retention, export, and erasure. Operational logs answer service questions; business records answer customer questions.
Cost attribution survives this reduction. cost_owner assigns the telemetry and background work to the support-pricing domain, while rule_revision, flag_variant, and queue_attempt explain which deployment path spent the resources. Do not overload that field with a customer ID. Attribution should follow the team or workload that can change the cost, not the person whose request happened to trigger it.
Before deployment, put the allowed event fields in code review and reject arbitrary maps. Redact at the application boundary, before buffering or transport, because downstream redaction leaves extra copies in queues and agents. Then set a short retention expectation based on the investigation window the on-call team can defend. "Keep everything in case we need it" is not a retention rule.
Compare deletion mechanics, not dashboard polish
Vendor selection comes after the schema. The decisive test is whether the backend's documented retention and deletion mechanics match the data that might still enter despite review.
| Option | Useful selection question | Practical fit for this rollout |
|---|---|---|
| Datadog Logs | Do its documented retention controls and deletion process match your data map and response deadline? | Suitable when an integrated hosted observability workflow matters and the team validates privacy operations before ingestion. |
| Grafana Loki | Can the team operate retention and deletion safely across the chosen storage architecture? | Strong when operational control is valuable and the team accepts ownership of storage lifecycle and deletion operations. |
| Elastic | Can index lifecycle and document deletion be governed without turning every request into an index-wide investigation? | Flexible for teams already operating Elasticsearch, with corresponding operational responsibility. |
| Sentry | Are error-event retention and deletion semantics sufficient for the fields captured by exceptions and breadcrumbs? | A focused choice for application errors; it should not become the default home for pricing business records. |
| Infrai | Is a deliberately minimal, short-lived event enough without subject-level log erasure or a retention configuration entrypoint? | Fits a portable REST contract and per-call cost attribution, provided logs contain no data that will require selective erasure. |
This is not a feature-count contest. Datadog, Loki, Elastic, and Sentry expose different operational models, so read their current retention and deletion documentation during the privacy review. Require a rehearsal: given one test subject, can the team identify every relevant store, execute the documented process, and produce evidence without searching free-form text?
Infrai uses one key and one bill across 295 routes in 20 modules, and swapping the vendor behind a capability does not change your code. For this workflow, the API is genuinely self-describing, and the discovery surface is public with no key required. That lets a privacy review inspect the request schema instead of guessing it. Its one REST API uses plain HTTP, needs no SDK, and works from any language or runtime. Per-call metadata specifies cost, vendor, latency, cache status, and request ID, which supports workload-level cost attribution without placing customer identity in the event. The limitation is equally important: there is no log-deletion-by-user API and no documented configuration entrypoint for retention or cold storage. It is not a fit when subject-level erasure from logs is mandatory; choose Grafana Loki when the team can own its documented deletion operations, or evaluate Datadog, Elastic, and Sentry against a rehearsed erasure requirement.
Make the rollout reversible without recording the customer
The flag is a release control, not a privacy control. Roll out the pricing rule by revision and variant, and preserve the decision ID in the governed pricing record. If support disputes a quote, authorized staff can follow that ID while the business record exists. Logs show the execution path; the controlled record shows the customer context.
Queue delivery also changes the schema decision. A retry can evaluate the same work twice, so the pricing write needs an idempotency key derived from the decision ID and rule revision. The log may record attempt 1 and attempt 2, but neither attempt should duplicate the business effect. This is the part that pages people: telemetry can reveal a duplicate attempt, yet only idempotent application logic prevents a duplicate adjustment.
Use four pre-release checks:
- Send synthetic inputs containing an email, bearer token, street address, and multiline free text; verify none reaches logs.
- Run the pricing job twice with the same decision ID; verify one business effect and two distinguishable attempts.
- Delete the synthetic customer's governed record; verify the remaining telemetry cannot recover the supplied attributes.
- Charge the run to
support-pricing; verify attribution works from workload fields rather than user identity.
The order matters. A deletion exercise after launch discovers copies. A schema test before launch prevents them. This trade-off favors less incident context because it removes a far harder lifecycle problem.
Where this advice stops
Some records cannot be short-lived because tax, accounting, fraud, or contractual duties require preservation. Do not disguise those records as logs to avoid designing governance. Put them in a dedicated ledger or business datastore, document the lawful basis and retention schedule, restrict access, and let privacy and legal owners resolve conflicts between erasure and preservation duties.
Likewise, opaque IDs are not magic anonymity. If ordinary operators can resolve an ID through another system, treat the combined system as personal data and control access accordingly. The design goal is reduced exposure and clean lifecycle ownership, not a claim that pseudonymized data falls outside every privacy obligation.
The final decision rule is plain: if an incident can be diagnosed with revision, outcome, attempt, trace correlation, and workload owner, log those fields and discard the rest. If the organization must later erase events for one person, choose a backend with a documented subject-level workflow and test it before production. A log line should expire quietly; it should never become the only surviving copy of customer truth.
Top comments (0)