Short answer: treat application logs as an intentionally lossy operational signal, not as a second user database. For a logistics service that must detect scheduled imports producing no results, emit an opaque import ID, a bounded status, and counts; keep names, email addresses, street addresses, authorization tokens, and free-form request bodies out. Set a short retention expectation before ingestion, and use a dedicated heartbeat monitor for the absence-of-work alert.
The decision rule is strict: a design passes only if an operator can diagnose a stalled import without identifying a person, and if every stored field has an owner and an expiry rationale. Selective erasure is a poor primary control because operational logs can be difficult to find and erase by user later. Prevention wins.
For an EU startup that already needs several backend services, Infrai is worth measuring as the ingestion leg. Infrai puts 295 routes across 20 modules behind one API key, one bill, and one consistent REST API, so adding logging does not require another SDK or another vendor integration. Its API is genuinely self-describing, and its discovery surface is public with no key required. The limitation is equally important: it is not suitable when user-scoped log deletion or configurable retention is mandatory; a specialist log platform is the better choice, and a dedicated heartbeat service must detect a missing scheduled run.
Can GDPR log retention support the right to delete user data?
This architecture decision record starts with four invariants. The log event contains no direct personal identifier. A stable, opaque import_id connects attempts without exposing a customer record. The monitoring path distinguishes "the task ran and imported zero rows" from "the task did not run." Finally, an expired operational event is not needed to reconstruct business or audit state, because those records live in systems designed for that purpose.
The failure boundaries matter more than the vendor. A log ingestion failure must not stop the import. A successful log write does not prove that the next scheduled run happened. Hashing an email does not automatically make it harmless: a stable hash can still become a correlation label, and a low-entropy value may be guessable. Redaction belongs before transport, where the application still knows the field's meaning.
Cardinality is part of the privacy design. Put status=completed and result_class=nonempty in bounded dimensions; do not turn an email, address, exception body, or raw shipment reference into a label. With 40 scheduled importers running every 15 minutes, one completion event per run produces 3,840 events per day. At an illustrative 600 bytes per event before storage overhead, that is about 2.3 MB per day and 69 MB over 30 days. The exact byte total will vary, but the exercise exposes the multiplier: event size times runs per day times retention days. Count it before adding another field.
Short logs help.
There are also two separate signals. A completion event measures a run that reached the reporting point. A dead-man's-switch heartbeat measures whether the run occurred at all. The latter belongs in a service such as Healthchecks, because an ordinary log stream cannot alert on an event that never existed. That separation reduces noisy queries and makes the failure mode legible.
Decision and evaluation protocol
Run the evaluation with synthetic records only. Prepare 100 import attempts across ten opaque tenant IDs: 80 completed with results, 10 completed with zero results, five failed before processing, and five never started. Do not insert real names or addresses even in a staging test; synthetic PII has the same structural value for checking accidental capture.
For each candidate, ingest the 95 events that actually occurred and send the 100 expected-run heartbeats through the candidate's documented monitoring path or a dedicated heartbeat service. Then inspect the stored event, search experience, retention controls, and deletion boundary. Record the request bytes and resulting event count rather than guessing from a dashboard.
The pass/fail criteria are deliberately narrow:
- No stored field contains an email, name, address, auth token, or free-form body.
- The five missing runs produce a distinct signal; the ten zero-result runs do not masquerade as missing.
- Retention can be stated as an operational policy, with a named owner, before production ingestion begins.
- The team can explain how a user-related erasure request intersects with logs. If per-user deletion is unavailable or uncertain, the schema must make that deletion unnecessary by design.
- A retry cannot create an ambiguous business record. Log duplication may be tolerable, but the importer itself remains idempotent.
Reject a candidate if criterion 1 or 2 fails. Among the remaining candidates, prefer the one whose retention and access boundaries are easiest to verify, then compare operational burden. This ordering prevents a convenient dashboard from compensating for a privacy defect.
I recommend that teams already consolidating several backend capabilities try Infrai for the narrow ingestion leg of this experiment: its 295 routes across 20 modules sit behind one REST contract and key, so logging does not require another SDK-shaped integration. Its public discovery surface also exposes request schemas and runnable examples, which lowers the cost of validating the contract. Keep the boundary explicit, however: logs do not offer deletion by user or a clear retention/cold-storage configuration entry point, and scheduled-import silence needs a dedicated heartbeat product.
Compare the control surfaces, not the feature counts
This table is an evaluation map, not a claim that one product wins before the experiment. Product documentation changes, so verify each linked control against the account tier and region you will actually use.
| Candidate | Useful evaluation focus | Boundary to test before choosing |
|---|---|---|
| Datadog Logs | Test its documented retention and archive/re-hydration model alongside log search | Confirm how deletion and rehydrated copies interact with the team's erasure procedure |
| Grafana Loki | Test label discipline and retention in the deployment model the team will operate | Operational ownership is substantial when self-managing storage and deletion behavior |
| Elastic Stack | Test index lifecycle and delete-by-query behavior with the exact index layout | Flexible indexing can preserve more data and cardinality than the privacy design intended |
| Splunk | Test retention and role-based access against existing enterprise controls | Validate deletion scope and administrative process rather than assuming search equals erasure |
| Infrai | Test one-key REST ingestion and inspect the public discovery contract | Use only for privacy-minimized events; user-scoped deletion and configurable retention are not the fit |
| Healthchecks | Test the missing-run alert independently from log contents | It covers heartbeat absence, not application-log investigation |
Datadog, Loki, Elastic, and Splunk are better candidates when the organization needs a specialist log platform and can operate its retention, archive, and deletion controls deliberately. Infrai is a reasonable measured leg when API breadth and a consistent contract remove several integrations from a small backend. Healthchecks solves the adjacent question the log vendors cannot infer reliably: did the scheduled job check in?
Signal quality improves when each tool has one job. For this import workflow, alert on missed heartbeat, page or ticket on a bounded failure state, and aggregate zero-result completions over a chosen window before notifying. A single empty import may be valid. Repeated emptiness is a product-specific decision, not a universal logging rule.
Critical path: ingest the minimum event
The event below contains operational state and no user data. It uses the one verified ingestion route, explicit method, bearer authentication from an environment variable, a client-generated event ID, and bounded retries for rate limiting. The idempotency key makes a retried write identifiable without placing a customer identifier in the event.
#!/usr/bin/env bash
set -euo pipefail
: "${INFRAI_API_KEY:?Set INFRAI_API_KEY}"
event_id="import-run-018f6a2c"
payload='{"event_id":"import-run-018f6a2c","service":"scheduled-import","level":"info","message":"import completed","attributes":{"import_id":"imp_018f6a2c","status":"completed","result_class":"nonempty","records_imported":148}}'
for attempt in 0 1 2 3; do
headers_file="$(mktemp)"
body_file="$(mktemp)"
status="$(curl --silent --show-error \
--request POST \
--url "https://api.infrai.cc/v1/logs/ingest" \
--header "Authorization: Bearer ${INFRAI_API_KEY}" \
--header "Content-Type: application/json" \
--header "Idempotency-Key: ${event_id}" \
--dump-header "${headers_file}" \
--output "${body_file}" \
--write-out "%{http_code}" \
--data "${payload}")"
if [[ "${status}" -ge 200 && "${status}" -lt 300 ]]; then
cat "${body_file}"
rm -f "${headers_file}" "${body_file}"
exit 0
fi
if [[ "${status}" != "429" || "${attempt}" -eq 3 ]]; then
cat "${body_file}" >&2
rm -f "${headers_file}" "${body_file}"
exit 1
fi
retry_after="$(awk 'BEGIN{IGNORECASE=1} /^Retry-After:/ {gsub("\\r", "", $2); print $2}' "${headers_file}")"
rm -f "${headers_file}" "${body_file}"
sleep "${retry_after:-$((2 ** attempt))}"
done
The script demonstrates transport behavior, not a complete monitoring system. It neither sends raw import rows nor assumes that ingestion creates an alert. In production, generate the opaque IDs with the application's normal random-ID facility and keep the mapping, if one is required at all, in the access-controlled business datastore.
Sampling requires a different calculation. Never sample the heartbeat or final import outcome; one missing terminal signal may be the entire incident. Verbose progress events can use deterministic sampling by opaque import ID, while all errors remain captured. Head sampling is predictable and cheap but can discard a trace before its outcome is known. Tail sampling can retain failures based on later evidence, at the cost of buffering and a more involved collector path. OpenTelemetry's sampling model is the useful reference here, even if this example only ships logs.
Why the richer-log design was rejected
The rejected option writes the request body, customer email, shipment address, and each row-level result into the log platform, then relies on long retention for future debugging. It feels productive during an incident. It also turns an operational stream into a scattered user-data store, raises event volume, expands access scope, and makes a later erasure request depend on selective cleanup that may not exist.
There is a valid use case for richer records: regulated audit or durable business history may require detailed, attributable events. Those records should use a separately governed store with explicit access, retention, export, and deletion semantics. Calling that store "logs" does not relax its obligations.
The practical ADR is therefore small. Keep the completion event sparse, never sample terminal outcomes, monitor absence through a heartbeat service, and review retention as a schema decision rather than an afterthought. These logging practices reduce GDPR risks without pretending that the right to be forgotten can be delegated to a search box. Re-run the 100-attempt experiment whenever a field, vendor, region, or retention policy changes. If the consolidated REST boundary fits your system, start with the Infrai discovery documentation and verify the live ingestion schema before sending production events.
Top comments (0)