A nightly media pipeline needs searchable failure evidence, but replay and rollback can quietly restore data that an erasure job already removed. That constraint changes the design: keep direct user identifiers out of application logs, separate the lookup needed for erasure from the log records, and make deletion state survive every rollback.
TL;DR: Treat log retention and user-data erasure as two different controls. Retention limits how long records exist. Erasure handles records that must disappear sooner. In Node.js, emit a versioned pseudonymous subject token, keep the reversible mapping outside the log system, delete that mapping and matching records through an auditable job, then test that a rollback cannot revive either one.
This is less convenient than logging an email address. Good. Convenience is how identifiers leak into exception strings, queue payloads, and copied debug fields.
How should GDPR log retention handle deleted user data?
The GDPR question is often phrased as a retention-period question: how long should application logs stay searchable, and how does the right to be forgotten apply? For an EU startup, the sharper operational question is where user data can return from after a valid erasure request. The log index is rarely the only copy. A media pipeline may read a nightly manifest, enqueue transformations, retry failed items, export an archive, and retain a replay input. Deleting a user's indexed events while leaving their identifier in any replayable input means the next run can emit the same data again. Good logging practices therefore start before retention: minimize fields at ingress, preserve a narrow lookup path for deletion, and enumerate every store or replay source that can reproduce an event. The legal basis, scope, exceptions, and response duties need qualified review for the actual service; an engineering design cannot decide those questions. It can make the resulting policy enforceable and testable.
The failure is temporal. Deployment version 42 removes a mapping and scrubs current records. A rollback to version 41 restores code that still logs raw ownerEmail, while an old queue message waits for retry. The software rollback worked; the privacy operation did not.
That is the risk.
Deletion state must move forward even when application code moves backward. Put tombstones and schema compatibility outside the application release artifact. An older release must either understand the current privacy envelope or refuse to process it. Never let it guess.
For the media pipeline, I would define four boundaries:
- Ingress replaces the account identifier with a scoped subject token before work reaches the queue.
- Workers log an allowlisted event shape, not arbitrary request objects or media metadata.
- Search storage and cold archives share an erasure manifest, including completion status.
- Replay checks the tombstone set before reading historical input or emitting fresh events.
Four is not a universal number. It is the smallest useful model for this pipeline because each boundary owns a different way deleted data can return. Schema version 2 in the example is also a local contract, not an industry standard.
Build the smallest logging surface
Start with types. A logger that accepts Record<string, unknown> is quick to ship and expensive to police. It lets every caller invent fields, so deletion becomes a hunt through unknown shapes.
The following TypeScript keeps the useful operational dimensions: pipeline run, stage, outcome, duration, and a scoped subject token. It deliberately omits filenames, URLs, free-form exception messages, and media captions. Those values can contain user data even when their field names look harmless.
import { createHash, randomUUID } from "node:crypto";
type PipelineStage = "ingest" | "transcode" | "index" | "publish";
type Outcome = "ok" | "retry" | "failed";
type PipelineEvent = Readonly<{
schemaVersion: 2;
eventId: string;
occurredAt: string;
runId: string;
subjectToken: string;
stage: PipelineStage;
outcome: Outcome;
durationMs: number;
errorCode?: string;
}>;
const subjectToken = (tenantId: string, userId: string, secret: string): string =>
createHash("sha256")
.update(`${secret}:${tenantId}:${userId}`)
.digest("hex");
export function pipelineEvent(
input: Omit<PipelineEvent, "schemaVersion" | "eventId" | "occurredAt">
): PipelineEvent {
return {
schemaVersion: 2,
eventId: randomUUID(),
occurredAt: new Date().toISOString(),
...input,
};
}
Hashing is not deletion, and a plain hash of a predictable user ID is weak pseudonymization. The secret scopes the token, while a separately controlled mapping or derivation-key lifecycle provides the erasure handle. The token remains personal data whenever it can still be linked back to a person, so do not treat this transform as an exemption from privacy controls.
Keep errors boring. Map known failures to stable codes such as SOURCE_UNREADABLE or TRANSCODE_TIMEOUT; send the original exception to a restricted diagnostic path only after applying the same field policy. A stack trace can carry request values. A generic serializer cannot know which substring matters.
Make erasure a state machine, not a delete button
A one-shot delete call gives weak evidence. The worker can time out after removing hot records but before updating an archive. Retrying must be safe, and operators need to distinguish “accepted” from “complete.”
I use an explicit state transition in the design, even if the first implementation has only one storage backend:
type ErasureState =
| { status: "requested"; subjectToken: string; requestId: string }
| { status: "tombstoned"; subjectToken: string; requestId: string }
| { status: "purging"; subjectToken: string; requestId: string }
| { status: "verified"; subjectToken: string; requestId: string };
interface LogStore {
deleteBySubject(token: string): Promise<void>;
countBySubject(token: string): Promise<number>;
}
async function purgeAndVerify(
state: Extract<ErasureState, { status: "tombstoned" }>,
stores: readonly LogStore[]
): Promise<ErasureState> {
for (const store of stores) await store.deleteBySubject(state.subjectToken);
const remaining = await Promise.all(
stores.map((store) => store.countBySubject(state.subjectToken))
);
if (remaining.some((count) => count !== 0)) {
return { ...state, status: "purging" };
}
return { ...state, status: "verified" };
}
Do not log the erasure request's raw identity from this code. Record its opaque request ID, state transition, policy version, and target-store result in a separate audit stream with its own access rules. Otherwise the mechanism designed to remove an identifier writes a new copy of it.
Retention still matters. It caps exposure and storage volume, but one global duration is usually a blunt instrument. Set periods by event class and documented purpose, then apply the shortest period that still supports the job. Security audit events, pipeline performance samples, and content-level diagnostics do not need identical fields or lifetimes.
The limitation is search fidelity. Removing free-form context makes novel failures slower to investigate, tokenization adds key-management work, and full-store verification can delay completion. This approach is not suitable when a team cannot inventory every replay source or keep old workers compatible with new tombstones; in that state, pause historical replay until the lifecycle is controlled. That is an explicit availability trade-off, not a hidden promise. It also does not determine which records a particular GDPR request legally requires a controller to erase. Policy owners must supply that decision, and the implementation must encode it without broadening the data collected.
Sampling does not solve this problem either. Head sampling decides before an operation completes; tail sampling can decide after observing more of the trace. Both reduce retained telemetry, but neither guarantees that a particular person's records were excluded or erased. Use sampling for signal volume. Use minimization, retention, and erasure for data lifecycle.
Test the rollback, not just the happy path
The valuable test begins after deletion. Seed two subjects across the live index, archive, retry queue, and replay input. Erase one. Then deploy the previous compatible worker version, replay the old batch, and assert three things: the erased token has zero searchable records, no new record is emitted for it, and the other subject remains searchable.
Fail closed on unknown schemas. If version 41 cannot interpret a version 42 tombstone, the queue item should park for review instead of running through legacy logging code. This costs availability for a narrow class of replay. I take that trade because a delayed media job is recoverable; resurrected personal data is much harder to contain.
Benchmark the controls separately. Measure ingestion overhead from tokenization, erasure completion latency across every store, replay throughput with tombstone checks, and query usefulness after fields are removed. Publish the test fixture and percentile method internally. A single average hides the exact tail where nightly batches and deletion jobs collide.
There is one nasty edge: key rotation. If token derivation changes without a version marker, the erasure job may search for today's token while old logs contain yesterday's token. Include a non-secret token version in the schema and retain only the key material needed by the active erasure window. Test every supported version.
What I would change at scale
At small volume, synchronous deletion across a few stores is understandable and easy to verify. At larger volume, use a durable erasure manifest, idempotent workers per storage class, and a final verifier that cannot mark completion until all required targets report zero matches. Partition work, but preserve one request-level state machine.
I would also move from per-event tombstone lookups to a local, frequently refreshed deny set with a strict freshness budget. That improves replay throughput, but introduces a delay between a request and enforcement at each worker. The acceptable budget is a policy decision, not a cache default.
Search quality will drop when raw filenames, URLs, and messages disappear. Compensate with stable enums and bounded dimensions, not a larger bag of strings. For this pipeline, stage, outcome, errorCode, runId, and duration buckets answer the operational questions without turning the index into a shadow user database.
The decision rule stays plain: a release is rollback-safe only when older compatible code cannot recreate erased telemetry from queues, archives, or replay inputs. Verify that property before optimizing ingestion or search. No dashboard can repair a broken lifecycle.
Sources and References
- OpenTelemetry, “Sampling”: https://opentelemetry.io/docs/concepts/sampling/
Top comments (0)