leiferiksson8493

Posted on Oct 23

Environment and Request Metadata to Attach for Healthtech Error Events and Rollbacks

#metadata #privacy #observability

Use a small, allowlisted event schema that can answer one operational question: did this release raise failures for a particular tenant cohort enough to roll back? The deciding constraint is rollback safety. A raw user identifier or request dump adds privacy exposure without making that decision clearer.

TL;DR: attach a random event ID, an opaque request correlation ID, release, environment, region, coarse tenant cohort, experiment variant, operation, error class, outcome, and timestamp. Keep direct identifiers, free-form request data, headers, URLs with query strings, and stack-local values out. Put identity resolution behind a separately controlled lookup only when an authorized investigation truly needs it.

What user metadata should I attach to release error events?

A healthtech experiment rarely has one global blast radius. Imagine a new appointment-confirmation flow enabled for 10% of tenants, split into small_clinic, hospital_group, and telehealth cohorts. A global error count can look tolerable while one cohort is failing. The rollback query needs dimensions that describe deployed code and exposure, not the patient.

This is the constraint that changed the choice: an event must support a cohort comparison without becoming a shadow medical record. Under the GDPR, pseudonymized data can still be personal data when it can be attributed to a person using additional information. Data minimization also requires personal data to be adequate, relevant, and limited to what is necessary. Calling a user ID "anonymous" does not settle either question.

For a solo SaaS, this matters beyond compliance. Every high-cardinality field expands storage, query, and incident-review work. I use a revenue-per-hour test: if a field cannot change a rollback, containment, or debugging decision, it does not earn a place in the default event. Ship weekly, but keep the schema boring.

The comparison set can stay deliberately narrow:

Dimension	Example	Decision it supports
`release`	`2026.10.09.1`	Compare the candidate with the prior build
`environment`	`production`	Prevent test traffic from contaminating the decision
`tenantCohort`	`small_clinic`	Find unequal impact across tenant groups
`experimentVariant`	`confirmation_v2`	Disable the exposed branch
`region`	`eu-west`	Contain a regional deployment
`operation`	`appointment.confirm`	Identify the failing workflow
`errorClass`	`dependency_timeout`	Separate retryable failures from validation outcomes
`requestId`	random opaque token	Correlate services for one request

Do not encode clinic name, patient status, diagnosis, email, IP address, or account number into those values. Even apparently harmless combinations can become identifying. Treat cohort definitions as reviewed configuration, with minimum group sizes chosen through a privacy risk assessment rather than guessed in application code.

No patient context. Ever.

The smallest working implementation

The useful pattern is an allowlist at the capture boundary. Do not construct an event by spreading a request object and deleting fields afterward. New framework fields will eventually bypass a denylist.

Keep it dull.

type ErrorEventInput = {
  eventId: string;
  requestId: string;
  release: string;
  environment: "production" | "staging";
  region: string;
  tenantCohort: "small_clinic" | "hospital_group" | "telehealth";
  experimentVariant: "control" | "confirmation_v2";
  operation: "appointment.confirm" | "appointment.cancel";
  errorClass: "validation" | "dependency_timeout" | "internal";
  outcome: "failed";
  occurredAt: string;
};

type ErrorEvent = Readonly<ErrorEventInput>;

const allowedEnvironments = new Set(["production", "staging"]);
const opaqueId = /^[A-Za-z0-9_-]{16,64}$/;

function buildErrorEvent(input: ErrorEventInput): ErrorEvent {
  if (!opaqueId.test(input.eventId) || !opaqueId.test(input.requestId)) {
    throw new Error("Correlation identifiers must be opaque random tokens");
  }

  if (!allowedEnvironments.has(input.environment)) {
    throw new Error("Unexpected environment");
  }

  return Object.freeze({
    eventId: input.eventId,
    requestId: input.requestId,
    release: input.release,
    environment: input.environment,
    region: input.region,
    tenantCohort: input.tenantCohort,
    experimentVariant: input.experimentVariant,
    operation: input.operation,
    errorClass: input.errorClass,
    outcome: input.outcome,
    occurredAt: input.occurredAt,
  });
}

This example excludes the exception message and stack because both can contain values copied from application inputs or local variables. That is a conservative default, not a claim that stacks are always prohibited. If stacks are operationally necessary, normalize them before export, test the normalizer against representative failures, restrict access, and define retention separately from lower-risk aggregate dimensions.

Request IDs deserve the same discipline. Generate them as opaque values, validate their shape at service boundaries, and never reuse an email, tenant slug, database key, or insurance identifier. They are correlation handles, not identity containers. If an on-call engineer can't look at a value and know whether it carries business meaning, the field name and schema documentation need to make the boundary explicit; guessing during an incident is a bad control.

Turn events into a rollback rule

Counts alone mislead because cohorts receive different traffic. Compare rates over the same bounded window: failed eligible operations divided by all eligible operations, grouped by release, cohort, and variant. Keep the denominator outside the error stream; a system recording only failures cannot calculate a failure rate.

A practical decision rule needs a predeclared minimum sample, an absolute rate limit, and a comparison with the control or previous release. The exact thresholds depend on clinical and operational risk, traffic, and the cost of a false rollback. There is no universal percentage in the cited standards, so inventing one would create false precision.

Make the action deterministic: stop expansion first, preserve the current cohort assignment, then roll back the affected variant if the reviewed threshold is crossed. Do not reshuffle tenants during diagnosis. That changes the denominator and destroys the comparison you need.

Also test absence. If a deployment stops emitting denominator events, the dashboard must say "insufficient telemetry," not "zero failures." Quiet can be broken.

Silence isn't success.

Privacy boundaries across EU and US operations

Region is useful for routing and containment, but eu and us are not legal conclusions. GDPR applicability depends on context, and US privacy obligations vary by jurisdiction and by the data and entity involved. HIPAA de-identification, where HIPAA applies, has specific methods and does not turn every health-related SaaS dataset into safe anonymous data by declaration. Get legal review for the actual data flow.

Engineering still has clear work to do. Document the purpose of each field, set retention by purpose, restrict raw-event access, and log access to sensitive diagnostic stores. Keep any identity lookup in a separate system with tighter authorization, then delete the mapping when its purpose ends. Test that boundary like code: send synthetic emails, bearer tokens, patient-like names, query strings, and long free-form values through error paths in CI, then assert that none reach the exported event. Exercise nested causes, arrays, and serializer fallbacks too. The easy trap is scrubbing the top-level message while a serializer preserves the original cause or request object several levels down. Run the same suite whenever the web framework, logger, tracing library, or exception serializer changes, because the capture surface can change even when the event type does not. These controls make minimization enforceable instead of aspirational, and they give a weekly ship cadence a repeatable gate instead of relying on somebody remembering a privacy checklist.

Outsource the undifferentiated parts where it makes sense, such as standards-compatible transport or storage, but keep schema ownership in the application. A collector cannot infer that tenantSlug names a clinic or that externalMemberId maps to a patient. The team that defines the domain must define the allowlist.

What I would change at scale

At higher volume, I would version the event schema, enforce it at every producer, and aggregate the rollback dimensions before long-term storage. Raw diagnostic events would have shorter retention and narrower access than cohort-level counters. I would also add automated cardinality limits so a supposedly coarse field cannot quietly become one value per tenant.

The trade-off is less exploratory freedom. An engineer cannot search arbitrary request context after an incident. That is acceptable when the default stream stays useful for safe rollback, while an explicit, audited escalation path handles the rare case that requires identity resolution. A one-person operation cannot review an endless data lake. The narrow contract is the feature.

Ship the schema beside the experiment. Review fields when the experiment ends, remove dimensions that no longer drive decisions, and keep the rollback query executable in a test environment. That turns error metadata into a controlled operational input rather than a permanent copy of whatever happened to be in memory.

DEV Community