When 'Don't Log Customer Content' Is a Type Guarantee, Not a Convention

The standard advice for keeping sensitive content out of your observability pipeline is: build a stripping function, call it before every log statement, and review carefully before shipping. That is an allowlist discipline; you maintain a list of fields that are safe to emit, and you trust every engineer who adds a field to a step's output to know the rule and apply it consistently. The problem with allowlist disciplines is that they erode. Under deadline pressure, in a late-night incident response, three months after the original author left; the log statement lands without the stripping call, and the incident you were preventing becomes the incident you are explaining to a customer.

The pipeline I was working on processes customer documents through a series of LLM-backed Mastra workflow steps. The observability requirement: structured logs to Datadog, faceted on outcome, failure class, step ID, latency, and token usage. The safety requirement: no customer document content in those logs, ever. These requirements sit in direct tension if you are not deliberate about where the boundary lives, because the same object that carries the LLM's output is the object you want to extract metrics from.

The design resolves that tension not by adding a stripping step, but by making it structurally impossible to log content from the metrics object at all.

The envelope design

export type LlmStepEnvelope = {
  stepId: string
  attempts: number
  failureClass?: LlmStepFailureClass
  modelVersion?: string
  promptBundleSha?: string
  latencyMs: number
  tokenUsage?: LlmStepTokenUsage
}

export type LlmStepResult<T> =
  | (LlmStepEnvelope & { outcome: 'OK'; value: T })
  | (LlmStepEnvelope & { outcome: 'DEGRADED'; value: T })
  | (LlmStepEnvelope & { outcome: 'ABSTAINED'; value: T })
  | (LlmStepEnvelope & { outcome: 'FAILED' })

LlmStepEnvelope holds step ID, attempt count, failure class, model version, a prompt template SHA, latency, and token usage. Every field is a scalar or a small nested record. None of it is the LLM's output. value: T (the actual content the model produced) only appears on the OK, DEGRADED, and ABSTAINED variants of the discriminated union. FAILED has no value at all. The envelope, which is the intersection type shared across all four variants, never holds content by construction.

This means any code path that has an LlmStepEnvelope in scope can log it unconditionally:

console.log(JSON.stringify({ ...envelope, outcome: result.outcome }))

No stripping call required because there is no content field to strip. TypeScript enforces this, not convention. A developer who tries to access .value on the envelope directly gets a compile error, because value is not a field of LlmStepEnvelope. It only appears on the intersection types that include the value variants, and you reach those only by explicitly accessing result (the full LlmStepResult<T>) and narrowing by outcome.

Contrast this with the alternative: a single result type where value is optional, and the safe-to-log pattern is "just omit value when you log." That is the allowlist. Every new field added to the step's output type is an unsafe field by default, and every log statement is a potential gap. The envelope design inverts the default: the logging-safe object is structurally separate from the object that holds content, and you have to explicitly reach for content, through result.value, available only after narrowing; when you are ready to use it, not when you are logging metrics.

promptBundleSha: the template identifier pattern

promptBundleSha looks like a candidate exception to the "no content" rule, because prompts often contain template variables. Once those variables are interpolated with a real customer document, the rendered prompt is sensitive content.

The SHA is the hash of the template; the uninstantiated string with variable placeholders, not the rendered version with actual document content substituted in. What ends up in Datadog is a value like a3f8bc1d..., which lets you correlate a model behavior regression to a specific prompt version in the repository without any of the input that was fed to the model appearing alongside it. If a model starts producing unexpected output across a class of documents and you need to know whether a recent prompt change is responsible, the SHA gives you that link. The rendered prompt, which would give you the link and the customer content, is never necessary for that diagnosis.

The generalizable form of this: log the identifier of a template rather than the instantiated template, whenever instantiation incorporates sensitive inputs. A git SHA, a content hash, a version tag. Any of these lets you correlate against the artifact in version control without logging what was fed into it.

Testing the bytes, not the call

The envelope design existed before the work described here. What did not exist were tests asserting that the structured logs emitted by each step actually contained only the safe fields. The work in INTEG-4366 added those tests, and how they are written is the part most worth stealing.

The obvious test approach: assert that the step calls a logEnvelope(result) helper, and test the helper in isolation. This confirms the wiring, that the step invokes the logging function. What it does not confirm is that the helper, when invoked in its actual call-site context, does not inadvertently include a field pulled from the step's local scope. A helper that takes an LlmStepEnvelope and logs it is correct in isolation. The same helper called with additional fields spread in { ...envelope, value, mappingPlan } is not, and the "was logEnvelope called" test passes either way.

The tests in INTEG-4366 go to the actual output channel:

const logSpy = vi.spyOn(console, 'log')

// run the step...

const rawLog = logSpy.mock.calls.find((call) => {
  const parsed = JSON.parse(call[0])
  return parsed.stepId === 'classify-tables'
})
const log = JSON.parse(rawLog![0])

// envelope fields present
expect(log).toHaveProperty('stepId', 'classify-tables')
expect(log).toHaveProperty('outcome', 'OK')
expect(log).toHaveProperty('attempts')
expect(log).toHaveProperty('latencyMs')

// content fields absent
expect(log).not.toHaveProperty('value')
expect(log).not.toHaveProperty('classifications')

expect(log).not.toHaveProperty('value') is a structural assertion on the parsed JSON object, not a string search on the raw log output. A step that serialized the content as an opaque [object Object] would pass a string-match check but fail this assertion. The test is inspecting what bytes actually reached the output channel, then asking whether those bytes have a specific key in their parsed structure.

Each step gets its own version of the "no content" assertion, naming the fields specific to that step's output type: classifications for the table-classification step, mappingPlan for the graph-review step. A generic not.toHaveProperty('value') catches the general case. Step-specific field assertions catch a developer adding a new field to that step's result type and inadvertently logging it. Both layers are necessary: the former is a floor, the latter is a canary.

The tests also assert that the log fires on both OK and non-OK outcomes. An implementation that only logs on success would pass every structural assertion while missing a significant chunk of the observable failure surface; which is exactly the data you most want in Datadog when something goes wrong.

Why the type boundary outlasts the allowlist

An allowlist requires every engineer who adds a field to know the rule, apply it, and not be in a hurry when they do it. A type boundary requires none of those things; the compiler enforces it, the rule is implicit in the type, and the only way to violate it is to change the type itself, which is an explicit, reviewable decision.

That asymmetry is the practical argument for the envelope design over a stripping function. It is not that stripping functions are impossible to write correctly, they are straightforward. It is that a stripping function is a gate that has to be passed through at every log call site, and the discipline of passing through it is exactly the kind of thing that erodes under conditions that are easy to imagine and hard to prevent. A type that cannot hold the field does not erode. It either compiles or it does not.

The test layer is the verification that the type guarantee actually propagated all the way to the output channel. The type tells you what the object can contain; the test tells you what the serialized bytes contain. Both matter, because a type error at compile time is not the same as an assertion about runtime JSON serialization behavior. Running both is the only way to confirm that the content boundary held end to end, not just at the TypeScript layer.

What I would generalize from this

• Make "safe to log" the default by type design, not by discipline. If the logging-safe subset of your result type is a structurally separate type that cannot hold unsafe fields, you cannot accidentally log content even if you try, because the compiler catches it first. Allowlists require the rule to be known and remembered; a type boundary requires neither.

• Log the template identifier, not the rendered template, whenever instantiation incorporates sensitive inputs. A SHA, a version tag, or any stable identifier gives you the correlation to the artifact in version control without requiring any of the sensitive content to travel with it.

• Test the bytes, not the call. A test that asserts a logging helper was invoked confirms the wiring. A test that spies on the output channel, captures the raw string, parses it, and runs structural assertions on the result confirms that the content boundary held at serialization time; which is the thing that actually matters when you are trying to guarantee what reaches your observability pipeline.

• Step-specific "no content" assertions are not redundant with generic ones. A generic not.toHaveProperty('value') is a floor. Per-step assertions that name the actual output fields of that step are a canary: the next developer who adds a reviewNotes field to a step result and wonders whether it is safe to log will find out immediately, not in an incident review.

None of this is specific to LLM pipelines; it applies to any system where a metrics object and a content object share a common ancestor and you need a hard guarantee that only the metrics object reaches your logging infrastructure. The shape is the same: separate the types at the point where you can enforce the boundary for free, test the serialized output at the point where the boundary has to actually hold, and prefer structural guarantees over anything that requires a human to remember a rule every time.

Comments

[Rahul S]

The envelope kills accidental logging on the happy path, but the leak that walks right past the type system is the error path — when something throws while processing value: T (a parse, a schema reject, a downstream 4xx), the content rides out inside the Error's message or the cause chain, and every error logger serializes the whole object with no idea your discriminated union exists. Types govern what you deliberately hand the logger; they say nothing about what an exception drags out of scope. The template-hash trick is the right instinct, but worth being precise about what it buys: the rendered prompt still has to exist to reach the model, so the guarantee is "not in our logs," not "not observable" — the model vendor's request logs and any auto-instrumenting APM that wraps fetch both sit outside your boundary. If you want the type wall to cover the paths types can't reach, give the content wrapper its own toJSON/toString that returns [redacted] — then even a thrown error or a stray console.log of the raw object emits nothing, which is the one channel you can't statically see.