Shadow-Mode Equivalence Testing: How to Safely Ship Pipeline Improvements Without Guessing

Shipping a better version of a pipeline step is not the hard part. The hard part is answering the question you have to answer first: how often does the new path produce different output than the old one, on real production data, before you let it become the primary path?

Unit tests tell you the new path is correct according to your fixtures. Integration tests tell you it runs. Neither tells you how much it diverges from the current behavior when fed the actual distribution of documents your users are processing. For a document-processing pipeline; in this case, a Google Docs → Contentful ingest that builds an EntryBlockGraph through a series of Mastra workflow steps; "how often does it produce different output" is exactly the question that gates whether a milestone is safe to ship.

The answer is a shadow-mode equivalence harness. The idea is not novel: run both the old and new code paths on every real request, compare their outputs, and measure the divergence rate. Do not change which result the caller gets. Do not alert on divergence. Just measure, because measurement is the prerequisite to setting a threshold, and a threshold is the prerequisite to shipping with confidence rather than with hope.

This post walks through the design of that harness, the three decisions that were non-obvious, and what generalizes.

The comparison harness

The harness has three pieces that compose into a single call at the step level.

runShadowModeComparison is the outer shell. It runs both paths in parallel, captures both results, runs the comparison, and returns the baseline result; the caller's behavior does not change:

export async function runShadowModeComparison<T>(
  baselineFn: () => Promise<T>,
  candidateFn: () => Promise<T>,
  compare: (baseline: T, candidate: T) => ShadowComparisonResult,
): Promise<T> {
  const [baseline, candidate] = await Promise.all([baselineFn(), candidateFn()])
  compare(baseline, candidate)
  return baseline
}

compareEntryBlockGraphs is the comparison function for the specific output type being gated. It canonicalizes both sides, runs a byte-level string comparison, and emits a metadata-only log:

export type ShadowComparisonResult = {
  didDiverge: boolean
  stepId: string
  milestoneId: string
}

export function compareEntryBlockGraphs(
  baseline: EntryBlockGraph,
  candidate: EntryBlockGraph,
  meta: { stepId: string; milestoneId: string },
): ShadowComparisonResult {
  const baselineStr = canonicalize(baseline)
  const candidateStr = canonicalize(candidate)
  const didDiverge = baselineStr !== candidateStr

  logger.info('shadow_mode_comparison', {
    stepId: meta.stepId,
    milestoneId: meta.milestoneId,
    didDiverge,
    // No content, no graph data - see the log-safety section below
  })

  return { didDiverge, ...meta }
}

computeDiffRate and assertDiffRateBelowThreshold are used in tests, not in production code, to gate each milestone:

export function computeDiffRate(comparisons: ShadowComparisonResult[]): number {
  if (comparisons.length === 0) return 0
  const diverged = comparisons.filter((c) => c.didDiverge).length
  return diverged / comparisons.length
}

export function assertDiffRateBelowThreshold(rate: number, threshold: number): void {
  if (rate > threshold) {
    throw new Error(
      `Shadow-mode diff rate ${(rate * 100).toFixed(1)}% exceeds threshold ${(threshold * 100).toFixed(1)}%`,
    )
  }
}

A milestone gate looks like this in a test:

it('M4 candidate diff rate is below 5%', async () => {
  const comparisons = await runShadowSuiteForMilestone('M4', testDocumentFixtures)
  assertDiffRateBelowThreshold(computeDiffRate(comparisons), 0.05)
})

Three non-obvious decisions made this work correctly.

Why canonicalize instead of JSON.stringify

JSON.stringify does not sort object keys. In JavaScript, object key order is insertion-order-dependent, which means two objects that are semantically identical can produce different JSON strings depending on how they were assembled:

JSON.stringify({ a: 1, b: 2 }) // '{"a":1,"b":2}'
JSON.stringify({ b: 2, a: 1 }) // '{"b":2,"a":1}'

In a pipeline that builds an EntryBlockGraph through multiple transformation steps, key insertion order varies across code paths. The baseline path assembles fields in one order; the candidate path, with a different internal implementation, may assemble them in another. Both produce structurally identical graphs. A naive JSON.stringify comparison would flag every one of those as a divergence; a constant stream of false positives, and a diff rate that tells you nothing about whether the candidate is actually producing different results.

canonicalize fixes this by recursing through the object and sorting keys at every nesting level before serializing:

export function canonicalize(obj: unknown): string {
  if (obj === null || typeof obj !== 'object') {
    return JSON.stringify(obj)
  }
  if (Array.isArray(obj)) {
    return '[' + obj.map(canonicalize).join(',') + ']'
  }
  const sortedKeys = Object.keys(obj as Record<string, unknown>).sort()
  const parts = sortedKeys.map(
    (k) => JSON.stringify(k) + ':' + canonicalize((obj as Record<string, unknown>)[k]),
  )
  return '{' + parts.join(',') + '}'
}

Two design choices here:

Arrays are not sorted. Only object keys are sorted. Array element order is semantically significant, the entries array in an EntryBlockGraph is ordered, and a candidate that swaps two entries has genuinely produced different output. Sorting array elements would hide that divergence.

The output is a stable string, not a hash. canonicalize produces a string rather than a digest. This makes it debuggable: if you need to diagnose why two graphs diverged during development, you can diff the canonical strings directly rather than comparing opaque hashes. The string is long, which is why you would not log it in production but it is invaluable in test environments.

Byte-level after canonicalization, not semantic

Once you have two canonical strings, the comparison is a strict equality check. Not a "do these fields have the same values" semantic comparison; a byte-level string equality, which means baselineStr !== candidateStr is the entire check.

This matters because EntryBlockGraph is consumed by downstream steps that care about more than field values; they care about ordering, nesting, and reference structure. A semantic diff that reports "these two entries have the same contentType" would miss the case where the entries appear in a different sequence, where a nested block is at a different depth, or where a reference edge points to the same node but is ordered differently in the edge list. Any of those divergences would affect the downstream step's behavior.

After canonicalization, the byte-level comparison is the semantic comparison for this type, because canonicalize has already normalized away the one dimension (key order) that is genuinely insignificant. Everything that remains in the string is load-bearing, a difference in the string is a difference in the graph that downstream steps would see.

The strictest useful comparison is what you want here. A lenient comparison would tell you the candidate is equivalent when it isn't. The goal of this harness is to find out whether the candidate ever diverges, so you want every real divergence to register, not just the ones that happen to survive a lenient filter.

The log-safety constraint

compareEntryBlockGraphs emits a log on every comparison. The log deliberately contains nothing from either graph:

logger.info('shadow_mode_comparison', {
  stepId: meta.stepId,
  milestoneId: meta.milestoneId,
  didDiverge,
  // NOT: baselineStr, candidateStr, baseline, candidate
})

The reason is the same reason LlmStepEnvelope does not contain the content the LLM produced: the documents flowing through this pipeline are customer data. Running both the baseline and candidate paths means you have two copies of potentially-sensitive output in memory at the same time. Logging either one to Datadog for comparison purposes means customer document content is now in your observability pipeline, regardless of whether it actually diverged.

The log carries only what is safe to carry: stepId, milestoneId, didDiverge. That is enough to compute a diff rate per milestone in Datadog, slice it by step, and track whether it trends toward zero as the candidate improves; all without a single byte of document content leaving the process via the log line.

This is a pattern worth making explicit in any pipeline that processes data with content sensitivity: the metric and the content it was measured from are two different things, and they need different handling. The metric is always safe to emit. The content almost never is. Build the boundary at the type level if you can, if the comparison function only returns { didDiverge, stepId, milestoneId } and never surfaces the graph objects themselves in its return type, callers cannot accidentally log them through this path.

Diff rate gates in tests, not in prod alerts

The assertDiffRateBelowThreshold function is used in tests, run against a representative fixture set, not in production code. This is intentional.

Production alerts on diff rate would mean every deployment triggers an alert investigation, and "what is an acceptable rate" would be a runtime policy decision made by whoever is on call rather than by the team that owns the milestone. Different milestones have genuinely different tolerances, a refactor that restructures internals without changing behavior should be at 0%, while a milestone that changes classification logic may produce deliberate divergence on a known subset of documents, and a 3–5% rate is expected and acceptable at that stage. That tolerance is a per-milestone engineering decision, not a global operational threshold.

Putting the gates in tests means:

• Each milestone gets its own threshold, set by the engineer who owns it, checked at CI time rather than discovered at alert-3am time.
• The fixture set is deterministic, the same documents, run the same way, every time, so a threshold failure is reproducible.
• A passing gate is a formal artifact in the PR: "M4 candidate runs at 2.1% divergence on the fixture set, below the 5% threshold, CI is green." That is a meaningful approval gate, not an inference from production noise.

The production logs are still emitted; they tell you how the candidate behaves on real documents that were never in the fixture set, and they let you validate that the fixture set's diff rate is representative. But they are measurement, not a gate. The gate lives in tests, where it can be reasoned about before anything ships.

What I'd generalize from this

The harness here is specific to Mastra workflow steps and an EntryBlockGraph type, but the design decisions are not:

• Canonicalize before comparing any structural output. Insertion-order variance in JavaScript object keys is a silent false-positive factory. Any comparison that uses JSON.stringify directly is going to produce noise that makes it impossible to distinguish real divergences from key-order differences. canonicalize is table stakes, not an optimization.
• Shadow mode is a measurement posture, not a feature flag. You are not branching behavior, you are running both paths and watching what they say. The caller always gets the baseline. The candidate result is captured, compared, and discarded. Making this explicit in the API (runShadowModeComparison always returns baseline) removes any ambiguity about which result is live.
• The log-safety constraint is the same problem as the LlmStepEnvelope pattern. The diff rate is safe to log. The content that diverged is not. Build a return type for the comparison function that structurally cannot carry content, so the boundary is enforced by the compiler and not by discipline at every log call site.
• Set diff rate thresholds per milestone in tests, not as global operational alerts. Different improvement milestones have different acceptable divergence rates at different stages. The engineer who owns the milestone is the right person to set that threshold, at the time the milestone is built, as a CI gate; not as a production alert that someone else has to interpret at runtime.

Comments

[Raju Dandigam]

The part that landed for me is that you treated divergence as a measurement problem before treating it as a rollout decision. Canonicalizing object keys but not arrays is exactly the kind of detail that separates a trustworthy shadow harness from one that just manufactures false positives, and the log-safety section is equally important because "compare everything" is how sensitive data quietly leaks into observability. I also agree with keeping diff-rate gates in tests instead of production alerts; milestone-specific tolerances belong with the engineers who understand the shape of intended change. Curious whether you also snapshot representative real-world divergences in a private fixture bank over time, so the shadow harness keeps learning from the production edge cases it discovers.