The Shadow Run: How I Test a New Workflow Version on Live Traffic Without Firing Side Effects

#automation #ai #webdev #programming

Fixtures prove the mechanism works. They don't prove the new version behaves the same on real traffic. The shadow run does: it replays live events through the new workflow version with every side effect stubbed out, then diffs "what v1 did" against "what v2 would have done." Here's the exact setup I run before any version touches production.

The fixture tests from my last post verify mechanics -- the gate drops unquoted items, the fallback fires, the crash rerun produces no duplicates. All deterministic, all necessary. But last week a prompt change passed every fixture and still would have double-emailed twelve customers in production. The fixtures never saw those transcripts because they were almost clean -- clean enough to pass, weird enough to trip the new wording.

That's the gap the shadow run closes: it tests the new version against live traffic, without letting it touch anything real.

The fork

Every one of my production workflows starts with a Webhook or Schedule trigger. The shadow version starts with the same trigger, plus one extra first step: the event is duplicated.

Branch A (live, v1): the current production workflow, unchanged. Writes real side effects -- sends the email, creates the task, updates the tracker.
Branch B (shadow, v2): the new workflow version, with every effectful node replaced by a stub.

The fork itself is one Code node right after the trigger:

// Fan-out: same event to live and shadow
const event = $input.first().json;
return [
  { json: { ...event, branch: 'live' } },
  { json: { ...event, branch: 'shadow' } }
];

Branch A runs the workflow as-is. Branch B runs the new version. Same input, two versions, one real world.

The stubs: effects become log rows

This is the whole trick. In the shadow branch, every node that would fire a side effect -- Gmail send, Slack post, Notion insert, HTTP POST to a third-party API -- is replaced with a stub Code node that writes a log row describing the intended effect, instead of performing it:

// Shadow stub: record the intent, fire nothing
const intent = {
  event_id: $input.first().json.event_id,   // same key as branch A
  workflow_version: 'v2',
  node: 'gmail-customer-reply',
  effect_type: 'email.send',
  target: $input.first().json.to,
  payload_hash: sha256(JSON.stringify($input.first().json.body))
};
// insert into shadow_log (event_id, workflow_version, node, effect_type, target, payload_hash)

Branch A writes the same shape of row for every effect it actually fires -- into live_log. Now I have two tables with the same schema: what v1 did, what v2 would have done.

The shadow stub must be incapable of firing the effect, not merely configured not to. I run my tag linter (previous post) against the shadow workflow before the run starts and assert: zero nodes tagged effectful. A stub with credentials attached is a bug waiting for a bad day.

The divergence query

After N live events flow through (I use 50 -- enough to see the weird cases, cheap enough to afford), one query:

-- per event: effects only in live, only in shadow, or changed
SELECT event_id,
       SUM(CASE WHEN l.event_id IS NULL THEN 1 ELSE 0 END) AS extra,    -- v2 would do something new
       SUM(CASE WHEN s.event_id IS NULL THEN 1 ELSE 0 END) AS missing,  -- v2 would drop something v1 did
       SUM(CASE WHEN l.payload_hash != s.payload_hash THEN 1 ELSE 0 END) AS changed
FROM live_log l FULL OUTER JOIN shadow_log s
  ON l.event_id = s.event_id AND l.node = s.node
GROUP BY event_id;

Three classes, three rules:

Missing (v2 drops something v1 did): always a bug. The new version must at least do everything the old one did, or the drop is an intentional, reviewed change. I have never once seen an intentional missing. It's always a bug.
Extra (v2 does something new): the highest-risk class. Every extra effect gets a manual review -- read the actual payload v2 would have sent -- before the version ships. The double-email incident above was an extra. The fixture pack didn't catch it because the triggering transcript pattern ("thanks, that covers it" followed by a new question in the same thread) wasn't in any fixture.
Changed (same target, different payload): usually a prompt improvement doing its job. Still sampled -- I read 10% of them -- because "different" includes "worse."

Promotion rule: 50 shadowed events, zero missing, every extra reviewed and signed off, changed sampled clean. Then v2 ships.

The cost bound

Shadow runs double your LLM spend while they run -- every event goes through two model stacks. So the run is bounded by design: the fork node counts events and stops duplicating after 50. At my volumes that's roughly $2 of extra model cost per version. A version that double-emails customers costs more than that in refunds and apology time. The bound is also enforced mechanically, not by memory: the counter lives in the fork node, and the shadow workflow deactivates itself when it hits the cap.

What the shadow run is not

It's not a replacement for fixtures. Fixtures are cheap, fast, and deterministic -- they run on every edit. The shadow run is expensive, slow, and nondeterministic -- it runs once per version, against the one thing fixtures can't fake: production reality. Fixtures test the mechanism. The shadow tests the version. Together they cover both axes, and neither one covers both.

It's also not A/B testing. Nothing in the shadow branch reaches a human. There is no metric on open rates or reply quality -- the divergence query only asks "would v2 have done the same things v1 did, and is every difference intentional?" Output quality is a review-queue question, and the review queue is where it stays.

I'm building AI Automation Lab -- n8n workflow templates for AI automation -- and keeping a free lab on GitHub. I also send occasional notes on n8n reliability patterns: get them here.

DEV Community