Shadowing Live VIN Decodes Against a Canary Path Without Double-Billing Users

A free VIN decode edge that ships a new DecodeVinValues parser, cache key, or upstream client wants proof in production without making users wait on two live calls -- or burning NHTSA quota twice per paste. Hedged requests (covered elsewhere) race parallel attempts for latency. Singleflight (elsewhere) merges identical in-flight VINs. Concurrency limits (elsewhere) cap sockets. This post is different: shadow (dark) traffic -- the user-facing path stays primary; a canary path runs in the background on a sample of requests; differences are logged; the user is never billed twice in time or in invented "premium canary" product framing.

The goal is narrow: compare primary vs canary decode outcomes safely, keep the primary response authoritative for the UI, and never double-charge the user experience (or imply a paid dual-decode) while you validate a change. Shadow traffic is a release-safety tool: it answers "does the canary disagree?" without turning every paste into two user-visible decodes.

Shadow vs hedge, singleflight, concurrency

• Hedge -- start a second live attempt to win latency; one result serves the user
• Singleflight -- share one in-flight promise per VIN key
• Concurrency limit -- bound simultaneous upstream sockets
• Shadow -- primary serves the user; canary runs async on a sample; results compared offline; canary must not block or replace the user response unless you explicitly promote it

Shadow is for correctness confidence, not for making the happy path faster. If the canary also calls live NHTSA, sample aggressively and prefer replaying cached primary bodies into the canary parser when the change is parser-only.

Sample, fork, compare

Serve primary first. Optionally enqueue a shadow job keyed by request id. Cap shadow concurrency separately so canaries cannot starve primary. Compare field-level diffs without inventing missing catalog values to "align" sides.

export type DecodeBody = { raw: string; fields: Record<string, string> };

export type ShadowDiff = {
  requestId: string;
  vinNormalized: string;
  mismatchedKeys: string[];
  primaryOnly: string[];
  canaryOnly: string[];
};

export type PrimaryDecode = (vin: string) => Promise<DecodeBody>;
export type CanaryDecode = (vin: string, primary: DecodeBody) => Promise<DecodeBody>;

export type ShadowGate = {
  sampleRate: number; // 0..1
  maxShadowInFlight: number;
  shadowInFlight: number;
};

export function shouldShadow(gate: ShadowGate, requestId: string): boolean {
  if (gate.shadowInFlight >= gate.maxShadowInFlight) return false;
  let h = 0;
  for (let i = 0; i < requestId.length; i++) {
    h = (Math.imul(31, h) + requestId.charCodeAt(i)) | 0;
  }
  const unit = ((h >>> 0) % 10_000) / 10_000;
  return unit < gate.sampleRate;
}

export function diffFields(
  primary: DecodeBody,
  canary: DecodeBody,
  requestId: string,
  vinNormalized: string,
): ShadowDiff {
  const pk = new Set(Object.keys(primary.fields));
  const ck = new Set(Object.keys(canary.fields));
  const mismatchedKeys: string[] = [];
  for (const k of pk) {
    if (ck.has(k) && primary.fields[k] !== canary.fields[k]) {
      mismatchedKeys.push(k);
    }
  }
  return {
    requestId,
    vinNormalized,
    mismatchedKeys,
    primaryOnly: [...pk].filter((k) => !ck.has(k)),
    canaryOnly: [...ck].filter((k) => !pk.has(k)),
  };
}

export async function decodeWithShadow(
  vinNormalized: string,
  requestId: string,
  primaryFn: PrimaryDecode,
  canaryFn: CanaryDecode,
  gate: ShadowGate,
  onDiff: (d: ShadowDiff) => void,
): Promise<DecodeBody> {
  const primary = await primaryFn(vinNormalized);

  if (shouldShadow(gate, requestId)) {
    gate.shadowInFlight += 1;
    void (async () => {
      try {
        const canary = await canaryFn(vinNormalized, primary);
        onDiff(diffFields(primary, canary, requestId, vinNormalized));
      } catch {
        // Shadow failures must not affect the user path
      } finally {
        gate.shadowInFlight -= 1;
      }
    })();
  }

  return primary; // always authoritative for the UI
}

export function userFacingNote(): string {
  return "Decoded via primary path -- canary shadow does not change this card";
}

Prefer parser-only canaries that reuse primary.raw so you do not double-hit NHTSA. When the canary must call live, keep sampleRate tiny and count canary calls against a separate concurrency/retry budget. Record shadow metrics (sampled, skipped by gate, diff count, canary error count) so you can raise or lower sampling without guessing. If diffs cluster on one field mapper, fix that mapper -- do not paper over disagreement by copying primary values into the canary result.

Forbidden upgrades

1. Blocking the user response on canary completion ("wait for both to be sure")
2. Replacing primary fields with canary fields silently when they differ
3. Inventing catalog values to make primary and canary "match" in diffs
4. Labeling the UI "dual verified decode" or implying a paid double lookup
5. Letting shadow traffic ignore concurrency / retry budgets and stampede NHTSA

Refuse those. Shadow is observability, not a second product SKU and not a license to invent specs.

Quick checks

import assert from "node:assert/strict";

const gate: ShadowGate = {
  sampleRate: 1,
  maxShadowInFlight: 2,
  shadowInFlight: 0,
};
let primaryCalls = 0;
let canaryCalls = 0;
const diffs: ShadowDiff[] = [];

const primaryFn: PrimaryDecode = async () => {
  primaryCalls += 1;
  return { raw: "{}", fields: { Make: "HONDA", Model: "Accord" } };
};
const canaryFn: CanaryDecode = async (_vin, primary) => {
  canaryCalls += 1;
  return {
    raw: primary.raw,
    fields: { Make: "HONDA", Model: "Civic" }, // intentional mismatch
  };
};

const body = await decodeWithShadow(
  "1HGCM82633A004352",
  "req-shadow-1",
  primaryFn,
  canaryFn,
  gate,
  (d) => diffs.push(d),
);
assert.equal(body.fields.Model, "Accord"); // primary wins
assert.equal(primaryCalls, 1);
await new Promise((r) => setTimeout(r, 20));
assert.equal(canaryCalls, 1);
assert.ok(diffs[0]?.mismatchedKeys.includes("Model"));
assert.ok(/primary path/i.test(userFacingNote()));
assert.ok(!/dual verified|premium canary/i.test(userFacingNote()));

Review rule: shadow modules must return primary to users and must not invent fields to force canary agreement.

Takeaway

Shadow traffic lets you validate DecodeVinValues changes against a canary path while the primary response stays authoritative. Sample, isolate concurrency, prefer replaying primary bodies into parser canaries, and log diffs without blocking users or double-billing their experience. Promote the canary only after diffs look boring -- never mid-request. Your free VIN edge stays honest when canaries illuminate disagreements -- without inventing catalog marketing or implying a paid dual decode.

I maintain VIN Lookup, a free VIN decode based on NHTSA data.