Deduplicating VIN Decode Jobs in a Queue Store Before Hitting NHTSA

Fair queues limit concurrency. Idempotent handlers survive double-clicks. Single-flight collapses concurrent awaits on one key. You still waste NHTSA budget when three workers dequeue three jobs for the same normalized VIN sixty seconds apart -- each written by a different upload, webhook, or retry. The missing piece is a dedupe store in front of the outbound call: one logical decode job per VIN (and policy version) until it finishes or expires.

This post sketches a small TypeScript pattern: normalize the VIN, claim a store key, enqueue once, and let waiters share the result.

Why queue fairness is not enough

A fair queue stops one tenant from owning every outbound slot. It does not stop five tenants from each enqueueing the same VIN. Without a store:

1. Burst uploads of the same spreadsheet create N identical DecodeVinValues calls
2. Webhook retries create a new job id even when the VIN is unchanged
3. Cache misses under stampede refill still fan out if the miss path always enqueues
4. Metrics show "jobs completed" rising while unique VINs stay flat

Deduping belongs before you pay upstream -- after normalize, before fetch.

Store shape

Keep three facts per key: status, result (or error), and a generation token so a stale worker cannot overwrite a newer claim.

export type DecodeJobStatus = "pending" | "done" | "failed";

export type DecodeJobRecord = {
  status: DecodeJobStatus;
  vin: string;
  createdAt: number;
  updatedAt: number;
  result?: Record<string, string>;
  error?: string;
  generation: number;
};

export interface DedupeStore {
  get(key: string): Promise<DecodeJobRecord | null>;
  claim(
    key: string,
    vin: string,
    now: number,
  ): Promise<"acquired" | "exists">;
  complete(
    key: string,
    generation: number,
    result: Record<string, string>,
    now: number,
  ): Promise<boolean>;
  fail(
    key: string,
    generation: number,
    error: string,
    now: number,
  ): Promise<boolean>;
}

claim is compare-and-set: insert pending only if absent (or expired). Redis SET key NX, Dynamo conditional puts, or a SQL unique index on dedupe_key all work. In-memory Maps are fine for a single process demo -- not for multi-instance production.

Key = normalize + policy

Hash or stringify after the same normalize path your cache uses: trim, upper-case, strip separators. Include a policy version so a decode schema change does not reuse a stale "done" record forever.

export function normalizeVin(raw: string): string {
  return raw.trim().toUpperCase().replace(/[\s-]/g, "");
}

export function dedupeKey(vin: string, policy = "vpic-dvv-1"): string {
  return `vin-decode:${policy}:${normalizeVin(vin)}`;
}

Do not key on client job ids. Two clients with different uuids must still collapse to one upstream call.

Enqueue path

export async function enqueueDecodeOnce(opts: {
  store: DedupeStore;
  vinRaw: string;
  enqueue: (vin: string, key: string, generation: number) => Promise<void>;
  now?: number;
}): Promise<{ key: string; enqueued: boolean }> {
  const vin = normalizeVin(opts.vinRaw);
  const key = dedupeKey(vin);
  const now = opts.now ?? Date.now();
  const outcome = await opts.store.claim(key, vin, now);
  if (outcome === "exists") {
    return { key, enqueued: false };
  }
  const rec = await opts.store.get(key);
  await opts.enqueue(vin, key, rec?.generation ?? 1);
  return { key, enqueued: true };
}

Callers that only need the payload should get in a loop or subscribe to a completion channel -- they should not call enqueueDecodeOnce again unless the record is missing or expired.

Worker path

export async function runDecodeJob(opts: {
  store: DedupeStore;
  key: string;
  generation: number;
  fetchVpic: (vin: string) => Promise<Record<string, string>>;
}): Promise<void> {
  const rec = await opts.store.get(opts.key);
  if (!rec || rec.generation !== opts.generation) return;
  if (rec.status === "done" || rec.status === "failed") return;
  try {
    const result = await opts.fetchVpic(rec.vin);
    await opts.store.complete(opts.key, opts.generation, result, Date.now());
  } catch (err) {
    const msg = err instanceof Error ? err.message : "decode failed";
    await opts.store.fail(opts.key, opts.generation, msg, Date.now());
  }
}

Ignore jobs whose generation no longer matches. That prevents a late retry from clobbering a successful re-claim after TTL expiry.

TTL and failure

Pending forever is a bug. Expire pending after a budget (for example 60s) so a crashed worker does not pin the VIN. Failed records need a shorter or longer TTL depending on whether the error was invalid VIN (cache negative) or upstream 503 (allow retry soon). Keep that policy explicit in the store layer -- not scattered in UI click handlers.

What this is not

• Not a substitute for response caching of successful payloads (keep a read-through cache too)
• Not single-flight alone (single-flight is in-process; the store spans workers)
• Not idempotent HTTP only (handlers still need safe retries; the store stops duplicate enqueue)

Observability for dedupe health

Export counters that make waste visible:

• decode_jobs_claimed -- successful store claims
• decode_jobs_deduped -- claim returned exists
• decode_jobs_upstream -- actual NHTSA (or proxy) calls
• decode_jobs_stale_ignored -- worker generation mismatch

Healthy burst traffic shows deduped rising faster than upstream. If upstream tracks claimed one-for-one under load, the store is not shared across instances, the key is unstable, or TTL is too aggressive. Alert when the ratio of upstream calls to unique VINs in a window exceeds a small threshold (for example 1.2) after you account for deliberate refreshes.

Takeaway

Deduplicate at the job store: one claim per normalized VIN and policy, workers honor generation, TTLs free stuck keys. You hit NHTSA once per logical decode window instead of once per upload, webhook, and impatient retry.

I maintain VIN Lookup, a free VIN decode based on NHTSA data.