Amazon Data API Node.js: Why a Job Dies at Block 17 and a Rerun Doubles Your Rows

A job that pulls Amazon data in Node.js died at block 17. The logs showed HTTP 200 for every request that ran. Nobody panicked; we reran the job from the start. The next morning the report showed double the rows for every ASIN in blocks 1 through 17, and the price column held stale numbers for the new rows. The rerun did not resume; it overwrote nothing and appended everything.

That failure is not a bug in your loop. It is three gaps that TypeScript types cannot see: the network fingerprint, the HTTP/2 concurrency model, and the missing idempotency key. This article walks through each gap with runnable TypeScript and explains where a managed Amazon Data API Node.js call removes the work.

What you will learn

• Why Node.js has no default answer that equals Python curl_cffi for TLS fingerprints, and what that costs in production.
• Why turning on HTTP/2 reshapes your concurrency model instead of just speeding it up.
• How to make a rerun idempotent so block 17 never writes a second row.
• How to assert data shape at the edge with Zod, split timeouts per phase, name Node errors, and shut down without losing rows.

Thread one: Node has no curl_cffi equivalent for fingerprints

In Python you reach for curl_cffi when a target inspects the TLS handshake. The library ships a BoringSSL build and replays the exact ClientHello a real Chrome sends, including the extension order and the HTTP/2 SETTINGS frame. Node.js has no default answer that does the same.

For a while the community pointed to got-scraping. The project is now unmaintained. Worse, it only rewrote request headers. It never touched the TLS handshake. Any checkpoint that keys on JA3 or JA4 sat upstream of what the library could change, so the gaps it claimed to close were never closed.

This is the part most teams miss: a fingerprint is not the User-Agent string. It is the bytes of the TLS ClientHello and the HTTP/2 frame settings. A header swap leaves those bytes untouched.

impit claims Chrome and speaks Rust

impit gets close and then stops. Under the hood it uses a Rust HTTP client built on rustls. Its HTTP/2 SETTINGS frame uses the underlying Rust library defaults. That frame omits HEADER_TABLE_SIZE, and it sends a MAX_FRAME_SIZE value that Chrome never puts on the wire. The result is a client that claims Chrome in its docs yet speaks Rust in its bytes. A fingerprint check that sorts and hashes the SETTINGS frame will read the request as a Rust client, not a browser.

For Amazon, the cost shows up as a block of ASINs that return a challenge page instead of a product object. Your code sees a 200, parses nothing, and writes a row with empty fields. The fingerprint is the first gate, and Node leaves you to build it by hand. A wrong fingerprint is invisible until the fill rate drops, at which point the rows are already in the table.

The 2026-08 benchmark, read as a shape not a verdict

Numbers shift with versions, so treat the table below as a snapshot from a public benchmark measured on 2026-08-06 on an M-series Mac, 300 serial requests against a local service. The point is the shape, not the exact figures, and the figures will move as the libraries ship updates.

Library	Engine	Latest Chrome	HTTP/2 fingerprint	req/s	Cold start
wreq-js	Rust wreq + BoringSSL	149	Correct	12842	7 ms
impers	curl-impersonate	146	Correct	8439	16 ms
node-wreq	Rust	149	Correct	6500	10 ms
impit	Rust reqwest + rustls	124	Not correct	6710	37 ms
CycleTLS	Go subprocess IPC	Not in run	Not in run	Not in run	IPC overhead

wreq-js leads on throughput and gets the HTTP/2 fingerprint right because it rides BoringSSL and copies Chrome's frame values. impit posts a high req/s but its fingerprint is wrong. For an Amazon data job, a wrong fingerprint costs more than a slow one, because the slow request still returns data while the wrong one returns a checkpoint page.

JA3 moved, so JA4 took over

JA3 is a hash of five ClientHello fields concatenated and run through MD5. After Chrome 110 the browser permutes the extension order on each connection, so the JA3 value moves between runs. Most risk systems now use JA4, which sorts the fields before hashing. The lesson for a Node shop is to pick a client whose ClientHello is stable and matches a real browser, or the values you send will never line up with what a browser sends.

Reuse the session or pay the handshake twice

A single TLS handshake costs about 15 ms when you reuse a session, against about 53 ms for a fresh one. The gap is the handshake, not the request. Reuse the session across the loop:

import { createSession } from "pangolinfo";

const session = await createSession({ browser: "chrome_149" });
try {
  for (const asin of asins) {
    await session.fetch(buildUrl(asin));
  }
} finally {
  await session.close();
}

That block runs on every exit path, so the warm handshake you paid for is not wasted on a thrown error.

Thread two: HTTP/2 changes the concurrency model

Flipping on HTTP/2 sounds like a free speed win. In Node.js it changes the rules of the game, and the two entry points disagree on the default.

Node's built-in fetch rides on undici. By default it does not negotiate HTTP/2. The team flags it as experimental and leaves it off. You must set it by hand:

import { Agent, setGlobalDispatcher } from "undici";

const dispatcher = new Agent({
  allowH2: true,
  connections: 8,
  pipelining: 0,
  maxConcurrentStreams: 100,
  bodyTimeout: 30_000,
  headersTimeout: 15_000,
  connect: { timeout: 5_000 },
});
setGlobalDispatcher(dispatcher);

Note the split: fetch keeps allowH2 off by default, while the undici Client keeps it on by default. The same library, two defaults. Miss the one for fetch and you stay on HTTP/1.1 without an error. Your code runs, your throughput does not move.

Streams replace pipelining

When HTTP/2 is on, a single connection opens many streams. The ceiling is maxConcurrentStreams, which defaults to 100. This replaces HTTP/1.1 pipelining. The flow control window initialWindowSize defaults to 262144 bytes. You are no longer racing one request per socket; you are filling a window across streams.

Most teams discover this the hard way. They turn on HTTP/2, watch the connection count drop, and expect throughput to climb. It does not, because the old per-socket mental model no longer applies. You now reason about streams per connection and the window size, not sockets. A pool of eight connections with 100 streams each is not eight workers; it is eight windows you must keep full without tripping the rate limit at the other end. Get the window wrong and you stall with the socket idle, which is the opposite of the problem you set out to fix.

Concurrency is not rate

p-limit(8) caps how many calls run at once. It does not cap the rate. At a 10 ms response you push about 800 requests per second. At a 5 second response the same cap yields about 1.6 requests per second. The concurrency number stayed the same; the rate swung by a factor of 500. If your target throttles by rate, a flat concurrency cap will not save you.

import PQueue from "p-queue";

const queue = new PQueue({
  concurrency: 8,
  interval: 1_000,
  intervalCap: 4,
});

for (const asin of asins) {
  queue.add(() => fetchProduct(asin));
}
await queue.onIdle();

Add intervalCap with interval and you cap both the in-flight count and the rate per window. That is the knob HTTP/2 work needs.

Thread three: idempotent reruns

Back to block 17. The rerun appended because nothing told the database that row 17 for an ASIN was the same row as the one from the first run. The fix is an idempotency key, not a WHERE NOT EXISTS guess.

Build the key from four fields: asin, marketplace, capturedAt, and contractVersion. The first three say which item and when. The fourth says which shape of the response you stored. When the contract changes, you want a new row, not an overwrite of old data with a new schema.

import { Pool } from "pg";

const pool = new Pool();
const sql = `
  INSERT INTO product_snapshot
    (asin, marketplace, captured_at, contract_version, title, price, currency, captured_at_ts)
  VALUES ($1, $2, $3, $4, $5, $6, $7, now())
  ON CONFLICT (asin, marketplace, captured_at, contract_version)
  DO UPDATE SET
    title = EXCLUDED.title,
    price = EXCLUDED.price,
    currency = EXCLUDED.currency;
`;

await pool.query(sql, [asin, marketplace, capturedAt, contractVersion, title, price, currency]);

Chunk the work and write a checkpoint after each chunk. A rerun that starts at the last good checkpoint never touches blocks 1 through 16 again. The 200 you got for those blocks stays as one row, not two.

The checkpoint is a small table of its own: block id, row count, and a timestamp. A rerun reads the last finished block and starts there. If the job dies at block 17, blocks 1 through 16 are marked done and never fetched again. The double-row bug dies at the root because the second run never touches the rows the first run wrote. Without the checkpoint, every rerun is a full replay, and a full replay is where duplicate rows come from.

Coverage is not fill rate

Two numbers get confused: coverage and fill rate. Coverage is the share of ASINs you attempted that returned a response. Fill rate is the share of those responses that held the fields your job needs. A job can hit 100 percent coverage and still ship a report where the price field is empty for half the rows. The gap between coverage and fill rate is where a 200 lies to you. We cover that split in a companion piece on empty fields in a daily job.

TypeScript types do not hold runtime data

TypeScript types describe the data you expect. They do not describe the data you receive at 03:14. The bytes from the wire hold whatever the source sent. Validate at the edge with Zod and use safeParse, not parse, so a bad row returns a result you can route instead of a thrown error that kills the chunk.

import { z } from "zod";

const ProductSnapshot = z.object({
  asin: z.string().regex(/^[A-Z0-9]{10}$/),
  marketplace: z.enum(["US", "DE", "JP", "UK"]),
  capturedAt: z.string().regex(/^\d{4}-\d{2}-\d{2}$/),
  contractVersion: z.string().default("2026-09-01"),
  title: z.string().min(1),
  price: z.number().positive().nullish(),
  currency: z.string().length(3).nullish(),
});

const P0_FIELDS = ["title", "price", "currency"] as const;

function assertP0(row: unknown) {
  const parsed = ProductSnapshot.safeParse(row);
  if (!parsed.success) {
    throw new Error(`contract violation: ${parsed.error.message}`);
  }
  for (const field of P0_FIELDS) {
    const value = parsed.data[field];
    if (value === null || value === undefined || value === "") {
      throw new Error(`P0 field ${field} missing on ${parsed.data.asin}`);
    }
  }
  return parsed.data;
}

Mark title, price, and currency as P0. A row that passes the shape check but drops a P0 field is still a failure for the report. Catch it at the edge, log it, and skip the row. Do not let it reach the upsert as a silent null.

This pattern moves the contract from a comment in your type file to a check that runs on every row. When the source changes a field from a number to a string, the safeParse result tells you which block broke and which ASIN, instead of a cryptic cast error two modules away. The type gave you no warning; the runtime check does.

Per-phase timeouts and the Node error names

A single fetch fails in stages: connect, then headers, then body. Set a timeout per stage so a stalled body does not hold a socket forever while connect and headers were fine.

import { Agent, setGlobalDispatcher, TimeoutError } from "undici";

const dispatcher = new Agent({
  connect: { timeout: 5_000 },
  headersTimeout: 15_000,
  bodyTimeout: 30_000,
});
setGlobalDispatcher(dispatcher);

async function fetchProduct(url: string, asin: string) {
  try {
    const res = await fetch(url, { signal: AbortSignal.timeout(45_000) });
    return await res.json();
  } catch (err) {
    if (err instanceof TimeoutError) {
      log.warn({ asin, reason: "timeout" });
    } else if ((err as any).code === "UND_ERR_CONNECT_TIMEOUT") {
      log.warn({ asin, reason: "connect_timeout" });
    } else if ((err as any).code === "UND_ERR_HEADERS_TIMEOUT") {
      log.warn({ asin, reason: "headers_timeout" });
    } else if ((err as any).code === "UND_ERR_BODY_TIMEOUT") {
      log.warn({ asin, reason: "body_timeout" });
    } else if ((err as any).code === "ENOTFOUND" || (err as any).code === "EAI_AGAIN") {
      log.warn({ asin, reason: "dns" });
    } else {
      log.error({ asin, err });
    }
    throw err;
  }
}

Know the error names: UND_ERR_CONNECT_TIMEOUT, UND_ERR_HEADERS_TIMEOUT, UND_ERR_BODY_TIMEOUT from undici; TimeoutError (a DOMException) from AbortSignal.timeout; and ENOTFOUND or EAI_AGAIN for DNS. Routing on the name lets you retry the right failures and drop the rest. AbortSignal.any is available from Node 22 if you need to combine two signals.

Graceful shutdown keeps the rows you pulled

A kill signal should stop new work and flush what you already pulled. Catch SIGTERM, set a flag, and let in-flight requests finish before you close the pool.

let shuttingDown = false;
const pending: Promise<void>[] = [];

process.on("SIGTERM", () => {
  shuttingDown = true;
  log.info({ event: "sigterm", pending: pending.length });
});

async function run(asin: string) {
  if (shuttingDown) return;
  const row = await fetchProduct(buildUrl(asin), asin);
  if (shuttingDown) {
    await flush([row]); // write the rows you already pulled
    return;
  }
  await store(row);
}

async function flush(rows: unknown[]) {
  for (const row of rows) await store(row);
}

Add structured logs as JSON, not prose strings. A log line that reads { "event": "body_timeout", "asin": "B0X…", "stage": "body" } is queryable; a line that reads timeout on B0X is not. When the scheduled run fails at block 17, you want to count failures by stage in seconds, not read a wall of text.

A hard exit that skips the flush loses every row in flight. On a 30 million call per day scale, even a small loss rate is thousands of missing rows by morning. The flag plus the flush turns a crash into a clean pause you can resume from the checkpoint. The rows already pulled are safe; the rows still queued wait for the next run.

What one managed request already includes

All of the above is work you do because the raw call returns a 200 with gaps. A single Amazon Data API Node.js request wraps the parts that break:

• Residential and mobile exit IPs with rotation.
• TLS and HTTP/2 fingerprints that match a real browser.
• Browser fingerprint coherence across the handshake, the headers, and the frames.
• JavaScript rendering when the page needs it.
• Challenge and intercept page handling with server-side retries.
• Geo alignment so the exit region matches the marketplace you query.

One request, one price. None of these are add-ons. You send a request, you get a structured real-time JSON back. The fingerprint work, the HTTP/2 tuning, and the idempotency key stay on your side of the line, in the data contract rather than in your retry loop.

In production the service holds a median latency around 3 seconds, a success rate near 99 percent, and more than 30 million calls each day, with sponsored slot capture at 91.4 percent across 13 markets.

See the pricing for what one call costs across those markets. If you want the cost view after the scrapers are gone, we wrote a piece on what an Amazon data pipeline costs you.