My pipeline parsed every page twice, and I blamed the parser

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

This story fits in one diff: memorySize: 2048 became memorySize: 512. Getting there took a week of optimizing the wrong thing, 404 lines of it, and one evening of actually reading the code between the functions.

The system

A product-data pipeline I ran for a client. In: a URL from any of tens of thousands of retailer domains. Out: a clean record with title, price, currency, images, availability. Eight figures of product pages refreshed every month, all of it on AWS Lambda and SQS.

Downloads went through a proxy waterfall. Plain HTTP request first, because it's free. If that fails, escalate: datacenter proxy, then residential, then a real browser. Each tier costs more than the one before it, so the economics of the whole pipeline depend on the cheap tiers winning as often as possible.

Extraction ran four strategies against every downloaded page: JSON-LD, microdata, OpenGraph, and a per-domain selector config. Parse the HTML once with cheerio, let the four extractors share the tree, merge results field by field. Boring, layered. It works. I've written before about why I like this shape: when a retailer redesigns, one strategy dies and the other three carry the record.

This is a story about the word "once" in that paragraph.

The symptom

Fashion-retailer product pages are enormous. A server-rendered page with a hundred color-and-size variants ships HTML by the megabyte, and a megabyte of HTML costs a multiple of itself in heap the moment it becomes a DOM tree. The download function had 2 GB of memory and still ran hot on heavy pages: slow invocations, memory climbing toward the ceiling, and enough background worry that the code had logMemoryUsage() calls sprinkled through it from earlier rounds of suspicion.

The graphs said memory. The parser is what uses memory. So the parser was guilty. Case closed, investigation over, off to optimize.

The wrong week

The commit is still in the repo, and the message alone is a confession: feat: EXPERIMENTAL cheerio memory reduction. Three files, 404 lines added, 53 removed.

What those lines contained:

• A pre-parse HTML stripper that removed scripts, styles, and comments before cheerio ever saw the document, complete with a proud log line, quoted verbatim from the diff: HTML optimized: ${html.length} -> ${optimizedHtml.length} characters.
• A brand-new utils/memory.ts with a whole toolkit: estimateHtmlMemoryUsage(), isHtmlTooLarge(), optimizeCheerio(), and the crown jewel, forceGarbageCollection(). Running Node with --expose-gc in production is the software equivalent of hitting the TV.
• A rewrite of the extractor runner from Promise.all to a sequential for-loop, on the theory that four extractors sharing one tree "held references too long". They share a reference to the same tree, so running them one at a time saves almost nothing. That detail should have tipped me off that I didn't understand where the memory was going. It didn't.

Here's the trap: it helped. Stripping megabytes of scripts before parsing genuinely shrinks the tree, and "it helped" feels like progress when it's actually anesthesia. Symptom optimization pays out just often enough to keep you at the table.

The read

Five days later I sat down and read the waterfall code end to end. Not the extractors. The plumbing between them.

A proxy waterfall needs to know whether a tier succeeded, and in scraping a 200 response means nothing. Bot walls return 200, empty client-side shells return 200, parked domains return 200 with a smile. So each tier's success check called a function named checkMinimumData: parse the response, run all four extraction strategies, and answer one question. Did we get at least a title and a price? Honest check, correct implementation.

Then, one function later, the success path took the response it had just validated... and parsed it. Ran all four strategies. Again. Same bytes, same cheerio load, same selector config, this time to produce the record we'd actually save.

Every page that succeeded anywhere in the waterfall was fully extracted exactly twice. Under the right timing, both DOM trees existed in memory at once. Which is how a 2 GB function chokes on a 3 MB page. The parser I'd spent a week optimizing was doing its job flawlessly. Twice per page.

Why nobody saw it

Both call sites were individually correct, which is exactly why the bug survived every review.

Validation genuinely needs extraction: checking that a page yields real product data is the only honest way to grade a scrape. Processing genuinely needs extraction: that's the product. Each function read sane on its own, each was tested on its own, and the duplication lived in the seam between them. Seams belong to nobody, so nobody was looking there.

It also explains why the optimization week "helped". Stripping scripts before parsing made both parses cheaper, so the numbers moved for a while.

The fix

Cache what validation computed, hand it to the processor:

// tier success check: run extraction ONCE, remember the result
const minimumDataResult = await checkMinimumData(
  parserConfig, extractor, response, pageDocument, url,
);
if (minimumDataResult.status === "Success") {
  validationResultCache = minimumDataResult;
}
return minimumDataResult.status === "Success";

// success path: extraction already happened, do not do it again
const minimumDataResult = (result as any).cachedValidationResult;

Twenty meaningful lines. And yes, as any - the cache rode on an object whose type didn't know about it, and I wanted the win in production the same day. The cast is still in the file. I checked this morning.

The receipts

The commit timestamps from that day are the most honest part of this story:

At 12:00, high on a week of optimization, I shipped this:

-    memorySize: 2048
+    memorySize: 512

At 18:33, the extraction cache landed. I'll let you reconstruct how the hours in between went, with a function that was still doing every parse twice on a quarter of its former memory.

The order was backwards, and it worked anyway, in the way that matters: quartering the allocation turned a hidden inefficiency into a visible emergency, and the emergency forced the read that the graphs never did.

Lambda bills memory times duration. Cutting allocation 4x cuts the price of every invocation by 75% before you even count duration, and duration dropped too, because half of all parsing work in the pipeline's hottest function simply stopped existing. No product change. No extraction-quality change. The savings came from deleting work, not from doing the work faster.

The 404-line optimization commit is still in the history. I left it as a monument. forceGarbageCollection is still exported from utils/memory.ts, still imported by the extractor builder. Never called from anywhere. I checked that this morning too. The most honest line of code in the repo.

What actually changed

Two habits came out of that week, and they've paid for themselves several times since.

First, any pipeline with a validate-then-process shape gets one specific review question now: what does validation compute that processing recomputes? The answer is almost never "nothing". Retry wrappers validate. Health checks validate. Queue consumers validate, then hand the raw input to a processor that starts from zero. The duplicated work hides between the two functions, in code no single reviewer owns.

Second, when the symptom is resource usage, the first question is "what work is happening", not "how do I make this work cheaper". A profiler answers the first. Optimizations answer the second. I had them in the wrong order for a full week while the answer sat in a function I'd read ten times without reading it.

Competitive programming taught me, years ago, to read the problem twice before typing. Production has the same rule. The difference is that in production, the problem statement is your own code.