Starting off with a full disclosure: I’m not an engineer. Together with my friend - Chapa - we run FourA, a data harvesting infrastructure via a single API. This is a founder’s post; if you wanted a deep dive into code — ping the one, wearing the CTO hat ;]. I ramble on about a pricing bottleneck developers kept complaining to me about, so I wanted to share how we dealt with it.
The thing that bugged us
Almost every scraping and data API bills you per request. You send a request, you pay for it. What actually comes back doesn't seem to matter to the invoice.
That means you pay for the 403 Forbidden. You pay for the Cloudflare challenge page. You pay for the empty body when a target site quietly serves you absolutely nothing. You even pay for the proxy that echoes your own request back as a fake response. Okay, maybe you're just the dev guy and the CDO/CTO pushes the invoice to the CFO, but that's beyond the point.
At low volumes, you don't really notice. At scale, a massive chunk of your bill is just dead weight—money spent on requests that yielded zero data. Ironically, the harder a target site fights back, the more you end up paying the scraping vendor to get blocked. The pricing model literally rewards the vendor when the product fails you.
That felt completely backwards to us. So we decided to change it: you should only pay when you get what you actually asked for. A failed request shouldn't be your financial problem.
Why "just don't bill failures" is a trap
It sounds incredibly simple on a landing page; as a slogan. In production, it’s a nightmare because "success" is highly subjective.
The naive assumption is that success equals an HTTP 200 OK. It doesn't.
Plenty of scraping pipelines legitimately need to record a 404 or 410 to know a product or page is gone.
Some APIs return a 202 or a redirect that contains the actual data you want.
Conversely, plenty of anti-bot walls will happily return a 200 OK status code while serving you a page that just says "Please prove you are human." Technically it's a success; actually, it's garbage.
If we just threw a naive status == 200 ? bill : don't_bill rule into our system, we’d penalise people who need non-200 responses, while still charging users for 200-coded junk.
Our fix: We shifted the definition of success to the user. Now, you send validation rules right along with your request—the status codes you accept, strings that must be present in the HTML, etc. If the response matches your criteria, it’s a success and it bills. If it doesn’t, it’s a failure and it’s free.
Now, a 404 you asked for counts, and a 200 CAPTCHA wall you didn't ask for doesn't. You draw the line, and the bill follows it.
As the non-technical person in the room, I walked away with two main takeaways:
The honest version of a feature is always the hardest one to build. "We don't charge for failures" is a great hook. But making it true meant building a validation engine so customers could define failure themselves. It’s way more complex than a clean marketing slogan, but the simple version would have been an accidental lie for half our users.
Stop guessing, start looking at the logs. Instead of guessing what developers wanted, we shipped the validation feature, looked at our actual internal billing feeds, and iterated based on how data was flowing. (Spoiler: our first version missed a ton of edge cases, which is the part nobody puts on a launch page).
Would love to engage the audience, that can speak from experience. If you run data pipelines at scale: how much of your current scraping bill is spent on responses your parsers just throw away?
Top comments (0)