One bad row failed a paying customer's whole run 🧯

The bug wasn't in the API call. It was in what I assumed people would type.

Hi! I'm Tim, a self-taught developer in Nairobi. I build small data tools that run as Apify Actors, which are serverless jobs that people pay for per result.

One of them takes a list of occupation codes and returns official US wage data from the Bureau of Labor Statistics (BLS) API. You give it 15-1252 and it gives you back what software developers earn, nationally or in one state.

This weekend it got its second paying user. They ran it three times. Two runs worked. One failed.

The failed run wasn't caused by BLS being down or by a timeout. My own code threw an error on one row, and that threw away every other lookup in the same run.

Let's look at how that happened! 👀

🤔 What I could see

Not much. On Apify, a developer can't open another user's runs unless that user chooses to share them, and this one hadn't.

So I had a failure rate (1 run of 3) and no input to reproduce it with.

Instead of guessing, I asked a different question:

What would a real person actually type into this form?

📋 The old code trusted the input

The tool has a simple input: one SOC code field for a quick lookup, plus an occupations JSON field for checking several at once. I added the single field a few weeks ago, after noticing that a raw JSON textarea scares people off. This weekend the JSON field bit back.

Here's what the old code did with each entry:

function buildSeriesId({ areaType, areaCode, socCode, dataTypeCode }) {
    const occupationCode = socCode.replace(/-/g, '');
    if (!/^\d{6}$/.test(occupationCode)) {
        throw new Error(`Invalid SOC code "${socCode}"`);
    }
    return `OEU${areaType}${areaCode}000000${occupationCode}${dataTypeCode}`;
}

And the run loop:

const results = await fetchOccupationWages({ occupations, apiKey });

for (const record of results) {
    await Actor.pushData(record);
    await Actor.charge({ eventName: 'occupation-checked' });
}

Two things are wrong here, and neither looks wrong on its own:

1. buildSeriesId throws on anything that isn't exactly a 6-digit code.
2. Nothing is saved until every lookup has finished.

Put together, one bad entry anywhere in the list means zero rows saved.

💥 Five inputs a real person might type

I wrote a quick script that called the lookup function directly with inputs I could imagine someone pasting in:

Input	Old result
["15-1252", "29-1141"] (a plain list of codes)	Cannot read properties of undefined (reading 'replace')
{ "socCode": "15-1252.00" }	Invalid SOC code "15-1252.00"
{ "socCode": "15-1252", "stateCode": "California" }	Unknown state code "California"
{ "socCode": "software developer" }	Invalid SOC code "software developer"
two good codes, then 15-1252.00	Whole run fails, 0 rows saved

All five crashed.

The second one hurts the most. 15-1252.00 is how O*NET writes the same code, and O*NET is one of the first places you land when you search for an occupation code. To a person, it's obviously the same occupation. To my regex, it was garbage.

The last row is the real lesson. The two good lookups had already been sent to BLS, so the tool did the work and then threw the results away.

So I wrote down the rule I should have started with:

In a batch tool, the unit of failure is the row, not the run.

🔧 Fix 1: understand more, and explain the rest

Each entry now goes through normalizeEntry before anything else. It accepts more shapes, and when it can't make sense of an entry, it returns an explanation instead of throwing:

export function normalizeEntry(entry) {
    const raw = typeof entry === 'string' || typeof entry === 'number'
        ? { socCode: String(entry) }
        : (entry ?? {});
    const rawSoc = raw.socCode ?? raw.soc ?? raw.code ?? raw.occupationCode;
    // ...
    const digits = String(rawSoc).trim().replace(/\.\d{2}$/, '').replace(/[\s-]/g, '');
    if (!/^\d{6}$/.test(digits)) {
        return { socCode: rawSoc, error: `"${rawSoc}" isn't a SOC code. Use the 6-digit code, e.g. "15-1252" for software developers.` };
    }
    // ...states: "TX", "tx", "Texas" and "texas" all work; "US" or empty means national
}

A plain string is treated as a code, the .00 suffix is dropped, and full state names map to their two-letter codes.

Anything that still doesn't make sense becomes a row with found: false and an error saying what to fix. The run carries on with the rest of the list.

💸 Fix 2: don't charge for the error rows

This is a pay-per-result tool, so "return an error row" raised a second question: should that row cost money?

No. Every lookup now carries a billable flag, and only the lookups BLS actually answered get charged:

for (const { billable, ...record } of records) {
    await Actor.pushData({ ...record, checkedAt: new Date().toISOString() });
    if (billable) await Actor.charge({ eventName: 'occupation-checked' });
}

While I was in there, I found an older billing bug. When BLS refuses a request, for example once its daily limit is used up, the reply comes back with a status like REQUEST_NOT_PROCESSED instead of data. The old code turned that into a found: false row and charged for it. Now those rows say what happened and add "You weren't charged for this lookup."

📦 Fix 3: batch the requests

That BLS limit deserved a closer look. Without a registration key, BLS allows 25 queries a day with up to 25 series per query.

Each occupation lookup needs 5 series: employment, mean and median annual wage, and mean and median hourly wage. The old code sent one request per lookup, so 5 series per request and 25 lookups a day before BLS started refusing.

Now valid lookups are grouped, 5 per request (10 with a key, where the limit is 50 series per query):

const perRequest = Math.floor((apiKey ? 50 : 25) / 5);

for (let start = 0; start < valid.length; start += perRequest) {
    const batch = valid.slice(start, start + perRequest);
    // one POST with every series for every lookup in the batch,
    // then map each returned series back to its lookup by series ID
}

That's 5 times more lookups out of the same daily allowance.

🧪 Testing it with a deliberately messy list

I ran the tool locally with the messiest input I could think of, seven entries in mixed shapes:

{
  "occupations": [
    "15-1252",
    { "socCode": "29-1141.00", "stateCode": "California" },
    { "socCode": "software developer" },
    { "socCode": "15-1252", "stateCode": "Narnia" },
    { "soc": "11-1021", "state": "tx" },
    "13-2011",
    "15-1211"
  ]
}

Result:

Entry	Found	Annual median wage	Error	Charged
15-1252 (national)	✅	135,980		yes
29-1141, CA	✅	140,270		yes
software developer	❌		isn't a SOC code	no
15-1252, Narnia	❌		unknown state	no
11-1021, TX	✅	103,230		yes
13-2011 (national)	✅	83,680		yes
15-1211 (national)	✅	105,850		yes

7 rows out, 5 charged, and all 5 valid lookups went to BLS in a single request.

I also tested an empty form. Apify runs every public Actor once a day with its default input, and flags it as under maintenance if that keeps failing. An empty form now looks up an example code instead of throwing.

🙀 One false alarm

My first local test runs exited with code 127, which made me think I'd broken the shutdown.

The log said:

Assertion failed: !(handle->flags & UV_HANDLE_CLOSING), file src\win\async.c, line 94

Before touching anything, I stashed my changes and ran the old code. It printed the same assertion.

It's Node 24 on Windows asserting while the process shuts down, after all the work is done. The cloud runs these on Linux with Node 20, so it never shows up there. If you only check exit codes on a Windows machine, it's worth knowing about.

🎯 What I learned

• Fail the row, not the run. In anything that takes a list, one bad item should produce one error row.
• Save as you go, or at least don't throw away finished work. The old code had done the good lookups and still saved nothing.
• Accept what people actually type. Codes copied from other sites, full state names and plain lists aren't edge cases. They're the input.
• An error row needs a reason and a price. Say what to fix, and don't charge for it.
• Read the upstream API's limits before they read you. Batching gave 5 times the headroom for free.

I still don't know exactly what that customer typed. But every input I could think of now gets either an answer or a clear reason why not.

Comments

[Oleksandr]

Assumptions about clean user input are the leading cause of production crashes in data pipelines. Having a single malformed row crash an entire paid batch execution—instead of isolating the failure and processing the remaining valid lookups—is a painful but necessary milestone when scaling serverless architectures like Apify Actors.
Your post highlights critical defensive engineering realities for data tools:

1. The All-or-Nothing Batch Trap: Wrapping an entire dataset loop in a monolithic execution block without granular row-level error boundaries means your system's uptime is completely dependent on perfect user data hygiene.
2. API Resilience vs. Code Robustness: It is incredibly common to heavily guard against external infrastructure failures (like BLS API rate limits or network timeouts) while leaving internal string mutation or parsing logic completely exposed to unhandled exceptions.
3. Monetization Friction: In a pay-per-result model, a full-run crash directly impacts user trust and revenue. A user paying for a batch expects the system to gracefully handle partial successes rather than discarding the entire operation. To ensure your data pipelines survive dirty user inputs and process large lists resiliently without dropping valid rows, here are two actionable patterns you can implement in Node.js: • Settled Promises for Independent Row Execution: Instead of resolving arrays concurrently using Promise.all()—which instantly rejects the entire batch if a single input throws an error—switch to Promise.allSettled(). Map each occupation code lookup to its own isolated asynchronous task wrapped in a internal try/catch block. This guarantees that every row is evaluated independently, allowing you to return the successful data points to the user alongside an explicit list of validation errors for the failed rows. • Decoupled Schema Validation via Zod: Never let raw input reach your business logic or external API handlers unverified. Implement a strict runtime validation layer using a tool like Zod right at the ingestion point. By defining a clear schema for your codes (e.g., enforcing string lengths, specific character masks like regex for the XX-XXXX format, and trimming white spaces), you can instantly filter out or log malformed rows before they ever enter the core lookup processing engine.

[Timothy Kelvin]

Thanks for reading! one small thing, the old loop was sequential (a plain for...of), so Promise.all wasn't what bit me. the throw came from parsing a code before its request went out, and nothing was saved until the end
on zod, I actually went the other way. a strict XX-XXXX check was the problem, because "15-1252.00" and "California" are reasonable things to type. so now it cleans the input first (drops the .00, maps state names) and only rows that still don't make sense get an error, with a reason and no charge

[Oleksandr]

That normalization-first approach makes complete sense for user experience—shifting from pure strict validation rejection to a robust data cleansing layer drastically drives down false friction.
You hit a key point regarding sequential parsing hooks: when parsing happens ahead of the execution pipeline inside a standard iteration loop, catching errors locally before state mutation prevents the entire block from discarding valid downstream data.
To keep this permissive ingestion layer clean as you expand formatting maps, look into structuring Zod's native z.preprocess() triggers. Encapsulating the text transformation logic (like the regex string cleanses and state mapping arrays) completely inside the pre-processing layer ensures your core schemas stay declaration-only, keeping the normalization pipeline decoupled from the final structure evaluation bounds.

[Timothy Kelvin]

Good shout, z.preprocess is pretty much what my normalizeEntry does by hand. for one small actor i'd rather not add a dependency, but if the cleanup rules keep growing i'll look at it