The bug wasn't in the API call. It was in what I assumed people would type.
Hi! I'm Tim, a self-taught developer in Nairobi. I build small data to...
Some comments have been hidden by the post's author - find out more
For further actions, you may consider blocking this person and/or reporting abuse
The line about still not knowing what that customer typed is the bit I'd go after next, since the next odd input will hit the same wall. Logs inside the user's run don't help you there, so I'd send the shape of each error row somewhere you can read, like an error tracker or a webhook you own. Shape, not value, so nothing private leaves the run:
15-1252.00comes through as99-9999.99and "software developer" asaaaaaaaa aaaaaaaaa. That's usually enough to write the next normalization rule without ever seeing the real data. Did you keep the.00stripping silent, or does the output tell people their code was cleaned up?Good catch, it’s silent right now. the row just comes back as 15-1252 and TX, in the same order as the input, but nothing says “you typed 15-1252.00, I used 15-1252”. I should probably put the original next to it
The shape trick is neat, 99-9999.99 tells me everything i’d need. the only thing holding me back is sending anything out of someone’s paid run to my own endpoint. i’d want that written in the readme before i do it
Silent cleanup is where the next support ticket comes from: the row looks fine and the customer still thinks they typed 15-1252.00. I'd keep the normalized values as the source of truth, and add a parallel field only when something changed, like
{ code: '15-1252', code_raw: '15-1252.00', fixed: ['strip_decimals'] }. That can stay inside their run until outbound reporting is ok. On the README point, I'd gate any shape webhook behind an opt-in flag defaulting to off, and document that it sends shapes not values. Do you already have a warnings slot in the output schema, or wouldfixedbe new?no warnings slot yet, just an error field on rows that couldn’t be looked up, so fixed would be new. i like only adding the raw value when something actually changed, keeps the clean rows clean. and opt-in, off by default is how i’d do the webhook too if i ever add it
Keeping
fixedseparate fromerroris worth it. A row can be looked up fine and still changed, and a client filtering onerrorshouldn't miss the ones you quietly rewrote. I'd make the values a small enum (strip_decimals,trim_space,upper_state) rather than free text, so customers can count them per run and you can see which cleanup fires most. Ifstrip_decimalsturns out to be most of them, that's a hint the input docs need one line about codes. Sorry this sat for ten days.No worries, ten days is fast for a dev.to thread lol. the enum makes sense. the cleanups it actually does right now are dropping the ONET .00 suffix, putting the dash back (151252 to 15-1252), trimming spaces, uppercasing the state code and turning full state names like California into CA. so the values would be something like strip_onet_suffix, add_dash, trim_space, upper_state, state_name. and yeah the counting part is the useful bit, if strip_onet_suffix is most of them that tells me people paste codes straight from ONET and the docs should say so. would you put the list on each row or in one summary row per run?
Assumptions about clean user input are the leading cause of production crashes in data pipelines. Having a single malformed row crash an entire paid batch execution—instead of isolating the failure and processing the remaining valid lookups—is a painful but necessary milestone when scaling serverless architectures like Apify Actors.
Your post highlights critical defensive engineering realities for data tools:
Thanks for reading! one small thing, the old loop was sequential (a plain for...of), so Promise.all wasn't what bit me. the throw came from parsing a code before its request went out, and nothing was saved until the end
on zod, I actually went the other way. a strict XX-XXXX check was the problem, because "15-1252.00" and "California" are reasonable things to type. so now it cleans the input first (drops the .00, maps state names) and only rows that still don't make sense get an error, with a reason and no charge
That normalization-first approach makes complete sense for user experience—shifting from pure strict validation rejection to a robust data cleansing layer drastically drives down false friction.
You hit a key point regarding sequential parsing hooks: when parsing happens ahead of the execution pipeline inside a standard iteration loop, catching errors locally before state mutation prevents the entire block from discarding valid downstream data.
To keep this permissive ingestion layer clean as you expand formatting maps, look into structuring Zod's native z.preprocess() triggers. Encapsulating the text transformation logic (like the regex string cleanses and state mapping arrays) completely inside the pre-processing layer ensures your core schemas stay declaration-only, keeping the normalization pipeline decoupled from the final structure evaluation bounds.
Good shout, z.preprocess is pretty much what my normalizeEntry does by hand. for one small actor i'd rather not add a dependency, but if the cleanup rules keep growing i'll look at it
Makes complete sense—avoiding unnecessary dependencies is always a smart move early on. If you ever feel like the manual maintenance is getting annoying, feel free to give it a spin!