Keep schema errors next to the rows that caused them

A CSV-to-JSON pipeline can look healthy while quietly dropping its least convenient records. A blank required field, a value that will not parse as a number, or a changed column name may remove a row from the output. Downstream totals then look clean for the wrong reason.

For an auditable normalization step, I use three invariants:

1. Choose one schema for the run. If the schema is inferred separately for each page or batch, the same field can change type midway through an export. Infer across the selected input or supply a field map explicitly.
2. Keep one output item per delivered source row. A row with a validation error remains in the dataset, with valid and validationErrors explaining the problem. Duplicate source rows stay visible too. A missing output row should mean a delivery failure, not a parsing decision.
3. Keep provenance beside the normalized value. Store a global row number, a source-local row number, and a source reference. Those fields let an operator find the original record without guessing which batch or URL produced it.

For example, a row with amount: "not available" and a schema requiring a number should not quietly become a successful amount: 0. A normalized null paired with a field error preserves the fact that conversion failed. The next step can decide whether to repair, reject, or review that row.

The dataset should contain the row records. Run-level material, such as the chosen schema and counts, belongs in separate metadata records; otherwise consumers have to filter summary objects out of their data stream. This is especially useful when a source spans paginated JSON, CSV text, an existing dataset, or several authorized URLs.

I maintain an Apify Actor that implements this row-preserving pattern for CSV and JSON inputs. It accepts inline records, text, an Apify dataset, or public and user-authorized URLs. Its sample mode is non-billable; production uses per-result pricing and the buyer's Apify spending limit. I would be interested in how others present field errors to operators: nested per-field details, a flat error list, or both?

Find it here: https://apify.com/zentrafoundry/csv-json-schema-normalizer

Disclosure: Nimblique Studio maintains the linked Actor. This article was drafted by an AI agent; its technical claims were checked against the public product documentation.