DEV Community

y4u
y4u

Posted on Originally published at uvp.y42u.net

The Morning the Monthly Batch Died — Four Realities of Running a Data Pipeline

#dataengineering #etl #devops #cli

line 48,213,996. That's all you got.

Can you look at that line right now?

The pipeline hands you a coordinate. The thing it points at is somewhere else entirely.

Pipeline failures don't behave like application failures. What breaks isn't your code — it's the input data, produced outside your control, and the evidence is one line inside tens of gigabytes. Four scenes here — reading around the line that died, hunting the odd rows in a training corpus, reopening the same monthly input every month, and looking at the original before you transform it — each starting from what actually jams.

Up front: In UwView Pro you type the line number and land on 48,213,996 — and you land in the original, not an extracted fragment, so you can widen the context as far as you like without re-reading the file. The index and the compression are saved to a sidecar, so next month the same input file reopens with line numbers still attached in 0.02–0.07 s (measured on a 47.73 GB text file; one specific setup, results vary). When you move on to fixing things, the original stays byte-identical: edits accumulate in a diff file, and you can stop halfway and resume tomorrow (details at the end)

This article is about reading the raw data on either side of a pipeline, on your own machine. What you may handle, move, or retain is governed by your organisation's policy and your client's.

1. All that's left is a line number

Situation

The monthly batch dies. The alert is waiting for you in the morning.

You open the log. There's one line in it.

ERROR: failed to parse record at line 48213996: unexpected field count (expected 12, got 13)
Enter fullscreen mode Exit fullscreen mode

That looks like enough. A line number, an expectation, an actual. And you still can't fix it.

Why there are thirteen fields is not something you can determine without seeing the line. A delimiter inside the data, an unclosed quote, a missing newline that welded two records together — all of these happen, and which one happened is decided by the bytes, not by reasoning.

The input file is 31 GB.

Why it happens

A parser is not built to report a broken line. It's built to stop.

What it can tell you is where it stopped and how reality differed from its expectation. Why the count became thirteen lies outside its concerns entirely.

Worse, the reported line number is not necessarily the line you want.

  • An unclosed quote makes the parser swallow dozens of lines as one record. The error surfaces where it finished swallowing; the cause is dozens of lines earlier
  • Mixed line endings make the parser's idea of a line differ from your editor's (part 13)
  • Header-skipping implementations report a number that's off by one
  • Parallel readers sometimes report a position relative to a chunk, not the file

So a line number is where you start looking, not where the answer is. Which is why "extract just that line" doesn't get you there — you need to be able to widen.

General tools, and where they stop

You go look at the line.

sed -n '48213976,48214016p' input.tsv

sed -n '48213976,48214016p;48214017q' input.tsv

awk -F'\t' 'NF!=12 {print NR": "NF}' input.tsv | head -50
Enter fullscreen mode Exit fullscreen mode

That third command is the one that matters. Whether there is one broken line or four thousand changes the entire response — one you fix by hand, four thousand means you go argue with whoever generated the file.

Three limits.

First, sed makes you choose the range before you know what you need. You look at ±20, realise the cause is further back, and reissue with ±200. Each attempt counts 48,213,976 lines from the top of a 31 GB file. Every wrong guess is a full pass.

Second, the original coordinates vanish the moment you excerpt. Line 1 of that sed output is line 48,213,976 of the original, and that mapping now exists only in your head. Find something suspicious inside the excerpt and you're doing arithmetic to say where it really lives. What goes in the handover document is only as good as that arithmetic.

Third, awk over the whole file means one pass per idea. Check field counts; then quote counts; then line lengths. Every new way of looking costs another 31 GB read. Your investigation runs at the speed of I/O, not at the speed of thought (part 15).

2. The odd rows in a 50 GB corpus

Situation

You're assembling training data. The concatenated corpus is 50 GB.

You write the preprocessing, kick off a run. Hours later the loss curve has a shape it shouldn't. Or the outputs contain strings you don't recognise.

The cause is somewhere in the data. But "somewhere" isn't something the training side can localise for you.

Why it happens

Because you cannot state the criterion in advance.

The pipeline failure in part 1 had an explicit one: twelve fields. Corpus validation has none. You're looking for "odd rows", and odd gets defined after you see them.

List what actually turns up and the impossibility of enumerating it ahead of time becomes obvious.

  • Mass-duplicated lines — a crawler's error page, tens of thousands of times, byte-identical
  • HTML and JSON debris — tags the extractor missed, un-unescaped &, raw script tags
  • One absurdly long line — a Base64-embedded image arriving as a single multi-megabyte row
  • Encoding accidents — one source concatenated in Shift_JIS, read back as UTF-8 (part 3)
  • Personal data that shouldn't be there — not a matter of counts; one occurrence is one too many

That last one is what makes "sampling is fine" indefensible. Ten thousand clean sampled rows tell you nothing about the rest.

General tools, and where they stop

Start with distributions.

awk '{print length}' corpus.txt | sort -n | tail -20

sort corpus.txt | uniq -c | sort -rn | head -20

grep -cP '[\x00-\x08\x0B\x0C\x0E-\x1F]' corpus.txt

grep -cE '</?(script|div|span|br)\b' corpus.txt
Enter fullscreen mode Exit fullscreen mode

Line length is the cheapest signal. A multi-megabyte line is itself the evidence.

Three limits.

First, sort does not survive 50 GB gracefully. Counting exact duplicates the honest way means sorting the file, which wants temp space on the order of the file itself plus the time to match. --parallel and -T help, but this is no longer in the "let me just check something" weight class.

Second, a count doesn't give you the thing. grep -c says 1,247. Whether those 1,247 are one problem or seven different problems is decided by looking. You can grep them out, but extraction erases their position in the corpus — and "a cluster from one source" versus "scattered throughout" is a judgement you can only make with position in hand.

Third, validation is a round trip. Find an anomaly, fix the preprocessing, rebuild, validate again. If one lap is forty minutes, every lap is forty minutes. The lap count is why corpus work never seems to end.

3. Reopening the same 31 GB every month

Situation

The monthly batch input is a large file with roughly the same shape every time.

You opened it last month. And the month before. Each time you do the same things: check the header, count records, look at the date range, search for the failure patterns you already know about.

The work itself takes thirty minutes. But opening takes several minutes, and each search costs tens of seconds. Every month, the same wait.

Why it happens

Because the wait is being accounted for as a one-time cost.

Three minutes is tolerable in the moment — it's a coffee. The problem is that it compounds across twelve months and however many people do this, and, more fundamentally, that a three-minute reopen makes people check less.

  • "While I'm here, let me also look at…" stops happening. Reopening is expensive, so you try to do it all in one go
  • You think of a search after closing the file, and you don't run it
  • The net effect: the thoroughness of your checking silently adjusts itself to the wait time

And one more thing. Monthly work carries no record of what last month's you actually checked. You rebuild the same queries from memory. Each rebuild differs slightly, and comparability quietly disappears — the same failure mode as an investigation procedure that lives only in one person's head.

General tools, and where they stop

It's routine work, so scripting it is the right instinct.

#!/bin/sh
f="$1"
echo "== header =="; head -1 "$f"
echo "== lines  =="; wc -l < "$f"
echo "== date range =="; head -2 "$f" | tail -1 | cut -f1; tail -1 "$f" | cut -f1
echo "== known bad patterns =="
grep -cE '\t\t|^\t|\t$' "$f"        # empty fields
awk -F'\t' 'NF!=12' "$f" | wc -l    # field count anomalies
Enter fullscreen mode Exit fullscreen mode

Now the checks are fixed and comparable month to month. Writing the checklist down is worth something on its own.

Two limits.

First, a script only does the checks you decided on. When a number differs from last month, what happens next is not in the script. From there you open the file and look — and the wait arrives exactly there. Scripting removes the routine part only.

Second, the read-from-scratch structure is untouched. The script above walks 31 GB three times: once for wc -l, once for grep -c, once for awk. It did the same last month. Nothing from the previous read survives anywhere.

4. Look at the original before you transform it

Situation

A partner sends you 258 GB of XML to ingest.

There's a spec. There's a schema. You write the transform, put it on the pipeline. It runs for hours and dies. Or it completes and the record count doesn't match what you expected.

Either way the next step is the same. You go look at the original.

And 258 GB of XML doesn't open in most editors.

Why it happens

Because a spec describes what should be, and the file describes what happened.

Between organisations, those two diverge essentially always.

  • Tags not in the spec — the sender extended something and your copy of the spec predates it
  • Different handling of absence — empty element or no element at all. The spec is silent; the parser is not
  • Character references and escaping — &amp;amp; is a fossil of a double-escaping step upstream
  • The format changes mid-file — receive several years in one delivery and you may be crossing a system replacement on the sender's side
  • A truncated tail — a transfer that stopped halfway looks perfectly fine at the head

That last one is the emblem of the whole problem. A defect you could see by glancing at the end of the file stays invisible because you can't open the file. You find out hours into the transform instead.

General tools, and where they stop

Inspect without opening.

head -c 2000 huge.xml
tail -c 2000 huge.xml

grep -oE '<[a-zA-Z_][a-zA-Z0-9_:.-]*' huge.xml | sort | uniq -c | sort -rn | head -30

xmllint --noout --stream huge.xml

grep -n -m 5 -A 10 '<UnexpectedTag' huge.xml
Enter fullscreen mode Exit fullscreen mode

tail -c is cheap and decisive. If the file doesn't end in a closing tag, the investigation is over and you're asking for a resend.

Three limits.

First, grep doesn't know XML. You can count tag names, but nesting depth and parentage are outside its world. "Which record is this tag inside?" needs either a structure-aware tool or a pair of eyes.

Second, xmllint --stream stops at the first error. A hundred defects report as one. Fix, rerun, learn about the next one — a hundred times, at tens of minutes a lap.

Third, the going-and-looking round trip is heavy. You're told line 18 million, you look, it isn't enough, you look earlier, then earlier still. Whether that loop is cheap or expensive is decided entirely by whether the original opens.

What the four had in common

Scene What you're given Where it jams General-tool approach What's left over
Batch died One line number Reported position ≠ cause position sed to excerpt ±N Range must be chosen first; excerpting kills coordinates
Corpus validation No criterion "Odd" gets defined by looking Length / duplicate / control-char distributions Counts without the rows or their positions
Monthly input check Last month's procedure Checking thins out to fit the wait Freeze the checks in a script Re-read from zero; nothing carries over
Pre-transform check A spec Spec and file diverge head / tail / tag inventory Heavy round trips; one error at a time

Four different stages, four different people. The right-hand column rhymes anyway, because all four require seeing a specific place in the actual file, with its context — and that is exactly the expensive part.

In a pipeline, data is something that passes through. Read, transformed, written. A human looking at the bytes is the exceptional case — and so nobody has tooling for the exceptional case. When it dies, you improvise a window out of sed and awk.

Three conditions, then.

  • Jump by line number, and land in the original — with an excerpt, widening the context means rebuilding it
  • Make the second open cheap — neither validation nor monthly checking is a one-shot job. If you can't reduce the number of laps, reduce the cost of a lap
  • Keep the original when you start fixing — migration and data repair need the baseline available to the very end

Back to line 48,213,996. The line number wasn't insufficient. The route to that line just stopped at the edge of 31 GB.

The tool I use

I build UwView (free), a viewer that makes huge text readable, scrollable and searchable from the moment it opens. It doesn't pull the file into memory, so files larger than RAM open fine. Indexing runs in the background and line numbers appear when it completes (most viewers show you only the head until indexing is done). It never splits or extracts, so the original stays one file, unmodified.

  • Jump by line number. Type 48,213,996 and you're there — in the original, so you widen the context as far as you want. No choosing a range up front, and a wrong guess costs nothing
  • Reach the tail in one move. Part 4's "does it end in a closing tag" is answerable on 258 GB right after opening — and unlike tail -c, you can walk back upward from there
  • Switch encodings without reopening (UTF-8 / Shift-JIS (CP932) / EUC-JP / UTF-16, auto-detected). Part 2's Shift_JIS contamination is confirmable without producing a converted copy

Beyond that is UwView Pro.

  • The index and compression are saved. This is part 3, directly. From the second open onward the file comes back with line numbers, instantly (0.02–0.07 s measured on a 47.73 GB text file; one specific setup, results vary). "Reopening is expensive, so I'll check less" stops being a decision you have to make
  • Save the queries, resume tomorrow. Against part 3's rebuilt-from-memory searches, the conditions themselves become an artefact. Sessions restore, so monthly work starts as a continuation
  • Drill-down search. Narrow a result set further, again. Part 2's "are those 1,247 one problem or seven?" gets sorted on screen without producing extract files. Tabs carry term (count) and the original line numbers survive to the last stage (drill-down article)
  • Tally (frequency ranking). Count occurrences captured by a regex and click a row to descend to it — close to part 2's distribution work, without walking the file once per idea. It does not sum or average; the lineage is grep -oE | sort | uniq -c
  • Store at roughly 1/9 and still search it. Keeping twelve monthly inputs a year becomes compatible with rechecking them (part 8)

And the part this series kept flagging — "once you've found it, fixing it means rewriting the original" — got an answer in v1.4.0 with Edit Upgrade. It lands squarely on migration and data-repair work.

  • The original is never rewritten. Edits and bulk replacements accumulate in a diff file (.ewvz). Fixing the broken row from part 1 leaves the baseline intact to the end
  • Repair cost doesn't scale with file size. In development measurement, 21,994 replacements across a 10 GB, 100-million-line file took 16.8 s (measurement article; macOS, external SSD, .uwvz-backed file, single run, results vary)
  • Stop halfway, resume tomorrow. Pause and continue without resaving the whole file — the property that multi-day work like part 2's corpus validation actually needs

Stated plainly: UwView is neither a validator nor an ETL tool.

  • Part 1's "pull every row with the wrong field count" is awk's job. This tool's turn comes after — putting the resulting line number onto the real thing
  • It does not understand CSV or XML structure. No sorting by column name, no type checking, no schema inference, no nesting validation. Expect a "huge CSV editor" or an "XML editor" and you'll be disappointed
  • Tally has no sum or average. Counting part 2's duplicates is sort and uniq
  • It does not run, schedule, retry, or monitor pipelines
  • Edit Upgrade operates on .uwvz-backed files, not on a raw CSV in place — you convert first, and it works as an add-on to a View License (it does not run standalone)
  • It handles text. Parquet and binary database dumps are out of scope

The same things, from the command line

v1.6.0 added a uvp command (and uvf for the free build). It uses the same .uwvz as the GUI, so an index built from the shell is already there when you open the file in the app. Mapped onto this article's four parts:

uvp input.tsv 'order_id=A1B2C3' -C 40

uvp input.tsv 'order_id=A1B2C3' -C 40 -open

uvp corpus.txt -uniq '^(.{0,30})' -head 20

uvp 2026-09.tsv 'ERROR' 'timeout' -out check-202609.txt.gz

uvp huge.xml.uwvz '<UnexpectedTag'
uvp huge.xml.uwvz -extract -out restored/   # restoring the original is free
Enter fullscreen mode Exit fullscreen mode

Exit codes are grep's — 0 found, 1 not found — plus 2 when the 1,000,000-hit cap is exceeded. if uvp input.tsv 'ERROR'; then works as written, so it drops straight into part 3's check script.

Stated honestly: on the first question ripgrep is 15–20% faster, because uvp builds its index first. On a 3 GB file that fits in RAM, rg stays ahead on the second question too. uvp pays off past 10 GB, when you ask the same file more than one question (the measurements; Mac M4, external USB SSD, OpenStreetMap XML — one setup, results vary).

What it does is find things, show you the real thing, and let you repair it without breaking the original. If you're doing monthly input checks or multi-day migration repairs, try it.

  • 14-day free trial (everything in View + Edit, no payment details)
  • UwView Pro (View) — persistent index, compressed-cache search and ~1/9 storage make both reopening and searching a step faster (all OS, one-time or monthly)
  • Edit Upgrade — non-destructive diff editing, bulk replace on huge files, pause and resume (add-on to a View License)

Links

From the developer: A full list of my apps, Kindle books and open-source work lives at GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. The error messages, line numbers, file sizes and counts are illustrative and do not refer to any real dataset or engagement. Quote handling in CSV/TSV, line-ending behaviour, and XML character-reference handling vary by parser and library version. Command examples may need adjusting for your environment (GNU vs BSD, differences among awk, sed, sort, grep, xmllint, availability of grep -P). Always confirm option names and defaults against your local man. Figures described as measured come from one specific setup and are not a guarantee of the same result; disk type, filesystem, fragmentation, encryption, page-cache state and concurrent processes change outcomes substantially. Handling of training data and migration data is governed by your organisation's policy, your client's, and applicable law. If you spot an error, please leave a comment and I'll check and correct it.

Top comments (0)