DEV Community

y4u
y4u

Posted on Originally published at uvp.y42u.net

Look at It Before You Feed It to jq — Four Times You Need the Raw Bytes

#logging #json #dataengineering #devops

jq ate 12 GB and died.

The input was an 8 GB JSON log. jq is supposed to stream records one at a time, so why did it exhaust memory?

Finding out took no clever analysis. It took looking at the first three lines of the file.

Structured-data tools are powerful. Every one of them also assumes the structure is what you think it is. When that assumption breaks, the tool doesn't get slower — it stops. And a tool that has stopped tells you very little about what it saw.

What follows are four moments when you need to look at the raw bytes before anything parses them: a jq that never returns, a one-object-per-line log, a giant SQL dump, and a raw TSV on its way to Parquet. All four jam on the same single point.

Up front: UwView opens JSON, SQL dumps, and TSV without parsing them — readable, scrollable and searchable from the moment they open, up to a measured 47.73 GB / ~890 million lines (free version, measured on one setup; results vary — details at the end)

1. jq never returns — check the shape before you parse

Situation

You want to aggregate an 8 GB events.json.

jq '.[] | select(.status >= 500)' events.json
Enter fullscreen mode Exit fullscreen mode

A few minutes in, memory keeps climbing, and then the process is gone. jq -c behaves the same. Trimming the filter changes nothing. The file is valid JSON; it's the tool that can't take it.

Why it happens

The top level is one enormous array.

By default jq reads its input as a single value, completely, before applying the filter. If the whole file is [ {...}, {...}, ... ], the value isn't finished until the closing bracket. Expanded into an in-memory tree, 8 GB of JSON swells to several times its raw size. That's where 12 GB comes from.

The same 8 GB as one object per line (NDJSON) costs almost nothing — jq handles each record and drops it. Same extension, same word "JSON", completely different memory profile.

The awkward part: the difference shows up only in the first byte of the file. Is it [ or {? We hand 8 GB to a tool without checking.

Working around it, and where that stops

Look at the head, then pick a reading strategy that matches the shape.

head -c 300 events.json          # first 300 bytes only
jq --stream -c '.' events.json   # read it as a stream of events
jq -c '.[]' events.json | head   # split the array into records (still reads it whole)
Enter fullscreen mode Exit fullscreen mode

--stream is a genuine answer here. Values arrive as path/value pairs, and memory stays flat.

Three limits. First, head only shows the head. The first 300 bytes can be perfectly healthy while a single malformed record at line 62,000,000 kills the run — and what you get back is parse error: ... at line 1, column 4831838192, a byte offset and nothing else. In a file with one physical line, a line number carries no information.

Second, --stream changes how you think. What was .status becomes matching on [["status"],500]. Rewiring your mental model mid-incident is itself a cost.

Third, every check costs another full read. Find the bad offset, look at what's around it, test another condition — 8 GB of scanning per iteration. Structurally identical to "the third zgrep still takes eight minutes" from Part 8.

2. Reading one-object-per-line logs with your eyes

Situation

Now it's NDJSON. One JSON object per line, forty keys, about 2 KB a line. Open it in less and the screen fills with wrapped braces.

Pretty-print it with jq and it becomes readable — but one record now occupies forty lines, so one and a half records fit on screen. You wanted to follow a sequence of events; you're scrolling slower than the events happened.

Why it happens

JSON is a format for machines. It promises nothing to human eyes.

Object key order is explicitly unordered in the spec. Implementations happen to preserve it, until a library version or a serializer setting changes. In practice that means the same field lands in a different column on different lines. People read aligned tables quickly; they do not read strings that move.

There's more: non-ASCII text is often escaped, so a Japanese word may sit in the file as エラー. Search for the word itself and you get zero hits — a close cousin of the traps in Part 3, where the encoding is correct and the file is still unreadable.

Working around it, and where that stops

Project the fields you care about into TSV.

jq -r '[.ts, .level, .service, .msg] | @tsv' app.ndjson | less
grep '"level":"ERROR"' app.ndjson          # filter the raw text
jq -r '.msg' app.ndjson | sort | uniq -c   # count the shapes of messages
Enter fullscreen mode Exit fullscreen mode

The first one works well. Columns line up and your eyes can move.

Three limits. First, changing which fields you want re-parses everything. You realize you also need .service, then .trace_id, and each realization costs another pass over tens of gigabytes. Investigation is supposed to be "look, then decide what to look at next" — and here that round trip takes minutes.

Second, raw grep depends on serialization. "level":"ERROR" and "level": "ERROR" are the same JSON and different strings. With more than one producer writing the file, half your hits vanish and the counts stop adding up.

Third, one broken line kills the pipe. jq exits non-zero on malformed input. 99,990,000 good lines and one bad one, and the output just stops — and you only learn where by inspecting the tail of what came out.

3. Pulling one table out of a 4 GB SQL dump

Situation

You have a 4 GB mysqldump. You want the INSERT statements for orders and nothing else.

Restoring the whole thing would be reliable, but there isn't room — and there are good reasons not to materialize production-shaped data locally in the first place.

Why it happens

A dump is an index-free script meant to be executed from the top.

There is no table of contents, and table order isn't guaranteed. Where your table sits is unknowable until you read.

Worse for line-based tooling: extended inserts. By default thousands of rows are packed into a single INSERT statement, so one physical line can run to hundreds of megabytes. grep decides per line, so a hit hands back hundreds of megabytes as "one match". sed and awk buffer that line too. The moment "line" stops being a human-scale unit, the whole line-oriented toolbox gets awkward at once.

Working around it, and where that stops

Extract by range.

grep -n 'Table structure for table' dump.sql        # map the table boundaries
sed -n '/Table structure for table `orders`/,/Table structure for table `payments`/p' dump.sql > orders.sql
awk '/^INSERT INTO `orders`/' dump.sql > orders_insert.sql
Enter fullscreen mode Exit fullscreen mode

The third is clean and widely used.

Three limits. First, you need to know the next table's name in advance. A sed range demands an end marker. Check the boundary, write the range, get it wrong, rewrite it — three or four attempts, each a full 4 GB scan.

Second, you'll want to eyeball what you extracted. Opening orders.sql puts you right back in the "open a huge file" problem, and a file containing a 300 MB line freezes most editors on load.

Third, the original position is gone. Nothing in the extract records which line of the dump it came from. When a colleague asks "are you sure that range was right?", answering means scanning 4 GB again. Had you confirmed it on the original, with position intact, there'd be no round trip.

4. Checking a schema guess against the raw TSV, before Parquet

Situation

A 60 GB TSV is headed for Parquet. Type inference reports column 23 as string. By design it's an integer.

The conversion succeeds. Because it succeeds, you find out weeks later, when a total doesn't reconcile.

Why it happens

Type inference only looked at the first N rows.

Most converters sample a few thousand to a few tens of thousands of rows and decide. Reading the file twice is expensive, so this is a defensible design. It also means the single N/A sitting at row 40,000,000 was never seen.

Other failure modes rhyme with it: a tab inside a value shifts the column count; mixed CRLF and LF produce the kind of line-counting disagreement covered in Part 3; a header row reappears mid-file because someone concatenated two exports. Every one of them is a single-line anomaly that corrupts the whole file's type and row count.

Working around it, and where that stops

One full pass that reports the offending line numbers is the reliable move.

awk -F'\t' 'NF!=23 {print NR": "NF}' data.tsv | head        # rows with the wrong field count
awk -F'\t' '$23 !~ /^-?[0-9]+$/ {print NR": "$23}' data.tsv  # rows where column 23 isn't an integer
sed -n '40123456,40123460p' data.tsv                         # look at that row in context
Enter fullscreen mode Exit fullscreen mode

That locates the problem, and awk does it in a single pass, so the check itself is cheap.

Three limits. First, knowing the line number doesn't get you there. That sed counts from the top to reach row 40 million: the moment you learn the number, you've committed to another full read. Want ten more lines of context? Read it again.

Second, you have to decide what "wrong" means before you can write the expression. awk assumes you already know the anomaly. Reality runs the other way — you usually see a strange line first and recognize it as wrong second. Validation starts as observation, not hypothesis testing.

Third, there's never only one kind of anomaly. Fix the field-count rows and mixed date formats surface next. Rewrite the awk, re-read 60 GB. Cost accumulates linearly in the number of times you check.

What the four had in common

Situation Tool used What the tool assumed When the assumption breaks
jq never returns jq the input is the JSON shape you expect memory exhausted, only a byte offset left behind
Reading NDJSON jq -r / grep key order and spelling are stable columns don't align, search terms aren't reliable
Extracting from a dump sed / awk a line is a human-readable length a 300 MB line breaks the line-oriented toolbox
Validating a TSV schema inference / awk the first N rows represent the file row 40,000,000 contradicts the guess

Read the third column downward and the shape appears. Every structured-data tool is optimized for a structure that is already known. That's exactly why they're fast. But the reason we want to open a file is almost always that the assumption has broken. The situation where the tools are strongest and the situation where we need help the most miss each other cleanly.

Collect the places things stop, and three requirements fall out.

  • Look first, parse later. You can't read something whose shape you don't know by first deciding its shape. The first byte, the first three lines, the one line at row 40 million — none of that should require building an 8 GB tree.
  • Jump by line number, and keep the position. Validation hands you line numbers. If a line number isn't a coordinate you can travel to, the validation stopped halfway. And the confirmation should happen on the original, with position intact, not on an extract.
  • Checking more often shouldn't cost more scans. Investigation and validation are inherently iterative — change the condition, widen the range, confirm from another angle. As long as one iteration costs a full read, the more carefully you check, the more time you lose.

None of the three is solved by better structured-data tools. A step is simply missing in front of them: open it as raw text and look. jq and type inference are tools that begin after that step is done.

Back to the opening. jq eating 12 GB wasn't a defect in jq. Nobody had looked at the first byte of an 8 GB file. Looking takes less than a second.

The tools I use

I build UwView (free), a viewer that displays, scrolls and searches a huge text file from the moment it opens. Indexing runs in the background, and line numbers appear when it completes (most other viewers show only the head until indexing finishes). No splitting: the original stays one file, unmodified. JSON, SQL dumps and TSV are all just "text" to it. Because it doesn't parse, it opens files that are broken. That is exactly the "look before parsing" step these four situations were missing.

  • The first byte, the tail, and row 40 million: you can move anywhere from the moment it opens, so checking whether the file starts with [ or { doesn't need a head command. Once line numbers are up, you jump straight to the number your validation produced — the "learning the number commits you to another full read" problem from section 4 doesn't arise.
  • Keep the position on the original: confirm inside the original rather than an extract, and section 3's "which line of the dump was this?" question never has to be asked. Cutting ±N lines for a report works without touching the original.
  • Search with context: search raw strings like "level":"ERROR" directly. When spelling varies, searching the common substring and reading the results is often faster than getting the pattern exactly right. → Drill-down search

To be straight about it: UwView does not interpret JSON. No pretty-printing, no key extraction, no schema validation. That is what jq and your converter are for, and there's no intent to replace them; this tool's job ends at "look, before that." Also, a file with a 300 MB physical line is one line to every viewer, UwView included — reading it comfortably isn't on offer. What works there is using search to pin down the position. And in the free version you get single-stage search, ±1 context and saving results; multi-stage search, sequence search, variable ±N, and persisting the index and compression are Pro features.

And if a huge pile of raw data is eating your disk and you want it compressed for storage while staying searchable at speed, give UwView Pro a look — persistent index, compressed-cache search, and ~1/9 storage make both reopening and searching a step faster (all OS, one-time or monthly). If you validate the same monthly input file over and over, every reopen after the first comes back with line numbers (measured at 0.02–0.07 s on a 47.73 GB text file; varies by environment).

There is also a license that adds non-destructive differential editing to Pro, but all four situations here live in "look, find, extract", so I'll leave it there.

Links

From the developer: A full list of my apps, Kindle books and open-source work lives at GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. jq memory behaviour, the sampling window used by type inference, and mysqldump defaults all vary by version, implementation and configuration. Measured figures come from one specific environment and are not a promise of the same result elsewhere. Command examples may need adjusting for your environment (GNU vs BSD, shell, and the implementation and version of jq, awk and sed). If you spot an error or something inaccurate, please leave a comment and I'll check and correct it.

Top comments (0)