DEV Community

Raknaos
Raknaos

Posted on

My JSONL tool printed null for a broken line and nothing in my pipeline noticed

#cli #json #python #tools

I put a JSON Lines parser in front of a pipeline that reads export dumps from other people's
services. Some of those dumps are not clean. Records get truncated by a proxy, a log shipper
writes half a line and dies, someone opens the file in a spreadsheet app and saves it back with
a stray tab. The failure I cared about was not "the file has one broken line" - it was "the file
has one broken line and my pipeline finished with exit code 0".

That is the exact behaviour the first version of jsonl-cli had, and I did not catch it in review.
I caught it because I ran the tool on a real export instead of on the fixture.

What the tool is

jsonl-cli is a single-file, stdlib-only Python 3 CLI for
JSON Lines (.jsonl) streams: validate, count, get, pretty. All four take a path or -
for stdin, and get resolves a dotted path into nested structures - user.profile.id for
objects, numeric steps for arrays, so user.roles.0 works. It is the first tool in the Raknaos
Tools Lab collection, published 08-09-2026, MIT, no third-party imports. The whole point of the
lab is tools you can drop onto a machine with curl and nothing else.

Here is the pipeline shape that broke the illusion, run against a four-record file with one
corrupt line:

$ python3 jsonl_cli.py get user.email data.jsonl
"a@x.io"
null
null
"b@x.io"
$ echo $?
0
Enter fullscreen mode Exit fullscreen mode

Four valid emails would have printed four quoted strings. I got two quoted strings and two
nulls and a clean exit. Reading that output as a human, I have no idea which of the two null
lines were records that genuinely had no email and which were records that never parsed at all.
Reading it as a shell, the situation is worse: exit 0 means "nothing to see", so the job that
wraps this command reports success and the count quietly drops.

The two errors are not the same error

The root cause was the shape of the loop: the parse, the field lookup and the print all sat
inside one try, and the handler printed a null.

# first release
def cmd_get(args):
    with _open_input(args.file) as f:
        for line in f:
            line_str = line.strip()
            if not line_str:
                continue
            try:
                obj = json.loads(line_str)
                val = _resolve_dotted_key(obj, args.key)
                print(json.dumps(val))
            except Exception:
                print("null")
    return 0
Enter fullscreen mode Exit fullscreen mode

return 0 is unconditional, so the function cannot report a problem even if it wanted to. And
print("null") conflates two completely different facts:

  1. The line parsed fine and the path is absent in it. That is data. The answer to "what is user.email in this record" genuinely is null, and the pipeline should carry on.
  2. The line is not JSON. That is an input error. Silently turning it into the same null as case one destroys the only signal that anything went wrong.

A null on stdout is a value. A corrupt line is not a value, it is a reason to distrust the rest
of the output. The second commit fixes exactly that separation.

What it does now

Invalid lines go to stderr with their line number, the offending line is skipped rather than
faked into a value, and the exit code flips to 1. A missing key still prints null and still
exits 0, because that is a legitimate answer:

# current
def cmd_get(args):
    had_invalid = False
    with _open_input(args.file) as f:
        for line_no, line in enumerate(f, start=1):
            line_str = line.strip()
            if not line_str:
                continue
            try:
                obj = json.loads(line_str)
            except Exception as e:
                # an invalid JSON line is an input error, NOT a null value:
                # report it on stderr and flip the exit code so pipelines notice
                sys.stderr.write(f"Line {line_no}: invalid JSON - {e}\n")
                had_invalid = True
                continue
            val = _resolve_dotted_key(obj, args.key)
            print(json.dumps(val))
    return 1 if had_invalid else 0
Enter fullscreen mode Exit fullscreen mode

Same file as before, same command:

$ python3 jsonl_cli.py get user.email data.jsonl
Line 3: invalid JSON - Expecting value: line 1 column 1 (char 0)
"a@x.io"
null
"b@x.io"
$ echo $?
1
Enter fullscreen mode Exit fullscreen mode

Three values on stdout instead of four, which is the honest answer: one of those four lines is
not a record. pretty got the same treatment, and count picked up a guard on
--valid-only --invalid-only - the first release accepted both flags and quietly let one win,
which is a 2 (usage error), not a 0. The exit-code contract is now written in the module
docstring: 0 clean, 1 at least one invalid input line, 2 usage error.

What it does not do

Honesty about the edge, because it will bite you:

  • Exit 1 is not a per-line contract. It says "at least one input line was bad somewhere in this stream". It does not tell you how many, and it is the same code whether one line out of a million failed or every line failed. If your pipeline needs the distinction, use count --invalid-only and branch on the number, not on the status.
  • A missing key is silent by design. get user.profile.id on a record without a profile prints null and exits 0. There is no flag that turns absent-path into an error, so a typo in your key path looks exactly like a record that lacks the field. validate will not help you either - it only checks that each line parses, never that a key exists.
  • Nothing is type-checked. get returns whatever is at the path, JSON-encoded: a string, an object, an array, a number. If you need the value usable downstream you parse it again.
  • Blank lines are skipped, not reported, in every subcommand. A file that lost half its records to a grep -v accident is not something this tool looks for.
  • stdin is read as text, one line at a time. There is no buffering of the whole stream, which is what makes it usable on a growing log, and there is no encoding auto-detection beyond open(path, "r", encoding="utf-8").
  • The BrokenPipeError in main() maps to sys.exit(0). That is deliberate - jsonl get k f | head -1 should not report failure because head stopped reading - but it does mean the exit code of a truncated pipeline is not meaningful. Check the status of the command you actually care about.

Trying it 

curl -sSL https://raw.githubusercontent.com/Raknaos/jsonl-cli/main/jsonl_cli.py > jsonl
chmod +x jsonl
./jsonl get user.email - < data.jsonl
Enter fullscreen mode Exit fullscreen mode

Repo: https://github.com/Raknaos/jsonl-cli - jsonl_cli.py, test_jsonl_cli.py, MIT, Python 3.8
and up, standard library only. The unit tests run under python3 -m unittest discover with no
install step, and the CI job for the repo is that one line.

Top comments (2)

Collapse
 
iamantonreznik profile image
Anton Reznik •

I would've missed the exit 0

Collapse
 
raknaos profile image
Raknaos •

Thanks — and same on my side, honestly: the exit 0 shipped in the first release precisely because I was reading the printed values and never the status. The null made the output look inspectable, which is exactly what made it dangerous.

The distinction that finally felt right: a missing key is an answer, a broken line is a complaint. Once the complaints go to stderr with line numbers and the exit code reflects them, get can stay a quiet query — and the wrapper job around it gets a real signal to alert on. The part I still get wrong occasionally is piping stderr into a log nobody reads, which quietly puts you back where you started.