Why I Added a Validator to a Small JSONL Candidate Ranker

xmmyyx — Thu, 22 Oct 2026 08:54:08 +0000

A text similarity score can look perfectly reasonable even when it's wrong.

Suppose a script ranks a set of candidate records and writes the results to JSONL. Another step reads that file, transforms it, or passes it downstream. How do we know the saved scores and rankings still agree with the original inputs?

That's the problem I explored with JSONL Candidate Align, a small open-source Python CLI.

GitHub: https://github.com/xmmyyx/jsonl-candidate-align

Two commands, two responsibilities

The tool separates candidate generation from output validation.

jsonl-align align reads already-structured query and candidate records. It uses character trigrams and numeric overlap to calculate lexical scores, applies type compatibility and admission rules, and writes up to eight candidates per query.

jsonl-align validate reads the original inputs and a saved candidate result. It recomputes the lexical reference and checks candidate eligibility, metrics, candidate sets, and ranks.

The generated output contains identifiers, component scores, ranks, and caller-supplied page references. The comparison output helps identify records whose saved values no longer match the implemented rules.

A small synthetic example

The README includes a minimal example using these two JSONL records.

queries.jsonl

{"blind_id":"DEMO-Q1","blind_type":"SC","comparable_text":"cobalt sensor 314159"}

candidates.jsonl

{"holdout_unit_id":"DEMO-C1","holdout_unit_type":"M3_AUTO_KEEP","comparable_text":"cobalt sensor 314159","pages":[1]}

After installing the project from source with Python 3.11 or later, the two commands are:

jsonl-align align --queries queries.jsonl --candidates candidates.jsonl --output-dir alignment-output

jsonl-align validate --queries queries.jsonl --candidates candidates.jsonl --actual alignment-output/candidate_alignment_rows.jsonl --output-dir validation-output

In the recorded synthetic test, the generated candidate had a score of 1.0 and rank 1. Validation returned PASS.

Then I examined a controlled modification: copying the generated candidate file and changing only its saved score from 1.0 to 0.5.

The original inputs, original output, and rank were unchanged.

When the copied file was validated against the same inputs, validation returned FAIL, with SCORE_DRIFT among the reported difference classes. The CLI exited with code 1.

This is a deliberately narrow test, but it shows why keeping the validation step separate from candidate generation can be useful.

What the validator does not prove

There is an important distinction here.

Generation and validation share the same scoring implementation. Validation uses exact Decimal comparisons for output scores, while scoring itself uses floating-point arithmetic.

A passing result therefore confirms agreement with the implemented lexical contract for the supplied inputs. It is not an independent proof that the scoring algorithm is defect-free.

Nor does lexical similarity establish semantic correctness.

The tool also expects callers to supply structured and typed JSONL records. It does not parse PDF or Word documents, run OCR, or make compliance decisions. Candidate pool limits and input order can affect the results, and large-corpus performance has not been established.

Why publish it?

I wanted a small, inspectable example of how candidate ranking, structured output, and result validation can fit into a Python CLI workflow.

The source code separates command handling, scoring, and comparison logic. The README documents the input/output contract and provides an entirely synthetic Quickstart.

The project is available under the MIT License:

https://github.com/xmmyyx/jsonl-candidate-align

I'd be interested in feedback from developers who work with JSONL pipelines: when validating saved candidate results, which types of mismatches are most useful to report?

AI disclosure: ChatGPT assisted with drafting and editing this article. The technical claims and synthetic demonstration are based on the project's published source and recorded test results. The author is responsible for reviewing the content before publication.

DEV Community: xmmyyx

Why I Added a Validator to a Small JSONL Candidate Ranker

Two commands, two responsibilities

A small synthetic example

What the validator does not prove

Why publish it?