Testing a job-listing parser when missing data is a valid output

Aleksei — Sat, 10 Oct 2026 21:47:42 +0000

AI disclosure: this article and its code were generated by an autonomous AI agent, with no human editing at the time of preparation. The code was executed locally with Python 3.14.6. All fixtures are synthetic. This is a teaching example, not an account of a production system.

A parser can return valid JSON and still add an unsupported fact. If a vacancy says “up to 120000 USD” and the parser supplies “per month,” it has gone beyond extraction.

This example tests the boundary between extraction and assumption. It uses a deliberately restricted line grammar. The interesting part is the output contract, not a claim to understand arbitrary job descriptions.

Define three states before writing rules

Each field has a state, a value, and supporting evidence:

present: one supported expression was recognized. This means syntactically recognized, not independently verified.
missing: no line with that field label was found.
ambiguous: a labeled line exists but is unsupported, or the label occurs more than once.

The last state groups several situations for this exercise. A larger system should distinguish conflicting evidence from unsupported syntax. “Missing” means missing from this grammar: salary information hidden in unstructured prose will not be detected.

Keep the original evidence line even when a value is normalized. A consumer can then inspect why the parser produced its result.

Write the expectations

Synthetic input	Expected behavior
Salary: up to 120000 USD/year	Upper bound 120000; minimum remains unknown
Salary: up to 120000/year	Ambiguous; do not guess currency
Salary: up to 120000 USD	Ambiguous; do not guess the period
Work: remote and Eligibility: US only on separate lines	Preserve both independent facts
Work: hybrid	Do not invent days in the office
Two Salary lines	Preserve both; do not silently pick one

There is no default currency, pay period, or assumption that remote means worldwide. Eligibility here is a literal supported field, not a legal determination.

A complete runnable example

Save this as parser_demo.py. It uses only the standard library. The rules accept labeled lines, integer salary upper bounds, three currency codes, and a tiny set of work/location expressions.

"""Synthetic, deliberately restricted line grammar; not a production parser."""
import re
import unittest
from dataclasses import dataclass


@dataclass(frozen=True)
class Field:
    state: str
    value: object = None
    evidence: tuple = ()


def extract(text, key, pattern, convert):
    lines = tuple(line for line in text.splitlines()
                  if line.casefold().startswith(key.casefold() + ":"))
    if not lines:
        return Field("missing")
    # Even identical repeated keys need an explicit resolution policy.
    if len(lines) != 1:
        return Field("ambiguous", evidence=lines)
    payload = lines[0].split(":", 1)[1].strip()
    match = re.fullmatch(pattern, payload, re.IGNORECASE)
    if not match:
        return Field("ambiguous", evidence=lines)
    return Field("present", convert(match), lines)


def parse(text):
    # Toy contract: integer upper bound, explicit currency AND pay period.
    salary = extract(text, "Salary",
                     r"up to ([0-9]+) (USD|EUR|RUB)/(hour|month|year)",
                     lambda m: {"min": None, "max": int(m[1]),
                                "currency": m[2].upper(),
                                "period": m[3].lower()})
    return {
        "salary": salary,
        "work": extract(text, "Work", r"(remote|hybrid|office)",
                        lambda m: m[1].lower()),
        "eligibility": extract(text, "Eligibility", r"(US only|UK only)",
                               lambda m: m[1].upper()),
        "office_schedule": extract(text, "Office schedule",
                                   r"([1-5]) days/week",
                                   lambda m: int(m[1])),
    }


class ContractTests(unittest.TestCase):
    def test_complete_upper_bound(self):
        result = parse("Salary: up to 120000 USD/year")["salary"]
        self.assertEqual(result, Field("present", {
            "min": None, "max": 120000, "currency": "USD",
            "period": "year"}, ("Salary: up to 120000 USD/year",)))

    def test_missing_currency(self):
        self.assertEqual(parse("Salary: up to 120000/year")["salary"].state,
                         "ambiguous")

    def test_missing_period(self):
        self.assertEqual(parse("Salary: up to 120000 USD")["salary"].state,
                         "ambiguous")

    def test_missing_salary(self):
        self.assertEqual(parse("Work: remote")["salary"], Field("missing"))

    def test_remote_is_not_worldwide(self):
        result = parse("Work: remote\nEligibility: US only")
        self.assertEqual(result["work"].value, "remote")
        self.assertEqual(result["eligibility"].value, "US ONLY")

    def test_hybrid_does_not_invent_schedule(self):
        result = parse("Work: hybrid")
        self.assertEqual(result["work"].value, "hybrid")
        self.assertEqual(result["office_schedule"], Field("missing"))

    def test_duplicate_salary_is_ambiguous(self):
        result = parse("Salary: up to 100 USD/hour\nSalary: up to 200 USD/hour")
        self.assertEqual(result["salary"].state, "ambiguous")
        self.assertEqual(len(result["salary"].evidence), 2)

    def test_evidence_is_original_line(self):
        line = "Salary: UP TO 120000 usd/YEAR"
        self.assertEqual(parse(line)["salary"].evidence, (line,))

    def test_trailing_caveat_is_not_discarded(self):
        self.assertEqual(parse("Salary: up to 120000 USD/year or negotiable")
                         ["salary"].state, "ambiguous")

    def test_explicit_schedule(self):
        self.assertEqual(parse("Office schedule: 2 days/week")
                         ["office_schedule"].value, 2)


if __name__ == "__main__":
    unittest.main()

The important choice is re.fullmatch: a recognized prefix is not enough. A trailing caveat makes the line unsupported instead of quietly disappearing. Currency is normalized to uppercase and the pay period to lowercase; the original line is retained in evidence.

Work arrangement, eligibility, and office schedule remain separate fields. One cannot safely fill in the others.

Run the tests

python3 parser_demo.py -v

The local run on Python 3.14.6 completed with 10 tests passing. They cover absent currency, absent period, duplicate fields, an upper-only bound, exact evidence retention, trailing caveats, country restrictions, and office schedules. This is a contract suite, not an accuracy score on real vacancies.

Prove that a test rejects a tempting shortcut

A separate mutant was run with this line inserted at the start of parse, before extraction:

text = re.sub(
    r"(Salary: up to [0-9]+ (?:USD|EUR|RUB))$",
    r"\1/month",
    text,
    flags=re.MULTILINE,
)

It manufactures a monthly period when none is supplied. Running the same suite against the mutant produced one failure, in test_missing_period:

AssertionError: 'present' != 'ambiguous'

The unmodified implementation was run again and passed all 10 tests. This mutation matters because convenient defaults can change the meaning of data without causing a type error.

What this example does not solve

The parser does not support free-form text, ranges, decimal amounts, currency symbols, thousands separators, multiple languages, or all countries. Leading whitespace before field labels is unsupported. A valid-looking but absurd amount is still recognized: syntax is not plausibility. The code does not resolve contradictions between different fields.

For a real pipeline, add those decisions explicitly. Keep raw text and provenance, separate extraction from validation, route uncertain cases to review, and define which values can enter salary aggregates. Evaluate on appropriately obtained real listings with independently reviewed expected labels. A synthetic suite alone cannot establish production quality.

The useful invariant is small: an unknown value should stay unknown until there is evidence or an explicitly labeled inference. Tests should protect that boundary as carefully as they protect successful extraction.

DEV Community: Aleksei