How to evaluate an AI-generated job shortlist

Panta — Tue, 10 Nov 2026 20:39:03 +0000

A job-search assistant returns five roles. The titles sound relevant, the links look plausible, and the summary says they match your requirements.

What has it actually established?

A useful evaluation separates three things: whether the assistant completed the task, whether the sources support its claims, and whether the roles fit the person's stated constraints. A fluent answer is not evidence for any of them.

This walkthrough adapts an existing job-shortlist review protocol into a small, repeatable evaluation. It works with any assistant or job-search tool. The worked examples below are fictional; they are not benchmark results.

1. Freeze the task and the first results

Make the request testable. For example:

Find five software engineering roles with a physical workplace in Switzerland. I work in English and cannot meet a mandatory German requirement. I need an advertised workload that includes 80%. Preserve original listing links and explain unresolved requirements.

Record the exact request, date, product, visible model setting and any relevant search settings. Keep that run record separately from the result rows. Avoid placing a CV or private employment history in a public evaluation file.

Freeze the first five results in their original order before checking sources. If the assistant returns three, record three. If a link is broken, keep that result in the review. Replacing it with a better result would change what you are evaluating.

Five results are enough to inspect a workflow. They are not enough to rank assistants generally.

2. Keep three different language questions separate

A source's language tag, the language of the written advert and the working-language requirement are different attributes.

For example, an advert written in English can explicitly require German. A German-written advert might describe English as the working language. A missing language statement settles neither question.

Evaluate the claim the assistant made against the actual requirement. Use explicit labels such as required, preferred, alternative, conflicting and unstated. Do not turn “German is an advantage” into a mandatory requirement, or “not mentioned” into an English-only guarantee.

The same distinction applies elsewhere: a Swiss headquarters is not evidence that a particular role is based in Switzerland, and a remote label is not evidence that the employer permits work from Switzerland.

3. Review each claim against its source

For every result, record the suggested URL, the source you could actually inspect and the check time. Then review these dimensions independently:

Dimension	Evidence to inspect	Common unsupported inference
Source access	Readable employer advert or authorized ATS page	An HTTP 200 response proves the advert is readable
Role	Role-specific duties and requirements	A matching title proves occupational fit
Workplace	Stated work location and explicit remote eligibility	Headquarters country determines workplace
Languages	Exact required/preferred/alternative wording	English-written means English-only
Workload	Advertised percentage or range	80–100% guarantees an agreed 80% contract
Schedule	Stated days, hours or shifts	80% means a four-day week
Contract and experience	Contract terms and accepted experience pathways	“Junior” means no experience is required
Availability	Explicit closure evidence and relevant application destination	An Apply button guarantees the job is still open

An inaccessible page should produce an access result: blocked, unavailable, redirect-only or not checked. It should not produce an invented answer about the job.

A source can support one claim and leave another unknown. A single “verified” badge hides that distinction.

4. Represent evidence before scoring it

Here is a fictional review record:

{
  "result_number": 1,
  "source_access": "readable",
  "role_match": "supported",
  "language_evidence": "Contradicted: advert explicitly requires German",
  "workload_evidence": "Advert states 80–100%",
  "schedule_evidence": "Unknown: no days stated",
  "availability_evidence": "no_explicit_closure",
  "decision": "exclude",
  "reason": "Mandatory German contradicts the declared essential constraint"
}

This result is readable and relevant to the occupation, but it fails one essential constraint. It should not be called either a fully verified match or a hallucinated vacancy.

For a second fictional result, imagine a source that cannot be read. Keep its requirement evidence unknown and its decision unresolved. A blocked page does not prove that the assistant invented the role.

no_explicit_closure means precisely that. It is not a positive verification that applications are accepted.

5. Report coverage, contradictions and unknowns

Start with counts rather than a single accuracy score. Report how many results were returned, how many sources were readable, which claims were contradicted and what remained unresolved.

The linked blank worksheet uses one row per result. After reviewing it, this standard-library Python script summarizes two useful dimensions without inventing an overall accuracy metric:

import csv
from collections import Counter
from pathlib import Path

path = Path("ai-job-shortlist-review.csv")
with path.open(newline="", encoding="utf-8") as handle:
    # Blank template rows do not count as returned results.
    rows = [row for row in csv.DictReader(handle)
            if row["suggested_url"].strip()]

print(f"Returned results recorded: {len(rows)}")
for field in ("source_access", "role_match", "decision"):
    counts = Counter(row[field].strip() or "unknown" for row in rows)
    print(f"{field}: {dict(counts)}")

readable = sum(row["source_access"] == "readable" for row in rows)
print(f"Readable original sources: {readable}/{len(rows)}")

These are descriptive counts of recorded review labels, not a semantic evaluator. The script cannot determine whether a human or model labeled the evidence correctly.

Keep unreadable sources in the returned-result denominator. If you also report a metric among readable sources, label that separate denominator explicitly. Otherwise a system with many inaccessible sources can look artificially strong.

6. Make repeated runs comparable

For a useful comparison, keep the task, result cutoff and review criteria stable. Record differences in date, access, tool availability and model settings. Live vacancies change, so a later result is not automatically evidence of a better assistant.

Preserve the original run and add later checks as new observations. Do not use today's readable page to rewrite yesterday's blocked result.

For a larger evaluation, use more tasks, independent review and explicit adjudication of disagreements. This protocol alone does not establish reviewer accuracy, causal effects or a representative model ranking. It also does not establish that an application was submitted or that anyone was hired.

The practical outcome is a shortlist a person can inspect: which claims are supported, which are contradicted, and which require follow-up.

Worksheet and provenance

Download the five-row review worksheet. The full review protocol documents the field definitions and limitations. Protocol version: panta-shortlist-review-v1.

Disclosure: I work on Panta, where this protocol was developed. This adaptation was prepared with AI assistance, including generated text. The examples are fictional, and no assistant-performance benchmark is claimed.

DEV Community: Panta