This is a submission for the Kaggle Benchmarking Challenge.
If a company uses an LLM to screen your job application, the outcome can depend on choices you never see: which model it uses, what instructions it receives, and how it is configured.
Those choices matter even when the requirements are straightforward. A role requires three years of relevant experience. An applicant has fewer than two. A human reviewer can check that mismatch directly. Yet in my tests, an LLM extracted the applicant’s experience correctly and still marked them as meeting the requirement.
This raises a practical question: how reliably do different LLMs apply the same hiring rules to the same applicants?
I built FairScreen to investigate, testing ten LLMs on fictional applications with known answers. The benchmark checks whether models recognize changes in qualifications, ignore irrelevant changes to names and presentation, and handle missing evidence appropriately.
The differences were substantial. Some configurations rejected qualified applicants; others approved applicants who failed explicit requirements. In an exploratory rerun, one model’s benchmark score rose from 39% to 98% under a different reasoning configuration.
These results show why evaluating the actual screening setup matters. A model’s reputation, a fairness instruction, or a plausible explanation does not tell us whether it applies the requirements correctly.
Kaggle benchmark · Interactive explorer · Code and data
What I Benchmarked
A screening model should change its decision when an applicant crosses a stated requirement. It should preserve the decision when only the applicant’s name or the document’s presentation changes.
Testing only the second property misses an obvious failure: a model that rejects everyone is perfectly consistent across names.
FairScreen therefore uses families of paired applications with known answers.
For example, suppose a Backend Developer policy requires at least three years of Python experience and a passed SQL assessment:
| Application | Iskandar bin Ahmad | Lee Zhen Yu |
|---|---|---|
| 4.6 years of Python; SQL assessment passed | MEETS | MEETS |
| 4.6 years of Python; SQL assessment failed | DOES_NOT_MEET | DOES_NOT_MEET |
Each of these four applications also appears in a changed presentation: reordered sections, paraphrased wording, an employer-background paragraph, or irrelevant details.
That produces eight decisions per core family:
- Two qualification states.
- Two identity variants.
- Two presentations.
A family passes only if all eight decisions are correct. I call the resulting score robust policy fidelity. A 50% score means half the families passed every check, not that half the individual answers were correct.
I compared four ways of making the decision:
| Configuration | Process |
|---|---|
| Direct | Read the policy and application, then decide |
| Fairness instruction | Add an instruction to ignore personal characteristics |
| Extract, then decide | Extract relevant facts with evidence quotations, then give a decision |
| Same extraction, code decides | Apply the executable policy to that exact extracted record |
The last comparison is useful because it holds the extraction fixed. It asks whether the model’s final decision agrees with what its own extracted values imply under the policy.
Separate diagnostic suites test missing and conflicting evidence, relevant versus irrelevant language skills, longer applications, and audit framing.
All applicants and policies are fictional. The identity variations use a Singapore setting, including names and community-organization markers. The intended group associations are part of the construction; they are not validated measurements of how every model perceives those strings.
Where the answers come from
The correct decision is computed from a hidden structured record before the application is rendered. An LLM does not assign the gold label.
Two policy implementations and a text-only parser agree on all 2,528 final variants. Blind checks with two separate Claude Sonnet sessions also helped identify wording defects before the retained confirmatory runs. These are useful consistency checks, although they do not replace independent human validation.
The distinctive part of FairScreen is the combination: qualification-changing pairs, identity and presentation changes, known policy labels, and replay of the same extracted record through code. The related-work notes explain its relationship to earlier resume-screening research.
Models Tested
I used two panels with different scopes.
On Kaggle, I ran five models on the 48-family core slice and the evidence-sufficiency task:
- Gemini 3.7 Flash
- Gemini 3.1 Flash-Lite
- GPT-5.4 nano
- gpt-oss-120b
- Gemma 4 31B
This panel covered different model tiers within the available hosted inference budget.
On the NUS School of Computing GPU cluster, I ran six open-weight models on 192 core families, plus the diagnostic suites, controls, and repeats:
- gpt-oss-120b
- gpt-oss-20b
- Qwen3-32B, with thinking off in the registered panel
- Gemma 3 27B
- Mistral Small 3.2 24B
- Llama 3.1 8B
The overlap, gpt-oss-120b, provided a cross-platform check. There are 10 distinct models across the two panels.
After the main analysis, I reran Qwen3-32B with thinking enabled. I report that separately as exploratory because it also changed sampling settings and the output budget.
Findings
1. Consistency, correctness, and configuration tell different stories
In the opening Technical Writer family, Qwen3-32B with thinking off approved all eight applications, including the four that did not meet the experience requirement.
It was perfectly consistent across names within that family. It was wrong on half the decisions.
Across the full core set, the difference between Qwen configurations was substantial:
| Qwen3-32B | Thinking off | Thinking-enabled rerun |
|---|---|---|
| Families with all eight decisions correct | 39% | 98% |
| Unqualified applicants incorrectly approved | 41% | 1% |
| Median output tokens per direct decision | 12 | 476 |
The family-score difference was +59.4 percentage points, with a 95% bootstrap interval of +53.1 to +65.6.
This is a configuration result. Because the rerun also changed sampling and output allowance, it does not isolate thinking mode as the sole cause. The roughly 40-fold difference in median output tokens is also not a measured 40-fold difference in price or latency.
The lesson I took from it is practical: a benchmark result needs a model configuration attached to it, not just a model name.
The fairness instruction addressed a different question. On the six-model cluster panel, it changed family accuracy by −0.3 percentage points, with a 95% interval from −1.2 to +0.6. On Kaggle, the estimate was +2.5 points, with an interval from −1.7 to +6.3.
That does not establish that fairness instructions never work. It says this particular instruction did not show a reliable improvement on these tests.
The name results were also inconclusive. In the 38 families with a same-group name control, identity swaps changed 2.6% of direct decisions, compared with 2.2% for another name from the same group.
Those controls cover only customer support and laboratory support. I cannot use that comparison to make a general claim about demographic bias across all occupations or models.
2. Code repaired decisions that contradicted the model’s own extraction
For the opening example, the decisive comparison was straightforward:
- Extracted experience: 1.8 years.
- Required experience: at least three years.
- Model decision: MEETS.
- Executable policy applied to the same extraction: DOES_NOT_MEET.
Across the cluster panel, replacing the model’s structured decision with code applied to the same extracted facts increased family accuracy by an average of 20.5 percentage points, with a 95% interval of 18.1 to 22.8.
| Cluster model | Model decides after extraction | Code decides from the same extraction |
|---|---|---|
| Llama 3.1 8B | 1% | 56% |
| Qwen3-32B, thinking off | 45% | 73% |
| Gemma 3 27B | 42% | 65% |
| Mistral Small 3.2 24B | 79% | 91% |
| gpt-oss-20b | 95% | 99% |
| gpt-oss-120b | 100% | 100% |
The average depends on the panel. Replacing Qwen’s thinking-off run with its thinking-enabled rerun reduces the average gain to 16.2 points. Models already following their extracted facts have little room to improve.
The orange segments show cases where the extracted values preserve every requirement’s correct status, but the final decision is wrong.
“Requirement status preserved” does not mean every extracted fact is literally accurate. A slightly incorrect duration can still fall on the correct side of a threshold. This figure also counts individual structured core outputs, including controls, rather than eight-item family passes.
Code replay has a clear limitation: it inherits the extraction’s mistakes.
When a role had no end date, an extractor sometimes returned the total from the dated roles instead of recording that the total was unknown. Code then treated that partial total as complete and rejected the applicant.
On the cluster sufficiency suite, gpt-oss-120b fell from 56% of families passed with its structured decision to 40% with code replay.
So the useful recommendation is specific: executable policies can repair decision inconsistencies, but the extraction must preserve missingness, conflict, and uncertainty. Evidence quotations also need validation; requesting a quote does not make the extracted value trustworthy.
3. A leaderboard can measure output handling as much as model behavior
The most instructive platform comparison involved gpt-oss-120b.
On the same 48 core families, it scored approximately 63% on Kaggle and 98% on the cluster. Inspecting the saved outputs showed hosted answers cut off mid-word.
On the 31 families where every hosted answer was complete, both platforms passed 30 of 31.
That comparison is conditional on completion, so it does not replace the full-run score. It does show why I would not interpret the original gap as a pure reasoning difference. The underlying cause of the cutoffs remains unresolved.
The structured task exposed a subtler issue. Fourteen hosted gpt-oss-120b responses were missing the closing JSON brace. The registered fallback parser could still read the final decision, but could not recover the structured facts needed by the code arm.
That gave the model decision and code replay different usable-output requirements.
| Kaggle code-minus-model comparison | Average difference |
|---|---|
| Registered parser | −2.1 points, 95% interval −5.0 to +1.2 |
| Sensitivity requiring valid JSON for both arms | +3.8 points, 95% interval +1.3 to +6.3 |
The strict-JSON result is a post hoc sensitivity check, not a replacement for the registered result. Its importance is that the sign changes when the output-validity rule is made consistent.
Missing responses matter too. On the hosted sufficiency task, Gemini 3.7 Flash answered all 169 returned items correctly, but 23 calls produced no answer after three attempts. Counting those as failures yielded 28 passing families out of 48.
That is a legitimate end-to-end reliability score. It is a different quantity from correctness conditional on receiving an answer.
For my next benchmark, I would report successful delivery, valid output format, and correct decisions separately from the beginning.
How I kept the interpretation bounded
The protocol specified three main comparisons: the fairness instruction, its behavior across presentation changes, and code replay versus the model’s decision on the same extraction. The registered tests use family-level bootstrapping and a multiple-testing correction within each panel.
The thinking-enabled rerun and strict-JSON analysis are labeled exploratory or sensitivity analyses. The original results remain available.
There are also limits that better statistics cannot remove:
- The applications are synthetic, in English, and drawn from one locale and a small set of policies.
- Several capable models reach or approach the core ceiling. Gemini 3.7 Flash and Gemma 4 pass all 48 hosted core families.
- A bag-of-words baseline passes about 41% of the larger core set, showing that some cases contain useful lexical shortcuts.
- The name controls are limited, and the hosted core slice has none.
- I wrote and checked the executable policies. This does not test automatic conversion of arbitrary policy prose into correct code.
- These results concern explicit eligibility rules, not real hiring outcomes or ranking candidates under scarcity.
What I would measure next
I would prioritize three extensions:
- Change the policy while keeping the applicant fixed. Moving a threshold or adding an exception would test whether a model actually follows the supplied rule.
- Balance the identity controls across occupations and hosted models. Same-group alternatives and exact repeats should appear on the same families.
- Make uncertainty harder and more realistic. Test partially dated experience and implicit contradictions, with evidence checks that preserve what is known and unknown.
I started by expecting the identity results to be the main story. The more useful finding was how many different failures can hide behind a stable answer: a misapplied requirement, incomplete extraction, a configuration choice, or an output the parser cannot use.
My Benchmark
The benchmark contains four tasks:
- Direct screening
- Screening with a fairness instruction
- Structured extraction and decision
- Evidence sufficiency
The structured task’s log also reports policy-code replay. The additional cluster experiments provide the broader diagnostic analysis described above.
The interactive explorer lets you inspect the opening example and the other 191 cluster core families using saved decisions.
The repository contains the code, data, protocol, deviations, raw outputs from both platforms, tests, and analysis scripts. Code is MIT-licensed; data is CC BY 4.0. Recomputing the saved results does not require new model calls.
If you inspect one example, start with the Technical Writer family: check the stated requirement, the extracted duration, and the final decision. That small comparison explains what FairScreen is trying to measure.
Top comments (0)