DEV Community

Jack Ma
Jack Ma

Posted on

Storing an interview rubric as data, and measuring how often scorers disagree

Two reviewers can watch the same one-way interview recording, use the same scorecard, and give the answer a 2 and a 4. They heard the same words, and one of them took those words as a specific example while the other took them as a general description. The Super Lede has a hiring-side piece on why two interviewers give the same answer a 2 and a 4: define each point of a 1-to-5 scale by what's in the answer, then calibrate reviewers on the first two or three recordings.

This post is the backend side of the same problem. At AnswerReel, candidates answer one-way interview questions by video, voice or text. When AI scoring is on, a model scores the answers about past experience against criteria the hiring team writes. Once software is one of the scorers, "the scorers disagree" becomes something you have to store, check and measure.

Decide what gets scored before you store anything

Every question has a type: experience, fact or file. Only experience questions carry criteria and get a score. A fact question like "When could you start?" isn't scored. The model is told to put the answer in the candidate's summary as stated, leaving out details about protected traits ("Can start in March", not "Can start in March, after maternity leave"). File uploads don't go to the model at all.

Some questions leave reviewers guessing what to score: a yes-or-no question asked on camera, two questions joined with "and", or "walk me through your career" with a two-minute limit. For a rewrite of each, see one-way interview questions to fix before you send the link. A schema can't tell a yes-or-no question from a good one. What it can do is cap how many criteria each question gets:

create table questions (
  id serial primary key,
  text text not null,
  type text not null check (type in ('experience', 'fact', 'file'))
);

create table criteria (
  id serial primary key,
  question_id int not null references questions (id),
  position int not null check (position between 1 and 3),
  text text not null check (length(text) between 1 and 200),
  unique (question_id, position)
);
Enter fullscreen mode Exit fullscreen mode

Each criterion can be 1 to 200 characters, and the editor asks hiring teams to write one sentence per criterion. Say a criterion reads "Handles complaints and finds upsell opportunities." If a candidate only talks about complaints, a single 3 hides which part they covered, and one reviewer may give a 2 for the missing half while another gives a 4 for the half that's there. Split it and complaint handling gets its own score, at the cost of two of the three slots.

The publish check adds two rules a column constraint can't express. Criteria are only allowed on experience questions (criteria_only_for_experience_questions). And before publishing, questions and criteria are checked against a list of phrases about protected traits such as age, family status and health. A match in a criterion blocks publication. A match in a question shows a warning. Phrase lists miss things. The scoring prompt also asks the model to flag any criterion that is about one of these traits, and when it does, the server stores that criterion as not scored whatever number came back.

The scale is data too

The five levels live in one constant, which the recruiter dashboard, the pricing page and the public page on AI scoring all render. The scoring prompt spells out the same scale separately, so when a level changes here, the prompt changes with it and gets a new version. This is the wording recruiters see. The cutoff for "too short" is 10 words for usable English answers and 20 characters for Japanese, and the database records two reasons the table leaves out: processing failures and criteria about protected traits.

export const RUBRIC = [
  { score: "1", meaning: "Not mentioned. An answer under 10 words also gets a 1, with the reason “answer too short”." },
  { score: "2", meaning: "Mentioned, but vague, with nothing specific." },
  { score: "3", meaning: "Explained, but without a specific example or method." },
  { score: "4", meaning: "Gives a specific example or method." },
  { score: "5", meaning: "Gives a specific example or method, and the result: a number, a change, or someone’s feedback." },
  { score: "Not scored", meaning: "No speech detected, too unclear to transcribe reliably, answered in a different language, or the quote could not be found word for word in the answer." },
] as const;
Enter fullscreen mode Exit fullscreen mode

"Not scored" is not a sixth level, and it is not a zero. In the table it's a null score that must come with a reason:

create table scores (
  criterion_id int not null references criteria (id),
  score int check (score between 1 and 5),
  unscorable_reason text check (unscorable_reason in
    ('no_speech', 'unclear', 'language_mismatch', 'quote_not_found')),
  quote text,
  check ((score is null) = (unscorable_reason is not null))
);
Enter fullscreen mode Exit fullscreen mode

Running a few inserts against it in PGlite:

ok     insert into scores values (1, 4, null, 'I called the courier before I replied')
ok     insert into scores values (1, null, 'unclear', null)
error  insert into scores values (1, 0, null, null)
        new row for relation "scores" violates check constraint "scores_score_check"
error  insert into scores values (1, null, null, null)
        new row for relation "scores" violates check constraint "scores_check"
error  insert into criteria (question_id, position, text) values (1, 4, 'A fourth criterion')
        new row for relation "criteria" violates check constraint "criteria_position_check"
Enter fullscreen mode Exit fullscreen mode

This matters for averages. If a failed upload is stored as 0, or quietly as 1, the candidate drops down the list for a reason that has nothing to do with their answer. Our totals skip nulls, and each result row stores scored_questions next to scorable_questions, so the review page says "2 of 3 questions scored" instead of showing a lower number with no explanation.

A spreadsheet needs the same count next to the average. The Calc sheet in the printable 1-to-5 interview scorecard for LibreOffice Writer and Calc shows how many criteria were scored beside each candidate's average, so you leave an unscored cell blank instead of typing a 0. It uses the same five levels, and the Writer file is the paper copy.

Make the top of the scale checkable

Scores moved most between 4 and 5 in our testing. Running the same answer three times would often give a 4 once and a 5 once. So a 5 now has a condition the server can check: the model has to copy the candidate's own words that state the result into a result_quote field. If that field is empty, or the words aren't in the answer, the score becomes a 4 and the reason changes to "Gives a specific example or method. Scored 4, not 5, because the result couldn't be found in the candidate's own words." The prompt also tells the model to pick the lower score when it's unsure between two.

Every score of 2 or more carries a quote, and the quote is checked before anyone sees it. If a quote fails the check, the server reruns the model once and keeps the scores that already passed. The rerun can return a different quote, or a 1, which needs none. If the new score still needs a quote and it can't be found in the answer, the criterion is stored as not scored with quote_not_found. The check is plain string matching after normalising both sides:

const WORD_CHAR = /[\p{L}\p{N}']/u;

// Lowercase, straighten apostrophes, collapse everything that isn't
// a letter, digit or apostrophe into single spaces.
function normalize(text: string): string {
  let out = "";
  for (const ch of text) {
    const c = ch.toLowerCase().replace(/[‘’ʼ]/g, "'");
    if (WORD_CHAR.test(c)) out += c;
    else if (out && !out.endsWith(" ")) out += " ";
  }
  return out.trimEnd();
}

// The quote has to appear as whole words, so "trained new hires"
// doesn't match inside "retrained new hires".
function quoteAppears(answer: string, quote: string): boolean {
  const hay = normalize(answer);
  const needle = normalize(quote);
  if (!needle) return false;
  for (let from = 0; ; ) {
    const i = hay.indexOf(needle, from);
    if (i === -1) return false;
    const before = hay[i - 1];
    const after = hay[i + needle.length];
    if ((before === undefined || before === " ") && (after === undefined || after === " ")) return true;
    from = i + 1;
  }
}

const answer = "Last spring I retrained new hires on refunds. Complaints dropped from 40 a week to 25.";
console.log(quoteAppears(answer, "dropped from 40 a week to 25"));   // true
console.log(quoteAppears(answer, "Dropped from 40 a week, to 25!"));  // true
console.log(quoteAppears(answer, "trained new hires"));               // false
console.log(quoteAppears(answer, "dropped from 40 a week to 20"));    // false
Enter fullscreen mode Exit fullscreen mode

node quote.ts on Node 26 prints true true false false. The production version also handles Japanese and Chinese, which have no spaces to mark word boundaries, and keeps a map back to the original text so the review page can highlight the exact span. For video and voice answers, the quote is also lined up with the transcript's word timings. When that works, the start time of the first word is stored with the score and clicking the quote starts playback there. When it doesn't, for example because the transcript text writes a number differently from the word-level tokens, the quote still counts as found and simply has no timestamp.

None of this proves a score is right. A quote that matches shows what the candidate said, not that 4 was the correct call. What it does is give two reviewers, or a reviewer and the model, the same words to argue about.

Measure two kinds of disagreement

There are two questions here, and they need different numbers. Does a scorer agree with itself when it sees the same answer again? And does it agree with someone else? This is roughly how our test harness computes both. The eight rows are made-up data for the example:

type Score = number | null; // null = not scored

// One row per criterion: three runs of the same scorer, plus one score
// from a second, independent scorer.
const rows: { runs: Score[]; second: Score }[] = [
  { runs: [4, 4, 4], second: 4 },
  { runs: [4, 5, 4], second: 3 },
  { runs: [2, 2, 2], second: 4 },
  { runs: [3, 3, 3], second: 3 },
  { runs: [null, null, null], second: null },
  { runs: [1, 3, 1], second: 1 },
  { runs: [5, 5, null], second: 5 },
  { runs: [3, 3, 3], second: 2 },
];

const pct = (n: number, d: number) => `${((100 * n) / d).toFixed(1)}% (${n}/${d})`;

// Same score on every run. All three "not scored" counts as the same;
// a mix of scored and not scored counts as different.
const identical = rows.filter((r) => new Set(r.runs).size === 1).length;

// Runs two or more points apart, looking only at runs that produced a score.
const wide = rows.filter((r) => {
  const s = r.runs.filter((x): x is number => x !== null);
  return s.length >= 2 && Math.max(...s) - Math.min(...s) >= 2;
}).length;

// First run vs the second scorer, within one point.
// Pairs where either side has no score are left out, not counted as misses.
const pairs = rows.filter((r) => r.runs[0] !== null && r.second !== null);
const close = pairs.filter((r) => Math.abs(r.runs[0]! - r.second!) <= 1).length;

console.log("same score on all runs:", pct(identical, rows.length));
console.log("runs 2+ points apart:  ", wide);
console.log("within 1 of second:    ", pct(close, pairs.length));
Enter fullscreen mode Exit fullscreen mode
same score on all runs: 62.5% (5/8)
runs 2+ points apart:   1
within 1 of second:     85.7% (6/7)
Enter fullscreen mode Exit fullscreen mode

Look at the third row. Three runs of 2, perfectly consistent, and the second scorer gave it a 4. A scorer can repeat itself every time and still read the criterion differently from everyone else, so a consistency number on its own tells you very little.

The null rules are choices, and they change the numbers. If "not scored" against a 3 counted as a miss, a transcription failure would show up as scorer disagreement, and you'd end up tuning the prompt to fix an audio problem.

We score with Claude Sonnet 5.5. For the English evaluation, an OpenAI model wrote all 235 test answers and also scored the experience answers against the same criteria without seeing Claude's scores. Those model ratings are the second column in the example above, so the comparison is model against model, not against human reviewers. We turned most answers into audio and ran them through the same transcription pipeline as candidate recordings. We tuned the prompt on 215 answers and kept 20 for one final evaluation, and ran Claude three times on each experience answer in both sets. Fact answers ran once, since they're only summarised:

Check 215 answers 20 kept back
Same outcome on all three runs (three "not scored" counts as the same) 79.5% 77.5%
Runs two or more points apart 3 criteria none
First run within one point of the OpenAI reference (pairs where both scored) 93.4% 95%

We had aimed for the same score on 90% of criteria across three runs and didn't get there with any model. The most consistent one, Claude Opus 5.5, reached 89.9% in an earlier round at twice the price. When runs disagreed, it was almost always by one point. So the release gates became these. On the held-back set, no criterion could have runs two or more points apart. In each set, at least 85% of criteria where both sides had a score had to come within one point of the OpenAI reference. The exact-match rate is recorded separately and doesn't block a release.

A one-point wobble matters less when the score only sets an order. By default, the review list in AnswerReel is sorted by score, and recruiters can switch it to submission time. The software doesn't move anyone forward or reject anyone; recruiters apply those labels themselves.

Freeze the rubric, version the scorer

Calibration only means something if the rubric holds still. Publishing locks a job's questions and criteria, and either the first invite or turning on the public application link publishes it. Both the draft save and the publish step take a row lock on the job first. If publishing gets the lock first, a concurrent draft save waits, then fails with job_locked. To change a criterion, you copy the job and edit the copy. Otherwise the first five candidates and the last five would be scored against different bars.

The scorer can't be frozen the same way, because we keep improving it. Each result row stores model_id and prompt_version. The prompt version changes whenever the prompt or output format does, and scores already on a job keep the version that produced them. If a job ends up with scores from more than one version, the candidate list shows "Scores in this job come from more than one model or prompt version, so compare them with care."

If you score by hand and change a criterion partway through, rescore the answers you've already reviewed.


I work on AnswerReel. The examples are simplified from our code, the outputs are from running them, and the English test results are from our release evaluation.

I used AI to help draft this post.

Top comments (0)