I lead AI engineering at a recruitment software company. Over the past two years, our team has built, iterated, and shipped an AI interview scoring pipeline that now evaluates tens of thousands of candidates a month. Here's what the actual architecture looks like — and what I wish someone had told me before we built it.
The naive first version (and why it failed)
Our first version of AI interview scoring did what most teams do when they're moving fast: keyword matching. If the candidate's response contained the right terms, the score went up. Simple, fast, and completely wrong.
The problem isn't hard to see in hindsight. A candidate who says "I collaborated closely with the engineering team" is demonstrating teamwork. A candidate who says "teamwork is really important to me" is not. Keyword presence and demonstrated behaviour are different things — and a recruiter knows the difference immediately, even if a naive scoring model doesn't.
How modern AI interview scoring actually works
Modern AI interview scoring pipelines use a combination of NLP on transcribed responses, speech analysis on delivery patterns, and structured rubric models to generate per-competency scores.
The pipeline broadly looks like this:
- Candidate records async video response
- Audio transcription (Whisper-class ASR)
- NLP analysis: response structure, evidence presence, specificity
- Speech signals: pace, filler word rate, confidence markers
- Rubric scoring: response mapped to observable competency levels
- Weighted aggregation → competency scores → overall score
- Score + transcript delivered to recruiter dashboard The critical layer is step 5. Without a well-designed rubric, steps 1–4 produce perfectly accurate measurements of the wrong things. This is the failure mode nobody talks about in AI hiring — not the model quality, but the evaluation design.
The rubric problem (and why it's harder than it looks)
A rubric that says "communication skills" is not a rubric. It's a category. A rubric that says "Level 3: Candidate provides a specific example with clear context, describes their personal action (not the team's), quantifies the outcome, and connects it to the role requirement" is actually scoreable.
Building good rubrics requires working closely with hiring managers to extract what they actually look for — not what sounds good in a job description. In my experience, the best rubric sessions involve asking "tell me about a candidate you hired who turned out to be great, and what did they say in the interview that made you confident?" That question surfaces the actual signal, not the official criteria.
Bias in training data — the problem nobody wants to own
This is the most uncomfortable part. If your scoring model was trained or calibrated on historical data from a non-diverse hiring cohort, it learns to score "like" your historical hires. At scale, that replicates past patterns to every future candidate.
We run adverse impact analysis on every scoring cohort — checking whether any demographic group's pass rate falls below 80% of the highest-passing group (the EEOC's four-fifths rule). This isn't optional when you're running AI at hiring scale. It's the minimum responsible standard.
The complete guide to AI powered interviews we published covers this in detail, including what to demand from vendors on training data transparency — which is honestly more important than any other technical spec when you're evaluating platforms.
Completion rates and the UX problem
If your completion rate is below 60%, it's almost never a technology problem. It's a communication problem. Candidates who don't understand what they're walking into abandon the process. A practice question, a clear time estimate, and an explanation of how the AI works — these consistently move completion rates 15–20 points.
We track completion rate as a first-class metric alongside time-to-shortlist and AI-to-human agreement rate. If completion drops, it usually means we changed something in the invitation flow. Fix the communication, not the model.
Where AI interviews genuinely break down
Senior roles. An async AI screen sent to a VP-level candidate is almost always a mistake. The signal it sends about your organisation's respect for their time is negative and hard to undo.
New role types without calibration. Never deploy a scoring model on a role type you haven't calibrated against. The rubric built for your SDR roles will produce misleading scores on your ops roles — even if the competency names look the same.
Automated rejections. We never auto-reject based on score. Scores filter for human review. A candidate within 10–15% of the cutoff threshold always gets a manual look. The legal and ethical exposure of fully automated rejections is not worth the marginal time saving.
What to measure
The metrics that actually tell you whether your AI interview pipeline is working:
Time-to-shortlist: application to ranked shortlist delivered. Target under 3 business days.
Completion rate: completions / invites. Target 70%+.
AI-to-human agreement: calibration match rate between AI scores and independent human scores. Target 75%+.
Adverse impact ratio: lowest group pass rate / highest group pass rate. Must be ≥ 0.80.
If you're building or evaluating AI recruitment software for your team, these are the four metrics to benchmark before and after deployment. They tell you far more than vendor demo stats.
The honest summary
AI interview scoring works. It works well for well-defined roles with well-built rubrics and a team that treats bias auditing as non-negotiable. It fails when teams treat it as a black box to plug in and trust.
The tech is the easy part. The hard part is the evaluation design, the calibration, and the ongoing audit. Get those right and you've genuinely solved one of recruiting's most persistent structural problems.
Top comments (0)