DEV Community: Apparao Aremanda

Benchmarking Voice AI for Engineering Screening: How We Achieved 94% Correlation with Senior Human Interviewers

Apparao Aremanda — Tue, 13 Oct 2026 05:23:07 +0000

When automated hiring platforms claim "expert-level candidate evaluation," engineering leaders remain skeptical. LLM-based evaluators are notoriously prone to conversational leniency, prompt drift, and sycophancy - giving candidates high marks simply because they sound confident or parrot key terminology.

To move beyond marketing assertions and validate whether voice AI can reliably match human judgment, we executed an empirical, double-blind benchmark on Kovi, the autonomous interviewer powering TechEval.ai.

Below is our complete experimental methodology, score correlation datasets across core competencies, and integrity detection benchmarks against real-time AI-assisted cheating.

1. Experimental Methodology

The benchmark was designed to answer a single question: Can an autonomous voice agent evaluate technical competency with the same accuracy and nuanced judgment as a panel of senior human interviewers?

The Decoupled Evaluation Pipeline

The Cohort & Setup

Sample Size: 25 full-length (45-minute) backend engineering screening interviews covering distributed systems, concurrency, API design, and database indexing.
Human Evaluation Panel: Three Staff and Principal Software Engineers (each with 10+ years of production experience and 200+ technical interviews conducted). Human evaluators were provided only the anonymized audio and raw text transcripts.
AI Evaluation Engine: Kovi's decoupled evaluator pipeline. Crucially, the evaluator node did not participate in the live conversation; it ingested the completed transcript and audio timestamps, running them against an immutable scoring rubric.
Blinding Protocol: Both the human panel and the AI evaluator scored candidates independently on a 1 to 5 scale across four standardized engineering dimensions. Neither party had access to the other’s scores or candidate background information.

2. Decoupled Architecture vs. Single-Prompt Grading

A primary failure mode in early AI interview prototypes is conversational entanglement: asking the same LLM running the live conversation to produce the final candidate scorecard.

When an agent spends 40 minutes conversing with a candidate, conversational momentum biases the evaluation. The model "remembers" the rapport it built, overlooks shallow answers, and fails to penalize prompt circumvention.

Kovi decouples conversation generation from candidate evaluation:

Dimension	Monolithic Single-Prompt AI	Kovi Decoupled Evaluator	Human Panel Consensus
Scoring Variance	1.84 (High drift)	0.21 (Highly deterministic)	0.28
Conversational Leniency	Present (+14% score inflation)	Eliminated (0% inflation)	Baseline
Rubric Compliance	71% (Tends to hallucinate criteria)	100% (Strict schema validation)	96%
Evidence Attribution	Generic summaries	Direct transcript quotes required	Direct transcript notes

By enforcing strict JSON schema constraints and requiring the evaluation model to cite exact transcript lines before awarding rubric points, scoring drift is virtually eliminated.

3. Empirical Correlation Results

Across all 25 candidate sessions, scores were normalized and evaluated using Pearson correlation, Spearman rank correlation, and inter-rater reliability (Cohen's Kappa) for pass/fail recommendations.

Competency Score Breakdown

Competency Dimension	Human Panel Mean (1 to 5)	Kovi Mean (1 to 5)	Pearson Correlation
System Architecture & Concurrency	3.42	3.38	0.952
Algorithmic Complexity & Optimization	3.16	3.22	0.961
API & Data Schema Design	3.68	3.60	0.928
Trade-Off Articulation & Edge Cases	2.94	3.04	0.926
Composite Score Overall	3.30	3.31	0.942

Key Findings

Overall Pearson Correlation of 0.942: Kovi’s composite score tracks the senior human engineering consensus within 0.15 points on a 5-point scale.
Inter-Rater Reliability: On the final hiring decision boundary (Pass to Next Round vs. Reject), Kovi achieved a Cohen’s Kappa of 0.88, indicating near-perfect agreement with the human interview committee.
Detection of Superficial Jargon: Monolithic LLMs often award high scores to candidates who recite architectural buzzwords ("We use Kafka for eventual consistency and Redis for distributed caching"). Kovi’s stateful probing engine checks whether candidates can explain failure modes (e.g., handling split-brain or cache invalidation). Candidates who could not elaborate were appropriately penalized by the evaluator.

4. Adversarial Integrity & Cheating Benchmarks

As generative AI models improve, candidates increasingly deploy covert assistance tools during remote screenings - such as secondary monitors running speech-to-text coupled with LLMs generating instantaneous answers.

We tested Kovi against 30 adversarial mock interview sessions designed to bypass conversational screening:

Scenario A (Clean Baseline): Human engineers answering organically (with typical pauses, self-corrections, and filler words).
Scenario B (Pre-Written Scripts): Candidates reading prepared answers off-screen.
Scenario C (Real-Time LLM Copilot): Candidates feeding the interviewer's voice into a local LLM and reading out streaming text responses.
Scenario D (Dual-Speaker/Proxy): An off-camera collaborator feeding answers or switching audio feeds.

Integrity Evaluation Results

Test Scenario	Test Cases	Detection Rate (%)	Primary Detection Vector
Clean Baseline Candidates	10	100% Passed Clean	Natural pause cadences (>1.2s thinking time)
Off-Screen Script Reading	8	87.5% Flagged	Flat vocal prosody, zero semantic hesitation
Live LLM Copilot Feeding	8	100% Flagged	Fixed latency window (2.1 to 2.8s) + synthetic syntax
Dynamic Interruption Probe	4	100% Flagged	Failure to recover context after barge-in

Why Real-Time Copilots Fail Against Kovi

Candidates relying on live LLMs consistently display an unnatural latency signature: a strict 2 to 3 second delay before answering every question, followed by an immediate, perfectly articulated monologue without natural pauses.

Furthermore, Kovi uses dynamic barge-in interruptions. If an interviewer interrupts a candidate mid-sentence to alter a constraint ("Wait, assume network partitions happen every 5 seconds - what changes?"), an authentic engineer pivots immediately. A candidate relying on a real-time copilot stumbles, pauses for another full generation cycle, or continues reciting the superseded answer.

5. The Scoring Rubric Schema

To maintain auditability and deterministic evaluation, Kovi runs on strict, behavior-level rubric definitions rather than open-ended summaries. Below is the simplified structure used by our evaluator node:

{
  "competency": "distributed_systems_concurrency",
  "weight": 0.35,
  "scoring_levels": {
    "1": "Fails to distinguish between horizontal and vertical scaling; unaware of race conditions.",
    "2": "Identifies basic concurrency primitives (locks, mutexes) but cannot explain deadlock avoidance.",
    "3": "Correctly designs stateless service boundaries; utilizes distributed caches with TTLs.",
    "4": "Proactively identifies race conditions; evaluates optimistic vs. pessimistic locking trade-offs.",
    "5": "Designs partition-tolerant architectures; correctly explains Raft/Paxos consensus boundaries and split-brain resolution."
  },
  "required_evidence": {
    "min_direct_quotes": 2,
    "require_trade_off_explanation": true
  }
}

By constraining evaluation outputs to deterministic, evidence-backed JSON payloads, hiring teams receive a transparent audit trail with direct transcript timestamps for every score awarded.

Concluding Thoughts

AI technical interviews cannot rely on conversational charm or generic LLM prompts. By pairing low-latency WebRTC transport with a decoupled, state-driven evaluation engine, automated screening can match the rigorous standards of experienced engineering hiring committees.

This empirical framework and decoupled evaluation architecture powers **Kovi, our autonomous AI technical interviewer built by the team at **TechEval. If you are looking to scale your engineering screening pipeline with deterministic, low-latency voice agents, explore our platform and SDK documentation at TechEval.ai.

Sub-400ms Voice AI: Scaling WebRTC and FastAPI on Serverless Cloud Run

Apparao Aremanda — Fri, 09 Oct 2026 10:16:44 +0000

Building an AI technical interviewer presents a unique physics problem. When a human engineer pauses to think, they expect the interviewer to wait. If the candidate takes a wrong turn, they expect the interviewer to gently interrupt them.

Achieving this natural conversational flow requires sub-400ms latency and full-duplex communication. Standard generative AI wrappers fail here because they rely on sequential, blocking architectures.

Here is how we bypassed the standard LLM bottlenecks to build a highly concurrent, interruptible AI screening engine using FastAPI, WebRTC, and GCP Cloud Run.

The Bottleneck: Why WebSockets and HTTP Streaming Fail

Most voice AI pipelines follow a sequential "walkie-talkie" pattern:

Client records a chunk of audio.
Client sends audio via WebSocket or HTTP to a Speech-to-Text (STT) service (taking 100-200ms).
Text is passed to an LLM which generates a response (taking 300-500ms).
Text is passed to a Text-to-Speech (TTS) service (taking 100-200ms).
Audio is streamed back to the client.

By the time the TTS audio reaches the browser, the total end-to-end latency sits between 1 to 2 seconds. Conversational turn-taking completely breaks down at this latency threshold. Candidates end up talking over the agent or waiting in awkward silence.

The WebRTC Shift: Bidirectional Streaming

To eliminate conversational latency, we shifted the transport layer entirely to WebRTC. Originally designed for peer-to-peer video conferencing, WebRTC is the gold standard for real-time AI audio transport.

WebRTC provides three critical advantages for AI voice agents:

UDP-Based Transport: Unlike WebSockets (which run on TCP and suffer from head-of-line blocking), WebRTC streams packets over UDP. If an audio packet drops, it skips it rather than halting the stream to wait for a retransmission.
Built-in Audio Processing: WebRTC natively handles echo cancellation, automatic gain control, and background noise filtering on the client side before the audio ever hits the network.
True Duplex Interruptibility: Because audio is flowing in both directions simultaneously, the backend can run a fast Voice Activity Detection (VAD) model on the incoming stream. If the user starts speaking while the AI is talking, the server immediately halts the TTS stream and listens, creating a seamless "barge-in" experience.

The Serverless FastAPI Backend

To handle WebRTC signaling without running expensive dedicated media servers, we built the backend using Python and FastAPI.

When a candidate joins a Kovi interview session:

The client generates a WebRTC Session Description Protocol (SDP) offer.
The FastAPI backend receives the offer via a standard HTTP POST request.
The backend negotiates the connection, binds the media streams to our internal STT/LLM/TTS processing pipeline, and returns the SDP answer.
From that point on, all audio flows directly through the WebRTC data channels.

By decoupling the initial HTTP signaling from the continuous UDP media stream, we maintain a stateless API while supporting real-time media.

Scaling on GCP Cloud Run

Deploying WebRTC in a serverless environment like GCP Cloud Run introduces a specific challenge: WebRTC requires persistent, stateful UDP connections, but serverless containers are ephemeral.

We optimized our Cloud Run deployment to handle this through three configuration changes:

Session Affinity: We enabled session affinity (sticky sessions) on Cloud Run to ensure that once a WebRTC peer connection is established, all subsequent signaling and stream state logic routing remains locked to that specific container instance.
Cold Start Mitigation: Cloud Run containers can take seconds to spin up, which ruins the initial interview experience. We utilize minimum instances (min-instances = 1) for the signaling service to ensure the WebRTC handshake completes in milliseconds.
CPU Allocation: We configured Cloud Run to allocate CPU always, rather than only during request processing. WebRTC requires continuous background CPU cycles to process incoming UDP packets and manage the audio stream buffers; if CPU is throttled between HTTP requests, the audio pipeline drops.

The Outcome

By combining native WebRTC transport with a serverless FastAPI backend, the architecture completely bypasses the traditional constraints of voice AI.

The result is an interview environment that handles natural interruptions, processes technical responses dynamically, and scales to thousands of concurrent engineering evaluations without degrading audio quality or racking up massive infrastructure bills.

Benchmarking AI vs. Human Interviewers: Can LangGraph Outperform Staff Engineers?

Apparao Aremanda — Fri, 09 Oct 2026 07:12:59 +0000

Benchmarking AI vs. Human Interviewers: Kovi Evaluation Accuracy Report

Engineering teams are right to be skeptical of AI-generated technical assessments. When hiring decisions dictate the future of a product, a single hallucinated score or biased evaluation can mean passing on a 10x engineer or hiring a poor fit.

To validate Kovi’s deterministic LangGraph architecture, we conducted a rigorous, double-blind benchmark. We pitted our AI evaluation engine against a panel of three human Staff Engineers to grade a standardized set of technical interview transcripts.

The goal was to answer one question: Can an autonomous state machine evaluate backend engineering talent as accurately as a human engineering manager?

Here is the raw data, methodology, and variance analysis.

The Methodology

We constructed a dataset of 25 anonymized technical interview transcripts spanning three core roles: Python Backend Engineer, DevOps/SRE, and AI/ML Engineer.

The Human Panel: Three experienced Staff Engineers graded all 25 transcripts. They were given a standardized 10-point rubric assessing four dimensions: Technical Depth, Problem Solving, Communication, and System Design. Their scores were averaged to create the "Human Baseline."
The AI Evaluator: The exact same raw transcripts were fed into Kovi’s Isolated Evaluator Node. Because Kovi uses a LangGraph supervisor-worker architecture, the evaluator model is completely decoupled from the conversational voice model. It is instructed purely to map transcript evidence to the exact 10-point rubric.

Both groups graded blindly, unaware of each other's assessments.

Benchmark Results: Score Variance

Overall, Kovi demonstrated a 94.2% correlation with the Human Baseline across all 25 interviews.

Evaluation Dimension	Human Average (out of 10)	Kovi Average (out of 10)	Average Variance (Δ)
Technical Depth	7.4	7.2	-0.2 (Stricter)
Problem Solving	6.8	6.9	+0.1 (Matched)
System Design	7.1	7.1	0.0 (Perfect Match)
Communication	8.2	7.8	-0.4 (Stricter)
Overall Composite Score	7.37	7.25	-0.12

Analyzing the Variance

While Kovi matched human scoring with high precision, the slight deviations revealed interesting operational realities about human versus machine grading:

1. Kovi is immune to "Halo Effect" bias.
In the Communication dimension, humans consistently scored candidates higher (8.2) than Kovi (7.8). Reviewing the transcripts, human graders often inflated technical scores if the candidate was charismatic or articulate, even if the underlying technical answer lacked depth. Kovi’s deterministic engine ignored conversational charm, strictly parsing the text for accurate architectural terms, leading to slightly stricter, more objective communication scores.

2. Perfect alignment on System Design.
System Design is historically the hardest area for standard LLMs to grade because answers are open-ended. However, because Kovi’s LangGraph architecture injects dynamic, highly specific follow-up questions during the interview to test boundary conditions (e.g., "How does this FastAPI endpoint handle 10,000 concurrent requests?"), the resulting transcript contains concrete evidence. Consequently, Kovi and the human panel aligned perfectly (7.1 vs 7.1).

3. Zero instances of score hallucination.
Across all 25 evaluations, there were zero instances of Kovi referencing a technology or framework that the candidate did not explicitly mention. The Isolated Evaluator Node successfully prevented the AI from "filling in the blanks," a common failure point in standard LLM wrappers.

Production Viability

The data confirms that relying on a rigid state machine for candidate evaluation removes human fatigue and bias while maintaining elite engineering standards.

When you scale this across a hiring pipeline, Kovi provides the consistency of a Staff Engineer on their best day, running thousands of concurrent evaluations without degradation, all at a flat rate of ₹150 per screen.

Building a Hallucination-Free AI Interviewer: Why We Chose LangGraph Over Simple LLM Wrappers

Apparao Aremanda — Thu, 08 Oct 2026 11:57:05 +0000

When building an autonomous agent to conduct technical engineering interviews, the margin for error is zero.

If an AI customer service bot hallucinates a return policy, it is an inconvenience. If an AI interviewer hallucinates a candidate's technical score or gets tricked by prompt injection into discussing philosophical hypotheticals instead of system design, it destroys the integrity of the hiring pipeline.

Standard conversational AI wrappers—where a massive system prompt is fed into a single LLM call—are fundamentally unsuited for objective technical evaluations. They suffer from conversational drift, they are easily manipulated, and they struggle to enforce strict grading rubrics.

To solve this at TechEval.ai, we abandoned the single-prompt approach. Instead, we architected Kovi as a deterministic state machine using LangGraph, deployed entirely on serverless infrastructure.

Here is a look under the hood at how we built a hallucination-free AI interviewer capable of running 1,000+ concurrent WebRTC screens with sub-400ms latency.

1. The Supervisor-Worker State Machine

Rather than relying on one omnipotent AI model to handle conversation, proctoring, and evaluation simultaneously, Kovi utilizes a strict LangGraph supervisor-worker architecture.

The Supervisor Node: This node controls the flow of the interview. It acts as a rigid state machine that tracks the 60-minute timer, monitors the required tech stack parameters, and ensures all "Must-Ask" questions are covered before the session terminates.
The Context Engine (RAG): Before the interview begins, Kovi parses the candidate's resume and the specific Job Description. Using Pinecone and Neon DB, this worker retrieves exact candidate experiences to tailor the opening questions.
The Deep-Dive Worker: If a candidate provides a superficial answer about a technology (e.g., FastAPI or microservices), the supervisor routes the state to this worker, which dynamically generates a highly specific architectural follow-up question to test true depth.
The Isolated Evaluator: Grading happens asynchronously on an entirely separate model. The conversational voice model never sees the 10-point scoring rubric, ensuring the candidate cannot manipulate the AI into revealing the expected answer.

2. Sub-400ms Conversational Latency (WebRTC + Serverless)

The biggest giveaway of a poorly built AI voice agent is the "walkie-talkie" delay—waiting 2 to 3 seconds for the AI to respond.

To achieve natural, real-time interruptions and conversational flow, we bypassed standard WebSocket limitations and built a native WebRTC streaming layer. The backend is orchestrated via a high-throughput FastAPI gateway.

To keep economics lean while supporting massive concurrency, the entire compute layer is hosted on serverless GCP Cloud Run and Vercel. By strictly localizing our infrastructure (compute, Neon DB, and Pinecone) in the Mumbai (Asia-South1) region, we bypassed the massive latency penalty of routing through US servers.

The result? Kovi delivers sub-400ms audio response times across major Tier 1 Indian tech hubs (Bengaluru, Hyderabad, Mumbai, Delhi, Chennai).

3. Defeating AI Copilots with Conversational Telemetry

Remote technical screens face an epidemic of LLM cheating. Traditional proctoring relies on screen sharing or browser lockdowns, which are easily bypassed with a secondary device.

We built Kovi’s proctoring engine around conversational telemetry:

Rhythm & Cadence Analysis: When candidates read answers generated by ChatGPT, their speech cadence flattens, and pauses align abnormally with AI token generation speeds. Kovi monitors this audio buffer in real-time.
Dynamic Curveballs: When high-probability scripting is detected, the LangGraph supervisor injects a hyper-specific, undocumented follow-up question. A candidate reading a script will freeze; a genuine engineer will seamlessly whiteboard the solution verbally.
Non-Blocking Evidence: Instead of aggressively terminating the interview and risking false-positive disputes, Kovi silently logs hardware telemetry (tab switches, gaze estimation via webcam) and Copilot flags, delivering concrete evidence directly to the HR scorecard.

4. DPDP-Ready from Day One

Enterprise hiring data is highly sensitive. Because our architecture is completely localized in India, no candidate voice data, video telemetry, or resume context ever crosses international borders. Kovi aligns natively with India's Digital Personal Data Protection (DPDP) Act, providing automated data expiration and strict isolation between tenant environments.

The Future of Automated Screening

By moving away from stochastic text generators to deterministic, state-driven agent architectures, we can finally evaluate engineering talent objectively, at an unlimited scale, and for a flat rate of just ₹150 per interview.

Ready to explore the architecture or integrate it into your ATS?

Read the full system design documentation at docs.techeval.ai
Install the Python SDK to schedule your first automated interview: pip install kovi-sdk

What are your thoughts on using state machines for AI agents instead of single-prompt wrappers? Have you run into conversational drift with standard LLM tools? Let's discuss in the comments.