The easiest mock interview product to build is "AI asks a question, you type an answer, AI comments on it". It works, but it's far from a real technical interview. When we started building EngMock, we wanted the interviewer to behave like a person: it talks, it listens to you explain, it watches you draw and write, and it follows up at the right moment.
This post covers a few engineering problems we ran into along the way and what we ended up choosing.
1. Why a question-asking AI isn't enough
In a real system design interview, the interviewer watches your whiteboard, looks at the boxes and arrows you draw, and listens to you at the same time. You say "I'll add a cache here" and they ask "which layer does it sit in? How do you handle invalidation?" In a coding interview they watch you type and run your tests, and when something fails they ask "which line do you think is the problem?"
All of that depends on three inputs being available together: voice, the whiteboard, and code (plus its run results). Without any one of them, the AI either chats in generalities or comments on a static block of text. So the goal was specific: a real-time interviewer with a voice, getting about the same information as a person sitting next to you.
2. Architecture
Browser (mic + VAD, whiteboard, code editor, audio playback)
| audio + turn boundaries, whiteboard snapshots,
| code workspace + run results (one WebSocket)
v
Go backend
- session and turn management <----> real-time voice model
- context building and validation ----> (voice stream comes back)
- event recording ----> recording/event storage ----> report generation
|
| voice stream / interrupt signal
v
Browser audio playback
A few choices worth calling out:
- WebSocket between the frontend and the backend. Audio, whiteboard, code, run results and control messages all go over the same connection, so message ordering is guaranteed. We lean on that a lot later.
- The browser never talks to the model directly. The Go backend sits in the middle. It handles auth, billing, validation, recording and recovery. The backend also keeps a long-lived connection to the real-time voice model, with audio going in and audio coming out.
- Next.js on the frontend, Excalidraw for the whiteboard, Monaco for the editor. All off the shelf. The real effort went into coordinating them with the voice.
3. The three hardest problems
3.1 Turns: when to speak and when to wait
The obvious approach is to use the model's built-in voice activity detection: it detects that you stopped talking and starts answering. We gave up on it pretty quickly, because pauses in a technical interview are nothing like pauses in casual conversation:
- When a candidate is thinking about a problem, they go quiet, say "um...", or stop halfway to look at the whiteboard.
- A sentence that ends on "and then..." or "because..." obviously isn't finished.
- People mumble while coding, with gaps between every sentence.
- When walking through a long design, people take a breath between sentences.
If every pause counts as "your turn", the interviewer keeps cutting in and the experience is terrible. It also gets expensive: every time the floor is handed to the model, it has to re-read the whole conversation, which costs both money and latency.
So we took turn-taking out of the model and manage it ourselves:
- Turn off the model's built-in voice activity detection. The frontend decides locally when an utterance starts and ends, and tells the backend explicitly: "this turn started", "this turn is committed".
- Treat a turn as one atomic commit. It contains that turn's audio, the full whiteboard or code at that moment, and the matching run results. The backend writes all of it to the model in order and only then declares the turn finished. When the model answers, it always sees the workspace as it was when you spoke, not a version you edited afterwards.
- Decide "are you done?" in layers. Clear cases (a question, an "I'm done", a sentence stuck on a conjunction, an obvious "let me think") are handled directly by local rules. Only the uncertain ones go to the model. The model also has a dedicated "wait" action: if it hears you and decides you're still thinking, it can choose to say nothing.
- Staying silent is a reply that needs proper handling. A turn where the model waits doesn't produce a made-up line from the AI, and a stale late reply can't take over the next turn's slot.
On barge-in, our position is fairly conservative: while the interviewer is speaking, the mic doesn't interrupt it. The reason is practical. If you're on speakers, the interviewer's own voice leaks into the mic, and voice activity detection can't tell an echo from you cutting in. We'd rather let it finish than have it interrupted by its own voice. Two cases still cut it off. First, if you start talking before the AI's reply has begun playing (it's still "thinking"), the new utterance replaces the queued reply. Second, if the model reports that it was interrupted, the backend immediately tells the frontend to stop playback and marks that line in the transcript as "interrupted", keeping only what was actually spoken. That way the transcript matches what the user really heard.
This tradeoff won't suit every product, but for technical interviews, where candidates need to think and talk continuously, we think "slower is better than cutting in" is the right call.
3.2 Letting the model "see" the whiteboard and code results
Whiteboard: a structured description instead of a screenshot. What we send the model is a structured description of the whiteboard: which elements exist, the text on them, their positions, which ones are connected by arrows, and what was added, changed or deleted since the last snapshot. This has a few advantages. The model reads the text in boxes and the connections accurately instead of guessing from an image. The description can be stored and replayed as-is when an interview is resumed. And it makes precise diffs easy.
Two design decisions we think matter:
- Every element carries a "creator" tag. Whiteboard content is split into what the candidate drew and what the AI drew. In our product the interviewer doesn't draw for you and the whiteboard is the candidate's own, but even if some AI-generated elements exist, they don't count as the candidate's work. That tag travels all the way into the scoring report.
- The backend doesn't trust the frontend. When the frontend sends a description, the backend checks it against the raw snapshot and rejects it if they don't match, instead of quietly fixing or truncating it.
Code: the whole workspace, plus run results tied to a specific version of the source. For coding questions we send the full workspace, not only the file that's currently open. A run result is bound to the exact version of the source that was run. If you edit the code afterwards, the old result no longer counts as evidence for the new version, and it won't be treated as "the code works". Also, if the execution environment itself fails, we record it as "execution environment unavailable" rather than as a compile error from the candidate, so our own problems don't get blamed on the user.
3.3 Context management in long conversations
An interview can run long, and each turn may carry a full copy of the whiteboard or workspace. If you resend everything every turn, the context fills up with N copies of the same whiteboard and the actual conversation gets pushed out of the window.
Our approach:
- Don't resend what hasn't changed. If this turn's whiteboard or code matches a full copy the model already holds, we send only a reference saying "same as that earlier full copy", and tell the model explicitly that this is not an empty whiteboard.
- Refresh when needed. When the content changes, when there's a new model connection, when the context is about to be compacted, or after some time has passed, we send a full copy again, so the latest copy in the window is always a complete one.
- Compact earlier to leave room for the conversation. We tuned the model-side context compaction to trigger sooner.
- The complete record stays on our side. What's sent to the model can be compacted or referenced, but the server keeps the full original of every turn, for reconnect recovery, pause and resume, and scoring.
Connection handling is part of the long-conversation problem too:
- The hosting platform caps how long a single connection can live, so a full-length interview might not fit in one. When the candidate isn't speaking and nothing is in flight, the frontend moves the interview over to a fresh connection ahead of the limit. Users barely notice.
- The model side may ask for session rotation, and the network can also go into a zombie state where the connection looks alive but nothing gets through. We added a heartbeat. When it detects a dead connection, we switch to a new one and recover automatically, so the user doesn't have to click "continue".
- On recovery, the earlier conversation is background, not a queue of pending requests. The first thing said after recovery responds to the current utterance, and old questions don't get answered a second time.
4. How the report ends up with evidence
The classic mistake with post-interview reports is a professional-sounding summary that nobody can verify. Our approach is to make every judgment traceable back to the raw record:
- We record an event stream, not just audio. Each interview stores timestamped transcripts, whiteboard snapshots and changes, code snapshots, and code run results. The report is generated from that full event stream, not from a text transcript alone.
- Only the candidate's own evidence counts. Things the interviewer said, hints it gave, and elements the AI drew don't count as proof of the candidate's ability. These rules are written explicitly into the scoring step.
- Conclusions must come with evidence. Each main conclusion in the report maps to a "conclusion + specific evidence + timestamp + source", and the user can jump back to that moment to see what was going on.
- If we didn't see it, we say we didn't see it. If a topic was never asked, or an environment failure meant we couldn't observe something, it's marked "not assessed" or "insufficient evidence" instead of getting a casual low score. For interviews with too little evidence, we skip calling the model entirely and say "too short to generate a report".
- System failures don't cost the candidate points. Gaps caused by an unavailable execution environment or a dropped connection go through their own handling path.
This makes reports slower and more conservative, but both users and we can check them.
Wrapping up
With a real-time voice interviewer, the hard part isn't getting the AI to talk. It's deciding when to talk, what it has seen, and whether what it said can be checked afterwards. Our approach is still changing. If you're building something similar, I'd love to hear how you handle these in the comments.
If you want to see it in action, you can start a free mock interview at engmock.com, or read the feature overview first.
Top comments (0)