When I started building my AI technical interviewer, I did what most people do: I threw a big system prompt at the LLM and told it to "act like an interviewer, ask coding questions, give hints when the candidate is stuck, and score them at the end."
It worked... about 80% of the time.
The other 20% is what this post is about.
The problem with trusting an LLM to run your whole flow
Here's what kept happening. Mid-interview, the model would:
- Randomly decide the interview was "done" after one question
- Give away the answer while trying to give a "gentle hint"
- Forget it already asked a question and ask it again
- Jump straight to scoring before the candidate even submitted code
None of this was a prompting skill issue. I rewrote that system prompt probably fifteen times. The real issue is structural: an LLM generating the next thing to say has no actual memory of "what phase are we in," unless you spoon-feed it that context perfectly, every single turn, forever. And even then, it can just... decide to do something else. It's a language model, not a state tracker. Treating it like one is where things fall apart.
For a casual chatbot, that's fine — mild chaos is charming. For a product where someone's actual hiring decision depends on the interview going through every stage correctly, "mild chaos" is a support ticket and an angry candidate.
What I did instead
I pulled the structure of the interview out of the LLM entirely and put it into a plain old finite state machine.
Something like:
SETUP → GREETING → QUESTION_ASKED → CANDIDATE_CODING →
HINT_CHECK → EVALUATING → SCORING → DONE
The state machine — not the model — decides what phase we're in and what's allowed to happen next. The LLM only gets called inside a state, to do the one thing that state needs: generate a question, generate a hint, or generate a scorecard. It never gets to decide "we're done now" or "let's skip to scoring." That's not its job anymore.
Practically, this meant:
- The LLM can't hallucinate a phase transition because it doesn't control phase transitions
- Hints only trigger off a real signal — 35+ seconds of no keystroke activity — not the model deciding "the candidate seems stuck"
- Scoring only fires after code is actually submitted and executed in the sandbox, not whenever the model feels like wrapping up
The LLM still does all the hard, "actually intelligent" work — writing a good question, phrasing a helpful hint, writing a fair evaluation. It's just not allowed to drive the car anymore. It's a really good passenger with really good opinions.
A simplified look at the transition logic
class InterviewState(Enum):
GREETING = "greeting"
QUESTION_ASKED = "question_asked"
CANDIDATE_CODING = "candidate_coding"
HINT_CHECK = "hint_check"
EVALUATING = "evaluating"
SCORING = "scoring"
DONE = "done"
def transition(current_state, event):
# The FSM owns "what happens next" —
# the LLM only fills in content within a state
if current_state == InterviewState.CANDIDATE_CODING:
if event == "idle_35s":
return InterviewState.HINT_CHECK
if event == "code_submitted":
return InterviewState.EVALUATING
# ...deterministic, testable, no hallucinated jumps
Compare that to letting the model implicitly track state through conversation history alone — there's no guarantee, no test coverage, and no way to catch a bad transition before it reaches the user.
The part that surprised me
I expected this to make the product feel more robotic. It did the opposite.
Because the structure is now guaranteed, I could actually let the LLM be more creative and natural within each state, without worrying about it going off the rails. Constraining the skeleton let me loosen up the muscle. Counterintuitive, but it checks out — a lot of "unpredictable AI" complaints aren't really about the model being too creative, they're about the model having too much control over things it was never designed to control.
The takeaway
If you're building something where an LLM is orchestrating a multi-step process — not just answering one-off questions — ask yourself: does the model actually need to decide what happens next, or does it just need to generate good content within a step someone else decides?
Most of the time, in my experience, it's the second one. And the moment I stopped asking the LLM to be both the actor and the director, my "why did the interview just end after one question" bugs basically disappeared overnight.
I write about the messier, more practical side of building AI products — the stuff that doesn't make it into the demo. If you're curious, I built this exact system as a live product: AI Technical Interviewer. Happy to talk through the architecture more in the comments.
Top comments (2)
Separating the transition logic from text generation makes debugging straightforward. When an LLM controls its own loop, a skipped phase or premature exit gets buried under prose, and you end up tweaking system prompts hoping the failure rate drops from 20% to 5%.
Moving the boundaries into explicit state nodes means token budget only goes toward generation tasks where variance is acceptable, like drafting a hint or phrasing a question. For the hint trigger, tying it to typing idle time rather than having the model guess hesitation is a reliable signal.
The 35-second idle trigger is the part I would steal. Hesitation has a shape in the input stream — long pauses, deletions, a line typed and erased — and reading it off keystrokes is far more reliable than asking a model to infer confidence from transcript text. Once that signal became deterministic, the hint path stopped needing a prompt at all.
What I am still unclear on is state loss. If the candidate refreshes mid-interview, or the socket drops while the machine sits in CANDIDATE_CODING, does the enum resume from persisted state or restart the flow? And how do you test transitions — table-driven cases per (state, event), or a snapshot of the whole graph? Curious whether the plain enum survived the first time a transition had to branch on something other than the event, like a time budget.