Why I Ditched "Just Let the LLM Handle It" for a State Machine (And Slept Better at Night)

Valipireddy Kowshik — Tue, 15 Sep 2026 02:56:27 +0000

When I started building my AI technical interviewer, I did what most people do: I threw a big system prompt at the LLM and told it to "act like an interviewer, ask coding questions, give hints when the candidate is stuck, and score them at the end."

It worked... about 80% of the time.

The other 20% is what this post is about.

The problem with trusting an LLM to run your whole flow

Here's what kept happening. Mid-interview, the model would:

Randomly decide the interview was "done" after one question
Give away the answer while trying to give a "gentle hint"
Forget it already asked a question and ask it again
Jump straight to scoring before the candidate even submitted code

None of this was a prompting skill issue. I rewrote that system prompt probably fifteen times. The real issue is structural: an LLM generating the next thing to say has no actual memory of "what phase are we in," unless you spoon-feed it that context perfectly, every single turn, forever. And even then, it can just... decide to do something else. It's a language model, not a state tracker. Treating it like one is where things fall apart.

For a casual chatbot, that's fine — mild chaos is charming. For a product where someone's actual hiring decision depends on the interview going through every stage correctly, "mild chaos" is a support ticket and an angry candidate.

What I did instead

I pulled the structure of the interview out of the LLM entirely and put it into a plain old finite state machine.

Something like:

SETUP → GREETING → QUESTION_ASKED → CANDIDATE_CODING →
HINT_CHECK → EVALUATING → SCORING → DONE

The state machine — not the model — decides what phase we're in and what's allowed to happen next. The LLM only gets called inside a state, to do the one thing that state needs: generate a question, generate a hint, or generate a scorecard. It never gets to decide "we're done now" or "let's skip to scoring." That's not its job anymore.

Practically, this meant:

The LLM can't hallucinate a phase transition because it doesn't control phase transitions
Hints only trigger off a real signal — 35+ seconds of no keystroke activity — not the model deciding "the candidate seems stuck"
Scoring only fires after code is actually submitted and executed in the sandbox, not whenever the model feels like wrapping up

The LLM still does all the hard, "actually intelligent" work — writing a good question, phrasing a helpful hint, writing a fair evaluation. It's just not allowed to drive the car anymore. It's a really good passenger with really good opinions.

A simplified look at the transition logic

class InterviewState(Enum):
    GREETING = "greeting"
    QUESTION_ASKED = "question_asked"
    CANDIDATE_CODING = "candidate_coding"
    HINT_CHECK = "hint_check"
    EVALUATING = "evaluating"
    SCORING = "scoring"
    DONE = "done"

def transition(current_state, event):
    # The FSM owns "what happens next" —
    # the LLM only fills in content within a state
    if current_state == InterviewState.CANDIDATE_CODING:
        if event == "idle_35s":
            return InterviewState.HINT_CHECK
        if event == "code_submitted":
            return InterviewState.EVALUATING
    # ...deterministic, testable, no hallucinated jumps

Compare that to letting the model implicitly track state through conversation history alone — there's no guarantee, no test coverage, and no way to catch a bad transition before it reaches the user.

The part that surprised me

I expected this to make the product feel more robotic. It did the opposite.

Because the structure is now guaranteed, I could actually let the LLM be more creative and natural within each state, without worrying about it going off the rails. Constraining the skeleton let me loosen up the muscle. Counterintuitive, but it checks out — a lot of "unpredictable AI" complaints aren't really about the model being too creative, they're about the model having too much control over things it was never designed to control.

The takeaway

If you're building something where an LLM is orchestrating a multi-step process — not just answering one-off questions — ask yourself: does the model actually need to decide what happens next, or does it just need to generate good content within a step someone else decides?

Most of the time, in my experience, it's the second one. And the moment I stopped asking the LLM to be both the actor and the director, my "why did the interview just end after one question" bugs basically disappeared overnight.