I Built GrillKit: A Self-Hosted AI Interview Trainer That Actually Runs Interviews

Cyril Vitchenko — Mon, 20 Jul 2026 05:46:46 +0000

Ask ChatGPT for a system design question and it'll give you one. Answer it, and you'll get a reasonable reply back. Ask a follow-up, and it'll follow up. What you won't get is anything close to an actual interview: no score, no memory of what you already covered last week, no real signal on whether you're ready or just having a pleasant conversation about load balancers.

That's the actual problem GrillKit tries to solve. It's an open-source, self-hosted technical interview trainer: point it at any OpenAI-compatible model, pick what you want to practice, and it runs an actual session — real questions from curated banks, real scoring, live coding against a real sandbox, and a history you can look back on afterward. I built it mainly for my own prep and kept going because the gap between "chatting about interview topics" and "practicing an interview" turned out to matter more than I expected.

This post is less about showing off code and more about what it's actually like to use — and since it's open source, an invitation to help with the part that scales worst as a one-person project, which is mostly the question banks.

Why practicing alone in a chat window doesn't really work

None of this is about the model being smart or dumb. It's about what a free-form chat can't give you no matter how good the model is.

You end up being your own interviewer. There's no curriculum, so you ask about whatever comes to mind that day — which tends to be whatever you already feel solid on, not your actual weak spots. Close the tab and the session is gone, so there's no way to tell if you're improving or just remembering your best answers. A general assistant is also polite by default: give it a shaky answer and it'll often soften the feedback or move on unless you explicitly demand a grade, which a real interviewer won't do for you. And code doesn't run — you paste a snippet, the model reasons about it, but nothing executes, so the kind of bug that shows up the second you hit "run" just sails through untested.

GrillKit is built around fixing each of those specifically. Questions come from curated tracks — Python, Database/SQL, System Design, Kafka, RabbitMQ, Docker, Kubernetes, Observability, Airflow — split by level, so you're pulling from an actual curriculum instead of improvising one. Every answer gets scored 1–5 against a rubric with up to two AI follow-ups, so a weak answer gets flagged instead of politely waved through. Sessions are saved, and you can mark any question "I already know this" so it stops coming up and you spend your time on what's still shaky — which, after a few sessions, feels genuinely different from grinding the same comfortable questions over and over. Coding tasks execute for real against a sandbox, so a bug shows up as a stack trace, not as something an AI quietly didn't notice. And there's an optional per-round timer if you want actual time pressure instead of unlimited thinking time.

A quick tour

The home page is a dashboard of recent sessions — enough to see what you've done and jump into a new one.

Setup walks you through session mode (theory only, coding only, or one then the other), tracks/levels/topics, how many questions or tasks, whether to exclude known questions, and the optional timer. During a theory section it's real-time Q&A: type an answer, dictate it, or record audio if your model accepts audio input. During a coding section you get a proper editor, a Run button against public tests, and Submit for the real evaluation. Once you finish, a results hub links out to full review pages for both sections — the whole conversation, the code you actually submitted, every score, still there whenever you want to look back.

Real scoring, real code

Two things make this feel like practice instead of a quiz. Scoring is structured: every answer gets a 1–5 score and specific feedback, checked against a rubric pulled from the question bank rather than freestyled, so the same answer gets held to the same standard whether it's your first attempt of the day or your fifth. And coding tasks genuinely execute — there's a real sandbox behind the editor (Judge0, running in its own optional set of containers), so if your fix still throws on an empty list, you find out from a stack trace. Tasks with a clean input/output contract get checked directly against test cases; open-ended ones like bug hunts get graded by the AI against a short checklist of what a correct fix needs to do, since "find the bug" doesn't have one canonical answer to diff against.

Self-hosted, and you pick the model

Everything stays on your machine — the database, your session history, your provider API key, any downloaded voice models. The only thing that leaves is whatever text you send to whichever model you've configured, and that's entirely your call: OpenAI, a local Ollama model, vLLM, anything that speaks the same API. Practically, that means you're not paying a subscription on top of your model costs, and you're not locked into one provider if you'd rather run something local and free. Voice practice, if you turn it on, is fully offline too — Whisper for dictating or recording answers, Piper for reading questions aloud, both downloaded once and then run entirely locally. Saying an answer out loud is a different skill than typing it, and it's worth being able to practice that way without an audio file going anywhere.

Help fill in the question banks

The part of GrillKit that scales worst as a one-person project is content, not code. Coverage right now is decent for Python, Database/SQL, System Design, Kafka, RabbitMQ, Docker, Kubernetes, Observability, and Airflow — but that's nowhere near everything people actually get asked. There's no Go, no Java, no frontend track, nothing ML-specific, and plenty of gaps even inside the tracks that exist.

Adding a question doesn't touch any Python. It's a YAML file under data/questions/{track}/{level}/{category}.yaml:

category: "Data Structures"
track: "python"
level: "junior"
description: "Lists, tuples, dicts, and sets"

questions:
  - id: "ds-001"
    type: "knowledge"
    difficulty: 1
    tags: ["list", "tuple"]
    question:
      text: "What is the difference between a list and a tuple?"
      code: null
    expected_points:
      - "Lists are mutable, tuples are not"
      - "Tuples can be used as dict keys, lists cannot"

category/track/level say where it lives, difficulty runs 1 to 5, and expected_points are the rubric bullets the AI checks your answer against when it scores you — you don't need to write follow-ups yourself, those get generated on the fly. Coding tasks work the same way, one folder over in data/coding/. If there's a track you wish existed or a category you know is thin, that's honestly the most useful contribution right now, more than a code PR would be. CONTRIBUTING.md has the full guidelines; the short version is match the existing schema, keep the wording clear, and open a PR.

Try it

git clone https://github.com/GrillKit/grillkit.git
cd grillkit
docker compose up --build

Open http://localhost:8000, add a model on /config, and start a session from /setup. Add --profile coding to the compose command if you want live coding tasks.

It's Apache-2.0. If something's broken, confusing, or just missing your interview topic, open an issue on the repo — and if you send a PR with a new question bank, even better.