DEV Community

Cover image for Running an AI interviewer fully on-device with Gemma 3 1B — and engineering around what a 1B model can't do
Alex DevOps engineer
Alex DevOps engineer

Posted on

Running an AI interviewer fully on-device with Gemma 3 1B — and engineering around what a 1B model can't do

#android #kotlin #ai #machinelearning

I built an Android app that runs a mock DevOps job interview with an AI interviewer that lives entirely on your phone. No cloud API, no OpenAI key, no round-trip to a server. The language model — Gemma 3 1B — is downloaded once and every inference after that runs fully offline, on the subway, on a plane, with airplane mode on.

The headline is "on-device LLM." But the honest story is the opposite of the hype: the model was the easy part. The real work was engineering a wall of determinism around the model — because a 1B model, running on a phone CPU, cannot be trusted to do the things you'd casually hand to GPT-4.

The project

DevOps Interview AI is a native Android app that puts you in a mock interview against one of four AI personas, each a different seniority level:

Interviewer Level Focus
Emma — Recruiter Easy Cultural fit, CV review, salary expectations
Olivia — HR Medium Behavioural questions, team dynamics, soft skills
Marcus — Senior DevOps Hard CI/CD, Kubernetes, IaC, observability, SRE
David — CTO Expert Architecture, scalability, engineering trade-offs

You pick an interviewer, join a Google Meet-style session (interviewer avatar tile, live subtitles, your camera in the corner), and answer five questions by voice. Speech-to-text turns your answer into text, the model asks the next question, text-to-speech speaks it back. At the end you get a scored evaluation. All of it offline.

The stack:

Layer Technology
Language Kotlin
UI Jetpack Compose + Material 3
AI engine Google MediaPipe Tasks GenAI 0.10.24
Model Gemma 3 1B, INT4 quantized (.task bundle)
Voice Android SpeechRecognizer (STT) + TextToSpeech (TTS)
Storage Room + DataStore
Architecture ViewModel + StateFlow, strict package boundaries

Why on-device at all?

The motivation is privacy, and it's not a marketing checkbox — it's the whole point. Interview practice is where you're most exposed: you fumble answers, you second-guess your seniority, you say things you'd never put in front of a recruiter. Sending that stream to a third-party API felt wrong. On-device means the interview data physically cannot leave the phone. No telemetry, no cloud sync, nothing to leak.

The side effects are nice too: zero inference cost, no rate limits, and it works with no connection at all. But privacy is the reason the whole thing runs the way it does.

Getting a gated model into a client app without leaking a token

Here's the first real problem. Gemma 3 1B is distributed on Hugging Face behind a license gate (litert-community/Gemma3-1B-IT). An anonymous download gets an HTTP 401 — you need an access token. And you cannot ship a personal Hugging Face token inside an APK: anyone who unzips the APK can extract it and reuse it.

The .task bundle is also ~530 MB, so bundling it as an app asset is a non-starter (nobody ships a half-gigabyte APK).

The solution splits distribution in two:

  • The source app is one repository.
  • The model bundle is published as a public GitHub Release asset in a separate repo. GitHub Release assets are served over plain HTTPS with no credential — no token to embed, no gate to pass.

On first launch, the app streams the model straight from that release into its private storage:

/**
 * Streams the Gemma 3 1B `.task` bundle from a GitHub Release asset straight
 * into Context.getFilesDir ...
 *
 * Hosted as a Release asset (not the original Hugging Face repo) because
 * litert-community/Gemma3-1B-IT is gated behind license acceptance — an
 * anonymous request gets HTTP 401, and embedding a personal HF access token
 * in a client APK would let anyone who unzips the APK extract and reuse it.
 *
 * Writes to a `.part` file first and renames atomically on success, so a
 * half-downloaded file is never mistaken for a ready model.
 */
Enter fullscreen mode Exit fullscreen mode

That last line matters more than it looks. The "is the model ready?" check is deliberately dumb — a single file-exists lookup:

const val MODEL_FILE_NAME = "gemma-2b-it-gpu-int4.task"

fun isModelReady(context: Context): Boolean =
    File(context.filesDir, MODEL_FILE_NAME).exists()
Enter fullscreen mode Exit fullscreen mode

If the download wrote directly to that filename and the process got killed at 60%, isModelReady() would return true for a corrupt file, and the engine would crash on load forever. Downloading to model.task.part and renaming atomically on completion means the real filename only ever exists as a complete file. Crash-safety from one rename().

(An aside on that filename: it says gemma-2b, but the model is Gemma 3 1B. MediaPipe's LlmInferenceOptions doesn't care what the model is called — it loads whatever .task sits at the path. So the filename is just a fixed interface contract that the downloader, the splash bootstrap check, and the engine all agree on. I left the legacy name rather than pretend it means anything.)

Loading the model: CPU, on purpose

MediaPipe makes the actual load short:

val options = LlmInferenceOptions.builder()
    .setModelPath(modelPath)
    .setMaxTokens(LlmConfig.MAX_TOKENS)        // 2048 — input + output combined
    .setMaxTopK(LlmConfig.TOP_K)
    .setPreferredBackend(LlmInference.Backend.CPU)
    .build()

inference = LlmInference.createFromOptions(context, options)
Enter fullscreen mode Exit fullscreen mode

Two things I learned the hard way:

  • MAX_TOKENS is the whole budget, not just output. MediaPipe rejects the entire request if the input alone (system prompt + conversation history) exceeds it. On a 2048-token budget, an interview's growing history is a real constraint you have to manage, not an afterthought.
  • I force the CPU backend. The GPU path was unstable on the devices I tested, and a 1B model runs perfectly well on CPU/XNNPACK. Chasing the GPU wasn't worth the crashes.

Streaming tokens so it doesn't feel frozen

A 1B model on a phone CPU is not instant. If you wait for the full response, the UI looks hung. So generation streams token-by-token through a callbackFlow, and the subtitles fill in live:

override fun generateResponseStream(prompt: String): Flow<String> = callbackFlow {
    val listener = ProgressListener<String> { partial, done ->
        if (partial != null) trySend(partial)   // emit each delta as it lands
        if (done) close()
    }
    val future = engine.generateResponseAsync(prompt, listener)
    future.addListener({
        runCatching { future.get() }.onFailure { close(it) }
    }, Runnable::run)
    awaitClose { }
}.flowOn(Dispatchers.Default)
Enter fullscreen mode Exit fullscreen mode

Measuring first-token latency (not total time) turned out to be the number that actually predicts whether the app feels alive.

The real lesson: don't trust the 1B model with anything you can compute yourself

This is the part I want other people building on small on-device models to hear.

A 1B model is not a small GPT-4. It's a text predictor that will happily forget which question number it's on, introduce itself with the wrong name, or invent a technical question that's off-topic. So the rule I converged on is: anything that has to be correct, compute in Kotlin. Only ask the model to do the part that genuinely needs generation.

Concretely:

The greeting is not generated. The interviewer's name and title must always be right, so a plain string builds it:

// A 1B model can't be trusted to introduce itself reliably, and the greeting
// is a hard product requirement.
fun buildGreeting(interviewer: InterviewerProfile): String =
    "Hello, I am ${interviewer.name}, your ${interviewer.title} interviewer today."
Enter fullscreen mode Exit fullscreen mode

The question counter lives in Kotlin, not in the model's head. The ViewModel owns "we're on question 3 of 5" and "all answers are in, now evaluate." Each turn injects that state as an explicit control block, because the model cannot reliably count its own turns:

// The question counter and the evaluation switch are decided in Kotlin, not by
// the model — a 1B model cannot reliably count its own turns. Each call injects
// the current state as an explicit control block so Gemma always knows exactly
// what to do next.
Enter fullscreen mode Exit fullscreen mode

Technical questions come from a bank, not the model. For the Hard/Expert interviewers, the exact question text is pulled from a curated QuestionBank and handed to the model to present (lightly rephrased at most). This keeps generations short, on-topic, and actually about CI/CD and Kubernetes instead of whatever a 1B model free-associates into.

And the prompt itself is hand-built in Gemma's turn format, priming the model with an acknowledgement turn so it stays in character:

sb.append("${SOT}user\n$systemPrompt$EOT\n")
sb.append("${SOT}model\nUnderstood. I am ready to conduct the interview.$EOT\n")
// ...conversation history, then the control block + user's answer
Enter fullscreen mode Exit fullscreen mode

The pattern generalizes: the model generates prose; deterministic code owns state, identity, and control flow. That division is what makes a 1B model feel like a product instead of a toy.

A small delightful detail: two interviewers, one voice — on purpose

Voices are on-device TTS. Marcus (Senior DevOps) and David (CTO) deliberately share the exact same male voice and pitch:

// Marcus shares David's exact male voice (MALE_SECONDARY + 0.78 pitch) by
// requirement — keep these two lines identical to David's below.
voicePitch = 0.78f,
voiceProfile = VoiceProfile.MALE_SECONDARY,
Enter fullscreen mode Exit fullscreen mode

It reads like a bug in a diff — two profiles with identical voice config — so the comment exists to stop a future me (or a reviewer) from "fixing" it. Product requirements that look like mistakes need a paper trail in the code.

Architecture & process

The package layout enforces clean boundaries — features / core (LLM, voice, download) / domain (models) / data (storage) — with no Android framework imports leaking into the domain models. Development follows a strict Git Flow (main → develop → feature/*), annotated release tags, task IDs on every commit, and GitHub Actions for CI. Currently at v1.7.2.

License & privacy

The app is licensed under the Business Source License 1.1 (© 2026 Alex Korchenko) — not MIT, and I'd rather be accurate than slap an "open source" badge on it. What's genuinely public is what a user needs: the APK and the model bundle are distributed openly, no login or token required. The privacy guarantee is the important promise, and it's structural, not a policy: interview data never leaves the device because there is no server for it to go to.

Takeaways

  • On-device LLM ≠ small cloud LLM. Budget your context window as input plus output, pin a stable backend, and measure first-token latency, not total time.
  • Never embed a credential in a client to reach a gated model. Re-host the artifact somewhere public (a GitHub Release asset works great) and stream it in — download to .part, rename atomically, and let a file-exists check be your "ready" signal.
  • Compute everything you can; only generate what you must. Names, counters, control flow, and canned questions belong in your own code. A 1B model is a prose engine wrapped in deterministic scaffolding — the scaffolding is the product.
  • Comment the requirements that look like bugs. Two identical voice configs, a legacy filename — a one-line comment saves the next person from "fixing" a deliberate choice.

The flashy line is "AI interviewer in your pocket, fully offline." The engineering truth is quieter: a small model plus a lot of boring, deterministic Kotlin around it.

About the author

I'm Alex — a DevOps/engineer who builds things end-to-end and writes about the parts that don't make it into the README. This project came out of wanting realistic interview practice without handing my worst answers to someone else's server.

Building on a small on-device model too? I'd love to compare notes on what you had to compute yourself — drop it in the comments.

Top comments (0)