DEV Community: Alex DevOps engineer

Running an AI interviewer fully on-device with Gemma 3 1B — and engineering around what a 1B model can't do

Alex DevOps engineer — Wed, 08 Jul 2026 20:35:35 +0000

I built an Android app that runs a mock DevOps job interview with an AI interviewer that lives entirely on your phone. No cloud API, no OpenAI key, no round-trip to a server. The language model — Gemma 3 1B — is downloaded once and every inference after that runs fully offline, on the subway, on a plane, with airplane mode on.

The headline is "on-device LLM." But the honest story is the opposite of the hype: the model was the easy part. The real work was engineering a wall of determinism around the model — because a 1B model, running on a phone CPU, cannot be trusted to do the things you'd casually hand to GPT-4.

The project

DevOps Interview AI is a native Android app that puts you in a mock interview against one of four AI personas, each a different seniority level:

Interviewer	Level	Focus
Emma — Recruiter	Easy	Cultural fit, CV review, salary expectations
Olivia — HR	Medium	Behavioural questions, team dynamics, soft skills
Marcus — Senior DevOps	Hard	CI/CD, Kubernetes, IaC, observability, SRE
David — CTO	Expert	Architecture, scalability, engineering trade-offs

You pick an interviewer, join a Google Meet-style session (interviewer avatar tile, live subtitles, your camera in the corner), and answer five questions by voice. Speech-to-text turns your answer into text, the model asks the next question, text-to-speech speaks it back. At the end you get a scored evaluation. All of it offline.

The stack:

Layer	Technology
Language	Kotlin
UI	Jetpack Compose + Material 3
AI engine	Google MediaPipe Tasks GenAI 0.10.24
Model	Gemma 3 1B, INT4 quantized (`.task` bundle)
Voice	Android `SpeechRecognizer` (STT) + `TextToSpeech` (TTS)
Storage	Room + DataStore
Architecture	ViewModel + StateFlow, strict package boundaries

Why on-device at all?

The motivation is privacy, and it's not a marketing checkbox — it's the whole point. Interview practice is where you're most exposed: you fumble answers, you second-guess your seniority, you say things you'd never put in front of a recruiter. Sending that stream to a third-party API felt wrong. On-device means the interview data physically cannot leave the phone. No telemetry, no cloud sync, nothing to leak.

The side effects are nice too: zero inference cost, no rate limits, and it works with no connection at all. But privacy is the reason the whole thing runs the way it does.

Getting a gated model into a client app without leaking a token

Here's the first real problem. Gemma 3 1B is distributed on Hugging Face behind a license gate (litert-community/Gemma3-1B-IT). An anonymous download gets an HTTP 401 — you need an access token. And you cannot ship a personal Hugging Face token inside an APK: anyone who unzips the APK can extract it and reuse it.

The .task bundle is also ~530 MB, so bundling it as an app asset is a non-starter (nobody ships a half-gigabyte APK).

The solution splits distribution in two:

The source app is one repository.
The model bundle is published as a public GitHub Release asset in a separate repo. GitHub Release assets are served over plain HTTPS with no credential — no token to embed, no gate to pass.

On first launch, the app streams the model straight from that release into its private storage:

/**
 * Streams the Gemma 3 1B `.task` bundle from a GitHub Release asset straight
 * into Context.getFilesDir ...
 *
 * Hosted as a Release asset (not the original Hugging Face repo) because
 * litert-community/Gemma3-1B-IT is gated behind license acceptance — an
 * anonymous request gets HTTP 401, and embedding a personal HF access token
 * in a client APK would let anyone who unzips the APK extract and reuse it.
 *
 * Writes to a `.part` file first and renames atomically on success, so a
 * half-downloaded file is never mistaken for a ready model.
 */

That last line matters more than it looks. The "is the model ready?" check is deliberately dumb — a single file-exists lookup:

const val MODEL_FILE_NAME = "gemma-2b-it-gpu-int4.task"

fun isModelReady(context: Context): Boolean =
    File(context.filesDir, MODEL_FILE_NAME).exists()

If the download wrote directly to that filename and the process got killed at 60%, isModelReady() would return true for a corrupt file, and the engine would crash on load forever. Downloading to model.task.part and renaming atomically on completion means the real filename only ever exists as a complete file. Crash-safety from one rename().

(An aside on that filename: it says gemma-2b, but the model is Gemma 3 1B. MediaPipe's LlmInferenceOptions doesn't care what the model is called — it loads whatever .task sits at the path. So the filename is just a fixed interface contract that the downloader, the splash bootstrap check, and the engine all agree on. I left the legacy name rather than pretend it means anything.)

Loading the model: CPU, on purpose

MediaPipe makes the actual load short:

val options = LlmInferenceOptions.builder()
    .setModelPath(modelPath)
    .setMaxTokens(LlmConfig.MAX_TOKENS)        // 2048 — input + output combined
    .setMaxTopK(LlmConfig.TOP_K)
    .setPreferredBackend(LlmInference.Backend.CPU)
    .build()

inference = LlmInference.createFromOptions(context, options)

Two things I learned the hard way:

MAX_TOKENS is the whole budget, not just output. MediaPipe rejects the entire request if the input alone (system prompt + conversation history) exceeds it. On a 2048-token budget, an interview's growing history is a real constraint you have to manage, not an afterthought.
I force the CPU backend. The GPU path was unstable on the devices I tested, and a 1B model runs perfectly well on CPU/XNNPACK. Chasing the GPU wasn't worth the crashes.

Streaming tokens so it doesn't feel frozen

A 1B model on a phone CPU is not instant. If you wait for the full response, the UI looks hung. So generation streams token-by-token through a callbackFlow, and the subtitles fill in live:

override fun generateResponseStream(prompt: String): Flow<String> = callbackFlow {
    val listener = ProgressListener<String> { partial, done ->
        if (partial != null) trySend(partial)   // emit each delta as it lands
        if (done) close()
    }
    val future = engine.generateResponseAsync(prompt, listener)
    future.addListener({
        runCatching { future.get() }.onFailure { close(it) }
    }, Runnable::run)
    awaitClose { }
}.flowOn(Dispatchers.Default)

Measuring first-token latency (not total time) turned out to be the number that actually predicts whether the app feels alive.

The real lesson: don't trust the 1B model with anything you can compute yourself

This is the part I want other people building on small on-device models to hear.

A 1B model is not a small GPT-4. It's a text predictor that will happily forget which question number it's on, introduce itself with the wrong name, or invent a technical question that's off-topic. So the rule I converged on is: anything that has to be correct, compute in Kotlin. Only ask the model to do the part that genuinely needs generation.

Concretely:

The greeting is not generated. The interviewer's name and title must always be right, so a plain string builds it:

// A 1B model can't be trusted to introduce itself reliably, and the greeting
// is a hard product requirement.
fun buildGreeting(interviewer: InterviewerProfile): String =
    "Hello, I am ${interviewer.name}, your ${interviewer.title} interviewer today."

The question counter lives in Kotlin, not in the model's head. The ViewModel owns "we're on question 3 of 5" and "all answers are in, now evaluate." Each turn injects that state as an explicit control block, because the model cannot reliably count its own turns:

// The question counter and the evaluation switch are decided in Kotlin, not by
// the model — a 1B model cannot reliably count its own turns. Each call injects
// the current state as an explicit control block so Gemma always knows exactly
// what to do next.

Technical questions come from a bank, not the model. For the Hard/Expert interviewers, the exact question text is pulled from a curated QuestionBank and handed to the model to present (lightly rephrased at most). This keeps generations short, on-topic, and actually about CI/CD and Kubernetes instead of whatever a 1B model free-associates into.

And the prompt itself is hand-built in Gemma's turn format, priming the model with an acknowledgement turn so it stays in character:

sb.append("${SOT}user\n$systemPrompt$EOT\n")
sb.append("${SOT}model\nUnderstood. I am ready to conduct the interview.$EOT\n")
// ...conversation history, then the control block + user's answer

The pattern generalizes: the model generates prose; deterministic code owns state, identity, and control flow. That division is what makes a 1B model feel like a product instead of a toy.

A small delightful detail: two interviewers, one voice — on purpose

Voices are on-device TTS. Marcus (Senior DevOps) and David (CTO) deliberately share the exact same male voice and pitch:

// Marcus shares David's exact male voice (MALE_SECONDARY + 0.78 pitch) by
// requirement — keep these two lines identical to David's below.
voicePitch = 0.78f,
voiceProfile = VoiceProfile.MALE_SECONDARY,

It reads like a bug in a diff — two profiles with identical voice config — so the comment exists to stop a future me (or a reviewer) from "fixing" it. Product requirements that look like mistakes need a paper trail in the code.

Architecture & process

The package layout enforces clean boundaries — features / core (LLM, voice, download) / domain (models) / data (storage) — with no Android framework imports leaking into the domain models. Development follows a strict Git Flow (main → develop → feature/*), annotated release tags, task IDs on every commit, and GitHub Actions for CI. Currently at v1.7.2.

License & privacy

The app is licensed under the Business Source License 1.1 (© 2026 Alex Korchenko) — not MIT, and I'd rather be accurate than slap an "open source" badge on it. What's genuinely public is what a user needs: the APK and the model bundle are distributed openly, no login or token required. The privacy guarantee is the important promise, and it's structural, not a policy: interview data never leaves the device because there is no server for it to go to.

Takeaways

On-device LLM ≠ small cloud LLM. Budget your context window as input plus output, pin a stable backend, and measure first-token latency, not total time.
Never embed a credential in a client to reach a gated model. Re-host the artifact somewhere public (a GitHub Release asset works great) and stream it in — download to .part, rename atomically, and let a file-exists check be your "ready" signal.
Compute everything you can; only generate what you must. Names, counters, control flow, and canned questions belong in your own code. A 1B model is a prose engine wrapped in deterministic scaffolding — the scaffolding is the product.
Comment the requirements that look like bugs. Two identical voice configs, a legacy filename — a one-line comment saves the next person from "fixing" a deliberate choice.

The flashy line is "AI interviewer in your pocket, fully offline." The engineering truth is quieter: a small model plus a lot of boring, deterministic Kotlin around it.

About the author

I'm Alex — a DevOps/engineer who builds things end-to-end and writes about the parts that don't make it into the README. This project came out of wanting realistic interview practice without handing my worst answers to someone else's server.

📱 App + model (public releases): AI-DevOps-Interview-Avatar
💼 LinkedIn: alex-d
🐦 X: @mainoceanm
📄 CV: résumé on OneDrive
✉️ mainoceanm@gmail.com

Building on a small on-device model too? I'd love to compare notes on what you had to compute yourself — drop it in the comments.

How a rotating refresh token turned into a 401 storm on Cloudflare Workers

Alex DevOps engineer — Wed, 08 Jul 2026 16:25:03 +0000

We run a Cloudflare Workers + D1 backend with rotating refresh tokens. One afternoon production started sending a steady stream of 401s on POST /v1/auth/refresh — not a spike, a storm: dozens of alerts, spread out, never stopping.

The obvious suspect (a rotation race between two browser tabs sharing one refresh cookie) turned out to only be part of the story. The actual fix took three small, independent changes:

The frontend called refresh on every page load, even with a still-valid access token — turning ordinary browsing into constant unnecessary rotation.
A rejected refresh never cleared the httpOnly cookie, so a browser with a dead session kept replaying the same doomed request forever.
Anonymous visitors (no session at all) were still calling refresh and getting a guaranteed 401 on every view.

None of this needed a new backend mechanism — just removing assumptions that had quietly stopped being true.

The project

This is Brewly Store, a coffee e-commerce platform built entirely on Cloudflare's edge:

Backend — a Cloudflare Workers API written in Hono with @hono/zod-openapi, backed by Cloudflare D1 (SQLite at the edge). It owns auth, users and products.
Frontend — a Nuxt 4 storefront (Vue 3, Pinia, TailwindCSS) deployed to Cloudflare Pages.
Bot — a Telegram bot, also on Workers, sharing the same D1 database.

Auth uses a short-lived access token (in memory on the client) plus a rotating refresh token kept in an httpOnly, cross-origin cookie. Because the cookie is httpOnly and set on a different origin than the storefront, the browser can send it but JavaScript can never read it. Remember that constraint — it's the seed of the whole storm.

The storm

The symptom was a flat, unrelenting line of 401 Unauthorized on POST /v1/auth/refresh. Not correlated with a deploy, not correlated with traffic peaks — just a constant background hum of failed refreshes, each one firing an alert.

Latency and error budgets were fine for real endpoints. The 401s were "successful failures": the Worker was doing exactly what it was told, rejecting refresh requests that could never succeed. The bug wasn't in the rejection — it was in who kept asking.

Fix 1 — stop refreshing on every page load

The Pinia auth store refreshed the session on initialization, unconditionally. Every navigation that re-hydrated the store fired a refresh, even when the access token had 40 minutes of life left. On a rotating-token scheme that's not just wasteful — every refresh rotates the token, so two quick navigations could race each other.

The fix is a threshold: only refresh when the access token is actually near expiry.

const REFRESH_THRESHOLD_MS = 60_000

function needsAccessTokenRefresh() {
  if (!expiresAt.value) return true
  return expiresAt.value - Date.now() <= REFRESH_THRESHOLD_MS
}

Now ordinary browsing rides the existing access token and only refreshes in the last minute of its lifetime, instead of on every mount.

Fix 2 — clear the dead cookie when refresh is rejected

Here's the nasty one. When the backend rejected a refresh (expired/rotated/revoked token), it returned a 401 — but left the httpOnly refresh cookie in place. The browser, having no way to inspect that cookie, had no idea it was dead. So on the next page load it dutifully sent the same doomed cookie again... and again... forever. One dead session = an infinite private 401 stream.

The backend now clears the cookie on the rejected path, so the browser stops resending it:

import { clearRefreshTokenCookie } from '../../lib/refresh-cookie'

// inside the refresh handler, on the failure branch:
if (!result.ok) {
  clearRefreshTokenCookie(c)          // tell the browser to drop the dead cookie
  return c.json({ error: 'Unauthorized' }, 401)
}

A Set-Cookie with an expired/Max-Age=0 value turns "replay forever" into "fail once, then go quiet."

Fix 3 — don't let anonymous visitors refresh at all

Because the refresh cookie is invisible to JavaScript, the client couldn't tell a logged-out visitor apart from a logged-in one at boot time — so it optimistically tried to refresh for everyone, including first-time anonymous visitors who never had a session. Each of those is a guaranteed 401.

The client can't read the cookie, but it can remember that a session once existed in this browser. A tiny localStorage marker does exactly that:

// The refresh cookie is httpOnly on a cross-origin API, so the client can't see
// it — this marker records "a session once existed in this browser" instead, so
// anonymous visitors never fire a doomed POST /v1/auth/refresh on page load.
const HAD_SESSION_MARKER_KEY = 'brewly-had-session'

function hadSessionInThisBrowser() {
  if (!import.meta.client) return false
  return localStorage.getItem(HAD_SESSION_MARKER_KEY) === '1'
}

The marker is set when a session is stored and removed in clearSession() (right next to fix #2 on the client side). At boot, the store only attempts a refresh if hadSessionInThisBrowser() is true. Anonymous visitors now make zero refresh calls.

The rotation race (the suspect that wasn't the whole story)

The original hypothesis — two tabs sharing one refresh cookie, each rotating it and invalidating the other — was real, but it was a contributor, not the root cause. Once fixes 1–3 collapsed the volume of refresh calls, the race window shrank dramatically: fewer calls, fewer concurrent rotations. (We also added a short rotation grace so a just-rotated token isn't instantly rejected for an in-flight second request — but that's a smaller, separate story.)

Takeaways

A 401 storm is a "who's asking" problem as often as a "why rejected" problem. The rejection logic was correct the whole time.
httpOnly cross-origin cookies are invisible to the client — design for that. If the client can't see the cookie, give it a separate signal (a marker) so it can reason about session state without guessing.
Always clear a credential the server just rejected. Otherwise the client will replay it until the heat death of the browser tab.
Refresh on need, not on mount. Rotation schemes punish over-eager refreshing.

Three small, independent changes — no new backend architecture, no schema migration. Just deleting assumptions that had quietly stopped being true.

About the author

I'm Alex — a DevOps engineer building Brewly Store, a coffee e-commerce platform that runs entirely on Cloudflare's edge (Workers, D1, Pages). I write about edge architecture, auth, and the debugging stories that come with shipping to production.

🌐 Live project: brewly.online
🎥 YouTube — building the Brewly Telegram bot on Cloudflare Workers: watch the walkthrough
💼 LinkedIn: alex-d
🐦 X: @mainoceanm
📄 CV: résumé on OneDrive
✉️ mainoceanm@gmail.com

Questions or war stories about edge auth? Drop them in the comments.

Shipping at the Edge: Migrating a Coffee Subscription Platform to Cloudflare Workers

Alex DevOps engineer — Fri, 22 May 2026 13:10:47 +0000

Building a D2C platform isn't just about the product; it's about the resilience of the delivery chain. I've been working on Brewly Store, a coffee subscription service, and recently decided to tear down our traditional setup to embrace a more modern, edge-first architecture.

brewly-frontend.pages.dev

The Tech Stack 🛠:

Runtime: Cloudflare Workers (using the Hono framework).

Language: TypeScript.

Infrastructure: AWS EKS for core services.

Delivery: A full GitOps pipeline using ArgoCD and GitHub Actions.

Why the Migration?
We moved from a containerized AWS ECR setup to Cloudflare Workers. Why?

Latency: Moving logic closer to the user.

Scalability: Handling subscription spikes without managing scaling groups.

Developer Experience: Hono provides a lightweight, expressive way to handle routing that feels incredibly fast.

The Security-First Mindset 🔒:
As a DevOps specialist, I believe infrastructure is only as good as its security. For Brewly, I’ve implemented:

GPG-signed commits to ensure the integrity of our code.

Strict secret management using tools like Kleopatra and PwPush.

GitOps-driven deployments where ArgoCD ensures the cluster state matches our repository exactly.

Lessons Learned from the Trenches:
The biggest challenge wasn't the code—it was the shift in thinking from "servers" to "distributed functions." Debugging edge cases in a serverless environment requires a robust observability stack, which we are currently refining within our EKS cluster.

DevOps Journey UA - YouTube

Welcome to the channel! I'm Alex, a DevOps Specialist. On this channel, I share hands-on experience in building modern IT infrastructure, automation, and implementing BigTech standards in real-world projects. What to expect: Cloud Solutions: Deep dives into AWS (EKS, ECR, Secrets Manager) and transitioning to Serverless architecture (Cloudflare Workers, D1). Containerization: Everything about Docker, Kubernetes, and CI/CD pipeline optimization. Security First: Practical advice on secret management, GPG-signed commits, and infrastructure security. IT Productivity: Reviews of task managers (Jira, Linear, ClickUp) and building effective workflows for developers. Why subscribe? I don't just talk about theory—I show the "end-to-end" process: how to migrate a real service from AWS to Cloudflare, how to set up automated NAS backups, or how to organize a team's work in Jira. All content is based on my daily work with Linux (Ubuntu) and CLI-driven environments.

youtube.com