Your First On-Device AI PR: Ship a Golden Set and Rehearse the Rollback

It is 9:40 on a Tuesday, and a junior engineer on your team has just cloned the notes app repository onto a laptop with 8 GB of RAM and no GPU. The ticket says: add a summary to the note detail screen, and make it work on the phone rather than round-tripping every keystroke to a server. She writes a first version, backgrounds the app mid-inference to answer a message, and comes back to a spinner that never resolves. That single lifecycle transition — backgrounding during inference — is what her first pull request has to survive, and it is the thing most onboarding checklists leave out.

This article is a proposed test plan, not a measured benchmark. I have not run the numbers below on your hardware, and the code samples are illustrative rather than copy-paste production code. What you get instead is a repeatable structure: one golden set, one decision table, and one rollback rehearsal you can finish in your first hour.

Record the environment before you write product code

You cannot debug an inference path if you do not know what you ran it on. Open a scratch file and fill in every row below before the first commit, then paste the same table into the PR description.

Field	Fill in
Device	model and, if you have one, the exact variant
OS	version and build number, not just "Android" or "iOS"
App	version, git SHA, and build type
Framework	runtime version plus every AI-related dependency pin
Permissions	microphone, network, storage — granted or revoked at test time
Network	Wi-Fi, cellular, throttled, or airplane mode
Power	battery percentage and whether the device was charging
Transition	the one lifecycle event the ticket is really about

Notice the last row. Pick exactly one transition for your first PR. Backgrounding during inference is a good default because it is cheap to reproduce and it exposes timeout handling immediately.

Three gates your first PR has to pass

A junior engineer's first inference PR is not judged on model quality. It is judged on whether a reviewer can answer three questions without opening the app.

1. With network present, does the request return a valid, schema-checked result?
2. With network absent, permission revoked, or the endpoint returning 5xx, does the feature fall back to something deterministic instead of a spinner?
3. Can the feature be turned off from a configuration endpoint without shipping a new build?

If you can only prove one of the three, prove the third. A kill switch is what lets the rest of the team merge your work without betting the release on it.

Where a free endpoint fits, and what you should not assume

Gates 1 and 3 both need a small endpoint you control: something to run the golden set against, and something to serve the feature flag. MonkeyCode is an open-source project whose operator states that it offers free model access, currently described as a 10M-token free allowance, together with a free server option. Treat both as vendor claims: sign in, read the current terms, and confirm the limits in your own account before you build a habit on top of them. Token allowances, available models, and server terms can change, and this article does not assert any specific model name, hardware spec, or performance number.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use the free model access for the golden-set runner and the free server for the flag endpoint and, if you like, a scheduled re-run of the same runner. Do not use it as the only path your app can take — see the rollback section below.

Build a 12-case golden set before you tune a prompt

Twelve cases is enough to catch schema drift and empty-output bugs, and small enough that a junior engineer can read every line in review. Store it as JSONL so the runner and any future harness can consume the same file.

{"id":"short-note","input":"Groceries: milk, eggs, bread.","max_chars":120,"must_be_single_paragraph":true}
{"id":"unicode-emoji","input":"Ideas 🎧 for the commute, plus a reminder to buy 🥛.","max_chars":120,"must_be_single_paragraph":true}
{"id":"empty-note","input":"","max_chars":120,"expect_fallback":true}
{"id":"very-long","input":"<paste 3000 characters of real notes from your own device>","max_chars":120,"must_be_single_paragraph":true}
{"id":"permission-revoked","input":"Summarize this with the microphone permission denied.","expect_fallback":true}
{"id":"offline","input":"Summarize this with radio off.","expect_fallback":true}

The important rows are the boring ones: empty input, revoked permission, and offline. Those are the states where a demo-grade implementation quietly returns null and the UI decides to show a spinner forever.

Run the golden set from a small server

The runner below is an unexecuted example. Set the base URL and key from your own project dashboard rather than hard-coding anything, and keep the API key in the server environment, never in the app bundle.

#!/usr/bin/env bash
set -euo pipefail

: "${MONKEYCODE_BASE_URL:?set from your project dashboard}"
: "${MONKEYCODE_API_KEY:?set this in the server environment}"
MODEL="${MODEL:?pick a model from your dashboard}"
OUT="golden-report.tsv"

printf 'id\tstatus\thttp\tttfb_ms\tschema_ok\tchars\n' > "$OUT"

while IFS= read -r line; do
  id=$(jq -r .id <<<"$line")
  prompt=$(jq -r .input <<<"$line")

  start=$(date +%s%3N)
  http=$(curl -sS -o /tmp/body.json -w '%{http_code}' \
    -X POST "$MONKEYCODE_BASE_URL/chat/completions" \
    -H "Authorization: Bearer $MONKEYCODE_API_KEY" \
    -H 'Content-Type: application/json' \
    -d "$(jq -nc --arg m "$MODEL" --arg p "$prompt" \
      '{model:$m,messages:[{role:"user",content:$p}],max_tokens:160,temperature:0}')")
  ttfb=$(( $(date +%s%3N) - start ))

  text=$(jq -r '.choices[0].message.content // empty' /tmp/body.json)
  chars=${#text}
  schema_ok=false
  if [ -n "$text" ] && [ "$(grep -c $'\n' <<<"$text")" -eq 0 ]; then schema_ok=true; fi

  printf '%s\t%s\t%s\t%s\t%s\t%s\n' \
    "$id" "$([ "$http" = 200 ] && echo ok || echo fail)" "$http" "$ttfb" "$schema_ok" "$chars" >> "$OUT"
done < golden_set.jsonl

column -t "$OUT"

Run it once, paste the table into the PR, and commit the script next to the app code so the next new hire inherits it. Two rules matter more than the code: keep temperature at 0 so diffs are readable, and record the wall-clock time of the run so a reviewer knows which endpoint revision produced it.

Pad the decision table with your own constraints

Before you defend a placement decision in review, write down what you actually need. The table below is a structure, not a recommendation.

Option	Latency you can promise	Offline behavior	Data egress	Rollback cost
Fully on-device	depends on your model size and device class	works	none	app release
Hosted endpoint	network round trip plus queueing	fallback needed	note text leaves the device	flag flip
Hybrid, flag-gated	hosted when healthy, local otherwise	degraded but deterministic	conditional	flag flip plus cache

Fill the latency column with measurements from your own device, not with numbers copied from someone else's blog post. If you cannot measure it, say so in the PR instead of guessing.

Rehearse the rollback in the first hour, not the first incident

A kill switch you have never exercised is a comment, not a control. Rehearse it once, in this order:

1. Enable the flag in your config endpoint and confirm the remote path is used.
2. Disable the flag, force-stop the app, and relaunch. The local fallback should appear with no crash and no empty state.
3. Disable the flag while the app is in the foreground and trigger a summary. Your code must read the flag per call, not once at startup.
4. Point the app at an unreachable endpoint and repeat step 3 to prove the timeout path, not just the flag path.
5. Enable airplane mode, then revoke the microphone permission, and repeat step 3 for each state.
6. Restore everything, and confirm the remote path returns without a reinstall.

Cache the last known flag value locally. If the config endpoint itself is down, an app that depends on a live fetch to decide whether to crash or fall back has simply moved the outage.

// Illustrative, not compiled. Gate the remote call on the cached flag
// and on a network check, and always keep the local path available.
class SummarizeNoteUseCase(
    private val flags: FeatureFlags,   // cached, refreshed in background
    private val remote: RemoteSummarizer,
    private val local: ExtractiveSummarizer,
) {
    suspend fun summarize(note: Note): Summary {
        if (!flags.noteSummaryEnabled || !remote.isReachable()) {
            return local.summarize(note)
        }
        return runCatching { remote.summarize(note) }
            .getOrElse { local.summarize(note) }
    }
}

Set your request timeout below the point where a user typically leaves the screen, and measure that interval on your own device rather than assuming it.

What goes in the PR description

Keep it short and evidence-shaped:

• The environment table from the top of this article.
• The golden-set report, including the offline and permission-revoked rows.
• Which lifecycle transition you tested, and what the app did after it.
• The rollback steps you ran, and whether the feature recovered, restarted, or silently disappeared.
• What you did not test, stated plainly.

Limitations and who should not copy this

This workflow is a starting point, and it has edges worth naming out loud. It will not tell you anything about model quality on long or adversarial inputs, because twelve cases cannot. It produces no battery or thermal numbers unless you add a measurement step and a consistent power state. It assumes you have some form of remote configuration; if your app has none, build the local fallback first and treat the flag as a later addition. And if your notes contain regulated data, a hosted endpoint is a compliance decision that a golden set cannot make for you.

Skip the hosted path entirely if your product needs deterministic sub-second behavior with no network dependency, if you cannot measure latency on a real device, or if you have no way to serve a flag. A local-only extractive summary is a perfectly respectable first PR.

Send me the environment, not the opinion

If you try this on your own repo, I am more interested in your setup than in your verdict. Post the device, OS version, the transition you tested, and whether the feature recovered, restarted, or silently disappeared after it. Those four fields are what turn an onboarding story into something the next junior engineer can reproduce.

If you need a model endpoint and a small server to run the golden set while you onboard, MonkeyCode's free model access and free server option is one place to start — check the current terms in your own account first, and keep the fallback path working regardless of what the free tier looks like next month.