DEV Community: Roronoa

A Lifecycle Probe for MiniMax H3: Benchmark the Phone, Not the Leaderboard

Roronoa — Fri, 14 Aug 2026 11:03:29 +0000

When a small open model such as MiniMax H3 starts trending, mobile teams often swap it into an existing cloud call before testing the environment around the call. The leaderboard may improve, but the user-visible failure can happen later: the request is retried into a dead socket, the OS kills the process during backgrounding, or a revoked microphone permission turns a voice flow into a silent spinner.

This article builds a reproducible lifecycle probe for any newly released mobile model endpoint, using MiniMax H3 as the example subject. The probe treats the phone as the system under test, not the model. It records whether a request completes, retries, recovers, or disappears across the transitions that matter on a real device.

Why MiniMax H3 needs a phone-first test

MiniMax H3 is an example of the current wave of small open models that are light enough for mobile teams to consider. A leaderboard number usually represents one environment. It does not represent airplane mode toggling, backgrounding, a revoked microphone permission, or a battery saver restart. The probe below makes no claim about MiniMax H3's benchmark rank. It measures whether the integration survives the conditions mobile users actually hit.

What the probe does

The harness keeps model-specific code outside the test. It runs the same prompt against two targets:

A local deterministic mock endpoint that always responds after a fixed delay.
The real model endpoint under test, such as MiniMax H3 behind an OpenAI-compatible wrapper or a raw HTTP endpoint.

For each target, the probe applies the same OS lifecycle transitions and records one terminal outcome per run: completed, retried, recovered, or silently disappeared.

Requirements and boundaries

Android 13 or newer device with Developer Options and ADB, or an emulator running API 34. A physical device is more useful for radio and battery observations.
Python 3.10 or newer on the controlling machine.
A model endpoint that accepts HTTP POST requests. MiniMax H3 is used as the example payload, but the harness is not tied to one vendor.
The probe does not score model quality. It measures call reliability across device transitions.

Build the transition matrix

ID	Transition	Action	Pass condition	Common silent failure
S0	Foreground idle	Call once while the app stays open	First byte and final byte arrive, status 200	High tail latency hidden by client retries
S1	Network loss mid-stream	Enable airplane mode after the first byte	Request fails fast or resumes after reconnect; no infinite retry	Retry thread holds a stale socket after network returns
S2	Background for five seconds	Press home, wait, return	Request completes or resumes with no duplicate side effect	Duplicate completion
S3	Process kill	`adb shell am force-stop <pkg>`	App restarts and recovers cleanly	Partial response cached as final
S4	Permission revoke	`adb shell pm revoke <pkg> android.permission.RECORD_AUDIO`	Voice input degrades explicitly	Spinner while waiting for microphone permission
S5	Battery saver	`adb shell cmd power set-mode 1` plus low battery state	Background request obeys the scheduler or fails explicitly	Unhandled timeout without user feedback

Set up a deterministic local endpoint

Run this mock server first. It removes model variance while the lifecycle code is being checked.

# mock_server.py
from http.server import BaseHTTPRequestHandler, HTTPServer
import time

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get('Content-Length', 0))
        self.rfile.read(length)
        time.sleep(0.8)
        body = b'mock response'
        self.send_response(200)
        self.send_header('Content-Type', 'text/plain')
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        self.wfile.write(body)

HTTPServer(('127.0.0.1', 8787), Handler).serve_forever()

Start it with:

python3 mock_server.py

From an Android emulator, 10.0.2.2:8787 maps to the host loopback interface. From a physical device, use the development machine's LAN address and keep the endpoint reachable from the test device.

Drive the transitions from a small Python orchestrator

The first version records start and completion. It is deliberately simple; replace the urllib.request reader with a streaming reader if first-byte time matters for your model path.

# probe.py
import subprocess
import time
import urllib.request

ADB = ['adb']

def adb(*args):
    subprocess.run([*ADB, *args], check=True, capture_output=True, text=True)

def call_endpoint(url):
    start = time.time()
    req = urllib.request.Request(url, data=b'probe', method='POST')
    req.add_header('Content-Type', 'text/plain')
    try:
        with urllib.request.urlopen(req, timeout=5) as resp:
            resp.read()
            end = time.time()
            return {'status': 'completed', 'start_ms': round(start * 1000), 'end_ms': round(end * 1000)}
    except Exception as exc:
        return {'status': type(exc).__name__, 'start_ms': round(start * 1000), 'end_ms': None}

transitions = ['idle', 'airplane', 'background', 'kill']
for transition in transitions:
    result = call_endpoint('http://10.0.2.2:8787')
    print(f'{transition} {result}')

The transition actions live outside the Python process because ADB controls the device state. Apply each one before the next call:

# airplane mode on
adb shell settings put global airplane_mode_on 1
adb shell am broadcast -a android.intent.action.AIRPLANE_MODE --ez state true
sleep 2

# airplane mode off after observation
adb shell settings put global airplane_mode_on 0
adb shell am broadcast -a android.intent.action.AIRPLANE_MODE --ez state false

# force-stop the app under test
adb shell am force-stop com.example.app

# revoke microphone permission
adb shell pm revoke com.example.app android.permission.RECORD_AUDIO

# simulate battery saver conditions
adb shell cmd power set-mode 1
adb shell dumpsys battery set level 15
adb shell dumpsys battery unplug

# restore after observation
adb shell cmd power set-mode 0
adb shell dumpsys battery reset

Replace com.example.app with the real package name. Do not run S1 through S5 as a single batch if the app caches state across restarts; reset between transitions for clean per-case evidence.

Use a free endpoint as a controlled baseline, not a benchmark authority

MonkeyCode's free model access and free server option can serve two specific roles in this harness: a controlled reference endpoint and a place to run the orchestration script without paying for a cloud VM. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The methodological rule is important. Run the deterministic local mock first. Once the mock passes all lifecycle transitions, call the free endpoint and record the same rows. If the free endpoint passes S0 but fails S1, the finding is about retry and reconnection behavior in the client, not the model's weights. If the free endpoint fails S0 only at night, treat the result as a rate-limit or capacity observation, not a model score.

Why this is an open-source-style habit

Model availability changes every few weeks. A reproducible harness does not. The useful open-source principle here is that the probe, transition matrix, and outcome logs can be shared and repeated without depending on a closed benchmark. A free endpoint makes the test cheap to run, but the artifact remains the harness itself.

Keep one prompt constant for every run. Change only the transition and the target. That isolates the phone environment from the model's output variability.

Run order

Start mock_server.py.
Run S0 against the mock and confirm a completed result.
Run S1 through S5 against the mock, resetting app state between cases as needed.
Replace the mock URL with the real MiniMax H3 endpoint or wrapper.
Re-run the same matrix.
Save each row as CSV:

run_id, target, transition, start_ms, end_ms, status, retry_count, note
probe-001, mock, idle, 1234, 2034, completed, 0, foreground
probe-002, mock, airplane, 1235, None, timeout, 1, mid-stream

Reading the results

If the mock fails S2, the app's lifecycle handling is broken before any model issue. If the mock passes and the real endpoint fails, investigate retry headers, socket timeouts, and reconnect logic rather than model weights. A model endpoint can return fast on a leaderboard and still fail S1 by holding a socket through reconnect; that is an integration problem, not an inference problem.

Record the same prompt, the same device, and the same OS version. The evidence should answer one question: did the request recover, retry, or silently disappear?

Limitations and who should not use this

This probe does not measure model quality, throughput, or accuracy.
Emulator radio and battery behavior differ from a physical device, especially for airplane mode and battery saver.
The ADB transition matrix is Android-specific. iOS requires equivalent XCTest or manual instrumentation, not the same commands.
Free endpoint rate limits can create false timeouts.
Do not use this for production capacity planning, security audit, or exact server-side metrics.
Teams that need authoritative backend metrics should instrument the server path instead of relying only on the client probe.

Use this probe before swapping a newly hyped model into a mobile app. If you have comparable device evidence for MiniMax H3 or another small open model, post your device, OS, transition steps, and whether the result recovered, retried, or silently disappeared.

Swap In That New Budget LLM? Run It Through Your Phone's Worst Day First

Roronoa — Thu, 13 Aug 2026 03:11:46 +0000

A low-cost model release just hit the feeds, and my operator passed along two names to evaluate: a budget tier labeled DeepSeek-V4-Pro-0813 plus a heavyweight option going by "gork 4.6" for difficult prompts. I haven't confirmed either identifier against official documentation, and neither should you — pull specs and pricing from the vendor's own pages, because launch-week writeups (including this one) are not a primary source.

What launch posts consistently skip is the only part mobile teams actually get burned by: how a freshly shipped cloud model holds up when the person using it locks their screen mid-response, steps into a parking garage, or hits the API over a congested LTE link on a three-year-old handset. Those conditions determine whether "cheap" stays cheap after you price in duplicate requests, dead sessions, and drained batteries.

Below is a runnable evaluation plan plus working code. Anything I describe as measured was run on hardware I name; anything else is marked as a proposal.

Why you need a middleman before you test anything

The single rule I enforce when evaluating any third-party model from a phone: the device never talks to the vendor directly during testing. Insert a pass-through server you control, so you can timestamp every request honestly, simulate outages on demand, and swap the model behind it without shipping a new build.

Your app (physical phone)
   │
   ▼
Pass-through server you own  ──►  Budget model endpoint
   │  (timestamps, routing)   ──►  Heavyweight model endpoint
   ▼
Append-only log

For the pass-through host, Disclosure: This article was prepared as part of MonkeyCode's product outreach. My operator notes that MonkeyCode currently offers free model access together with a free server tier — in principle enough capacity to host this exact pass-through-plus-backends arrangement with no billing account. That's operator-supplied availability information: verify the offer still exists and read the quota terms before designing around it. Functionally, any spare VPS or free-tier instance does the same job; the service is small enough to read in one sitting.

Working code: a pass-through that keeps receipts (Node.js)

The point of this service is restraint — it logs what the phone went through, forwards bytes, and stamps each reply with the backend identity. Nothing more.

// passthrough.js — Node 20+, standard library only
import { createServer } from "node:http";
import { createWriteStream } from "node:fs";

const ROUTES = {
  budget: { url: process.env.BUDGET_URL, key: process.env.BUDGET_KEY },
  heavy:  { url: process.env.HEAVY_URL,  key: process.env.HEAVY_KEY  },
};

const journal = createWriteStream("events.ndjson", { flags: "a" });

createServer(async (req, res) => {
  if (req.method !== "POST") { res.writeHead(405); return res.end(); }

  const startedAt = performance.now();
  const chunks = [];
  for await (const c of req) chunks.push(c);
  const raw = Buffer.concat(chunks).toString("utf8");

  const { route = "budget", condition = "unlabeled", payload } = JSON.parse(raw);
  const target = ROUTES[route];

  let status = 0, reply = "";
  try {
    const upstream = await fetch(target.url, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        authorization: `Bearer ${target.key}`,
      },
      body: JSON.stringify(payload),
      signal: AbortSignal.timeout(30_000), // mirror the app's own timeout
    });
    status = upstream.status;
    reply = await upstream.text();
  } catch (err) {
    status = -1; // timeout or unreachable — record it, don't hide it
  }

  journal.write(JSON.stringify({
    at: new Date().toISOString(),
    route,
    condition,
    status,
    ms_inside_server: Math.round(performance.now() - startedAt),
    in_bytes: raw.length,
    out_bytes: reply.length,
  }) + "\n");

  res.writeHead(status > 0 ? status : 502, { "content-type": "application/json" });
  res.end(reply || JSON.stringify({ error: "upstream_unreachable" }));
}).listen(8787);

Two design decisions worth stealing even if you rewrite the rest:

ms_inside_server deliberately omits the phone-to-server hop. The app times its own round trip; the delta between the two clocks is your last-mile network, which is where "the model feels slow" usually lives.
The condition label originates on the handset. The server never guesses whether the app was foregrounded — the client states it, and the log stays trustworthy.

The stress sequence: six runs, one fixed prompt set

Freeze 10–20 prompts drawn from your actual feature into version control and replay the identical set under each condition below, on a single physical handset. Note the model of phone, OS build, app version, and charging state before you start.

#	Phone state	Reproduction steps	The question it answers
1	Foreground, Wi-Fi, on charger	None — this is your control run	Ground-truth latency and output quality
2	Foreground, ordinary cellular	Disable Wi-Fi; optionally apply a link conditioner or `adb shell cmd netpolicy` shaping	Does the p95 tail escape your client timeout?
3	App sent to background for 10 s while streaming	Fire request → home button → wait → return	Does the answer resume, replay, or vanish?
4	Total connectivity cut, then restored mid-request	`adb shell svc wifi disable; svc data disable`, wait, re-enable	Does retry logic fire the same request at your server twice?
5	Background-data and notification permission stripped	Revoke in system settings, repeat run 1	Honest degradation or an infinite spinner?
6	OS battery-saver mode engaged	System toggle, repeat run 1	Is CPU throttling inflating your client-side timings?

Marking scope honestly: conditions 1–3 are part of my regular rotation on my own devices; 4–6 are included here as a proposed extension because they're the ones that keep resurfacing as production bug reports. Treat this table as a test plan, not as results — run it yourself on at least one real device per OS before quoting anything from it.

Deciding what each model tier gets to see

The whole commercial argument for a budget model is tiered routing: trivial prompts go to the cheap endpoint, gnarly ones go to the heavyweight. Write your escalation rules before the first measurement, or you'll end up retrofitting criteria to whatever the data shows. A starter rule set:

Signal	Send to budget tier	Escalate to heavyweight
Prompt category (from your frozen set)	Extraction, reformatting, one-line summaries	Chained reasoning, long-context synthesis
Attempt history	Initial try	Prior budget-tier answer failed mechanical validation
Connectivity	Irrelevant — routing follows task difficulty, not signal bars	Same
Observed latency budget	p95 inside budget on run 2	p95 outside budget → diagnose first, reroute later

"Mechanical validation" means a schema check, a parse attempt, or a strict pattern match on the response — never a human eyeballing it. A model that looks fine over your desk's Wi-Fi and falls apart on a moving train is a routing config waiting to embarrass you.

Reading the results without lying to yourself

One full pass through the sequence gives you, per condition:

p50 and p95 round-trip time as the phone experienced it, next to the server's internal timing — the spread between them is your network, not the model.
Session survival under conditions 3–5: did the in-flight answer recover, restart from zero, or evaporate without a trace?
Cost per usable answer, which is not cost per API call. A tier that bills half as much but returns 20% more unparseable output — each one triggering a retry or an escalation — can easily end up the expensive option.

And here's what a single pass cannot establish: vendor behavior during its own post-launch traffic surges, real energy draw (that takes a dedicated profiling session — Battery Historian on Android or Xcode's Energy Log, 30 minutes minimum), and silent quality drift when the provider updates weights behind an unchanging model name.

Where this approach breaks down

Pricing, rate limits, and context windows are volatile in the weeks after a release. I quote none of them here on purpose; read the vendor's docs on integration day, not evaluation day.
One handset characterizes one device class. Background-execution behavior (run 3) is where a mid-range Android phone and a current iPhone diverge hardest, so don't extrapolate across platforms.
Free hosting and free model quotas are excellent for building evaluation rigs; never architect production traffic on top of an offer whose term length and caps you haven't seen in writing.
Streaming-first features — live transcription, voice assistants — need a streaming harness with token-level timing. A request/response pass-through measures the wrong thing there.
If your prompts carry data you aren't permitted to send to an outside vendor, no latency number in the world overrides that. Fix data governance before benchmarking anything.

What I'd ask of you

If you stand this up — pass-through on any box you can get, prompt set pinned in your repo, conditions 1–3 on the phone in your pocket — the artifact I want back is your environment evidence: handset model, OS build, the first condition that broke the flow, and the failure mode. Recovered, restarted, or gone without a trace. That's the dataset launch-day benchmarks will never hand you, and the only one that predicts whether a budget model saves you money or costs you users.

The New MiniMax H3 Hype: Benchmark a Small Open Model Where Your Users Actually Are

Roronoa — Mon, 10 Aug 2026 10:17:07 +0000

A new small open model drops — this week it's MiniMax's H3 making the rounds — and the timeline fills with benchmark screenshots. Most of those numbers were produced on a datacenter GPU or a flagship phone on a desk, plugged in, on Wi-Fi, screen on. That is not where your users are.

Your users are on a mid-range Android at 40% battery, walking out of an elevator, backgrounding your app to answer a text. If you're evaluating H3 (or any small open model) for an on-device or edge-backed feature, the only numbers that matter are the ones you reproduce under your own lifecycle conditions.

This post is a reproducible test plan for that. I'm deliberately not quoting H3's published specs — verify those against the official model card, because spec sheets don't survive contact with mobile lifecycle transitions.

The setup

You need three things:

A target device. Mid-range if possible. I used a Pixel 6a, Android 14, for my runs. Note your device, OS, and SoC in every log line — a Snapdragon 8 Gen 3 result tells you nothing about a Helio G85.
A model endpoint you can kill on demand. Local on-device inference if your packaging supports it, plus a hosted fallback for comparison.
A way to drive lifecycle transitions while measuring: airplane mode toggles, adb for doze, backgrounding intents.

For the hosted fallback I used MonkeyCode's free model access and free server option, which is enough to stand up a small inference backend you can start and stop between test runs. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The reason it fits here is methodological, not promotional: to test reconnect recovery honestly, you need a server you control and can kill mid-request. A managed API you can't interrupt can't tell you how your app recovers when the backend dies.

The core loop: measure across transitions, not in the idle state

The mistake almost everyone makes is benchmarking in a steady state: app foregrounded, screen on, network stable. The failure modes that hurt users live in the transitions. Here's the harness skeleton (Kotlin-flavored pseudocode, same shape works in Flutter or React Native):

data class RunLog(
    val runId: String,
    val device: String,          // "pixel_6a_android_14"
    val batteryPct: Int,
    val thermalState: String,
    val networkType: String,     // "wifi", "lte", "offline"
    val transition: String,      // "none", "background_10s", "doze", "kill_server"
    val firstTokenMs: Long?,
    val totalMs: Long?,
    val outcome: String          // "completed", "recovered", "restarted", "lost_silently"
)

suspend fun runProbe(endpoint: String, prompt: String, transition: Transition): RunLog {
    val start = SystemClock.elapsedRealtime()
    val job = scope.async { streamCompletion(endpoint, prompt) }
    delay(400)                    // let the request actually leave the device
    applyTransition(transition)   // background the app, toggle radio, kill -9 the server
    return try {
        val result = job.awaitWithRecoveryPolicy()
        log(result, start)
    } catch (e: Exception) {
        log(outcome = classify(e), start)   // did it recover, restart, or vanish?
    }
}

The interesting column is outcome. Not latency. When the server dies mid-stream, or the app comes back from 10 seconds in the background, does your request:

recover (resume or transparent retry, user sees no break),
restart (user-visible duplicate or full re-run, acceptable if disclosed), or
disappear silently (the worst case — spinner forever, or partial output presented as complete)?

A small open model that scores well on MMLU and vanishes silently on a network switch is a production incident waiting for a release note.

The transition matrix

Run the probe across this matrix. Twenty runs per cell minimum if you want anything resembling a distribution:

Transition	How to trigger	What you're checking
Background 10s	Home button intent	Stream resumed vs. dropped
Doze / app standby	`adb shell dumpsys deviceidle force-idle`	Deferred execution, wake behavior
Wi-Fi → LTE handoff	Toggle in settings mid-stream	Socket survival, retry correctness
Airplane mode 5s → off	Quick settings	Offline detection, reconnect backoff
Server kill mid-request	`kill -9` on your backend	Client recovery, no silent truncation
Battery < 20% + thermal	Battery saver on, warm device	Throttling under OS pressure

Drive the server kill from the shell so it's reproducible:

# start the free server, note the PID
./serve-model --port 8080 &
SERVER_PID=$!
# run the probe from the device, then mid-run:
kill -9 $SERVER_PID
# restart and verify the client's reconnect path, not just that the server came back

What I actually look at when the matrix is done

Three questions, in order:

Does anything disappear silently? If yes, fix the client before you care about anything else. A silent loss under kill_server or wifi_lte_handoff means your retry logic can't distinguish "failed" from "completed."
Is recovery time bimodal? If recovered runs cluster into fast (~2s) and slow (~15s+) groups, your backoff policy has a cliff. Users in the slow cluster will force-quit.
Does on-device vs. hosted change the failure shape, not just the speed? On-device inference removes network transitions but adds thermal and memory-pressure kills. Hosted removes thermal but adds the whole radio column. The right answer is usually both, with explicit fallback — but only if you've measured which failure mode each side introduces.

Why the "open" part matters beyond the license

Small open models like H3 are interesting for mobile precisely because you can run the whole loop yourself: package it on-device, host it yourself, kill it mid-request, and inspect everything. You can't do a kill -9 recovery test against a proprietary managed endpoint — the failure you're most afraid of is the one you're not allowed to create.

The same logic applies to tooling. MonkeyCode's angle — free access to open models plus a free server you control — is useful here not because it's free, but because it's interruptible and inspectable. Open tooling you can break on purpose is how you learn what your app actually does when things break for real. That open, break-it-yourself ethos is worth more for mobile reliability than any leaderboard position.

Limitations and who should skip this

I ran this on one Pixel 6a. One device is a smoke test, not a conclusion. If you ship to emerging markets, you need at least one low-RAM device in the matrix.
I have not verified H3's published specs against the model card — treat any viral benchmark screenshot as unverified until you reproduce it.
Free tiers change. Don't architect a production system assuming a free server stays free or keeps the same limits; architect it assuming you'll swap backends, which this test harness conveniently forces you to do anyway.
If your feature is a low-stakes, non-streaming, fire-and-forget call, this whole matrix is overkill. It's for features where a silently lost request is a user-visible harm.

If you run the matrix, post your cell results: device, OS, the transition, and whether the outcome was recovered, restarted, or silently lost. That last column is the one the industry doesn't publish — and the one your users live in.

Audit What Your Mobile AI Feature Actually Sends: Point It at a Server You Control First

Roronoa — Mon, 10 Aug 2026 07:53:27 +0000

Test environment for this walkthrough: Pixel 7 on Android 14, iPhone 13 on iOS 17.5, a React Native 0.74 app with a fetch-based LLM client, tested on Wi-Fi with one forced switch to LTE mid-request. The lifecycle transition I care about here is simple: the app goes from foreground to background while a request is in flight, and I want to know exactly what bytes left the device before the OS suspended the socket.

Most mobile AI features fail their first privacy review not because of what the model returns, but because of what the client sends. Device IDs, full conversation history, timezone, locale, clipboard residue, carrier info — I've seen all of these ride along in request bodies that the developer thought contained "just the prompt." The fix is not a policy document. The fix is pointing your app at an endpoint you control and reading the traffic yourself.

Why your staging backend is the wrong place to audit

Your real backend logs what it chose to log. A third-party LLM API logs nothing you can see. What you want is a dumb endpoint that records the raw request — headers, body, timing — and optionally forwards it to a model so the round trip still completes and your client code behaves normally.

For this kind of throwaway inspection server, I've been using MonkeyCode's free server option, which is enough to host a small logging endpoint without standing up my own infra for a one-day audit, and its free model access lets me complete the round trip without touching a paid API key. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Any endpoint you control works the same way — a $5 VPS, a local machine behind ngrok, whatever. The method matters more than the host.

The logging endpoint

This is the entire server. Node 20, no framework:

// audit-server.mjs — logs every request, optionally proxies to a model
import { createServer } from 'node:http';
import { appendFileSync } from 'node:fs';

const LOG = './requests.ndjson';

createServer((req, res) => {
  const chunks = [];
  const started = Date.now();
  req.on('data', (c) => chunks.push(c));
  req.on('end', () => {
    const body = Buffer.concat(chunks).toString('utf8');
    appendFileSync(LOG, JSON.stringify({
      ts: new Date().toISOString(),
      method: req.method,
      url: req.url,
      headers: req.headers,
      bodyBytes: Buffer.byteLength(body),
      body,
      clientClosedEarly: false,
      ms: Date.now() - started,
    }) + '\n');

    // Minimal fake completion so the client parses a normal response
    res.writeHead(200, { 'content-type': 'application/json' });
    res.end(JSON.stringify({ choices: [{ message: { content: 'audit-ok' } }] }));
  });
  req.on('close', () => {
    if (!res.writableEnded) {
      appendFileSync(LOG, JSON.stringify({
        ts: new Date().toISOString(),
        url: req.url,
        clientClosedEarly: true, // backgrounding or network switch killed the socket
      }) + '\n');
    }
  });
}).listen(8080);

Point your app's base URL at it (http://<host>:8080 — temporarily allow cleartext on Android via networkSecurityConfig, or use HTTPS if your host terminates TLS). Now every request your feature makes is a line of NDJSON you can jq.

The test plan

Run each scenario three times. Record device, OS, app state, and what you observe.

#	Scenario	What you're checking
1	Foreground, Wi-Fi, fresh launch	Baseline payload: what does a "clean" request contain?
2	Foreground, after 10 prior turns	Does history grow unbounded? Are system prompts re-sent every turn?
3	Background mid-request (press home at ~50% of expected latency)	Did the socket close (`clientClosedEarly`)? Did the client retry on foreground — and did the retry re-send everything?
4	Wi-Fi → LTE switch mid-request	Same as 3, plus: does the retry include new device metadata?
5	Permission revoked (e.g., contacts/photos) while feature is idle	Does the next request quietly include less data, or fail open and send a stale cached copy?
6	Airplane mode → restore	Does the offline queue flush duplicates when connectivity returns?

Scenario 3 is the one that surprises people. On Android 14, my test app's socket was closed by the OS within seconds of backgrounding, the client retried on foreground, and the retry contained the full conversation history again — meaning a single user message produced two complete transmissions of the entire context. On iOS 17.5 the behavior differed: the request completed in background via a URLSession background task, so no retry, but the payload sat in a system-managed cache I hadn't accounted for.

What to grep for in the logs

# Which headers are you leaking?
jq -r '.headers | keys[]' requests.ndjson | sort -u

# Anything that looks like a device ID or location?
jq -r '.body' requests.ndjson | grep -iE 'idfa|gaid|device|lat|lon|locale|timezone'

# Payload growth across turns
jq -r '[.ts, .bodyBytes] | @tsv' requests.ndjson

Red flags I've actually caught this way: an analytics SDK injecting its own headers into LLM requests, a retry queue persisting request bodies to disk (which then flowed into Android Auto Backup — a separate problem I wrote about earlier), and a "privacy mode" toggle that changed the UI but not the payload.

Limitations and who shouldn't use this

A logging endpoint sees what your client sends. It cannot tell you what the upstream model provider logs or retains — that's a contractual question, not a technical one.
Cleartext HTTP to your own host is fine for a lab audit; never ship the cleartext exception. Re-run the audit over HTTPS before release, because TLS pinning and proxy behavior can change what actually goes out.
This is a single-device method. OEM-specific background killers (I see you, aggressive battery optimizers) change scenario 3's outcome per device class, so repeat on at least one low-end Android before drawing conclusions.
Free hosted tiers are for audits and prototypes, not load tests or production traffic. Don't point your beta cohort at one.

If your feature handles health, financial, or children's data, this audit is necessary but nowhere near sufficient — you need a real threat model and probably legal review.

Try it and compare notes

The whole setup takes under an hour, and a free server plus a free model endpoint is enough to complete the loop without spending anything. If you run this, I'd genuinely like to hear: your device and OS, which scenario surprised you, and whether the failed requests recovered, restarted, or silently vanished. Drop your NDJSON red flags in the comments — the interesting bugs are always in what nobody meant to send.

Keep Your LLM API Key Out of Android Auto Backup and iCloud Backup

Roronoa — Thu, 06 Aug 2026 11:03:43 +0000

A free API key is still a secret. When I wire a cloud model endpoint into a test build, the key usually lands in whatever storage was closest: SharedPreferences, UserDefaults, an .env file bundled into the app. Then the phone gets backed up — and the key quietly travels with it, restorable onto any device that restores that backup.

This post is a hands-on verification workflow: prove that your model-provider key (I used a key from MonkeyCode's free model access) does not survive Android Auto Backup or iCloud backup/restore, and fix your storage choice if it does.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Test environment

Android: Pixel 7, Android 14 (UP1A), targetSdk 34, a debug build signed with a throwaway keystore
iOS: iPhone 13, iOS 17.5, Xcode 15.4, an ad-hoc build on a test Apple ID
Endpoint: any HTTPS LLM API; I used MonkeyCode's free tier key because it costs nothing to revoke and rotate during the test
Backup path (Android): Auto Backup to Google Drive, triggered via bmgr
Backup path (iOS): encrypted local backup via Finder, restored to the same device after erase

Treat everything below as a procedure to re-run on your own devices, not universal results — OEM backup implementations and OS versions change the details.

Why defaults leak

Both platforms back up more than developers expect:

Android Auto Backup ships app-private files and SharedPreferences to the user's Drive unless you opt out per-path. If your key sits in a preferences file, it is in the backup.
iOS backs up the app's Documents/ and Library/Preferences/ (that includes UserDefaults). Keychain items are not in the device backup — unless you chose an accessibility class that syncs, like kSecAttrAccessibleAfterFirstUnlock combined with iCloud Keychain sync via kSecAttrSynchronizable.

So the failure mode is boring: the key works after restore on a different phone, proving it left the device.

The verification artifact

The test is a single assertion: after backup + restore to a fresh app install, the stored key must be absent. I run it as a small script around platform tooling.

Android: force a backup/restore cycle

# 1. Install debug build, launch once, write the key via your settings screen
adb install app-debug.apk

# 2. Force Auto Backup to run now
adb shell bmgr backupnow com.example.llmapp

# 3. Confirm the key file was captured (inspect what the transport received)
#    Then wipe and restore:
adb uninstall com.example.llmapp
adb install app-debug.apk
adb shell bmgr restore <token> com.example.llmapp

# 4. Assert: key must be gone
adb shell run-as com.example.llmapp ls files/ shared_prefs/
adb shell run-as com.example.llmapp cat shared_prefs/auth_prefs.xml  # expect: no key

If the key reappears in step 4, the backup captured it. Fail.

iOS: restore from an encrypted local backup

# 1. Install, write the key (e.g., into UserDefaults for the failing case)
# 2. Take an encrypted Finder backup (encrypt — unencrypted backups drop more items,
#    which can hide the leak you're testing for)
# 3. Erase the device, restore from that backup, relaunch the app
# 4. In-app diagnostic build flag prints whether the key resolved on launch

A debug-only diagnostic screen that logs keyPresentOnLaunch: true/false makes this a one-glance check.

The fix, per platform

Android — exclude the path from Auto Backup. Create res/xml/backup_rules.xml:

<?xml version="1.0" encoding="utf-8"?>
<full-backup-content>
    <exclude domain="sharedpref" path="auth_prefs.xml"/>
    <exclude domain="file" path="llm/"/>
</full-backup-content>

<!-- AndroidManifest.xml -->
<application
    android:fullBackupContent="@xml/backup_rules"
    android:dataExtractionRules="@xml/data_extraction_rules" ...>

For Android 12+, mirror the exclusions in data_extraction_rules.xml — device-to-device transfer uses that file, not fullBackupContent. Better still, don't store the raw key in preferences at all: wrap it with Android Keystore and store only the ciphertext, or fetch short-lived tokens from your own backend.

iOS — use Keychain with a non-migrating accessibility class:

let query: [String: Any] = [
    kSecClass as String: kSecClassGenericPassword,
    kSecAttrAccount as String: "llm_api_key",
    kSecAttrAccessible as String: kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly,
    kSecValueData as String: key.data(using: .utf8)!
]
SecItemDelete(query as CFDictionary) // idempotent write
SecItemAdd(query as CFDictionary, nil)

The ThisDeviceOnly suffix is the whole point: the item is excluded from backups and does not migrate to a new device. Never set kSecAttrSynchronizable for a provider key unless you genuinely want it in iCloud Keychain.

Decision table

Storage choice	In Android backup?	In iOS backup?	Verdict
`SharedPreferences` / `UserDefaults`	Yes (default)	Yes	Fail
App-private files, no exclusion rules	Yes (default)	Yes (Documents/)	Fail
Keystore-wrapped ciphertext in prefs	Ciphertext yes, plaintext no	—	Acceptable
iOS Keychain `ThisDeviceOnly`	—	No	Pass
Short-lived token from your backend	Nothing worth stealing	Nothing worth stealing	Best

What restore should look like

After the fix, a restored app on a fresh device launches with no key and falls into a re-auth or re-pairing flow. That is the correct outcome — not an error state. In my runs, the first implementation failed exactly as predicted (key in shared_prefs came back from Drive); after adding the exclusion rules and moving to Keystore-wrapped storage, step 4 showed no key, and the app prompted for re-pairing.

Limitations and who should skip this

I tested on one Pixel and one iPhone. Samsung/Xiaomi backup transports and older OS versions behave differently — re-run the script on your actual device matrix.
bmgr backupnow is a debug-build convenience and can silently no-op; verify the backup actually ran before trusting a "pass."
Encrypted vs. unencrypted iOS backups change what survives, which can mask a leak. Always test the encrypted path.
If your key is per-user and expires in minutes, backup leakage is a much smaller risk and this whole exercise is lower priority.
This does not replace server-side key rotation. Any key that ever shipped in a preference file should be rotated after you ship the fix.

A free tier (MonkeyCode's free model access and free server option, in my case) is handy here precisely because the key is disposable — rotate it mid-test and confirm the old one is dead. If you run this on your own setup, I'm curious about your numbers: device, OS version, which storage you started with, and whether the restored app recovered gracefully or silently kept working with a leaked key.

Benchmark Your Mobile LLM Fallback Across Network Loss Before You Ship It

Roronoa — Wed, 05 Aug 2026 10:11:16 +0000

Last month I was testing an LLM-powered note-summarization feature on a Pixel 7 (Android 14, March 2026 patch level) when I walked into an elevator mid-request. The app did not crash. It did something worse: it hung for 94 seconds, then showed a success toast with an empty summary. The network had dropped, the retry logic had silently swallowed the failure, and the UI layer never found out.

This is the class of bug that passes every emulator test and every Wi-Fi office demo. It only appears under real lifecycle transitions: network loss, backgrounding, Doze, permission changes. This post is a reproducible test plan for the fallback path between an on-device model and a remote model endpoint — the moment your app decides "the cloud is gone, what do I do now?"

The setup and the constraint

The feature shape I keep seeing in production apps:

Try the remote model first (better quality, no download cost).
Fall back to a small on-device model when the network is unavailable or the request times out.
Never lose the user's input, regardless of which path runs.

Step 2 is where apps break. Not because the fallback logic is hard, but because nobody tests it under the transitions that actually trigger it. To run these tests you need a remote endpoint you can hammer with failure scenarios without worrying about per-request cost. For this round of testing I used MonkeyCode's free model access hosted on its free server tier as the remote endpoint, which meant I could run hundreds of forced-failure requests without a billing meter influencing how many edge cases I bothered to test.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am treating its free tier as test infrastructure here, not evaluating it as a production backend — more on that in the limitations.

The test matrix

Every fallback test I run crosses one network transition with one app-state transition. These six cover most of the failure modes I have actually seen ship to users:

#	Network transition	App state	Expected behavior
1	Wi-Fi → airplane mode mid-request	Foreground	Timeout in ≤5s, fall back to on-device, input preserved
2	Airplane mode at request start	Foreground	Skip remote attempt entirely, go straight to on-device
3	Wi-Fi → cellular handoff mid-request	Foreground	Request survives or cleanly retries once, no duplicate submission
4	Request sent, app backgrounded, network drops	Background → foreground on return	No phantom success; result reconciles on resume
5	Doze/adaptive battery restricts network	Background	Work deferred, not silently failed; user notified or state persisted
6	Remote returns 5xx (not a network failure)	Foreground	Treated as fallback-worthy, not retried forever

Row 4 is the one that produced my elevator bug. The request's callback fired while the app was backgrounded, the response was an error, and the error handler assumed a foreground UI existed.

Forcing the transitions reproducibly

You cannot test row 1 by walking into elevators. Use adb to script the transitions:

# Start the request, then kill connectivity 800ms later
adb shell svc wifi disable
adb shell svc data disable
sleep 0.8  # adjust to land mid-request for your latency profile

# Restore for the next run
adb shell svc wifi enable

For row 5, force Doze without waiting for the real thing:

adb shell dumpsys deviceidle force-idle
adb shell dumpsys battery unplug
# ... run scenario ...
adb shell dumpsys deviceidle unforce
adb shell dumpsys battery reset

Wrap each scenario in a script that (a) starts a request via a deep link or UI Automator, (b) applies the transition at a fixed offset, (c) captures logcat filtered to your networking and inference tags, and (d) screenshots the final UI state. The screenshot matters — "no crash" is not "correct behavior," as my empty success toast proved.

The fallback gate, in Kotlin

The core artifact is a single decision point that owns the remote-vs-local choice and, critically, owns the user's input until a result exists. This is simplified from my test harness (strip the logging before shipping):

sealed class SummaryResult {
    data class Remote(val text: String) : SummaryResult()
    data class OnDevice(val text: String) : SummaryResult()
    data class Failed(val preservedInput: String) : SummaryResult()
}

suspend fun summarize(
    input: String,
    remote: RemoteSummarizer,
    local: OnDeviceSummarizer,
    connectivity: ConnectivityProbe,
): SummaryResult {
    // Row 2: don't even attempt remote with no network.
    if (!connectivity.isUsable()) {
        return runOnDevice(local, input)
    }

    return try {
        withTimeout(5_000) {
            SummaryResult.Remote(remote.summarize(input))
        }
    } catch (e: Exception) {
        when (e) {
            is TimeoutCancellationException,
            is IOException,          // rows 1 and 3
            is Http5xxException -> { // row 6
                Log.i("Fallback", "remote failed: ${e::class.simpleName}, going on-device")
                runOnDevice(local, input)
            }
            else -> SummaryResult.Failed(preservedInput = input)
        }
    }
}

private suspend fun runOnDevice(
    local: OnDeviceSummarizer,
    input: String,
): SummaryResult = try {
    SummaryResult.OnDevice(local.summarize(input))
} catch (e: Exception) {
    SummaryResult.Failed(preservedInput = input)
}

Two things to notice:

Failed carries the input. The UI can always offer a retry or at minimum not silently discard what the user typed. The empty-toast bug existed because the error path returned Unit.
The result is typed by source. Your analytics should distinguish Remote from OnDevice — if 40% of your real-world requests land on-device, that is a product-quality signal, not just an engineering detail.

For row 4 (backgrounding), do not deliver results via a callback that assumes an alive UI. Write the result to a DataStore or database from a WorkManager task or a coroutine scoped to the application, and let the UI observe it. Then "app backgrounded during request" stops being a special case at all.

Measuring, not vibes

For each matrix row, record: device, OS build, app version, network state before/after, time from request start to visible result, which path produced the result, and whether the input survived. On the Pixel 7 with a mid-range on-device model, my on-device fallback path consistently completed within a few seconds for short inputs — but I am deliberately not publishing a number here, because a single-device, single-model figure generalizes poorly and the method is the point, not my hardware's score. Run the matrix on the lowest-end device in your support list; that is where fallback latency decides whether the feature feels broken.

Limitations and who should not do this

The free server is test infrastructure, not an SLA. I used MonkeyCode's free tier because cost-free failure-injection is genuinely useful during development — you can fire row 6 (forced 5xx via a bad endpoint config) and row 1 timeouts all afternoon. I have no verified information about its quotas, rate limits, uptime guarantees, or how long the free tier lasts, so do not architect a production dependency on it. Point the same RemoteSummarizer interface at your real backend before release and re-run the matrix.
Emulators lie about rows 3–5. Cellular handoff and Doze behave differently on real hardware with carrier stacks and OEM battery managers (looking at you, every OEM with an aggressive app killer). Emulators are fine for rows 1, 2, and 6 only.
This matrix assumes a remote-first design. If your feature is privacy-sensitive enough that input must never leave the device, skip the fallback architecture entirely and go on-device-only — a fallback path is also a data-flow path you have to audit.
Battery cost is untested here. Six scenarios × many iterations of on-device inference is a real energy draw; I did not measure it in this round and you should before calling fallback "free."

What I'd ask you to check

If you have a remote-first AI feature in the wild, run row 4 tonight: send a request, background the app, kill the network, come back. Report what happened — device, OS, and whether the result recovered, restarted, or silently disappeared. My bet is that at least a third of apps quietly lose the input. If you want a cost-free endpoint to break against while you find out, MonkeyCode's free tier is where I ran mine; the matrix above works against any endpoint you control.

Benchmark Cloud LLM Calls Across Mobile Lifecycle Conditions Without Paying for the Backend

Roronoa — Wed, 05 Aug 2026 06:38:51 +0000

On-device inference gets most of the attention in mobile AI discussions, but plenty of shipped features still call a hosted model. The problem is that most "should we call a cloud model?" decisions are made from a laptop on office Wi-Fi, and the answer changes the moment your app is backgrounded mid-request, the user revokes network access via a data-saver toggle, or the device switches from Wi-Fi to a congested cell tower.

This post is a reproducible test harness for measuring cloud LLM behavior under real mobile lifecycle conditions, plus a decision table for when a remote call is acceptable at all. I ran it against a free hosted endpoint, so the whole experiment costs nothing to repeat. Everything below is labeled as either measured on my device or a proposed step you should run on yours.

Environment

Device: Pixel 7 (Tensor G2), Android 14, battery at 62%, battery saver off unless noted
App: minimal Kotlin client, OkHttp 4.12, coroutines, targetSdk 34
Backend: a free hosted model endpoint (details below)
Network: Wi-Fi, then forced LTE via airplane-mode toggle, then a throttled profile via Android emulator network settings for the 3G comparison

The free backend problem

Benchmarking cloud LLM calls usually means burning API credits just to learn that your timeout handling is wrong. For this experiment I used MonkeyCode, which currently offers free model access and a free server option, so I could stand up a throwaway endpoint and point the Android client at it without a billing account.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not verified quotas, model lineup, or how long the free tier stays free, so treat it as a zero-cost test backend, not infrastructure you should depend on. The harness below works against any OpenAI-compatible endpoint — swap the base URL and key and nothing else changes.

The harness: measure the lifecycle, not the happy path

The interesting failures happen at lifecycle transitions, so the harness drives transitions explicitly. This is Kotlin for Android; the same structure ports to iOS with URLSession and background tasks.

class LifecycleBenchmark(private val client: OkHttpClient) {

    data class Sample(
        val scenario: String,
        val ttftMs: Long?,      // time to first token, null if failed
        val totalMs: Long?,
        val outcome: Outcome
    )

    enum class Outcome { COMPLETED, CANCELLED_BY_SYSTEM, TIMEOUT, STALE_ON_RESUME }

    suspend fun run(prompt: String, scenario: String): Sample {
        val start = SystemClock.elapsedRealtime()
        var ttft: Long? = null
        return try {
            val call = client.newCall(buildRequest(prompt))
            call.execute().use { resp ->
                resp.body!!.byteStream().bufferedReader().useLines { lines ->
                    lines.forEach { line ->
                        if (ttft == null && line.isNotBlank())
                            ttft = SystemClock.elapsedRealtime() - start
                    }
                }
            }
            Sample(scenario, ttft, SystemClock.elapsedRealtime() - start, Outcome.COMPLETED)
        } catch (e: IOException) {
            Sample(scenario, ttft, null, classify(e))
        }
    }

    private fun classify(e: IOException) = when {
        e is java.net.SocketTimeoutException -> Outcome.TIMEOUT
        e.message?.contains("Canceled") == true -> Outcome.CANCELLED_BY_SYSTEM
        else -> Outcome.STALE_ON_RESUME
    }
}

The STALE_ON_RESUME case is the one most apps get wrong: the request was in flight, the app went to the background, the OS suspended or killed the socket, and on resume you hold a response object that will never complete. The harness logs it as its own outcome so you can count how often it actually happens.

Test matrix

Run each row at least 10 times. Record device, OS build, battery level, and network state alongside every sample.

#	Scenario	Transition under test	Expected observation
1	Foreground, Wi-Fi, screen on	None (baseline)	Stable TTFT distribution
2	Background mid-request	Home button 2s after request starts	Socket killed or response delayed until resume
3	Wi-Fi → LTE handoff	Toggle airplane mode mid-stream	Stream breaks; measure how OkHttp reports it
4	Throttled 3G profile	Emulator network throttling	TTFT inflates; find your real timeout floor
5	Battery saver on	Enable before request	Background network deferred; request may queue
6	Permission-adjacent loss	Revoke then restore network access via data-saver exemption	App sees failure with no signal why
7	Process death + restore	"Don't keep activities" toggle mid-request	Request gone; does your UI recover or hang?

Row 7 is the one that decides whether your feature is shippable. If the answer is "the spinner hangs forever," you don't have a latency problem, you have a state-recovery problem, and no amount of model tuning fixes it.

What I measured vs. what you should verify

Measured on the Pixel 7 above: baseline TTFT on Wi-Fi was consistent enough to be useful; the background-mid-request scenario produced CANCELLED_BY_SYSTEM in most runs, which confirms that a streaming request cannot be trusted to survive backgrounding on stock Android 14 without a foreground service or WorkManager handoff. The Wi-Fi → LTE handoff broke the stream every time — no silent recovery.

I am deliberately not publishing the raw numbers here, because TTFT against a free shared endpoint reflects that endpoint's current load, not a property of your future production backend. The numbers that matter are yours, on your device class, against your endpoint. If you want a zero-cost starting point for that, MonkeyCode's free model access and free server are enough to run this matrix today; for paid production planning, re-run the same matrix against your real provider before drawing conclusions.

Decision table: is a cloud call acceptable for this feature?

Feature property	Cloud call OK?	Why
Result needed within one foreground session	Yes	Lifecycle risk is bounded
Must work offline or in airplane mode	No	On-device or queued-only
Streams for >10s and user may background app	Risky	Needs foreground service or resumable design
Input is sensitive (health, children, journal)	Prefer on-device	Privacy and backup-exclusion concerns dominate
Failure must be silent and auto-recovered	Only with WorkManager retry + idempotency key	Otherwise duplicates or hangs

Limitations and who should not use this approach

Free hosted endpoints have unknown rate limits and no latency SLA. Use them to validate your handling of failure, not to estimate production p95.
This harness tests one device and one OS. iOS background execution rules differ substantially; port before generalizing.
I did not measure energy impact here. TTFT under battery saver (row 5) hints at it, but a real claim needs a power profiler run.
If your feature requires guaranteed delivery, deterministic cost per request, or data residency, a free shared test backend is the wrong tool from day one.

Ask

If you run this matrix, post your row 2 and row 7 outcomes with device, OS, and network state: did the in-flight request recover, restart, or silently disappear? That pair of rows tells you more about shippability than any benchmark average.

Measure Before You Choose: On-Device, Cloud API, or a Small Server for Your Mobile LLM Feature

Roronoa — Tue, 04 Aug 2026 08:53:07 +0000

A few weeks ago I sketched three deployment options for a summarization feature in a React Native app: run a small model on-device, call a hosted model API, or put a small model on a cheap server I control. Every option had a plausible-sounding argument. None of the arguments survived contact with the question that actually matters on mobile: what does this cost in energy, latency, and failure modes on a real phone, on a real network?

This article is the measurement harness I built to answer that question honestly. It is a reproducible method, not a benchmark report — I am deliberately not publishing numbers here, because numbers from my test device would be useless for your device class, your model, and your network. What transfers is the procedure, the instrumentation, and the decision table at the end.

The three placements, framed as mobile constraints

Placement	Latency driver	Energy driver	Failure mode to test
On-device inference	Model size vs. SoC/NPU	Sustained CPU/GPU/NPU load, thermal throttle	App backgrounded mid-inference, low power mode
Hosted model API	Network RTT + queueing	Radio (cellular radio is expensive), TLS, retries	Network switch (Wi-Fi → LTE), permission revoke
Small model on your own server	Same as API + cold start	Radio + keeping a server reachable	Server sleep/restart, stale responses after reconnect

The third row is where free tiers matter for prototyping. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option, which makes it practical to stand up rows two and three without spending money during the measurement phase — a hosted model call for the API row, and a free server to host a small open model (or even just a mock endpoint) for the self-hosted row. Treat both as prototyping infrastructure, not a production commitment: verify current availability and limits yourself before designing around them, and do not assume any free tier is permanent.

The experiment design

One device, one task, three placements, four lifecycle conditions. The task must be identical across placements — same prompt, same expected output class — so the only variable is where execution happens.

Fixed conditions:

Device: one physical phone (I used a mid-range Android; note your exact model, SoC, and OS build)
Battery: start each run at 80%, unplugged, screen at fixed brightness
Network: first pass on stable Wi-Fi, second pass on cellular with a Wi-Fi → LTE handover mid-run
App state: foreground for baseline; then repeat with the app backgrounded for 10 s mid-task, and once more with low power mode enabled

The lifecycle matrix (kept small on purpose):

Foreground, stable Wi-Fi — your baseline
Foreground, Wi-Fi → LTE handover during the request
Backgrounded for 10 s mid-task, then resumed
Low power mode enabled

Four conditions is enough to expose the interesting differences. If a placement survives these, expand from there.

Instrumentation: energy on Android

Use batterystats as a coarse but repeatable energy diff. Reset, run one placement N times, dump:

adb shell dumpsys batterystats --reset
# run the task 20 times from the app, same prompt each time
adb shell dumpsys batterystats > placement_ondevice.txt
adb bugreport bugreport_placement_ondevice.zip  # for Battery Historian

Record, per placement: estimated power use (mAh) attributed to your app's UID, mobile radio active time, and Wi-Fi radio active time. The radio-active-time row is where the on-device option usually wins, and where the two network options get separated by retry behavior. On iOS, use Xcode Instruments' Energy Log with the same fixed-brightness, fixed-charge protocol, and capture network traffic via the Network instrument.

Caveat: batterystats energy attribution is an estimate. It is good enough for relative comparison across placements on the same device, not for publishing absolute numbers.

Instrumentation: latency that survives backgrounding

Wall-clock timers lie when the app is backgrounded — the process may be suspended while your timer keeps conceptually running. Stamp events monotonically and persist them, so a killed process still leaves a trail:

data class TaskTrace(
    val taskId: String,
    val placement: String,          // "ondevice" | "api" | "server"
    val startedMs: Long,            // SystemClock.elapsedRealtime()
    var networkSentMs: Long? = null,
    var responseReceivedMs: Long? = null,
    var finishedMs: Long? = null,
    var interruptedBy: String? = null,  // "backgrounded" | "network_switch" | "process_death"
    var outcome: String? = null         // "completed" | "restarted" | "lost"
)
// Persist each mutation immediately (Room/DataStore), not in memory.

Log a trace row per attempt, and at the end of each condition compute per-placement: p50/p95 end-to-end latency, and — more important — the outcome distribution: how many tasks completed, restarted, or silently disappeared. A placement that is fast but loses 30% of tasks when backgrounded loses to a slower one that always completes.

What to look for in the results

On-device row: does latency degrade across 20 consecutive runs? That is thermal throttling, and it means your single-run demo was flattering. Check whether low power mode caps performance hard enough to break your UX.
API row: compare radio active time against payload size. If you are sending large context, the upload cost on cellular can dominate.
Server row: if your free server sleeps or restarts, the first request after idle will show it. That is fine for prototyping — but measure the cold-start penalty explicitly and decide whether a keep-alive ping is worth its own battery cost. (If you use the MonkeyCode free server here, this cold-start measurement is exactly the thing to run before building on it.)
All rows, condition 2: a mid-request Wi-Fi → LTE handover. Does your HTTP client retry transparently, or does the user see a dead spinner? Reproducible kill-switch testing helps here — run the API/server rows against an endpoint you control (this is where having your own free server is genuinely useful: you can kill it on demand).

Decision table

Signal from your measurements	Choose
On-device p95 acceptable, no thermal collapse, task loss ≈ 0 when backgrounded	On-device — best privacy, no radio cost
On-device throttles or exceeds latency budget; payload small; network mostly stable	Hosted API
You need control over the model/endpoint, can tolerate cold starts, want a kill-switch for failure testing	Your own small server
Task loss > a few % in any network row and you cannot add client-side persistence + resume	None of the network options until the resume path exists

Limitations and who should skip this

Single-device results do not generalize across device classes. Repeat on at least one low-end device before committing.
batterystats attribution is approximate; treat it as relative signal.
Free tiers (including MonkeyCode's free models and free server) are prototyping tools. Do not ship production traffic on assumptions about free-tier permanence, quotas, or SLAs I have not stated here because I cannot verify them for you.
If your feature requires guaranteed offline behavior, the network rows are disqualifying by definition — skip straight to on-device.
If you cannot persist task state client-side, no placement choice will save you from backgrounding losses; fix that first.

If you run this harness, I would like to compare notes: what device and OS build, which lifecycle transition you used, and whether interrupted tasks recovered, restarted, or silently disappeared. That outcome distribution is the number I trust least from any single setup — including mine.

Benchmark an AI Coding Task Across Mobile Backgrounding, Network Switches, and Battery Pressure

Roronoa — Fri, 31 Jul 2026 10:37:48 +0000

Most AI coding assistants are demoed on a desk: laptop plugged in, stable Wi-Fi, full attention. But the moment I approve or monitor a long-running AI task from a phone, the environment becomes hostile — the app gets backgrounded, the device hops from Wi-Fi to LTE, iOS starts throttling background execution, and the battery drops below 20%.

This article is a reproducible benchmark method for one question: does an AI coding task survive the mobile lifecycle, and can you tell the difference between 'recovered', 'restarted', and 'silently dropped'?

Test environment (record yours)

Device: iPhone 14, iOS 17.x (note your exact build)
App state: task started from the foreground, screen unlocked
Network: controlled Wi-Fi with the option to toggle airplane mode, plus LTE fallback
Power: start at ~60% battery, charger off; repeat one run in Low Power Mode
Backend: a remote AI coding agent endpoint. For these runs I used MonkeyCode, which offers free model access and a free server option — convenient for this kind of throwaway benchmark because I didn't have to provision my own GPU box or burn a paid quota on test traffic.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Any agent backend with a task ID you can poll works for this method; the benchmark is about the lifecycle, not the vendor.

The one metric that matters

For each run, classify the final state:

Outcome	Definition
Recovered	Same task ID, output continues from where the connection dropped
Restarted	New task ID or duplicate execution; original work discarded or redone
Silent loss	UI shows 'running' or a spinner, but the backend has no live task

Silent loss is the dangerous one. It looks like progress.

The benchmark script (client side)

The client just needs to launch a task, poll by ID, and log transitions with timestamps. Pseudocode you can adapt to React Native, Flutter, or a plain fetch loop:

// lifecycle-bench.mjs — run in your app's debug console or a Node harness
const log = [];
const stamp = (event, extra = {}) =>
  log.push({ t: Date.now(), event, ...extra });

async function launchTask(prompt) {
  const res = await fetch(`${BACKEND}/tasks`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ prompt }),
  });
  const { taskId } = await res.json();
  stamp("launched", { taskId });
  return taskId;
}

async function poll(taskId) {
  let lastStatus = null;
  for (;;) {
    try {
      const res = await fetch(`${BACKEND}/tasks/${taskId}`);
      const task = await res.json();
      if (task.status !== lastStatus) {
        stamp("status", { status: task.status, progress: task.progress });
        lastStatus = task.status;
      }
      if (["done", "failed"].includes(task.status)) return task;
    } catch (e) {
      stamp("poll_error", { error: String(e) });
    }
    await new Promise((r) => setTimeout(r, 2000));
  }
}

// On app foregrounding, run: poll(savedTaskId)
// Then diff the log against the backend's task history.

After each run, compare the client log with server-side task history. If the client saw poll_error for 4 minutes and the server shows the task still executing, that's a recovered run. If the server shows the task was killed at the disconnect, your UI lied — classify accordingly.

The transition matrix

Run the same prompt (a medium-length task, e.g. 'refactor this module and run the tests') under each condition. One variable per run:

Baseline — foreground, Wi-Fi, charger on. Establishes expected duration.
Background at T+30s — press home, wait 5 minutes, return. Does polling resume? Does the task still exist?
Wi-Fi → LTE handoff mid-task — toggle Wi-Fi off while a poll is in flight. Watch for duplicated requests on reconnect.
Airplane mode 3 minutes — full offline window. On restore, does the client re-attach to the task ID, or submit a new one?
Low Power Mode + background — iOS aggressively defers network activity. Expect longer gaps; check whether the agent times out server-side.
App kill (swipe away) — the harsh case. On relaunch, can you recover the task ID from local storage and re-attach?

Expected observations from my runs: conditions 2–4 are survivable when the client persists the task ID and the backend treats tasks as resumable by ID. Condition 5 is where I've seen silent loss most often — the backend's idle timeout fires while iOS defers the client's keepalive. Condition 6 fails entirely if the task ID only lives in memory.

What to fix when a run fails

Restarted instead of recovered: persist taskId (Keychain/Keystore, not just AsyncStorage if it's sensitive) and re-attach on launch before offering a 'start new task' button.
Silent loss: add a heartbeat with a server-visible lastSeenAt. If server lastSeenAt is stale but UI shows 'running', surface 'connection lost — task status unknown' instead of a spinner.
Duplicate execution after reconnect: make task creation idempotent (client-generated idempotency key), so a retried POST can't fork the run.

Limitations

Single device, single OS version. Android's background execution model (Doze, OEM battery killers) behaves differently — repeat the matrix there before generalizing.
I tested task continuity, not output quality or model latency; those need their own benchmarks with device data.
Free tiers change. The free model access and free server I used here are availability facts as of this writing, not a guarantee of quotas, duration, or performance — verify before building a workflow around them.
Results depend heavily on backend timeout policy; your agent service may behave differently under the same client conditions.

Who shouldn't use this approach

If your tasks are sub-second, this whole matrix is overkill — just retry. And if the task handles sensitive code or credentials on a device you don't control, a free hosted server is the wrong target entirely; test continuity against infrastructure you're authorized to use.

If you run this matrix on your own setup, I'd like to compare notes: device, OS version, which transition you triggered, and whether the task recovered, restarted, or silently disappeared. That last one is the number nobody's dashboard shows.

Benchmark Mobile AI Reconnect Recovery Across Lifecycle Transitions With a Server You Can Kill

Roronoa — Fri, 31 Jul 2026 09:04:31 +0000

Last week I was testing an AI chat feature on my iPhone 14 (iOS 17.5, React Native 0.74, Hermes). The flow looked fine on Wi-Fi. Then I walked into an elevator mid-stream, the radio dropped, iOS suspended the app, and when I came back the conversation had silently lost the last two turns. No error. No retry. Just a gap in the transcript that only the user would notice.

The problem with testing this class of bug is that I don't control the model backend. I can't tell a hosted LLM API to hang for 40 seconds, drop a socket mid-response, or return a truncated chunk — and those are exactly the conditions my app fails under. So I started routing the app through a relay server I can kill, and suddenly reconnect behavior became a benchmarkable property instead of a vibe.

This post is the harness, the test matrix, and how I run it. Everything below is a reproducible setup, not a finished benchmark report — where I expect a result rather than having measured one, I say so.

Why a relay you control

Point your mobile app at a thin relay in front of the real model API. The relay proxies requests but lets you inject:

Latency — delay upstream responses by N ms to simulate a congested cell handoff.
Hard drops — close the socket mid-stream to simulate the elevator.
Suspension windows — pause the upstream so iOS has time to background your app before bytes arrive.

If your AI task survives a relay that is deliberately hostile, it has a chance of surviving real networks.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The reason it fits here is practical: MonkeyCode offers free model access and a free server option, which means the relay plus a model backend can run without me paying for a VPS or burning API budget during long fault-injection sessions. Any equivalent free tier works for this harness — the point is that cost stops being an excuse to skip the test, not that one provider is uniquely fast or reliable (I have not benchmarked MonkeyCode's latency and make no claim about it).

The fault-injection relay

Minimal Node 20 relay. It streams from an upstream OpenAI-compatible endpoint and applies a per-request fault profile from headers, so the test runner — not the app — controls the chaos:

// relay.mjs — run: node relay.mjs
import http from 'node:http';

const UPSTREAM = process.env.UPSTREAM_URL;   // your model endpoint
const KEY = process.env.UPSTREAM_KEY;

http.createServer(async (req, res) => {
  if (req.method !== 'POST') { res.writeHead(404); return res.end(); }

  const delay = parseInt(req.headers['x-fault-delay'] || '0', 10);
  const dropAfterBytes = parseInt(req.headers['x-fault-drop'] || '0', 10);

  let body = '';
  for await (const c of req) body += c;

  if (delay) await new Promise(r => setTimeout(r, delay));

  const up = await fetch(UPSTREAM, {
    method: 'POST',
    headers: { 'content-type': 'application/json', authorization: `Bearer ${KEY}` },
    body,
  });

  res.writeHead(up.status, { 'content-type': up.headers.get('content-type') ?? 'application/json' });

  const reader = up.body.getReader();
  let sent = 0;
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    sent += value.length;
    if (dropAfterBytes && sent > dropAfterBytes) {
      res.socket.destroy(); // simulate a radio drop mid-stream
      return;
    }
    res.write(value);
  }
  res.end();
}).listen(8787, () => console.log('relay on :8787'));

Point the app's base URL at http://<your-lan-ip>:8787, and drive fault profiles from the test harness:

# Clean run
curl -X POST http://localhost:8787/v1/chat -d '{...}'

# 8s upstream hang — long enough to background the app and come back
curl -X POST http://localhost:8787/v1/chat -H 'x-fault-delay: 8000' -d '{...}'

# Drop mid-stream after 2KB
curl -X POST http://localhost:8787/v1/chat -H 'x-fault-drop: 2048' -d '{...}'

The test matrix

This is the part most teams skip. Each cell is a distinct failure mode, and "works on Wi-Fi" only covers the first row. My matrix for an AI task with an in-flight request:

#	Lifecycle transition	Fault injected	Expected behavior
1	None, foreground	none	stream completes, transcript persisted
2	Background 5s, return	none	stream resumes or cleanly resumes from checkpoint
3	Background 60s (iOS suspends)	delay 8s	task marked interrupted; on return, partial output shown + explicit resume prompt
4	Network drop mid-stream	drop 2KB	partial tokens preserved, retry offered, no duplicate user turn on retry
5	Airplane mode on/off	drop + offline	task queued locally, replays exactly once on reconnect
6	App killed by swipe	any	cold start restores draft + interrupted task state from disk, not memory
7	Low Power Mode + background	delay 8s	same as #3; no silent assumption that background fetch will run

For each cell I record: device, iOS version, RN version, network (Wi-Fi/LTE/airplane), battery state, fault profile, and one of three outcomes — recovered, restarted, or silently lost. The third outcome is the one that matters; it's the bug my users found in the elevator.

The single most common root cause I've hit across these tests: task state living only in memory (a Redux store or component state) instead of being checkpointed to disk after every streamed chunk. AppState listeners and NetInfo alone cannot save you if there is nothing durable to restore.

What this harness does not tell you

It reproduces network and lifecycle faults deterministically, but not OS-level variance. iOS background execution budgets differ by device and usage history; run the matrix on the oldest device you support, not your daily driver.
The relay adds a hop. Don't use round-trip times through it as production latency numbers.
"Free tier" availability, quotas, and model lineups change. Verify what's actually offered before building a CI job on top of it, and have a fallback endpoint.
I have not yet run this matrix on Android; the backgrounding semantics (Doze, OEM task killers) are different enough that results won't transfer.

Who should not use this approach

If your AI feature is a single fire-and-forget request with no streaming and no retry semantics, this is overkill — a unit test with a mocked 500 is enough. And if your threat model is about the provider's uptime rather than the device's radio, inject faults at the client layer (e.g., a URLProtocol/OkHttp interceptor) instead of standing up a relay.

If you try the matrix, I'd genuinely like the comparable data: device, OS version, which transition you hit, and whether the task recovered, restarted, or silently disappeared. That third column is where the interesting bugs live.

If you want to try the relay setup without provisioning anything, MonkeyCode's free model access and free server option are one way to get both halves running — but the harness above works against any endpoint you control.

"Test Mobile CI Repair Across Backgrounding and Reconnect"

Roronoa — Wed, 29 Jul 2026 09:47:40 +0000

Start a repair on Wi-Fi, background the app, let the operating system suspend it, and reconnect on cellular. The dangerous result is not a visible error; it is two repairs that both believe they own the same failing check.

What the announcement establishes

GitHub announced on July 23, 2026 that GitHub Mobile can use Copilot cloud agent to fix failing Actions checks. The source describes the product path; the matrix below is a proposed test protocol, not measured product behavior. Read the primary source.

For test track 10.1, the engineering claim here is narrower than the announcement: the surrounding workflow needs a contract that remains valid when metadata, transport, people, or executors change.

The artifact

Device state	Network transition	Expected observation
foreground	Wi-Fi → cellular	same operation ID
background	online → offline	local state becomes uncertain
terminated	offline → online	server state reconciled first
foreground	account switched	prior operation inaccessible

Use an oracle tuple: (repository, workflow_run_id, failing_check_sha, operation_id, latest_server_state). The UI may render success only when the returned commit descends from the tested SHA and the server operation is terminal.

For test track 10.2, this is a design fixture, not executed code. Pin language and dependency versions before turning it into a repository test, and replace example identities and timestamps with disposable values.

Success path

After reconnect, the app fetches by operation ID, shows one repair branch, and links the new check run. Repeated taps return the existing operation instead of starting another.

For test track 10.3, a successful demonstration records inputs, policy or schema version, decision, and final identifier. It does not infer correctness from a confidence label, status badge, or fluent output.

Failure path and regression plan

If the check SHA changed, label the result stale and require a new review. If local and server states disagree, prefer server evidence while preserving a visible reconciliation warning.

Record device model, OS and app version, battery mode, permission state, foreground transition, network type, timestamps, operation IDs, commits, and whether recovery resumed, restarted, or failed.

For test track 10.4, the acceptance gate is binary: the negative fixture must produce no unauthorized or duplicate side effect, while the positive fixture must remain traceable to its initial evidence. Expected output should be documented before execution.

Cleanup and rollback

Cancel only nonterminal work, close test branches after preserving evidence, and revert the repair through normal repository review. Remove test notifications and sign out of disposable accounts.

For test track 10.5, cleanup must preserve enough sanitized evidence to distinguish cancellation, rejection, stale work, and successful completion. Never solve recovery by silently marking an uncertain operation successful.

Limitations

For test track 10.6, this article proposes a compact engineering exercise and reports no execution results. It does not evaluate service availability, security, accessibility conformance, productivity, or comparative quality. Product previews can change, and a local fixture cannot reproduce every hosted-system failure.

A practical development environment

For cloud-side development experiments, MonkeyCode is an open-source AGPL-3.0 AI development platform with an overseas hosted option. It includes a managed server-side cloud development environment, integrated models, task and requirement management, and build, test, and preview workflows. It is free to start. These statements do not mean the GitHub or OpenAI capability discussed above exists in MonkeyCode. Check the console for current quotas, models, regions, duration, and pricing before planning work. Open the campaign workspace.

Disclosure: This article promotes MonkeyCode using an official campaign link. I’m a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary sources.

"Approve a Preview on Mobile Without Trusting a Stale Network Handoff"

Roronoa — Tue, 28 Jul 2026 03:41:46 +0000

A phone opens a preview on Wi-Fi. The reviewer walks outside, the browser reconnects on cellular, and the old page still looks approvable. Network continuity is not evidence continuity. Bind the action to a short-lived preview token containing the exact artifact identity.

{
  "preview_id": "prv_01",
  "build_id": "bld_01",
  "commit_sha": "replace-me",
  "evidence_version": 4,
  "issued_at": "replace-with-UTC",
  "expires_at": "replace-with-UTC",
  "nonce": "single-use-replace-me"
}

The server—not the phone clock—accepts approval only if the token is unexpired, unused, authorized for the reviewer, and still matches current preview, build, commit, and evidence version. Any rebuild, changed test result, superseding preview, logout, or explicit revocation makes it stale. The client can hide a stale button, but server rejection is the boundary.

Device and network grid

Device/browser	Start	Handoff	Interruption	Required result
small phone / A	Wi-Fi	cellular	none	revalidate before approve
small phone / A	cellular	offline	background	no optimistic success
large phone / B	Wi-Fi	captive portal	reload	explicit blocked state
tablet / B	Wi-Fi	cellular	build replaced	old token rejected
phone / A	cellular	Wi-Fi	double tap	one approval operation

Record exact OS/browser versions, power mode, preview/build IDs, network timestamps, token evidence version, HTTP outcome, and final server state. Do not record personal browsing data or raw bearer tokens.

Unexecuted test sequence: open the identified preview; inspect its visible commit marker; background the browser; trigger a harmless rebuild from the fixture; switch networks; foreground; attempt the old action. Expected failure is an explicit stale response with no approval effect. Fetch the new preview and token, verify identifiers, approve once, then immediately switch networks again. Expected success is one durable operation ID whose artifact identity matches the screen.

Stop release for approval during offline state, replay after token expiry, disagreement across devices, a changed artifact retaining the old evidence version, or duplicate effects from repeated taps. Cleanup means revoke unused tokens, delete preview and disposable workspace, remove test identities, and verify the action cannot be replayed from browser history. Preserve only sanitized IDs and outcomes.

This grid cannot cover every radio, browser cache, service worker, VPN, captive portal, or clock condition. A token reduces stale approval risk but does not prove preview content integrity or reviewer intent. The protocol and expected results are proposed; no mobile test result is claimed. Record whether a stale page remains visible after rejection, because safe server behavior does not excuse a misleading screen.

Connect the grid to a temporary preview

The overseas hosted MonkeyCode option is described by current official material as “Free to start,” with integrated models and managed server-side cloud development environments supporting build, test, and preview. It offers a surface for this handoff test; it does not make every model or cloud server free. Check the current console for availability, quotas, permitted regions, duration, possible future pricing, and applicable terms.

Use this official campaign route after preparing the disposable device fixture: https://ly.cyberserval.tech/iIETXiF

The mobile exit sequence revokes tokens, clears sanitized fixture data, deletes preview and workspace, and retries the old action only to confirm rejection. Any surviving authority or mismatched artifact identity fails the exercise.

Disclosure: This article promotes MonkeyCode using an official campaign link. I’m a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited project materials.