DEV Community

Roronoa
Roronoa

Posted on

Benchmark Mobile AI Reconnect Recovery Across Lifecycle Transitions With a Server You Can Kill

#mobile #ai #testing #ios

Last week I was testing an AI chat feature on my iPhone 14 (iOS 17.5, React Native 0.74, Hermes). The flow looked fine on Wi-Fi. Then I walked into an elevator mid-stream, the radio dropped, iOS suspended the app, and when I came back the conversation had silently lost the last two turns. No error. No retry. Just a gap in the transcript that only the user would notice.

The problem with testing this class of bug is that I don't control the model backend. I can't tell a hosted LLM API to hang for 40 seconds, drop a socket mid-response, or return a truncated chunk — and those are exactly the conditions my app fails under. So I started routing the app through a relay server I can kill, and suddenly reconnect behavior became a benchmarkable property instead of a vibe.

This post is the harness, the test matrix, and how I run it. Everything below is a reproducible setup, not a finished benchmark report — where I expect a result rather than having measured one, I say so.

Why a relay you control

Point your mobile app at a thin relay in front of the real model API. The relay proxies requests but lets you inject:

  • Latency — delay upstream responses by N ms to simulate a congested cell handoff.
  • Hard drops — close the socket mid-stream to simulate the elevator.
  • Suspension windows — pause the upstream so iOS has time to background your app before bytes arrive.

If your AI task survives a relay that is deliberately hostile, it has a chance of surviving real networks.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The reason it fits here is practical: MonkeyCode offers free model access and a free server option, which means the relay plus a model backend can run without me paying for a VPS or burning API budget during long fault-injection sessions. Any equivalent free tier works for this harness — the point is that cost stops being an excuse to skip the test, not that one provider is uniquely fast or reliable (I have not benchmarked MonkeyCode's latency and make no claim about it).

The fault-injection relay

Minimal Node 20 relay. It streams from an upstream OpenAI-compatible endpoint and applies a per-request fault profile from headers, so the test runner — not the app — controls the chaos:

// relay.mjs — run: node relay.mjs
import http from 'node:http';

const UPSTREAM = process.env.UPSTREAM_URL;   // your model endpoint
const KEY = process.env.UPSTREAM_KEY;

http.createServer(async (req, res) => {
  if (req.method !== 'POST') { res.writeHead(404); return res.end(); }

  const delay = parseInt(req.headers['x-fault-delay'] || '0', 10);
  const dropAfterBytes = parseInt(req.headers['x-fault-drop'] || '0', 10);

  let body = '';
  for await (const c of req) body += c;

  if (delay) await new Promise(r => setTimeout(r, delay));

  const up = await fetch(UPSTREAM, {
    method: 'POST',
    headers: { 'content-type': 'application/json', authorization: `Bearer ${KEY}` },
    body,
  });

  res.writeHead(up.status, { 'content-type': up.headers.get('content-type') ?? 'application/json' });

  const reader = up.body.getReader();
  let sent = 0;
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    sent += value.length;
    if (dropAfterBytes && sent > dropAfterBytes) {
      res.socket.destroy(); // simulate a radio drop mid-stream
      return;
    }
    res.write(value);
  }
  res.end();
}).listen(8787, () => console.log('relay on :8787'));
Enter fullscreen mode Exit fullscreen mode

Point the app's base URL at http://<your-lan-ip>:8787, and drive fault profiles from the test harness:

# Clean run
curl -X POST http://localhost:8787/v1/chat -d '{...}'

# 8s upstream hang — long enough to background the app and come back
curl -X POST http://localhost:8787/v1/chat -H 'x-fault-delay: 8000' -d '{...}'

# Drop mid-stream after 2KB
curl -X POST http://localhost:8787/v1/chat -H 'x-fault-drop: 2048' -d '{...}'
Enter fullscreen mode Exit fullscreen mode

The test matrix

This is the part most teams skip. Each cell is a distinct failure mode, and "works on Wi-Fi" only covers the first row. My matrix for an AI task with an in-flight request:

# Lifecycle transition Fault injected Expected behavior
1 None, foreground none stream completes, transcript persisted
2 Background 5s, return none stream resumes or cleanly resumes from checkpoint
3 Background 60s (iOS suspends) delay 8s task marked interrupted; on return, partial output shown + explicit resume prompt
4 Network drop mid-stream drop 2KB partial tokens preserved, retry offered, no duplicate user turn on retry
5 Airplane mode on/off drop + offline task queued locally, replays exactly once on reconnect
6 App killed by swipe any cold start restores draft + interrupted task state from disk, not memory
7 Low Power Mode + background delay 8s same as #3; no silent assumption that background fetch will run

For each cell I record: device, iOS version, RN version, network (Wi-Fi/LTE/airplane), battery state, fault profile, and one of three outcomes — recovered, restarted, or silently lost. The third outcome is the one that matters; it's the bug my users found in the elevator.

The single most common root cause I've hit across these tests: task state living only in memory (a Redux store or component state) instead of being checkpointed to disk after every streamed chunk. AppState listeners and NetInfo alone cannot save you if there is nothing durable to restore.

What this harness does not tell you

  • It reproduces network and lifecycle faults deterministically, but not OS-level variance. iOS background execution budgets differ by device and usage history; run the matrix on the oldest device you support, not your daily driver.
  • The relay adds a hop. Don't use round-trip times through it as production latency numbers.
  • "Free tier" availability, quotas, and model lineups change. Verify what's actually offered before building a CI job on top of it, and have a fallback endpoint.
  • I have not yet run this matrix on Android; the backgrounding semantics (Doze, OEM task killers) are different enough that results won't transfer.

Who should not use this approach

If your AI feature is a single fire-and-forget request with no streaming and no retry semantics, this is overkill — a unit test with a mocked 500 is enough. And if your threat model is about the provider's uptime rather than the device's radio, inject faults at the client layer (e.g., a URLProtocol/OkHttp interceptor) instead of standing up a relay.

If you try the matrix, I'd genuinely like the comparable data: device, OS version, which transition you hit, and whether the task recovered, restarted, or silently disappeared. That third column is where the interesting bugs live.

If you want to try the relay setup without provisioning anything, MonkeyCode's free model access and free server option are one way to get both halves running — but the harness above works against any endpoint you control.

Top comments (0)