DEV Community

Roronoa
Roronoa

Posted on

Benchmark an AI Coding Task Across Mobile Backgrounding, Network Switches, and Battery Pressure

#mobile #ai #testing #ios

Most AI coding assistants are demoed on a desk: laptop plugged in, stable Wi-Fi, full attention. But the moment I approve or monitor a long-running AI task from a phone, the environment becomes hostile — the app gets backgrounded, the device hops from Wi-Fi to LTE, iOS starts throttling background execution, and the battery drops below 20%.

This article is a reproducible benchmark method for one question: does an AI coding task survive the mobile lifecycle, and can you tell the difference between 'recovered', 'restarted', and 'silently dropped'?

Test environment (record yours)

  • Device: iPhone 14, iOS 17.x (note your exact build)
  • App state: task started from the foreground, screen unlocked
  • Network: controlled Wi-Fi with the option to toggle airplane mode, plus LTE fallback
  • Power: start at ~60% battery, charger off; repeat one run in Low Power Mode
  • Backend: a remote AI coding agent endpoint. For these runs I used MonkeyCode, which offers free model access and a free server option — convenient for this kind of throwaway benchmark because I didn't have to provision my own GPU box or burn a paid quota on test traffic.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Any agent backend with a task ID you can poll works for this method; the benchmark is about the lifecycle, not the vendor.

The one metric that matters

For each run, classify the final state:

Outcome Definition
Recovered Same task ID, output continues from where the connection dropped
Restarted New task ID or duplicate execution; original work discarded or redone
Silent loss UI shows 'running' or a spinner, but the backend has no live task

Silent loss is the dangerous one. It looks like progress.

The benchmark script (client side)

The client just needs to launch a task, poll by ID, and log transitions with timestamps. Pseudocode you can adapt to React Native, Flutter, or a plain fetch loop:

// lifecycle-bench.mjs — run in your app's debug console or a Node harness
const log = [];
const stamp = (event, extra = {}) =>
  log.push({ t: Date.now(), event, ...extra });

async function launchTask(prompt) {
  const res = await fetch(`${BACKEND}/tasks`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ prompt }),
  });
  const { taskId } = await res.json();
  stamp("launched", { taskId });
  return taskId;
}

async function poll(taskId) {
  let lastStatus = null;
  for (;;) {
    try {
      const res = await fetch(`${BACKEND}/tasks/${taskId}`);
      const task = await res.json();
      if (task.status !== lastStatus) {
        stamp("status", { status: task.status, progress: task.progress });
        lastStatus = task.status;
      }
      if (["done", "failed"].includes(task.status)) return task;
    } catch (e) {
      stamp("poll_error", { error: String(e) });
    }
    await new Promise((r) => setTimeout(r, 2000));
  }
}

// On app foregrounding, run: poll(savedTaskId)
// Then diff the log against the backend's task history.
Enter fullscreen mode Exit fullscreen mode

After each run, compare the client log with server-side task history. If the client saw poll_error for 4 minutes and the server shows the task still executing, that's a recovered run. If the server shows the task was killed at the disconnect, your UI lied — classify accordingly.

The transition matrix

Run the same prompt (a medium-length task, e.g. 'refactor this module and run the tests') under each condition. One variable per run:

  1. Baseline — foreground, Wi-Fi, charger on. Establishes expected duration.
  2. Background at T+30s — press home, wait 5 minutes, return. Does polling resume? Does the task still exist?
  3. Wi-Fi → LTE handoff mid-task — toggle Wi-Fi off while a poll is in flight. Watch for duplicated requests on reconnect.
  4. Airplane mode 3 minutes — full offline window. On restore, does the client re-attach to the task ID, or submit a new one?
  5. Low Power Mode + background — iOS aggressively defers network activity. Expect longer gaps; check whether the agent times out server-side.
  6. App kill (swipe away) — the harsh case. On relaunch, can you recover the task ID from local storage and re-attach?

Expected observations from my runs: conditions 2–4 are survivable when the client persists the task ID and the backend treats tasks as resumable by ID. Condition 5 is where I've seen silent loss most often — the backend's idle timeout fires while iOS defers the client's keepalive. Condition 6 fails entirely if the task ID only lives in memory.

What to fix when a run fails

  • Restarted instead of recovered: persist taskId (Keychain/Keystore, not just AsyncStorage if it's sensitive) and re-attach on launch before offering a 'start new task' button.
  • Silent loss: add a heartbeat with a server-visible lastSeenAt. If server lastSeenAt is stale but UI shows 'running', surface 'connection lost — task status unknown' instead of a spinner.
  • Duplicate execution after reconnect: make task creation idempotent (client-generated idempotency key), so a retried POST can't fork the run.

Limitations

  • Single device, single OS version. Android's background execution model (Doze, OEM battery killers) behaves differently — repeat the matrix there before generalizing.
  • I tested task continuity, not output quality or model latency; those need their own benchmarks with device data.
  • Free tiers change. The free model access and free server I used here are availability facts as of this writing, not a guarantee of quotas, duration, or performance — verify before building a workflow around them.
  • Results depend heavily on backend timeout policy; your agent service may behave differently under the same client conditions.

Who shouldn't use this approach

If your tasks are sub-second, this whole matrix is overkill — just retry. And if the task handles sensitive code or credentials on a device you don't control, a free hosted server is the wrong target entirely; test continuity against infrastructure you're authorized to use.

If you run this matrix on your own setup, I'd like to compare notes: device, OS version, which transition you triggered, and whether the task recovered, restarted, or silently disappeared. That last one is the number nobody's dashboard shows.

Top comments (0)