DEV Community: Todd Sullivan

The CoreML Model Was Fine. The Photo Was Too Big.

Todd Sullivan — Tue, 13 Oct 2026 08:03:32 +0000

I hit a very ordinary computer-vision failure this week: the model was doing what I asked, but the image pipeline was making the task impossible.

The app counts livestock on-device with a small YOLO/CoreML model. No network, no account, no server-side fallback. That constraint is useful, but it means the phone pipeline has to be honest about what the model actually sees.

The first photo path looked sensible:

let request = VNCoreMLRequest(model: model)
request.imageCropAndScaleOption = .scaleFit
handler.perform([request])

One image in, detections out.

Then I tested it on a real paddock photo with roughly 250 sheep in it. The count came back as 14.

That was not a model mystery. A 12MP photo was being squeezed into the model input. By the time the flock reached the network, distant sheep were just a few pixels each. There was nothing clever for the detector to recover.

The fix was boring and much better than changing the model:

run the full image once
run overlapping tiles over the same image
map tile-relative boxes back into image coordinates
drop boxes cut by an inner tile edge
merge the remaining detections with class-agnostic NMS

The important bit is that tiling changes the object size the model sees without changing the model.

let regions = [CGRect(x: 0, y: 0, width: 1, height: 1)]
    + Self.tiles(aspect: aspect, across: 3, overlap: 0.25)

for region in regions {
    request.regionOfInterest = region
}

On a 4:3 photo that becomes 12 tiles plus the full-frame pass. More inference, but still on-device and still only a fraction of a second for a still photo.

The same paddock photo went from 14 to 173 detections. Not perfect, but suddenly useful. Close-up hen photos held steady, which mattered just as much: the fix improved recall where scale was the problem without inflating easy counts.

The non-obvious part was the merge step.

Default IoU-only NMS is not enough when tiling. A partial head box can sit inside a whole-animal box and have low IoU, so I also merge on intersection-over-smaller. I also make the merge class-agnostic because the same animal can come back under two labels from different passes. Count the animal, not the label argument.

The lesson was simple: before blaming the model, check what pixels you actually gave it.

For on-device AI, the model is often the least flexible part of the system. The image pipeline around it is where the cheap wins hide.

My AI Cron Job Failed Because .env Wasn't a Shell Script

Todd Sullivan — Tue, 06 Oct 2026 08:03:11 +0000

I hit a small failure in an AI publishing cron job this morning that is exactly the kind of boring edge that makes agents unreliable.

The job needed one secret:

DEVTO_API_KEY=...

The lazy version was to source the whole env file and let the shell export everything.

set -a
. ~/.hermes/.env
set +a

That is fine until the env file stops being a shell script.

This one had unrelated values in it. One contained an app path with spaces. Another looked enough like shell syntax to be treated as a command. The job still found the API key later, but the logs had the real problem:

Chrome.app/Contents/MacOS/Google: No such file or directory
fyqi: command not found

Nothing about the Dev.to API was broken. The agent had just widened the trust boundary for no reason.

The fix was not to make the env parser clever. It was to stop parsing the file as code.

import re
from pathlib import Path

text = Path.home().joinpath(".hermes/.env").read_text(errors="ignore")
match = re.search(r"^DEVTO_API_KEY=(.*)$", text, re.M)
api_key = match.group(1).strip().strip('"\'')

That is deliberately boring. It reads exactly the value the job needs and ignores everything else.

The useful rule for agent jobs is:

if a task needs one credential, load one credential.

Do not source a shared env file just because it is convenient. AI agents already have enough ways to do surprising things. Giving a scheduled job a file full of executable shell lines is a cheap way to add another one.

This is especially true for personal agent infrastructure, where the same .env tends to collect API keys, browser paths, tunnel tokens, feature flags, old experiments, and comments that were never meant to be interpreted by sh.

The failure mode is annoying rather than dramatic, which is why it is worth fixing. The job does not need a config framework. It just needs to stop executing config.

Small boundary. Small parser. Fewer weird mornings.

The 8B Local Model Was Worse Than the 4B One

Todd Sullivan — Tue, 29 Sep 2026 08:02:49 +0000

I tried the obvious local-LLM upgrade on a small Mac mini: move from a 4B model to an 8B model.

It was worse.

The machine is an 8GB Apple Silicon mini running a personal assistant stack. The useful path was already working with mlx-community/Qwen3-4B-4bit through Swama's OpenAI-compatible endpoint:

4B weights: ~2.3GB
warm plain reply: ~1s
warm tool-call round: ~3-4s model time
full tool response: ~7-18s, depending on external APIs

So I pulled the 8B quantized model to see if the extra capability was worth it.

The first trivial request was enough:

8B weights: ~4.3GB
"say hello": 14s
throughput: 6.8 tok/s
swap churn: ~1.6GB in/out
free memory: basically zero

That was not an edge case. That was the cheapest possible prompt.

A real assistant request is heavier: system prompt, recent conversation, tool schemas, maybe a tool round, then a final answer. If the 8B model swaps on hello, it is not going to survive a weather/tool/home-status request while the rest of the app is running.

The annoying part is that the model choice looked sensible on paper. 8B should be smarter. It probably is, in isolation. But local AI work is not model leaderboard work. It is memory pressure, cold starts, prompt prefill, tool reliability, and whether the box is still usable after the request.

The fix was not glamorous:

MAX_HISTORY_MESSAGES = 10

@lru_cache(maxsize=1)
def get_ai_handler():
    return SwamaAIHandler()

Cap the prompt history. Reuse the handler. Keep long-term memory as distilled facts, not raw transcript replay. Use the 4B model that stays resident and answers quickly enough.

I also had to add boring production guardrails:

run the real Swama binary, not the symlink, because first inference crashed loading MLX's metallib
keep the model warm because idle eviction made cold starts painful
strip <think> blocks from streamed responses because /no_think was not enough after tool calls
bias the prompt toward tool use for live data, because small models will confidently answer from stale weights if you let them

The lesson was simple: on tiny local hardware, the best model is the one that leaves enough machine behind for the product.

The 4B model shipped. The 8B model got deleted.

The AI Coding Agent Wasn't the Risk. The Missing Deploy Gate Was.

Todd Sullivan — Tue, 22 Sep 2026 08:03:37 +0000

I had a small reminder this week that AI-assisted development does not really change the old rule: if a human or an agent can edit a deployment path, the deployment path needs to prove the important thing still works.

The bug was not dramatic. A routing service sits behind a tiny Caddy gate because the upstream server has no auth of its own. Requests without ?key= should get 403; requests with a valid key should route.

That was already checked after deploy:

check 403 "without key" "https://$FQDN/route?$ROUTE"
check 200 "with key"    "https://$FQDN/route?$ROUTE&key=$KEY"

Good enough for the live app. Not good enough for App Review.

The app also needs a separate review key. A reviewer opening a fresh install cannot use a key already typed into my own devices. So Caddy accepts two values for the same query param:

@authorized query key=LIVE_KEY key=REVIEW_KEY

The failure mode was easy to miss: a deploy could ship a Caddyfile that still accepted the live key but silently dropped the review key. The health check would pass. The app would still work for me. Then days later App Review would open the build, hit the first-run screen, and reject it.

That is the worst kind of infrastructure bug: green deploy, delayed failure, wrong human paying the cost.

The fix was nine lines:

# The key issued to Apple's Beta App Review, when this machine holds one.
# Absent file, no review key in play, nothing to check.
if [ -f .secret-review ]; then
  check 200 "review key" \
    "https://$FQDN/route?$ROUTE&key=$(cat .secret-review)" || ok=1
fi

No framework. No secret-management migration. Just make the deploy assert every key path that matters.

Claude co-authored this change, which is the bit that made it worth writing down. Not because the agent did something dangerous. Because agent-written code has the same problem as human-written code, only faster: it can make a local patch that looks right while missing the operational contract around it.

The useful pattern is simple:

let the agent edit the boring deployment code
make the deploy script test the real external behaviour
fail before the broken state becomes somebody else's problem

For anything with AI in the development loop, I care less about whether the diff looks plausible and more about whether the system can prove the side effect survived.

In this case the proof is three HTTP codes:

without key -> 403
with key    -> 200
review key  -> 200

That is not a big security architecture. It is just a tripwire in the place I am guaranteed to step before shipping.

The On-Device People Counter Only Worked After I Made It Boring

Todd Sullivan — Tue, 15 Sep 2026 08:02:58 +0000

I built a tiny line-crossing people counter for a Seeed XIAO ESP32S3 Sense this week. Camera on the board, model on the board, counts on the board. No laptop in the loop unless I want the more accurate host-mode version.

The first useful constraint was admitting what the hardware could actually do.

The host detector runs at 416×416 and is about 1.1 GFLOPs. That is fine on a laptop. On an ESP32-S3 it is fantasy. The on-device detector is roughly 15× smaller and still takes about 118 ms per frame.

At QVGA the budget ended up like this:

JPEG decode      ~30 ms
preprocess        9 ms
model           118 ms
postprocess       2 ms
---------------------
total          ~160 ms

So the real target is not “real time” in the desktop-CV sense. It is about 6 fps, which is enough for someone walking through a doorway if the rest of the pipeline does not get clever.

The counter is deliberately plain:

capture the newest frame
detect people
use the bottom-center of each box as the foot point
match points with a nearest-centroid tracker
count when a track crosses a configured line
log the event to serial, optionally expose /counts over WiFi

The foot point mattered more than I expected. Box centroids move around with posture and crop shape. If the counting line is on the floor, the bottom-center of the box is a better approximation of the thing that actually crosses the line.

The tracker is also intentionally boring:

// ponytail: no motion model, no re-id. Fine for a doorway at ~6 fps;
// swap in ByteTrack if people cross paths in frame.
class Tracker {
  // id -> centroid, max distance, max age
};

No Kalman filter. No re-identification. No cloud fallback. Just keep a track alive for a few missed frames and drop it when it gets too stale.

That is not because those techniques are bad. It is because this is a doorway counter on a microcontroller. People mostly move in one direction, the camera points down, and the useful failure mode is obvious: if people cross paths in frame, this tracker is too small and should be replaced.

That shape is easier to operate than a “smart” pipeline pretending it has more signal than it does.

The board can serve its own tiny monitoring page when WiFi is configured:

GET /counts
{"in":14,"out":11,"occupancy":3}

But it also works serial-only with empty WiFi settings. That was another good constraint. The core counting path should not depend on a dashboard, a backend, or a phone app being alive.

The bit I like most is that the counting logic has a host self-check separate from the ESP-IDF build:

make -C firmware/test
# ok: 6 counter tests

For edge AI, that split is worth doing. The model and camera are the annoying parts to test. The line-crossing rules do not need to be. Pull the state machine into boring C++, beat it up on the host, then let the board worry about frames and inference.

The lesson was the same one on-device AI keeps teaching me: the model is only one budget. Latency, memory, camera format, mounting angle, testability, and failure honesty matter just as much.

A smaller model plus a boring counter beats an impressive pipeline that cannot keep up with someone walking through a door.

The Credit Counter Race That Let 50 AI Requests Through

Todd Sullivan — Tue, 08 Sep 2026 08:03:40 +0000

I fixed a boring little backend bug this week that would have been expensive in exactly the way AI features like to be expensive: quietly, under concurrency, while every individual request looked valid.

The app has Claude-backed generation behind a monthly credit allowance. Simple shape:

authenticate the user
check their subscription and monthly allowance
call Claude
record the generation and token cost
increment credits_used

That looks fine until you remember the model call is the slow part.

The old gate read credits_used, saw the user was under the cap, then let the request continue. Only after the multi-second Claude call did it increment the counter.

So a user sitting at 29 / 30 credits could fire 50 parallel requests. All 50 read 29. All 50 passed the gate. All 50 spent tokens. Then the counter landed at 79.

No exotic attack. No clever prompt injection. Just check-then-act around a slow side effect.

The fix was to stop treating the credit check as application logic and make Postgres do the one thing it is very good at: serialize writes to the row.

create or replace function public.reserve_ai_credit(
  p_user_id uuid,
  p_month text,
  p_limit integer
) returns integer language plpgsql security definer as $$
declare
  v_used integer;
begin
  insert into public.ai_usage (user_id, month, credits_used)
  values (p_user_id, p_month, 1)
  on conflict (user_id, month) do update
    set credits_used = ai_usage.credits_used + 1
    where ai_usage.credits_used < p_limit
  returning credits_used into v_used;

  return v_used;
end;
$$;

That where ai_usage.credits_used < p_limit on the conflict update is the whole trick. Concurrent requests now queue on the same row. The first one increments. The next sees the updated value. Once the cap is reached, returning gives nothing back and the request stops before the model call.

The Edge Function now reserves a credit before calling Claude:

const { data: used, error } = await supabase.rpc("reserve_ai_credit", {
  p_user_id: user.id,
  p_month: month,
  p_limit: allowance,
});

if (used === null || used === undefined) {
  return json({ error: "Credit limit reached" }, 429);
}

That changes the failure shape, so the other half of the fix was refunds. If the model call fails, or the structured response violates the schema, the function calls refund_ai_credit. Failed generations should not bill the user, but they also should not leave the gate open until after the expensive work is done.

While I was in there, I removed a duplicate credit gate from one function and made it call the shared runCreditGate path like the others. That cut about 52 lines and made the concurrency fix apply everywhere instead of only to the function I happened to be staring at.

A few adjacent bugs fell out of the same pass:

uncapped text fields could push huge strings into prompts
avgSleepHours.toFixed() could throw when the client sent a string
an unknown Claude model name logged cost as $0 silently
one suggestion endpoint was returning 401 for “no subscription”, which signs the user out instead of showing the paywall

None of those needed a grand framework. Just input clamps at the boundary, one shared gate, and loud logs when accounting metadata drifts.

The useful lesson: if an AI feature has a quota, the quota has to be enforced before the model call, in the database, atomically. Anything else is a polite suggestion with a cloud bill attached.

I ran the Supabase Edge Function helper tests after the fix: 62 Deno tests, 0 failed.

The Tracking Bug Wasn't the Model. It Was the Dot Lying.

Todd Sullivan — Tue, 01 Sep 2026 08:07:11 +0000

I spent part of this week debugging an on-device livestock counter in a field-video mode. The model was fine. The count was not.

The symptom from the real device run was simple: a tracking dot would leave one animal, drift onto its neighbour for a few frames, then snap back. In a dense mob that is exactly the kind of tiny lie that ruins the user's trust. If the UI says that sheep is being tracked, it had better mean it.

The first version of the counter was deliberately boring:

detect animals with a small CoreML model
maintain short-lived tracks between frames
count when a track crosses a tripwire line
let missed tracks coast for a few frames so one bad detection does not drop the animal

That worked in still-ish tests. It failed once the phone and the sheep were both moving.

Prediction is not presentation

The lazy assumption was that the predicted track position could also be the drawn dot position.

Bad assumption.

A tracker is allowed to guess. A UI marker is a claim. Those are different contracts.

A track can go unseen for up to 15 frames, because dropping it too quickly loses animals behind brief occlusion. During that time prediction uses the last observed heading plus camera-motion compensation. Good for matching. Bad for display. On a slow pan, a neighbouring sheep is about one body length away, so a few frames of coasting was enough to draw the dot onto the wrong animal.

The fix was smaller than the bug report made it sound:

let visibleCoastLimit = 2
let isDisplayPrediction = track.missedFrames > 0

if track.missedFrames <= visibleCoastLimit {
    drawDot(at: track.predictedCenter,
            opacity: isDisplayPrediction ? 0.45 : 1.0,
            scale: isDisplayPrediction ? 0.7 : 1.0)
}

Matching can keep the longer memory. Display gets the honest one.

Centroid distance breaks in a flock

The second bug was association cost.

I had already moved past purely greedy matching, but the cost was still mostly centroid distance. That sounds reasonable until you put the phone in front of a packed group of sheep. Every animal has another animal a body length away. Centroids are not enough signal.

The better signal is overlap.

Two boxes on the same animal overlap heavily between adjacent frames. Two boxes on neighbouring animals usually do not. So the association cost now prefers overlap first and only falls back to distance when there is no meaningful overlap:

if iou > 0 {
    cost = 1 - iou
} else {
    cost = 1 + normalisedDistance
}

That one change makes the tracker much less willing to jump to a neighbour just because its centre is nearby. It also means tracks need to keep the whole observed box, not just a centre point and longest side. Slightly more state, much better truth.

The useful lesson

Most ML product bugs are not model bugs.

The detector can be good enough and the product can still feel broken because the tracking layer lies, the UI overstates certainty, or the recovery path optimizes for the wrong thing.

In this case the fix was not a new model, a heavier tracker, or a dependency. It was separating prediction from presentation, then using the geometry the video already had.

Boring, local, testable. My favourite kind of AI fix.

Development Notes: Navigation UX Is a State Problem

Todd Sullivan — Mon, 31 Aug 2026 17:21:46 +0000

Recent FlatNav development has been a useful reminder that navigation UX is mostly a state-management problem pretending to be a design problem.

The visible part is the map, route, buttons, and labels.

The real work is deciding what the user needs to know right now, what can wait, and what should never be shown at all.

Good navigation software has to stay calm while the user is not. That means the interface needs to handle uncertainty without pushing it onto the person using it.

The useful questions are simple:

what state is the journey in?
what is the next useful action?
has the user changed context?
does the UI need to interrupt, confirm, or stay quiet?
is this information actionable now, or just noise?

Most improvements come from reducing interpretation.

If the user has to stop and decode the screen, the app has already leaked complexity. If a feature adds more choices than it removes, it probably made the system feel smarter while making the user slower.

That is why the best development cycles often look small from the outside: clearer states, fewer edge-case dead ends, tighter feedback after actions, better defaults, less UI competing with the next step.

There is a lazy engineering lesson in that.

Before adding another control, another setting, or another explanation, ask whether the state model is doing its job. Often the fix is not more interface. It is making the existing state clearer, deleting an unnecessary decision, or moving logic into the product so the user does not have to carry it.

Navigation products get better when they get quieter.

Development Notes: Building AI for Menopause Support

Todd Sullivan — Mon, 31 Aug 2026 17:21:45 +0000

Recent development work on a menopause intelligence app has been a good reminder that useful AI in health-adjacent software is mostly not about model novelty.

The interesting part is the product layer around the intelligence.

Menopause data is messy in the way real human data is messy: symptoms overlap, patterns take time to emerge, user input is incomplete, and the app has to be careful not to imply certainty where there is none.

That creates a few practical engineering constraints:

keep the language plain
separate tracking from interpretation
avoid over-confident output
design for partial information
make escalation to human support obvious where needed
treat privacy and consent as part of the core UX, not paperwork

The useful technical work is in the guardrails and flow design.

A lot of AI demos assume the best input: a clean prompt, a clear goal, and a user who knows exactly what to ask. Real apps get the opposite. Users arrive with fragments: symptoms, worries, timings, medication changes, sleep disruption, mood changes, and half-remembered appointment notes.

So the system has to help structure the problem before it tries to answer anything.

That means boring but important work: tighter onboarding, clearer state, better defaults, careful copy, and prompts that make uncertainty explicit rather than hiding it behind confident wording.

The App Store release process also forces discipline. Privacy wording, support flows, screenshots, review metadata, and boundary-setting all push the same question: does this behave like something people can safely understand, or just like a clever demo?

That is the real development lesson here.

In sensitive domains, the intelligence layer should be small, explainable, and restrained. The product around it does most of the trust-building.

The RLS Policy Was Correct. The Write Was Still Wrong.

Todd Sullivan — Tue, 25 Aug 2026 08:08:16 +0000

The RLS Policy Was Correct. The Write Was Still Wrong.

I spent a chunk of last week doing the least glamorous kind of security work: reading database policies line by line and asking, "what can the client still put in this row?"

The useful bug pattern was simple:

Row-level security said which rows a user could touch, but not always which values they were allowed to write.

That difference matters a lot in offline-first apps, because the mobile client necessarily syncs whole records back to the server. If the database policy only checks tenant scope, a malicious or buggy client can sometimes write fields that were supposed to be server-owned.

A simplified version:

CREATE POLICY org_isolation ON inspections
  FOR ALL
  TO authenticated
  USING (org_id = my_org_id());

Looks reasonable. A user can only see and edit inspections in their own org.

But for writes, that policy is incomplete. Without a WITH CHECK, the database is mostly answering "is this row in your org?" It is not necessarily answering "are you allowed to set approval_status = 'approved'?"

That distinction turned into a real class of findings:

an org member could mark an inspection approved without an approval record
approval rows could be forged with someone else's reviewer id/name/role
onboarding could attach a fresh account to an existing org id if the RPC accepted caller-supplied ids too literally
client-writable billing/quota columns were one PATCH away from entitlement nonsense

None of those are exotic exploits. They are boring trust-boundary mistakes. The client had too much ability to describe facts the server should have decided.

The fix was not "add more app-side checks." App-side checks are UI. They help honest users, not the boundary.

For approval decisions, the server now adjudicates the thing that matters:

CREATE OR REPLACE FUNCTION record_approval(
  p_inspection_id uuid,
  p_decision text,
  p_comment text DEFAULT NULL
)
RETURNS inspection_approvals
LANGUAGE plpgsql
SECURITY DEFINER
AS $$
DECLARE
  v_org_id uuid;
  v_role text;
BEGIN
  SELECT org_id, role
  INTO v_org_id, v_role
  FROM profiles
  WHERE id = auth.uid();

  IF v_role NOT IN ('supervisor', 'admin') THEN
    RAISE EXCEPTION 'Supervisor or admin role required';
  END IF;

  INSERT INTO inspection_approvals (
    inspection_id, org_id, reviewer_id, reviewer_role, decision, comment
  )
  VALUES (
    p_inspection_id, v_org_id, auth.uid(), v_role, p_decision, p_comment
  );

  UPDATE inspections
  SET approval_status = p_decision
  WHERE id = p_inspection_id AND org_id = v_org_id;
END;
$$;

The caller can request a decision. It cannot choose the reviewer, role, org, or audit identity. Those come from auth.uid() and server-side profile state.

For the offline sync path, I kept the boring path boring. Inspectors still need to create and submit inspections with no signal, then sync later. So the value gate became narrower: clients can write draft/submitted states, but approved/rejected must match the latest server-recorded approval decision.

That's the part I like. The fix did not require turning the whole app into RPC soup. Only the authority-bearing transition moved server-side.

The takeaway: RLS is not one switch. Tenant isolation is only the first question.

For every writable field, ask:

Can the client choose this value?
If yes, is tenant scope enough?
If no, where is the server-side writer?

Most bugs I found were hiding between question 1 and question 2. The policy knew whose row it was. It did not always know whose decision it was.

I Used Claude to Fill Inspection Forms by Voice, Without Building a Voice Stack

Todd Sullivan — Tue, 18 Aug 2026 08:03:21 +0000

I Used Claude to Fill Inspection Forms by Voice, Without Building a Voice Stack

I just added voice form fill to GroundCheck, and the most useful decision was the boring one: I did not build a voice assistant.

The feature is scoped very tightly. An inspector is already inside one inspection form. They tap Fill with Voice, dictate something like:

all four guys had hard hats and vests on, scaffold tags were current, one trip hazard by the east gate, no incidents

The app sends that transcript plus the visible form schema to a Supabase Edge Function. Claude maps the narration onto the matching fields. The app shows a review sheet. The inspector accepts or rejects each proposed value.

That is it. No chat UI. No cross-report memory. No “AI assistant for the whole day.” Just spoken notes → structured form fields.

The client-side rule is deliberately small:

const FILLABLE_TYPES = new Set(["text", "checkbox", "multiple_choice", "rating"]);

export function fillableQuestions(section, responses) {
  return section.questions.filter(
    (q) =>
      FILLABLE_TYPES.has(q.type) &&
      (!q.show_when || evaluateCondition(q.show_when, responses))
  );
}

Photos and signatures are excluded because narration cannot produce them. Conditional questions hidden by show_when are excluded too. That matters on a whole-form fill: if the user says the vest check passed, Claude should not even see the “why did hi-vis fail?” follow-up field.

The Edge Function builds a JSON schema dynamically from the fields in scope. Choice fields become enums. Rating fields become enums from the score map. Unknown fields and illegal values are dropped again server-side after the model returns, because this eventually lands on a safety-compliance record and “the schema probably handled it” is not good enough.

I also kept the API key entirely server-side. The mobile app never sees the Anthropic key. The function checks auth, org tier, and a per-org monthly cap before making the paid call, then logs usage rows with input/output token counts.

The nice lazy bit: speech capture uses the iOS keyboard dictation button in an autofocused multiline text field. No speech-recognition dependency, no native module, no new permission strings, no rebuild just to test the first version. If field testing proves people need a big in-app hold-to-talk button, the transcript box can be swapped later. The contract downstream is still just text.

The review step is non-negotiable. Claude proposes; the human applies. Already-answered fields show as changes instead of silent overwrites. Identical values are dropped. Everything is displayed in form order so it feels like reviewing the inspection, not debugging JSON.

The part I like most is that the whole feature is smaller because it refused to be a general assistant. It only knows about the form on screen, only sees fields that can currently be answered, and only writes after a human says yes.

That shape is usually where LLM features get useful: narrow input, constrained output, boring gates, human review at the boundary.

My On-Device ML Feature Was Registered, But Never Scheduled

Todd Sullivan — Tue, 11 Aug 2026 08:06:21 +0000

I spent part of this week fixing a mobile AI feature where the model code was not the problem.

The app has a small on-device personalization loop: collect daily logs, merge in HealthKit metrics, retrain a local model periodically, and use that state to drive predictions and nudges. Nothing huge. Just the kind of background ML plumbing that makes a feature feel alive instead of static.

The bug was simpler and more annoying:

I had registered the background tasks.

I had handlers for them.

I had rescheduling inside each handler.

But the first task was never submitted.

So nothing ever ran.

Registration is not scheduling

This is the shape that matters with BGTaskScheduler:

func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]? = nil
) -> Bool {
    registerBackgroundTasks()
    submitBackgroundTasks()
    return true
}

That second line is the one I was missing.

registerBackgroundTasks() tells iOS which identifiers this process can handle. It does not put work on the queue. Each handler can reschedule itself after it runs, but something has to submit the first request or the chain never starts.

The final version is deliberately boring:

private func submitBackgroundTasks() {
    scheduleHealthRefresh()
    scheduleMLRetrain()
    scheduleEngagementDaily()
    scheduleWeeklyInsight()
}

And the retrain request is just a normal BGProcessingTaskRequest:

func scheduleMLRetrain() {
    let request = BGProcessingTaskRequest(
        identifier: AppConstants.mlRetrainTaskID
    )
    request.earliestBeginDate = Date(
        timeIntervalSinceNow: 86400 * Double(AppConstants.retrainingIntervalDays)
    )
    request.requiresExternalPower = true
    request.requiresNetworkConnectivity = false
    try? BGTaskScheduler.shared.submit(request)
}

That is the whole fix. Not a new scheduler abstraction. Not a custom job runner. Just submit the work at launch and let iOS deduplicate by identifier.

The other trap: background queues and SwiftData

The second issue was actor isolation. BGTaskScheduler invokes handlers on a background queue. My persistence layer uses container.mainContext, so every handler that touches SwiftData needs to hop back to the main actor.

private func handleMLRetrain(task: BGProcessingTask) {
    task.expirationHandler = { task.setTaskCompleted(success: false) }

    Task { @MainActor in
        let context = PersistenceController.shared.container.mainContext
        let descriptor = FetchDescriptor<DailyLog>(
            sortBy: [SortDescriptor(\DailyLog.date)]
        )
        let logs = (try? context.fetch(descriptor)) ?? []
        try? await MLModelManager.shared.trainPersonalisedModel(from: logs)
        scheduleMLRetrain()
        task.setTaskCompleted(success: true)
    }
}

Again, boring. But this is exactly where AI features usually break: not in the model, but in the lifecycle around the model.

One shared path for HealthKit updates

I also pulled the HealthKit merge into one shared function so the scheduled refresh and HealthKit background delivery cannot drift apart:

@MainActor
static func fillTodaysLogFromHealthKit() async {
    let metrics = await HealthKitManager.shared.fetchDailyMetrics(for: Date())
    let context = PersistenceController.shared.container.mainContext
    let log = DailyLog.findOrCreate(for: Date(), in: context)

    if log.hrv == nil               { log.hrv = metrics.hrv }
    if log.restingHeartRate == nil  { log.restingHeartRate = metrics.restingHeartRate }
    if log.stepCount == nil         { log.stepCount = metrics.stepCount }
    if log.sleepDuration == nil     { log.sleepDuration = metrics.sleepDuration }

    try? context.save()
}

The important detail is that manual user-entered values win. Background sync should fill gaps, not overwrite what someone typed by hand.

The tests that caught the boring failures

A few regression tests now pin the things that looked too small to test before:

one DailyLog row per calendar day, because duplicate daily rows poison the training set
future-dated rows must not be mistaken for today
CloudKit-backed SwiftData schema must load, because a bad relationship can crash the app on launch
engagement notifications default to enabled, because UserDefaults.bool(forKey:) returns false for an unset key
pending Watch entries drain by payload type instead of clearing the whole queue

That last one matters because the Watch queue contains two shapes in one array: symptom logs and wellbeing check-ins. The old reconciliation path could drain symptoms and silently delete check-ins that had not been processed yet.

Silent data loss in a health-adjacent logging app is not a UX bug. It is a trust bug.

The takeaway

On-device AI features are mostly normal mobile engineering with a model in the middle.

The model can be fine while the product is still broken because:

the background task was registered but never scheduled
the handler touched persistence from the wrong execution context
sync overwrote user-entered data
duplicate daily rows polluted the training input
a queue drain deleted the wrong payload type

None of that is glamorous. All of it matters.

If a model retrains locally but the scheduler never wakes up, you did not ship personalization. You shipped a nice code path nobody calls.

That is the bar I keep coming back to with practical AI work: not "does the model exist?" but "does the system around it actually keep running after the app leaves the foreground?"