DEV Community: Todd Sullivan

The Tracking Bug Wasn't the Model. It Was the Dot Lying.

Todd Sullivan — Tue, 01 Sep 2026 08:07:11 +0000

I spent part of this week debugging an on-device livestock counter in a field-video mode. The model was fine. The count was not.

The symptom from the real device run was simple: a tracking dot would leave one animal, drift onto its neighbour for a few frames, then snap back. In a dense mob that is exactly the kind of tiny lie that ruins the user's trust. If the UI says that sheep is being tracked, it had better mean it.

The first version of the counter was deliberately boring:

detect animals with a small CoreML model
maintain short-lived tracks between frames
count when a track crosses a tripwire line
let missed tracks coast for a few frames so one bad detection does not drop the animal

That worked in still-ish tests. It failed once the phone and the sheep were both moving.

Prediction is not presentation

The lazy assumption was that the predicted track position could also be the drawn dot position.

Bad assumption.

A tracker is allowed to guess. A UI marker is a claim. Those are different contracts.

A track can go unseen for up to 15 frames, because dropping it too quickly loses animals behind brief occlusion. During that time prediction uses the last observed heading plus camera-motion compensation. Good for matching. Bad for display. On a slow pan, a neighbouring sheep is about one body length away, so a few frames of coasting was enough to draw the dot onto the wrong animal.

The fix was smaller than the bug report made it sound:

let visibleCoastLimit = 2
let isDisplayPrediction = track.missedFrames > 0

if track.missedFrames <= visibleCoastLimit {
    drawDot(at: track.predictedCenter,
            opacity: isDisplayPrediction ? 0.45 : 1.0,
            scale: isDisplayPrediction ? 0.7 : 1.0)
}

Matching can keep the longer memory. Display gets the honest one.

Centroid distance breaks in a flock

The second bug was association cost.

I had already moved past purely greedy matching, but the cost was still mostly centroid distance. That sounds reasonable until you put the phone in front of a packed group of sheep. Every animal has another animal a body length away. Centroids are not enough signal.

The better signal is overlap.

Two boxes on the same animal overlap heavily between adjacent frames. Two boxes on neighbouring animals usually do not. So the association cost now prefers overlap first and only falls back to distance when there is no meaningful overlap:

if iou > 0 {
    cost = 1 - iou
} else {
    cost = 1 + normalisedDistance
}

That one change makes the tracker much less willing to jump to a neighbour just because its centre is nearby. It also means tracks need to keep the whole observed box, not just a centre point and longest side. Slightly more state, much better truth.

The useful lesson

Most ML product bugs are not model bugs.

The detector can be good enough and the product can still feel broken because the tracking layer lies, the UI overstates certainty, or the recovery path optimizes for the wrong thing.

In this case the fix was not a new model, a heavier tracker, or a dependency. It was separating prediction from presentation, then using the geometry the video already had.

Boring, local, testable. My favourite kind of AI fix.

Development Notes: Navigation UX Is a State Problem

Todd Sullivan — Mon, 31 Aug 2026 17:21:46 +0000

Recent FlatNav development has been a useful reminder that navigation UX is mostly a state-management problem pretending to be a design problem.

The visible part is the map, route, buttons, and labels.

The real work is deciding what the user needs to know right now, what can wait, and what should never be shown at all.

Good navigation software has to stay calm while the user is not. That means the interface needs to handle uncertainty without pushing it onto the person using it.

The useful questions are simple:

what state is the journey in?
what is the next useful action?
has the user changed context?
does the UI need to interrupt, confirm, or stay quiet?
is this information actionable now, or just noise?

Most improvements come from reducing interpretation.

If the user has to stop and decode the screen, the app has already leaked complexity. If a feature adds more choices than it removes, it probably made the system feel smarter while making the user slower.

That is why the best development cycles often look small from the outside: clearer states, fewer edge-case dead ends, tighter feedback after actions, better defaults, less UI competing with the next step.

There is a lazy engineering lesson in that.

Before adding another control, another setting, or another explanation, ask whether the state model is doing its job. Often the fix is not more interface. It is making the existing state clearer, deleting an unnecessary decision, or moving logic into the product so the user does not have to carry it.

Navigation products get better when they get quieter.

Development Notes: Building AI for Menopause Support

Todd Sullivan — Mon, 31 Aug 2026 17:21:45 +0000

Recent development work on a menopause intelligence app has been a good reminder that useful AI in health-adjacent software is mostly not about model novelty.

The interesting part is the product layer around the intelligence.

Menopause data is messy in the way real human data is messy: symptoms overlap, patterns take time to emerge, user input is incomplete, and the app has to be careful not to imply certainty where there is none.

That creates a few practical engineering constraints:

keep the language plain
separate tracking from interpretation
avoid over-confident output
design for partial information
make escalation to human support obvious where needed
treat privacy and consent as part of the core UX, not paperwork

The useful technical work is in the guardrails and flow design.

A lot of AI demos assume the best input: a clean prompt, a clear goal, and a user who knows exactly what to ask. Real apps get the opposite. Users arrive with fragments: symptoms, worries, timings, medication changes, sleep disruption, mood changes, and half-remembered appointment notes.

So the system has to help structure the problem before it tries to answer anything.

That means boring but important work: tighter onboarding, clearer state, better defaults, careful copy, and prompts that make uncertainty explicit rather than hiding it behind confident wording.

The App Store release process also forces discipline. Privacy wording, support flows, screenshots, review metadata, and boundary-setting all push the same question: does this behave like something people can safely understand, or just like a clever demo?

That is the real development lesson here.

In sensitive domains, the intelligence layer should be small, explainable, and restrained. The product around it does most of the trust-building.

The RLS Policy Was Correct. The Write Was Still Wrong.

Todd Sullivan — Tue, 25 Aug 2026 08:08:16 +0000

The RLS Policy Was Correct. The Write Was Still Wrong.

I spent a chunk of last week doing the least glamorous kind of security work: reading database policies line by line and asking, "what can the client still put in this row?"

The useful bug pattern was simple:

Row-level security said which rows a user could touch, but not always which values they were allowed to write.

That difference matters a lot in offline-first apps, because the mobile client necessarily syncs whole records back to the server. If the database policy only checks tenant scope, a malicious or buggy client can sometimes write fields that were supposed to be server-owned.

A simplified version:

CREATE POLICY org_isolation ON inspections
  FOR ALL
  TO authenticated
  USING (org_id = my_org_id());

Looks reasonable. A user can only see and edit inspections in their own org.

But for writes, that policy is incomplete. Without a WITH CHECK, the database is mostly answering "is this row in your org?" It is not necessarily answering "are you allowed to set approval_status = 'approved'?"

That distinction turned into a real class of findings:

an org member could mark an inspection approved without an approval record
approval rows could be forged with someone else's reviewer id/name/role
onboarding could attach a fresh account to an existing org id if the RPC accepted caller-supplied ids too literally
client-writable billing/quota columns were one PATCH away from entitlement nonsense

None of those are exotic exploits. They are boring trust-boundary mistakes. The client had too much ability to describe facts the server should have decided.

The fix was not "add more app-side checks." App-side checks are UI. They help honest users, not the boundary.

For approval decisions, the server now adjudicates the thing that matters:

CREATE OR REPLACE FUNCTION record_approval(
  p_inspection_id uuid,
  p_decision text,
  p_comment text DEFAULT NULL
)
RETURNS inspection_approvals
LANGUAGE plpgsql
SECURITY DEFINER
AS $$
DECLARE
  v_org_id uuid;
  v_role text;
BEGIN
  SELECT org_id, role
  INTO v_org_id, v_role
  FROM profiles
  WHERE id = auth.uid();

  IF v_role NOT IN ('supervisor', 'admin') THEN
    RAISE EXCEPTION 'Supervisor or admin role required';
  END IF;

  INSERT INTO inspection_approvals (
    inspection_id, org_id, reviewer_id, reviewer_role, decision, comment
  )
  VALUES (
    p_inspection_id, v_org_id, auth.uid(), v_role, p_decision, p_comment
  );

  UPDATE inspections
  SET approval_status = p_decision
  WHERE id = p_inspection_id AND org_id = v_org_id;
END;
$$;

The caller can request a decision. It cannot choose the reviewer, role, org, or audit identity. Those come from auth.uid() and server-side profile state.

For the offline sync path, I kept the boring path boring. Inspectors still need to create and submit inspections with no signal, then sync later. So the value gate became narrower: clients can write draft/submitted states, but approved/rejected must match the latest server-recorded approval decision.

That's the part I like. The fix did not require turning the whole app into RPC soup. Only the authority-bearing transition moved server-side.

The takeaway: RLS is not one switch. Tenant isolation is only the first question.

For every writable field, ask:

Can the client choose this value?
If yes, is tenant scope enough?
If no, where is the server-side writer?

Most bugs I found were hiding between question 1 and question 2. The policy knew whose row it was. It did not always know whose decision it was.

I Used Claude to Fill Inspection Forms by Voice, Without Building a Voice Stack

Todd Sullivan — Tue, 18 Aug 2026 08:03:21 +0000

I Used Claude to Fill Inspection Forms by Voice, Without Building a Voice Stack

I just added voice form fill to GroundCheck, and the most useful decision was the boring one: I did not build a voice assistant.

The feature is scoped very tightly. An inspector is already inside one inspection form. They tap Fill with Voice, dictate something like:

all four guys had hard hats and vests on, scaffold tags were current, one trip hazard by the east gate, no incidents

The app sends that transcript plus the visible form schema to a Supabase Edge Function. Claude maps the narration onto the matching fields. The app shows a review sheet. The inspector accepts or rejects each proposed value.

That is it. No chat UI. No cross-report memory. No “AI assistant for the whole day.” Just spoken notes → structured form fields.

The client-side rule is deliberately small:

const FILLABLE_TYPES = new Set(["text", "checkbox", "multiple_choice", "rating"]);

export function fillableQuestions(section, responses) {
  return section.questions.filter(
    (q) =>
      FILLABLE_TYPES.has(q.type) &&
      (!q.show_when || evaluateCondition(q.show_when, responses))
  );
}

Photos and signatures are excluded because narration cannot produce them. Conditional questions hidden by show_when are excluded too. That matters on a whole-form fill: if the user says the vest check passed, Claude should not even see the “why did hi-vis fail?” follow-up field.

The Edge Function builds a JSON schema dynamically from the fields in scope. Choice fields become enums. Rating fields become enums from the score map. Unknown fields and illegal values are dropped again server-side after the model returns, because this eventually lands on a safety-compliance record and “the schema probably handled it” is not good enough.

I also kept the API key entirely server-side. The mobile app never sees the Anthropic key. The function checks auth, org tier, and a per-org monthly cap before making the paid call, then logs usage rows with input/output token counts.

The nice lazy bit: speech capture uses the iOS keyboard dictation button in an autofocused multiline text field. No speech-recognition dependency, no native module, no new permission strings, no rebuild just to test the first version. If field testing proves people need a big in-app hold-to-talk button, the transcript box can be swapped later. The contract downstream is still just text.

The review step is non-negotiable. Claude proposes; the human applies. Already-answered fields show as changes instead of silent overwrites. Identical values are dropped. Everything is displayed in form order so it feels like reviewing the inspection, not debugging JSON.

The part I like most is that the whole feature is smaller because it refused to be a general assistant. It only knows about the form on screen, only sees fields that can currently be answered, and only writes after a human says yes.

That shape is usually where LLM features get useful: narrow input, constrained output, boring gates, human review at the boundary.

My On-Device ML Feature Was Registered, But Never Scheduled

Todd Sullivan — Tue, 11 Aug 2026 08:06:21 +0000

I spent part of this week fixing a mobile AI feature where the model code was not the problem.

The app has a small on-device personalization loop: collect daily logs, merge in HealthKit metrics, retrain a local model periodically, and use that state to drive predictions and nudges. Nothing huge. Just the kind of background ML plumbing that makes a feature feel alive instead of static.

The bug was simpler and more annoying:

I had registered the background tasks.

I had handlers for them.

I had rescheduling inside each handler.

But the first task was never submitted.

So nothing ever ran.

Registration is not scheduling

This is the shape that matters with BGTaskScheduler:

func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]? = nil
) -> Bool {
    registerBackgroundTasks()
    submitBackgroundTasks()
    return true
}

That second line is the one I was missing.

registerBackgroundTasks() tells iOS which identifiers this process can handle. It does not put work on the queue. Each handler can reschedule itself after it runs, but something has to submit the first request or the chain never starts.

The final version is deliberately boring:

private func submitBackgroundTasks() {
    scheduleHealthRefresh()
    scheduleMLRetrain()
    scheduleEngagementDaily()
    scheduleWeeklyInsight()
}

And the retrain request is just a normal BGProcessingTaskRequest:

func scheduleMLRetrain() {
    let request = BGProcessingTaskRequest(
        identifier: AppConstants.mlRetrainTaskID
    )
    request.earliestBeginDate = Date(
        timeIntervalSinceNow: 86400 * Double(AppConstants.retrainingIntervalDays)
    )
    request.requiresExternalPower = true
    request.requiresNetworkConnectivity = false
    try? BGTaskScheduler.shared.submit(request)
}

That is the whole fix. Not a new scheduler abstraction. Not a custom job runner. Just submit the work at launch and let iOS deduplicate by identifier.

The other trap: background queues and SwiftData

The second issue was actor isolation. BGTaskScheduler invokes handlers on a background queue. My persistence layer uses container.mainContext, so every handler that touches SwiftData needs to hop back to the main actor.

private func handleMLRetrain(task: BGProcessingTask) {
    task.expirationHandler = { task.setTaskCompleted(success: false) }

    Task { @MainActor in
        let context = PersistenceController.shared.container.mainContext
        let descriptor = FetchDescriptor<DailyLog>(
            sortBy: [SortDescriptor(\DailyLog.date)]
        )
        let logs = (try? context.fetch(descriptor)) ?? []
        try? await MLModelManager.shared.trainPersonalisedModel(from: logs)
        scheduleMLRetrain()
        task.setTaskCompleted(success: true)
    }
}

Again, boring. But this is exactly where AI features usually break: not in the model, but in the lifecycle around the model.

One shared path for HealthKit updates

I also pulled the HealthKit merge into one shared function so the scheduled refresh and HealthKit background delivery cannot drift apart:

@MainActor
static func fillTodaysLogFromHealthKit() async {
    let metrics = await HealthKitManager.shared.fetchDailyMetrics(for: Date())
    let context = PersistenceController.shared.container.mainContext
    let log = DailyLog.findOrCreate(for: Date(), in: context)

    if log.hrv == nil               { log.hrv = metrics.hrv }
    if log.restingHeartRate == nil  { log.restingHeartRate = metrics.restingHeartRate }
    if log.stepCount == nil         { log.stepCount = metrics.stepCount }
    if log.sleepDuration == nil     { log.sleepDuration = metrics.sleepDuration }

    try? context.save()
}

The important detail is that manual user-entered values win. Background sync should fill gaps, not overwrite what someone typed by hand.

The tests that caught the boring failures

A few regression tests now pin the things that looked too small to test before:

one DailyLog row per calendar day, because duplicate daily rows poison the training set
future-dated rows must not be mistaken for today
CloudKit-backed SwiftData schema must load, because a bad relationship can crash the app on launch
engagement notifications default to enabled, because UserDefaults.bool(forKey:) returns false for an unset key
pending Watch entries drain by payload type instead of clearing the whole queue

That last one matters because the Watch queue contains two shapes in one array: symptom logs and wellbeing check-ins. The old reconciliation path could drain symptoms and silently delete check-ins that had not been processed yet.

Silent data loss in a health-adjacent logging app is not a UX bug. It is a trust bug.

The takeaway

On-device AI features are mostly normal mobile engineering with a model in the middle.

The model can be fine while the product is still broken because:

the background task was registered but never scheduled
the handler touched persistence from the wrong execution context
sync overwrote user-entered data
duplicate daily rows polluted the training input
a queue drain deleted the wrong payload type

None of that is glamorous. All of it matters.

If a model retrains locally but the scheduler never wakes up, you did not ship personalization. You shipped a nice code path nobody calls.

That is the bar I keep coming back to with practical AI work: not "does the model exist?" but "does the system around it actually keep running after the app leaves the foreground?"

The mAP50 That Lied to Me: A Debugging Story About On-Device Safety AI

Todd Sullivan — Tue, 04 Aug 2026 08:02:43 +0000

I'm building GroundCheck, an offline-first field inspection app for construction safety managers. One of the things we bet on early was on-device hazard detection — a YOLO model that runs entirely on the phone, no internet required, spotting missing hardhats and safety vests in a photo the moment you take it. No competitor in this space does that; everyone else ships cloud-only detection that falls over the second you lose signal, which on a job site is often.

This is the story of a metric that looked great, was actually badly broken, and what it took to find out — plus a training run that got derailed twice by things that had nothing to do with the model.

A good score, hiding a broken detector

Our first real model, trained on a 10-class taxonomy (hardhats, vests, masks, cones, and a few others), hit 0.510 mAP50. That cleared our internal bar of 0.50. Model shipped, feature declared done, on to the next thing.

Then I ran it against a simple test photo — six construction workers, standard job-site shot, some in hardhats, all in hi-vis vests. The model found zero vests. Not "low confidence," zero. It also hallucinated a hardhat on someone who wasn't wearing one.

That's a bad look for a safety app, and it directly contradicted the "0.510, above target" number sitting in my training log. So what happened?

mAP50 is a metric that's easy to be misled by, if you only ever look at the macro-average. Ours was averaged across 10 classes with wildly different underlying performance:

Class	mAP50
Hardhat	0.924
Face-guard	0.850
Safety Vest	0.343
Mask	0.267

A model that nails hardhats and face-guards can drag a broken vest detector up to a respectable-looking average. The number wasn't wrong, exactly — it just wasn't answering the question I actually cared about, which was "does this work for the classes that matter most."

The lesson: a single aggregate metric is a summary, not a verdict. If your classes aren't roughly equally easy and equally important, look at the per-class breakdown before you trust the headline number — and if you can, run the model against a real example and look at it. The zero-vest failure took me thirty seconds to spot with my own eyes; it was invisible in the metric that was supposed to be measuring exactly that.

Fixing it

Two changes: I dropped a pair of classes (NO-Safety Vest / NO-Mask) that had too little consistent training data to ever work — the app now infers "no vest" from a person detected with no vest box overlapping them, rather than trying to detect the absence directly, which turned out to be a much harder learning problem than it needed to be. Then I went and found three new open datasets specifically to beef up the underrepresented classes, roughly tripling the validation volume for the worst offenders.

Retrained. Same architecture (YOLOv8s), same target: beat 0.510, and this time, verify the classes that actually matter.

The training run that fought back

This part has nothing to do with the model and everything to do with running long training jobs on hardware that occasionally has other ideas. Two things went wrong that weren't ML problems at all:

First, the machine rebooted mid-training (unrelated maintenance), killing the process at epoch 134 of a planned 200. Recoverable — checkpoints survive a kill, just resume from last.pt.

Second, and more interesting: the auto-resume script I'd written specifically to handle crashes like that had a bug. yolo detect train resume=True model=<checkpoint> is supposed to pick up exactly where you left off. In my environment, it silently didn't — it fell back to Ultralytics' built-in 4-image toy dataset instead of my real 32,000-image training set, and happily reported success. No error, no warning. Just a training run quietly doing nothing useful for however long it took me to notice the loss curves looked suspiciously clean.

I caught it because the process count looked wrong — one training job should mean one Python process, and I was staring at twenty-five. Followed that thread back to the argument parser silently discarding my dataset config and substituting its own defaults.

The fix that actually worked: stop trusting the "smart" resume, and do a plain, boring, explicitly-specified restart every time — spell out every argument, verify the log shows my actual dataset path before walking away, don't rely on a flag that's supposed to infer everything for you. Boring and verified beats clever and silent, every time a background process fails quietly instead of loudly.

Knowing when to stop

The retrained model plateaued around epoch 177 at 0.682 mAP50, still short of a stretch goal of 0.80. I tried a short "continuation" run — warm-starting a fresh 40-epoch pass from the final weights, hoping for a few more easy points.

It made things worse, immediately and consistently — accuracy dropping every single epoch. Resetting the full learning-rate schedule on top of a model that had already converged over 178 epochs was too big a shock; instead of fine-tuning, it started forgetting. Killed it after 4 epochs once the trend was unambiguous, and kept the original checkpoint. Sometimes the right move with a training run — like with a lot of optimization problems — is recognizing you've already found a good stopping point, and further tinkering is more likely to hurt than help.

Validating for real, this time

Same two checks as before, run properly this time:

Per-class mAP50 on the full validation set:

Class	mAP50	Before
Hardhat	0.855	0.924
Safety Vest	0.684	0.343
Mask	0.555	0.267
vehicle	0.425	(new weakest)

The test photo. Same six workers as before. This time: 4 of the visible vests correctly detected, all 3 worn hardhats correctly detected, bare heads correctly flagged. Not perfect — a couple of vests still missed, confidence scores are moderate rather than sky-high — but a real, visible fix. The classes that were quietly broken before are now genuinely functional.

The takeaway

Two unglamorous habits saved this retrain from shipping a broken detector twice: refusing to trust a single aggregate metric without a per-class breakdown, and refusing to trust an automated recovery mechanism without checking, concretely, that it actually did what it claimed to do. Neither is exciting. Both are the difference between a safety feature that works and one that just looks like it does in a dashboard nobody double-checked.

GroundCheck is an offline-first inspection app for construction safety managers, built around on-device AI hazard detection that works with zero connectivity — because job sites don't always have signal, and safety checks shouldn't wait for a cloud API.

My Local AI Assistant Got Worse When I Remembered Too Much

Todd Sullivan — Tue, 21 Jul 2026 08:06:35 +0000

My Local AI Assistant Got Worse When I Remembered Too Much

I moved a personal AI assistant onto a small local model last week and immediately hit a boring problem: the model was fine, but my memory layer was not.

The old version persisted raw conversation history and replayed it back into the prompt. That worked well enough with hosted models. Then I pointed the same app at a local OpenAI-compatible server running Qwen3-4B-4bit through Swama on the Mac mini.

After 196 accumulated messages, the assistant started doing the classic small-model failure mode: parroting its own previous replies, over-weighting stale context, and sounding less useful the more “memory” I gave it.

The fix was not a vector database. It was deleting most of the memory.

I split memory into two different things:

short-term conversation state
long-term user facts

Short-term history now stays in RAM only. It resets after an idle gap, and it has a hard cap so a marathon session cannot poison every future turn.

self.session_memory: Dict[str, ConversationBufferMemory] = {}
self._last_activity: Dict[str, float] = {}

idle_limit = int(os.getenv("SESSION_IDLE_MINUTES", "120")) * 60
last = self._last_activity.get(user_id)

if last is not None and now - last > idle_limit:
    del self.session_memory[user_id]

Long-term memory is not chat logs. It is a small list of distilled facts: preferences, people, devices, recurring activities, that kind of thing. Maximum 30 facts per user.

_FACT_EXTRACTION_PROMPT = """
Update the fact list. Add only stable, personal facts worth remembering across
conversations: preferences, interests, people, pets, places, devices, recurring
activities. Ignore small talk, one-off requests, and anything the assistant said
about itself.

Return ONLY a JSON array of strings.
"""

The fact extraction runs in a background thread after each exchange. The chat path should not wait for memory housekeeping.

threading.Thread(
    target=self._extract_facts,
    args=(user_id, msgs[-2].content, msgs[-1].content),
    daemon=True,
).start()

I also deliberately use a hosted model for the distillation step. The local 4B model is good enough for fast interaction, but long-term memory cleanup is one of those places where quality matters more than latency. It is off the response path anyway.

The other local-model tweak was tool bias. Small models are much more likely to answer from stale weights even when tools exist, especially if the system prompt says anything like “use your knowledge first.” So the Swama handler adds a blunt override for live data:

_TOOL_BIAS = (
    " IMPORTANT OVERRIDE: for anything happening NOW - weather, sea or"
    " kitesurfing conditions, device/home status, prices, news, live data"
    " of any kind - you MUST call the matching tool. Never answer those"
    " from memory. /no_think"
)

Qwen3 also emits <think> blocks even when asked not to, including mid-stream after tool calls, so the streaming handler strips those tags incrementally. Not glamorous, but necessary if you do not want raw reasoning markup leaking into a voice/chat UI.

The useful lesson was this:

Memory is not “more previous tokens.”

For a personal assistant, raw transcript replay is the cheapest thing to build and one of the easiest ways to make the system worse. The assistant needs enough recent context to hold the current conversation, plus a tiny set of stable facts that survive across sessions.

Everything else is prompt pollution with a better name.

Source: Recent personal assistant backend work: Swama local model support, Qwen3-4B-4bit via an OpenAI-compatible endpoint, RAM-only session history, 2-hour idle reset, 200-message cap, background fact extraction, and 30 persisted user facts.
Tags: ai, python, llm, devops
Status: published

I Made My Voice Agent Feel Faster by Streaming Sentences, Not Audio

Todd Sullivan — Tue, 14 Jul 2026 08:05:39 +0000

The annoying thing about voice agents is that “the model is fast” does not mean the experience is fast.

I had a small voice assistant running on a local device, talking to a hosted chat backend. The actual LLM call was only one part of the wait. The full path looked more like this:

wake word detection
speech recognition
authenticated /chat call
model response
local TTS synthesis
audio playback

If you wait for step 4 to finish before starting step 5, the user hears nothing until the entire reply is done. That feels dead, even when the backend is technically fine.

So I changed the contract. The hardware client now calls the chat endpoint with stream_tts: true:

response = self.session.post(
    f"{CHAT_API_BASE}/chat",
    json={"message": message, "stream_tts": True},
    timeout=30,
    stream=True,
)

The backend yields text chunks as they arrive from the model. The device keeps a small buffer, splits complete sentences, and starts synthesizing each sentence immediately:

_SENTENCE_BOUNDARY_RE = re.compile(r'(?<=[.!?])\s+')

def split_complete_sentences(buffer: str) -> tuple[list[str], str]:
    *sentences, remainder = _SENTENCE_BOUNDARY_RE.split(buffer)
    return [s.strip() for s in sentences if s.strip()], remainder

That is deliberately boring. Not phoneme streaming. Not a custom audio protocol. Just sentence-level pipelining.

The next useful bit was overlapping synthesis and playback. A single background worker waits for synthesized WAV files and plays them in order, while a one-worker ThreadPoolExecutor starts rendering the next sentence as soon as it is complete.

for sentence in sentences:
    future = synth_executor.submit(self.tts_engine.synthesize, sentence)
    playback_queue.put((sentence, future))

That removed the worst gap: “sentence one finished playing, now start thinking about sentence two’s audio.” The hardware now does the obvious thing a human expects — keep talking.

I also cut backend time-to-first-token by doing less. For this conversational path, I turned off extended model thinking:

_NO_THINKING = types.ThinkingConfig(thinking_budget=0)

And I stopped advertising Google Search on every request. The search tool is only added when the prompt smells like it needs current/external information. Most turns do not.

if self._needs_search(built_contents):
    all_functions.append(self.google_search)

The result was about a 5x cut in chat time-to-first-byte for the common path, plus a much better perceived response because speech starts before the full answer exists.

The lesson was not “stream everything.” It was smaller than that:

stream at the boundary the product can actually use
overlap the slow local work with the slow network work
do not give the model tools or reasoning budget unless the turn needs them
log chunks, sentence counts, gaps, and total time so you can see where the pause moved

Voice agents do not need heroic architecture to feel better. Sometimes the fix is a regex, a queue, and deleting the expensive defaults.

Silent Data Loss Is Worse Than a Failed Export

Todd Sullivan — Tue, 07 Jul 2026 08:04:31 +0000

This week I fixed a set of bugs in a mobile export pipeline where the scary part was not that the export could fail.

It was that it could succeed while quietly omitting data.

The pipeline builds an internal assessment model, validates it, serializes it into an RdSAP XML export, packages evidence, writes a checksum, and records an audit log. The deterministic exporter work was already in place. Same assessment in, same bytes out.

But deterministic output only helps if the model you serialize has not already lost meaning.

A code review turned up a few places where valid survey answers were being collapsed into “nothing to export”.

The dangerous `null`

The first bug was a token mapper:

fuelToken(value): string | null

A valid answer, Other, was returning null. That meant mandatory fields like main fuel and water heating fuel could disappear from the XML instead of being exported as the existing fallback token.

The fix was boring:

Other -> "other"

Boring is good. Boring means the field is present and downstream validation can do its job.

The second version of the same bug was draught proofing. Unknown was a valid answer in the survey UI, but there is no clean numeric RdSAP value for it. The old behavior silently omitted the value. The fixed behavior raises a non-blocking validation warning instead.

That distinction matters:

missing because the assessor never answered: omit
present but not representable in this export format: warn
present and representable: export

Those are three different states. Treating them all as null is how data loss gets a polite API.

`false` is not “unanswered”

The more subtle bug was boolean handling.

The original helper effectively collapsed explicit false and “never answered” into the same output. So a recorded “no” for fields like cylinder present, immersion heater, or insulation could become indistinguishable from a question that was never asked.

That is a bad trade. In an assessment/export system, “no” is data.

The helper now returns a tri-state value:

boolean | null

And the serializer emits true/false for real answers, only omitting the element when the value is genuinely unanswered.

The test is tiny, which is exactly the point:

it("distinguishes an explicit false answer from never-answered", () => {
  const a = buildAssessment({
    inspection,
    responses: mkResponses({ ...validValues, epc_cylinder_present: false }),
    media: [],
  });

  expect(a.hotWater.cylinderPresent).toBe(false);
});

No giant test harness. Just one assertion that prevents the bug from coming back.

Do not guess from raw JSON

There was also a LiDAR floor-area parser that tried too hard.

If the JSON had a total, use it. Fine.

If total was missing, the old fallback scanned the raw JSON for a number. That could accidentally grab an individual room area and treat it as the property's total floor area.

That is worse than failing.

The fixed version returns null when total is missing. No heroic guessing. No “probably 20m² because I found 20 somewhere in the blob”.

expect(a.property.floorAreaM2).toBeNull();

Again: boring test, useful guarantee.

Audit failures before they escape

One more bug was outside serialization. getExporter() could throw before the main try/catch, which meant a format lookup failure skipped the audit-trail write that other failures recorded.

That is the kind of edge case that makes production debugging annoying. The export failed, but the failure path forgot to leave evidence.

The fix was just moving the lookup inside the guarded path so the audit log is written consistently.

The AI-assisted part

This is where AI-assisted development is actually useful for me.

Not “write the whole exporter and hope”. More like:

build the deterministic seam
have Claude review the boring mapping logic
turn every discovered ambiguity into a narrow test
keep the deliberate deferrals explicit

This pass added 6 focused tests. The export suite moved from 119 to 125 tests, with the targeted export tests passing locally: 3 suites, 23 tests.

There is still one known bigger issue left alone: validation is currently too RdSAP-specific for every export format. That needs per-format validation profiles. I did not jam it into this fix because it is a different change.

That is another useful rule when working with AI in real codebases: fix the silent data loss now, leave the larger design change visible, and do not let the model turn a bug fix into a rewrite.

The best export pipeline is not the cleverest one.

It is the one that refuses to quietly lie.

The Exporter Was Easy. Making It Deterministic Was the Work.

Todd Sullivan — Tue, 30 Jun 2026 08:02:27 +0000

The Exporter Was Easy. Making It Deterministic Was the Work.

I spent this week building a mobile export pipeline for UK energy assessment data.

The output is not glamorous: take completed survey responses, turn them into an RdSAP XML file, package the evidence, write a manifest, calculate a checksum, and keep a local audit trail.

The interesting part was not generating XML. The interesting part was making the whole thing deterministic and testable before adding more export formats.

The shape ended up like this:

responses
  -> buildAssessment()
  -> validateAssessment()
  -> exporter.serialize(assessment)
  -> sha256
  -> write package
  -> persist audit log

The exporter itself is deliberately boring:

serialize(assessment): string

No file system. No clock. No database. No random IDs. Same assessment in, same bytes out.

That one rule makes the rest of the pipeline much easier to reason about. If the XML changes, it changed because the input changed or the serializer changed. Not because a timestamp moved, a native module behaved differently, or a test environment had a different document directory.

Side effects at the edge

The orchestrator is the only layer allowed to do side effects:

const body = exporter.serialize(assessment);
const checksum = await deps.sha256Hex(body);
const pkg = await writeExportPackage(
  { inspectionId, exporter, assessment, body, checksum },
  deps.fs
);
await deps.persistLog(log);

Those dependencies are ports:

export type ExportDeps = {
  fs: FileSystemPort;
  sha256Hex: (input: string) => Promise<string>;
  persistLog: (entry: ExportLogEntry) => Promise<void>;
};

In production, they map to Expo file system, Expo crypto, and a local SQLite audit table.

In tests, they are an in-memory file system, Node crypto, and an array log sink.

That was not architectural theatre. Under jest-expo, native modules are often partially unavailable. documentDirectory can be undefined. Crypto enums can be missing. If the export logic directly imports and touches those modules, the “end-to-end” test either becomes a mock festival or stops covering the actual path.

With ports, the test runs the real pipeline:

const { deps, files, logs } = memoryDeps();
const run = await runExport(input(), deps);

const body = files.get(expectedPath)!;
expect(run.result.checksum).toBe(
  createHash("sha256").update(body, "utf8").digest("hex")
);

That test proves a few things at once:

the package path is correct
the written XML is exactly the pure exporter output
the checksum is a real SHA-256 of the bytes
the manifest references the same checksum
one audit entry is written

There are also tests for evidence attachment copying, missing attachment sources, validation-blocked exports, checksum reproducibility, and a second JSON archive exporter running through the same pipeline.

Validation blocks, but still logs

One design choice I care about: failed validation writes an audit record too.

If a mandatory field is missing, the export does not create files. But the attempt is still logged with success: false, validation counts, schema version, and no checksum.

That makes the export feature behave like a real operational system rather than a button that either emits a file or silently refuses.

Why this matters for AI-built software

This is the seam I want when using AI heavily in engineering work.

LLMs are very good at producing a first version of “convert this shape into that schema.” They are much less reliable if the codebase lets schema mapping, file IO, logging, hashing, and UI state collapse into one blob.

The fix is not to use less AI. The fix is to give the code stronger boundaries:

pure transformations for the model or human to edit safely
deterministic outputs that can be snapshot-tested or checksummed
injected IO so tests exercise the pipeline without native dependencies
audit trails for both success and blocked paths
format-specific exporters behind a small registry

Once those seams exist, adding the next export format is not a rewrite. It is another serializer plus a few tests.

That is the part I keep finding in real AI-assisted development: the model can help move fast, but the architecture has to make fast changes safe.

Source: Recent mobile export module work: RdSAP XML export pipeline, pure serializers, injectable IO ports, SHA-256 packaging, local audit logging, and 24 export tests.
Tags: ai, testing, typescript, mobile
Status: published

AI Features Need Product Edges, Not Just Better Prompts

Todd Sullivan — Tue, 23 Jun 2026 08:03:02 +0000

Most AI features don't fail because the model is bad.

They fail because everything around the model is treated like a demo.

This week I was tightening an iOS workout app that uses Claude through Supabase Edge Functions. The model part is straightforward: send training context, get a structured exercise or plan back, validate it, write it into SwiftData.

The less glamorous work was the part that makes it feel like an actual product:

monthly AI credit balance
offline handling
auth token storage
disabled states while generation is running
different rules for “add exercise” vs “swap this exercise”
tests for the boring edge cases

That is where most AI app quality lives.

One button, several states

The UI has a small “fill with AI” affordance on the exercise editor. Underneath it, the button is not just “call endpoint”. It has to know whether suggestion is currently allowed:

var canSuggest: Bool {
    guard !isLoadingAI,
          NetworkMonitor.shared.isOnline,
          (creditsRemaining ?? 1) > 0
    else { return false }

    // Swap mode can use the original exercise as context.
    // Add mode needs the typed name as a hint.
    return isSwapMode || !name.trimmingCharacters(in: .whitespaces).isEmpty
}

That little predicate is doing a lot of product work.

If the user is offline, don't let them tap into a doomed network request.
If a generation is already running, don't double-spend.
If they have zero credits, don't pretend the feature is available.
If they are swapping an existing exercise, don't force them to type a name because the old exercise is already useful context.

The model does not care about any of this. The user does.

Credits should be part of the response

The suggestion response includes the updated credit count:

struct SuggestedExercise: Decodable {
    let name: String
    let briefDescription: String
    let muscleGroup: String
    let sets: Int
    let repTargetLow: Int
    let repTargetHigh: Int
    let restSeconds: Int
    let isDualDumbbell: Bool
    let creditsRemaining: Int
}

Then the view model updates local state immediately after a successful fill:

creditsRemaining = result.creditsRemaining
aiFilledFields = true

That avoids the classic AI-product weirdness where the backend knows the user has spent a credit but the UI keeps showing stale allowance until the next refresh.

It is also easier to test. In the app tests, the credit transition is explicit:

func testAIDisabledWhenCreditsReachZero() {
    let vm = ExerciseEditViewModel(mode: .add(makeDay()), context: container.mainContext)
    vm.name = "Row"
    vm.creditsRemaining = 1
    XCTAssertTrue(vm.canSuggest)

    vm.creditsRemaining = 0
    XCTAssertFalse(vm.canSuggest)
}

There are 13 tests just around the exercise edit view model, plus separate coverage for offline error mapping. Not because this is academically interesting, but because this is the stuff that breaks in front of real users.

Offline is not a server error

Another small detail: connectivity failures are mapped separately from backend failures.

static let connectivityCodes: Set<URLError.Code> = [
    .notConnectedToInternet,
    .networkConnectionLost,
    .timedOut,
    .cannotConnectToHost,
    .dataNotAllowed,
]

The resulting message is intentionally plain:

"You're offline. Reconnect to use AI features."

No “unexpected server response”. No fake intelligence. Just tell the user what happened.

The actual lesson

Shipping AI features is mostly normal software engineering with a probabilistic dependency in the middle.

The model call matters, but the surrounding contract matters more:

Can the user invoke it right now?
What happens if the network dies?
Is usage counted consistently?
Does the UI reflect server state immediately?
Are the edge cases testable without calling the model?

Once those pieces are in place, the AI feature stops feeling like a prompt wired to a button and starts feeling like part of the app.

That is the bar I keep coming back to: not “does the model answer?” but “does this survive normal product reality?”

DEV Community: Todd Sullivan

The Tracking Bug Wasn't the Model. It Was the Dot Lying.

Prediction is not presentation

Centroid distance breaks in a flock

The useful lesson

Development Notes: Navigation UX Is a State Problem

Development Notes: Building AI for Menopause Support

The RLS Policy Was Correct. The Write Was Still Wrong.

The RLS Policy Was Correct. The Write Was Still Wrong.

I Used Claude to Fill Inspection Forms by Voice, Without Building a Voice Stack

I Used Claude to Fill Inspection Forms by Voice, Without Building a Voice Stack

My On-Device ML Feature Was Registered, But Never Scheduled

Registration is not scheduling

The other trap: background queues and SwiftData

One shared path for HealthKit updates

The tests that caught the boring failures

The takeaway

The mAP50 That Lied to Me: A Debugging Story About On-Device Safety AI

A good score, hiding a broken detector

Fixing it

The training run that fought back

Knowing when to stop

Validating for real, this time

The takeaway

My Local AI Assistant Got Worse When I Remembered Too Much

My Local AI Assistant Got Worse When I Remembered Too Much

I Made My Voice Agent Feel Faster by Streaming Sentences, Not Audio

Silent Data Loss Is Worse Than a Failed Export

The dangerous null

false is not “unanswered”

Do not guess from raw JSON

Audit failures before they escape

The AI-assisted part

The Exporter Was Easy. Making It Deterministic Was the Work.

The Exporter Was Easy. Making It Deterministic Was the Work.

Side effects at the edge

Validation blocks, but still logs

Why this matters for AI-built software

AI Features Need Product Edges, Not Just Better Prompts

One button, several states

Credits should be part of the response

Offline is not a server error

The actual lesson

The dangerous `null`

`false` is not “unanswered”