DEV Community: Todd Sullivan

My Local AI Assistant Got Worse When I Remembered Too Much

Todd Sullivan — Tue, 21 Jul 2026 08:06:35 +0000

My Local AI Assistant Got Worse When I Remembered Too Much

I moved a personal AI assistant onto a small local model last week and immediately hit a boring problem: the model was fine, but my memory layer was not.

The old version persisted raw conversation history and replayed it back into the prompt. That worked well enough with hosted models. Then I pointed the same app at a local OpenAI-compatible server running Qwen3-4B-4bit through Swama on the Mac mini.

After 196 accumulated messages, the assistant started doing the classic small-model failure mode: parroting its own previous replies, over-weighting stale context, and sounding less useful the more “memory” I gave it.

The fix was not a vector database. It was deleting most of the memory.

I split memory into two different things:

short-term conversation state
long-term user facts

Short-term history now stays in RAM only. It resets after an idle gap, and it has a hard cap so a marathon session cannot poison every future turn.

self.session_memory: Dict[str, ConversationBufferMemory] = {}
self._last_activity: Dict[str, float] = {}

idle_limit = int(os.getenv("SESSION_IDLE_MINUTES", "120")) * 60
last = self._last_activity.get(user_id)

if last is not None and now - last > idle_limit:
    del self.session_memory[user_id]

Long-term memory is not chat logs. It is a small list of distilled facts: preferences, people, devices, recurring activities, that kind of thing. Maximum 30 facts per user.

_FACT_EXTRACTION_PROMPT = """
Update the fact list. Add only stable, personal facts worth remembering across
conversations: preferences, interests, people, pets, places, devices, recurring
activities. Ignore small talk, one-off requests, and anything the assistant said
about itself.

Return ONLY a JSON array of strings.
"""

The fact extraction runs in a background thread after each exchange. The chat path should not wait for memory housekeeping.

threading.Thread(
    target=self._extract_facts,
    args=(user_id, msgs[-2].content, msgs[-1].content),
    daemon=True,
).start()

I also deliberately use a hosted model for the distillation step. The local 4B model is good enough for fast interaction, but long-term memory cleanup is one of those places where quality matters more than latency. It is off the response path anyway.

The other local-model tweak was tool bias. Small models are much more likely to answer from stale weights even when tools exist, especially if the system prompt says anything like “use your knowledge first.” So the Swama handler adds a blunt override for live data:

_TOOL_BIAS = (
    " IMPORTANT OVERRIDE: for anything happening NOW - weather, sea or"
    " kitesurfing conditions, device/home status, prices, news, live data"
    " of any kind - you MUST call the matching tool. Never answer those"
    " from memory. /no_think"
)

Qwen3 also emits <think> blocks even when asked not to, including mid-stream after tool calls, so the streaming handler strips those tags incrementally. Not glamorous, but necessary if you do not want raw reasoning markup leaking into a voice/chat UI.

The useful lesson was this:

Memory is not “more previous tokens.”

For a personal assistant, raw transcript replay is the cheapest thing to build and one of the easiest ways to make the system worse. The assistant needs enough recent context to hold the current conversation, plus a tiny set of stable facts that survive across sessions.

Everything else is prompt pollution with a better name.

Source: Recent personal assistant backend work: Swama local model support, Qwen3-4B-4bit via an OpenAI-compatible endpoint, RAM-only session history, 2-hour idle reset, 200-message cap, background fact extraction, and 30 persisted user facts.
Tags: ai, python, llm, devops
Status: published

I Made My Voice Agent Feel Faster by Streaming Sentences, Not Audio

Todd Sullivan — Tue, 14 Jul 2026 08:05:39 +0000

The annoying thing about voice agents is that “the model is fast” does not mean the experience is fast.

I had a small voice assistant running on a local device, talking to a hosted chat backend. The actual LLM call was only one part of the wait. The full path looked more like this:

wake word detection
speech recognition
authenticated /chat call
model response
local TTS synthesis
audio playback

If you wait for step 4 to finish before starting step 5, the user hears nothing until the entire reply is done. That feels dead, even when the backend is technically fine.

So I changed the contract. The hardware client now calls the chat endpoint with stream_tts: true:

response = self.session.post(
    f"{CHAT_API_BASE}/chat",
    json={"message": message, "stream_tts": True},
    timeout=30,
    stream=True,
)

The backend yields text chunks as they arrive from the model. The device keeps a small buffer, splits complete sentences, and starts synthesizing each sentence immediately:

_SENTENCE_BOUNDARY_RE = re.compile(r'(?<=[.!?])\s+')

def split_complete_sentences(buffer: str) -> tuple[list[str], str]:
    *sentences, remainder = _SENTENCE_BOUNDARY_RE.split(buffer)
    return [s.strip() for s in sentences if s.strip()], remainder

That is deliberately boring. Not phoneme streaming. Not a custom audio protocol. Just sentence-level pipelining.

The next useful bit was overlapping synthesis and playback. A single background worker waits for synthesized WAV files and plays them in order, while a one-worker ThreadPoolExecutor starts rendering the next sentence as soon as it is complete.

for sentence in sentences:
    future = synth_executor.submit(self.tts_engine.synthesize, sentence)
    playback_queue.put((sentence, future))

That removed the worst gap: “sentence one finished playing, now start thinking about sentence two’s audio.” The hardware now does the obvious thing a human expects — keep talking.

I also cut backend time-to-first-token by doing less. For this conversational path, I turned off extended model thinking:

_NO_THINKING = types.ThinkingConfig(thinking_budget=0)

And I stopped advertising Google Search on every request. The search tool is only added when the prompt smells like it needs current/external information. Most turns do not.

if self._needs_search(built_contents):
    all_functions.append(self.google_search)

The result was about a 5x cut in chat time-to-first-byte for the common path, plus a much better perceived response because speech starts before the full answer exists.

The lesson was not “stream everything.” It was smaller than that:

stream at the boundary the product can actually use
overlap the slow local work with the slow network work
do not give the model tools or reasoning budget unless the turn needs them
log chunks, sentence counts, gaps, and total time so you can see where the pause moved

Voice agents do not need heroic architecture to feel better. Sometimes the fix is a regex, a queue, and deleting the expensive defaults.

Silent Data Loss Is Worse Than a Failed Export

Todd Sullivan — Tue, 07 Jul 2026 08:04:31 +0000

This week I fixed a set of bugs in a mobile export pipeline where the scary part was not that the export could fail.

It was that it could succeed while quietly omitting data.

The pipeline builds an internal assessment model, validates it, serializes it into an RdSAP XML export, packages evidence, writes a checksum, and records an audit log. The deterministic exporter work was already in place. Same assessment in, same bytes out.

But deterministic output only helps if the model you serialize has not already lost meaning.

A code review turned up a few places where valid survey answers were being collapsed into “nothing to export”.

The dangerous `null`

The first bug was a token mapper:

fuelToken(value): string | null

A valid answer, Other, was returning null. That meant mandatory fields like main fuel and water heating fuel could disappear from the XML instead of being exported as the existing fallback token.

The fix was boring:

Other -> "other"

Boring is good. Boring means the field is present and downstream validation can do its job.

The second version of the same bug was draught proofing. Unknown was a valid answer in the survey UI, but there is no clean numeric RdSAP value for it. The old behavior silently omitted the value. The fixed behavior raises a non-blocking validation warning instead.

That distinction matters:

missing because the assessor never answered: omit
present but not representable in this export format: warn
present and representable: export

Those are three different states. Treating them all as null is how data loss gets a polite API.

`false` is not “unanswered”

The more subtle bug was boolean handling.

The original helper effectively collapsed explicit false and “never answered” into the same output. So a recorded “no” for fields like cylinder present, immersion heater, or insulation could become indistinguishable from a question that was never asked.

That is a bad trade. In an assessment/export system, “no” is data.

The helper now returns a tri-state value:

boolean | null

And the serializer emits true/false for real answers, only omitting the element when the value is genuinely unanswered.

The test is tiny, which is exactly the point:

it("distinguishes an explicit false answer from never-answered", () => {
  const a = buildAssessment({
    inspection,
    responses: mkResponses({ ...validValues, epc_cylinder_present: false }),
    media: [],
  });

  expect(a.hotWater.cylinderPresent).toBe(false);
});

No giant test harness. Just one assertion that prevents the bug from coming back.

Do not guess from raw JSON

There was also a LiDAR floor-area parser that tried too hard.

If the JSON had a total, use it. Fine.

If total was missing, the old fallback scanned the raw JSON for a number. That could accidentally grab an individual room area and treat it as the property's total floor area.

That is worse than failing.

The fixed version returns null when total is missing. No heroic guessing. No “probably 20m² because I found 20 somewhere in the blob”.

expect(a.property.floorAreaM2).toBeNull();

Again: boring test, useful guarantee.

Audit failures before they escape

One more bug was outside serialization. getExporter() could throw before the main try/catch, which meant a format lookup failure skipped the audit-trail write that other failures recorded.

That is the kind of edge case that makes production debugging annoying. The export failed, but the failure path forgot to leave evidence.

The fix was just moving the lookup inside the guarded path so the audit log is written consistently.

The AI-assisted part

This is where AI-assisted development is actually useful for me.

Not “write the whole exporter and hope”. More like:

build the deterministic seam
have Claude review the boring mapping logic
turn every discovered ambiguity into a narrow test
keep the deliberate deferrals explicit

This pass added 6 focused tests. The export suite moved from 119 to 125 tests, with the targeted export tests passing locally: 3 suites, 23 tests.

There is still one known bigger issue left alone: validation is currently too RdSAP-specific for every export format. That needs per-format validation profiles. I did not jam it into this fix because it is a different change.

That is another useful rule when working with AI in real codebases: fix the silent data loss now, leave the larger design change visible, and do not let the model turn a bug fix into a rewrite.

The best export pipeline is not the cleverest one.

It is the one that refuses to quietly lie.

The Exporter Was Easy. Making It Deterministic Was the Work.

Todd Sullivan — Tue, 30 Jun 2026 08:02:27 +0000

The Exporter Was Easy. Making It Deterministic Was the Work.

I spent this week building a mobile export pipeline for UK energy assessment data.

The output is not glamorous: take completed survey responses, turn them into an RdSAP XML file, package the evidence, write a manifest, calculate a checksum, and keep a local audit trail.

The interesting part was not generating XML. The interesting part was making the whole thing deterministic and testable before adding more export formats.

The shape ended up like this:

responses
  -> buildAssessment()
  -> validateAssessment()
  -> exporter.serialize(assessment)
  -> sha256
  -> write package
  -> persist audit log

The exporter itself is deliberately boring:

serialize(assessment): string

No file system. No clock. No database. No random IDs. Same assessment in, same bytes out.

That one rule makes the rest of the pipeline much easier to reason about. If the XML changes, it changed because the input changed or the serializer changed. Not because a timestamp moved, a native module behaved differently, or a test environment had a different document directory.

Side effects at the edge

The orchestrator is the only layer allowed to do side effects:

const body = exporter.serialize(assessment);
const checksum = await deps.sha256Hex(body);
const pkg = await writeExportPackage(
  { inspectionId, exporter, assessment, body, checksum },
  deps.fs
);
await deps.persistLog(log);

Those dependencies are ports:

export type ExportDeps = {
  fs: FileSystemPort;
  sha256Hex: (input: string) => Promise<string>;
  persistLog: (entry: ExportLogEntry) => Promise<void>;
};

In production, they map to Expo file system, Expo crypto, and a local SQLite audit table.

In tests, they are an in-memory file system, Node crypto, and an array log sink.

That was not architectural theatre. Under jest-expo, native modules are often partially unavailable. documentDirectory can be undefined. Crypto enums can be missing. If the export logic directly imports and touches those modules, the “end-to-end” test either becomes a mock festival or stops covering the actual path.

With ports, the test runs the real pipeline:

const { deps, files, logs } = memoryDeps();
const run = await runExport(input(), deps);

const body = files.get(expectedPath)!;
expect(run.result.checksum).toBe(
  createHash("sha256").update(body, "utf8").digest("hex")
);

That test proves a few things at once:

the package path is correct
the written XML is exactly the pure exporter output
the checksum is a real SHA-256 of the bytes
the manifest references the same checksum
one audit entry is written

There are also tests for evidence attachment copying, missing attachment sources, validation-blocked exports, checksum reproducibility, and a second JSON archive exporter running through the same pipeline.

Validation blocks, but still logs

One design choice I care about: failed validation writes an audit record too.

If a mandatory field is missing, the export does not create files. But the attempt is still logged with success: false, validation counts, schema version, and no checksum.

That makes the export feature behave like a real operational system rather than a button that either emits a file or silently refuses.

Why this matters for AI-built software

This is the seam I want when using AI heavily in engineering work.

LLMs are very good at producing a first version of “convert this shape into that schema.” They are much less reliable if the codebase lets schema mapping, file IO, logging, hashing, and UI state collapse into one blob.

The fix is not to use less AI. The fix is to give the code stronger boundaries:

pure transformations for the model or human to edit safely
deterministic outputs that can be snapshot-tested or checksummed
injected IO so tests exercise the pipeline without native dependencies
audit trails for both success and blocked paths
format-specific exporters behind a small registry

Once those seams exist, adding the next export format is not a rewrite. It is another serializer plus a few tests.

That is the part I keep finding in real AI-assisted development: the model can help move fast, but the architecture has to make fast changes safe.

Source: Recent mobile export module work: RdSAP XML export pipeline, pure serializers, injectable IO ports, SHA-256 packaging, local audit logging, and 24 export tests.
Tags: ai, testing, typescript, mobile
Status: published

AI Features Need Product Edges, Not Just Better Prompts

Todd Sullivan — Tue, 23 Jun 2026 08:03:02 +0000

Most AI features don't fail because the model is bad.

They fail because everything around the model is treated like a demo.

This week I was tightening an iOS workout app that uses Claude through Supabase Edge Functions. The model part is straightforward: send training context, get a structured exercise or plan back, validate it, write it into SwiftData.

The less glamorous work was the part that makes it feel like an actual product:

monthly AI credit balance
offline handling
auth token storage
disabled states while generation is running
different rules for “add exercise” vs “swap this exercise”
tests for the boring edge cases

That is where most AI app quality lives.

One button, several states

The UI has a small “fill with AI” affordance on the exercise editor. Underneath it, the button is not just “call endpoint”. It has to know whether suggestion is currently allowed:

var canSuggest: Bool {
    guard !isLoadingAI,
          NetworkMonitor.shared.isOnline,
          (creditsRemaining ?? 1) > 0
    else { return false }

    // Swap mode can use the original exercise as context.
    // Add mode needs the typed name as a hint.
    return isSwapMode || !name.trimmingCharacters(in: .whitespaces).isEmpty
}

That little predicate is doing a lot of product work.

If the user is offline, don't let them tap into a doomed network request.
If a generation is already running, don't double-spend.
If they have zero credits, don't pretend the feature is available.
If they are swapping an existing exercise, don't force them to type a name because the old exercise is already useful context.

The model does not care about any of this. The user does.

Credits should be part of the response

The suggestion response includes the updated credit count:

struct SuggestedExercise: Decodable {
    let name: String
    let briefDescription: String
    let muscleGroup: String
    let sets: Int
    let repTargetLow: Int
    let repTargetHigh: Int
    let restSeconds: Int
    let isDualDumbbell: Bool
    let creditsRemaining: Int
}

Then the view model updates local state immediately after a successful fill:

creditsRemaining = result.creditsRemaining
aiFilledFields = true

That avoids the classic AI-product weirdness where the backend knows the user has spent a credit but the UI keeps showing stale allowance until the next refresh.

It is also easier to test. In the app tests, the credit transition is explicit:

func testAIDisabledWhenCreditsReachZero() {
    let vm = ExerciseEditViewModel(mode: .add(makeDay()), context: container.mainContext)
    vm.name = "Row"
    vm.creditsRemaining = 1
    XCTAssertTrue(vm.canSuggest)

    vm.creditsRemaining = 0
    XCTAssertFalse(vm.canSuggest)
}

There are 13 tests just around the exercise edit view model, plus separate coverage for offline error mapping. Not because this is academically interesting, but because this is the stuff that breaks in front of real users.

Offline is not a server error

Another small detail: connectivity failures are mapped separately from backend failures.

static let connectivityCodes: Set<URLError.Code> = [
    .notConnectedToInternet,
    .networkConnectionLost,
    .timedOut,
    .cannotConnectToHost,
    .dataNotAllowed,
]

The resulting message is intentionally plain:

"You're offline. Reconnect to use AI features."

No “unexpected server response”. No fake intelligence. Just tell the user what happened.

The actual lesson

Shipping AI features is mostly normal software engineering with a probabilistic dependency in the middle.

The model call matters, but the surrounding contract matters more:

Can the user invoke it right now?
What happens if the network dies?
Is usage counted consistently?
Does the UI reflect server state immediately?
Are the edge cases testable without calling the model?

Once those pieces are in place, the AI feature stops feeling like a prompt wired to a button and starts feeling like part of the app.

That is the bar I keep coming back to: not “does the model answer?” but “does this survive normal product reality?”

I Replaced My Old AI Agent Stack with Hermes — and It Finally Feels Like Infrastructure

Todd Sullivan — Fri, 19 Jun 2026 11:16:31 +0000

I recently moved my personal AI automation setup over to Hermes.

Not because I wanted another chatbot. I already had one of those.

The useful shift was that Hermes feels less like “an AI you talk to” and more like a small operating layer for personal automation: persistent memory, scheduled jobs, skills, gateway routing, tool access, and enough system awareness to actually do maintenance work instead of just narrating it.

That distinction matters.

Most AI assistant demos stop at the prompt. Real usage starts after the third week, when you need the assistant to remember your preferences, avoid stale work contexts, run scheduled checks, post to the right channel, and not accidentally resurrect some old service you migrated away from months ago.

That last one is not theoretical.

After the migration I noticed stale references to my old agent stack still appearing in messages. So I asked Hermes to verify the machine state rather than just reassure me.

It checked:

the live launchd services
the running gateway process
old LaunchAgent plists
cron job delivery metadata
process names and command lines
stale references in config/state

The important part: it found the actual answer, not just the obvious one.

Hermes itself was running cleanly through launchd as ai.hermes.gateway. The old gateway labels were disabled. But there were still two leftover MCP-related LaunchAgents from the previous setup:

com.clawmcp.server
com.clawmcp.tunnel

One of them was still using an old SSH key path from that previous world. Not catastrophic, but exactly the kind of residue that turns into weird behaviour later.

Hermes stopped them, disabled the launchd labels, moved the plist files into a timestamped disabled folder, and re-verified that only the Hermes gateway remained active.

That is the bit I care about.

A useful AI agent is not just a text generator. It needs to be able to close the loop:

inspect the real system
distinguish stale text from live processes
make a reversible change
verify the change actually worked
remember the durable lesson for next time

Hermes also gives me a cleaner mental model than my previous setup.

Skills are reusable procedures. Memory is for stable facts. Cron jobs are durable scheduled agents. The gateway lets the same assistant operate from Discord, email, terminal, or wherever I am. Profiles keep different contexts isolated. Toolsets make capabilities explicit.

That sounds boring, which is why I like it.

Boring infrastructure is what makes automation trustworthy.

The more I use AI agents, the less interested I am in “look what this model can say” and the more interested I am in whether the surrounding system can behave predictably over time.

Can it run every morning and only notify me when something needs attention?
Can it publish an article without leaking secrets?
Can it check the actual machine state before making a claim?
Can it avoid old work contexts I no longer want referenced?
Can it improve its own operating instructions when a workflow changes?

That is where the productivity gain is.

The model matters, obviously. But the scaffolding around the model matters more than people admit: state, tools, memory, permissions, scheduling, logs, profiles, and recovery paths.

Hermes is starting to feel like the right layer for that: not a magic assistant, not a toy, not a demo — just a practical agent runtime I can keep shaping around my own workflows.

That is probably the highest compliment I can give an automation system:

I am starting to trust it with boring jobs.

I Put TestFlight Behind a Preflight Gate Before It Can Touch the Repo

Todd Sullivan — Tue, 16 Jun 2026 08:04:53 +0000

I added a preflight gate to an iOS + watchOS app build this week because the build script had a small but annoying property: it mutated the repo before it proved the repo was safe to build.

The script stamps CFBundleVersion with a timestamp before generating the Xcode project and archiving for TestFlight. That is useful because every upload gets a monotonically increasing build number.

It is also exactly the wrong thing to do before quality checks.

If formatting or tests fail after that stamp, the working tree is now dirty for a reason unrelated to the change I was trying to ship. Small thing, but this is how build scripts become suspicious. You run them, they fail, then you have to separate your actual diff from the build system's leftovers.

So I moved the build into a fail-fast shape:

echo "==> Running preflight checks..."
"$PROJECT_DIR/scripts/preflight.sh"

BUILD_NUMBER=$(date +%Y%m%d%H%M)
sed -i '' "s/CFBundleVersion: \"[^\"]*\"/CFBundleVersion: \"$BUILD_NUMBER\"/" "$PROJECT_DIR/project.yml"

The important bit is ordering. preflight.sh runs before the build number is touched.

The gate currently has four checks:

# 1. no local surprises
git -C "$PROJECT_DIR" status --porcelain

# 2. formatting is enforced, not auto-mutated
swiftformat --lint \
  "$PROJECT_DIR/SessionzApp/" \
  "$PROJECT_DIR/SessionzKit/Sources/" \
  "$PROJECT_DIR/SessionzKit/Tests/" \
  "$PROJECT_DIR/SessionzWatch/" \
  --config "$PROJECT_DIR/.swiftformat"

# 3. local Swift package tests
swift test --package-path "$PROJECT_DIR/SessionzKit" --parallel

# 4. backend edge-function unit tests
deno test "$PROJECT_DIR/supabase/functions/_lib/"

That last line is the one I care about most.

The app has a Claude-backed plan generation flow. The mobile app sends goals, equipment, available weights, and a small training context to a backend function. The backend validates auth/subscription state, calls Claude, then returns structured JSON that becomes local SwiftData models.

I do not want the only tests for that boundary to live in an app simulator.

So I pulled the pure logic out of the Edge Function handlers into _lib/ modules and tested that directly with Deno:

// subscription.ts
export function isActiveStatus(status: string | null): boolean {
  return status === "active" || status === "trialing";
}

export function validateProductId(productId: unknown): string | null {
  return typeof productId === "string" && productId.trim().length > 0
    ? productId
    : null;
}

Same idea on the Swift side. The AI coach needs compact summaries of recent training, not an unbounded dump of every logged set. PerformanceSummary condenses the last 42 days into short lines like:

Dumbbell Floor Press: best 20 kg × 8, trained 3×
Push-up: best 20 reps (bodyweight), trained 2×

That now has regression coverage for the edge cases that tend to rot quietly: bodyweight vs weighted sets, zero-weight entries, sorting by training frequency, limiting output size, and ignoring sets outside the six-week window.

The result is not fancy CI. It is just a local quality gate that refuses to make a TestFlight archive unless the repo is clean, formatted, and both sides of the AI boundary pass tests.

This is the kind of infrastructure I like for AI features: boring checks around the non-boring part.

The model can be probabilistic. The system around it should not be.

Source: Recent Sessionz commits: scripts/preflight.sh, TestFlight build integration, Swift regression tests for plan/performance logic, and Deno tests for backend Edge Function helpers.

Tags: ai, swift, testing, devops

Status: published

I Replaced Hardcoded Workouts with a Claude-Generated Plan System

Todd Sullivan — Tue, 09 Jun 2026 08:03:48 +0000

I spent this week ripping out the fixed-routine part of a SwiftUI workout app and replacing it with a small AI planning system.

Not a chatbot bolted onto the side. The app asks for goals, reads a compact set of real constraints, calls Claude through a backend function, then stores the resulting 7-day plan locally in SwiftData.

The old version was simple: two hardcoded push/pull routines and a fixed exercise library. Fine for a prototype, but too rigid. If the user only has dumbbells up to 25kg, trains three days a week, and had a rough sleep week, the app needs to know that before it suggests anything.

The new flow is:

user configures equipment once
user enters goals, capped at 300 chars
app reads a small HealthKit summary if permission exists
Supabase Edge Function verifies auth and subscription state
Edge Function calls Claude
app parses the returned JSON into SwiftData models
watchOS session tracking uses the generated plan day

The important bit is the prompt shape. I did not want prose back from the model. I wanted app data.

const userMessage = `Goals: ${(goals as string).slice(0, 300)}
Equipment: ${(equipment as string[]).join(", ")}
Available weights (kg): ${(weights as number[]).join(", ")}${healthContext}
Schema: ${PLAN_JSON_SCHEMA}`;

The system prompt is deliberately boring:

Output ONLY valid JSON matching the schema provided.
No explanation. No markdown fences.
Exercise descriptions: 1 sentence max.
Metric units only.
Include warmup and rest seconds.

That boring prompt is the product decision. The UI does not need motivational filler. It needs PlanDay, WarmupExercise, and PlannedExercise rows that can be rendered, edited later, and tracked on the watch.

The response schema is inline and maps directly onto the local model:

{
  "planName": "string",
  "days": [
    {
      "dayNumber": 1,
      "isRestDay": false,
      "warmup": [{ "order": 1, "name": "string", "duration": "string" }],
      "exercises": [
        {
          "order": 1,
          "name": "string",
          "sets": 3,
          "repTargetLow": 8,
          "repTargetHigh": 12,
          "suggestedWeightKg": 20.0,
          "restSeconds": 90
        }
      ]
    }
  ]
}

I also put the API call behind a server-side gateway instead of calling Claude directly from iOS. That gives me three useful controls:

JWT verification before generation
subscription check before spending tokens
monthly usage tracking per user

The current limit is 10 plan generations per month. With the compact prompt, the target budget is roughly 800 input tokens and 600 output tokens per generation. That keeps AI cost predictable instead of letting a mobile text field become an unbounded invoice generator.

On-device still matters here. The generated plan, session logs, equipment inventory, and active plan state live locally in SwiftData. The network is only used to create a new plan. Running a workout should not depend on an API being online.

The lesson: AI features get much easier to ship when you stop treating the model response as content and start treating it as a typed boundary.

Prompt in. JSON out. Validate, store, render, track.

That is less magical than a chat UI, but it is much closer to software.

Source: Recent SwiftUI/watchOS app work: replaced hardcoded routines with a Claude-backed 7-day plan generator, HealthKit summary input, Supabase Edge Function gateway, SwiftData local storage, and monthly usage limits.

Tags: ai, swift, ios, claude

Status: published

The iOS 26 Deep-Link Bug That Only Happened When the App Was Already Open

Todd Sullivan — Tue, 02 Jun 2026 08:04:23 +0000

I spent a chunk of this week chasing a bug that looked like auth, then routing, then Supabase, then Expo Router.

Actual root cause: iOS 26 was receiving the magic link, but the running app never got the warm-start URL in JavaScript.

Cold start worked:

app not running
tap myapp://auth/callback?code=...
iOS launches the app
Expo Router mounts auth/callback
PKCE code exchange succeeds

Warm start failed:

app already open
tap the same magic link
app comes foreground
no JS Linking event
user is still logged out

No crash. No useful exception. Just a login flow that works from killed state and fails from running state, which is exactly the kind of mobile bug that eats a day.

The Native Part: SceneDelegate Still Matters

On iOS 26, warm-start URLs are delivered through the scene lifecycle:

func scene(_ scene: UIScene, openURLContexts URLContexts: Set<UIOpenURLContext>) {
  for context in URLContexts {
    forwardURL(context.url)
  }
}

The Expo app already had an AppDelegate, but warm-start delivery needed a UIWindowSceneDelegate as well. The slightly non-obvious bit was how to forward the URL.

Calling RCTLinkingManager directly looked reasonable, but it only posts the React Native URL notification. expo-linking was not listening to that path in this app. Routing through AppDelegate.application(_:open:options:) let ExpoAppDelegate forward the URL to all of its subscribers, including Expo's linking module:

private func forwardURL(_ url: URL) {
  let app = UIApplication.shared
  if let appDelegate = app.delegate as? AppDelegate {
    _ = appDelegate.application(app, open: url, options: [:])
  } else {
    RCTLinkingManager.application(app, open: url, options: [:])
  }
}

That got the URL into JS.

Then iOS 26 added one more trap.

The Key Window Trap

The system log had the clue: key window is null.

The app creates its UIWindow in didFinishLaunchingWithOptions, before the UIWindowScene exists. startReactNative calls makeKeyAndVisible(), but at that point the window is not attached to a scene yet.

On iOS 26, that meant scene:openURLContexts: never fired for warm-start links.

The fix was to associate the existing window with the scene and call makeKeyAndVisible() again after windowScene is set:

func scene(
  _ scene: UIScene,
  willConnectTo session: UISceneSession,
  options connectionOptions: UIScene.ConnectionOptions
) {
  guard let windowScene = scene as? UIWindowScene else { return }

  if let appDelegate = UIApplication.shared.delegate as? AppDelegate,
     let window = appDelegate.window {
    window.windowScene = windowScene
    window.makeKeyAndVisible()
  }
}

After that, scene:openURLContexts: fired reliably. Verified on the iOS 26 simulator: native SceneDelegate log, AppDelegate openURL log, then the JS linking handler.

The JS Part: Don't Route During Foreground Animation

Once the URL reached JS, the first instinct was to navigate to /auth/callback and let that screen do the exchange.

That was fragile for two reasons.

First, during warm start the app is still moving through UISceneActivationStateForegroundInactive. I saw the URL arrive at 10:56:09.749; foreground transition finished at 10:56:10.140. Navigation commands during that window can be silently dropped.

Second, auth/callback may already be mounted. A useEffect([]) on that screen will not re-run just because another link arrived.

So I moved warm-start exchange into the always-mounted root layout:

const linkSub = Linking.addEventListener("url", ({ url }) => {
  const parsed = Linking.parse(url);
  const code = typeof parsed.queryParams?.code === "string"
    ? parsed.queryParams.code
    : null;

  if (code) {
    supabase.auth.exchangeCodeForSession(code).catch(() => {});

    const unsub = useAuthStore.subscribe(({ session, signingIn }) => {
      if (session && !signingIn) {
        unsub();
        router.replace("/");
      }
    });

    setTimeout(unsub, 30_000);
  }
});

The important part is the split:

native code guarantees warm-start URLs reach Expo linking
root layout handles every warm-start URL event
auth callback remains as the cold-start/race fallback
navigation waits until the auth store has a settled session

The Takeaway

Deep links are not one flow. They are at least two:

cold start: launch app into a URL
warm start: deliver URL into an already-running app

If you only test the cold path, your auth flow can look perfect while the common real-world case is broken.

For Expo + React Native apps on newer iOS versions, I now treat warm-start deep links as their own integration test: app open, link tapped, native URL delivery observed, JS Linking event observed, auth state settled, navigation after state settle.

That is more ceremony than a one-line Linking.addEventListener, but it is the difference between a demo login flow and a production one.

Source: Recent Expo / React Native auth work: iOS 26 warm-start deep-link fix, SceneDelegate URL forwarding, PKCE exchange moved to root layout.
Tags: ios, reactnative, devops, javascript

The Silent Code Path: When Your AI Runs on Camera But Not on Gallery

Todd Sullivan — Fri, 29 May 2026 08:17:16 +0000

Here's a bug that's easy to miss and harder to debug: your AI runs perfectly on one input path, silently does nothing on another, and there's no error — just missing results.

I hit this recently in an on-device inspection app. The flow is straightforward: capture a photo (camera or photo library), run AI hazard detection, overlay bounding boxes, trigger violation alerts if needed.

Camera captures worked great. Gallery picks saved the photo fine. But no bounding boxes appeared, no alerts fired. The AI was just... not running.

What Actually Happened

Here's the stripped-down structure:

// Camera capture handler
const handleRequestCapture = async (uri: string) => {
  await savePhoto(uri);
  await detectAndSave(uri);   // ✅ AI runs
  checkViolationAlerts();
};

// Library pick handler  
const handleLibraryPick = async (uri: string) => {
  await savePhoto(uri);
  // ❌ detectAndSave was never called
};

The library path was added later, modelled on the save logic but not the full inference pipeline. No crash. No warning. Just missing detections.

The fix was six lines — copy the detectAndSave + alert block into the library handler. But finding it took longer than writing the fix.

Why This Pattern Is Easy To Ship

On-device AI inference tends to get wired in during the happy path. You build the camera flow first, the AI gets integrated there, everything works in demos. Then you add the gallery pick as a "nice to have" — and because you're only thinking about the file I/O, you copy the save logic but not the inference call.

There's no type error. No lint warning. The function name (handleLibraryPick) doesn't imply inference should happen. The photo saves successfully, so from the app's perspective, nothing broke.

Lessons From The Fix

1. Treat every input path as a first-class citizen.
If AI inference is core to your feature, it should run regardless of how the image arrived. Camera, gallery, deep link, background upload — all of them.

2. Extract inference into a shared pipeline.
After the fix I refactored toward a single processPhoto(uri) function that both handlers call. Now if the pipeline changes, it changes in one place.

const processPhoto = async (uri: string) => {
  await savePhoto(uri);
  await detectAndSave(uri);
  checkViolationAlerts();
};

const handleRequestCapture = (uri: string) => processPhoto(uri);
const handleLibraryPick = (uri: string) => processPhoto(uri);

3. End-to-end tests that cover input variants.
Unit tests on detectAndSave wouldn't have caught this — the function worked fine, it just wasn't being called. What you need is an integration test that exercises each entry point and asserts that inference results exist.

4. Visibility into what ran.
This is the sneaky part: when AI silently doesn't run, you need observability to know it happened. Adding a log line (AI inference: skipped | ran on <uri>) to your inference call sites makes this class of bug immediately obvious in development.

The Broader Pattern

This isn't unique to AI. Any side-effect that only gets wired to one code path — analytics events, permission checks, cache invalidation — can silently miss inputs added later. But with AI inference it's particularly painful because the failure mode is invisible and the debugging surface is narrow (you're looking at model output, not a thrown error).

Build the pipeline once, call it everywhere.

Wrapping Apple's LiDAR Room Scanner as a Native Expo Module

Todd Sullivan — Wed, 27 May 2026 08:08:52 +0000

Most property and field-service apps still ask users to manually enter room dimensions. Tape measure, pen, back to the app, mistype, repeat. It's tedious and it's lossy.

I've been building a property inspection app and this week I shipped a feature that replaces manual floor-area entry with a LiDAR scan. Walk around a room for 30 seconds, hit Done — the app returns floor area, wall count, door count, window dimensions, and a confidence score for each surface. No internet connection required. No server round-trips. All on-device.

Here's how I wired Apple's RoomPlan framework into a React Native / Expo app as a native module.

The Basic Idea

Apple's RoomPlan framework uses the LiDAR sensor on iPhone 12 Pro and later to build a structured 3D model of a room in real time. It gives you back typed objects: floors, walls, doors, windows — each with dimensions and a confidence rating (high, medium, low).

The challenge is surfacing this in a cross-platform Expo app without hacking around the JS/native boundary.

Building the Module

Expo's native module API makes this cleaner than the old React Native bridge approach. The module exposes two functions:

isAvailable() — synchronous check, returns true on LiDAR + iOS 17+
scanRoom() — async, presents a full-screen capture UI and resolves with structured results

AsyncFunction("scanRoom") { (promise: Promise) in
  guard RoomCaptureSession.isSupported else {
    promise.reject("UNSUPPORTED", "Requires a LiDAR-equipped device")
    return
  }
  // present RoomCaptureViewController...
}

The view controller runs captureSession.run(configuration:), delegates back through RoomCaptureSessionDelegate, and when the user hits Done it hands raw CapturedRoomData to RoomBuilder. The builder does a cleanup pass (with .beautifyObjects) and returns a CapturedRoom.

What Comes Back

let floorArea = room.floors
  .map { Double($0.dimensions.x) * Double($0.dimensions.y) }
  .reduce(0, +)

I map the result into a plain dictionary — area in m², per-surface dimensions, confidence string — and resolve the Expo promise. The JS side gets a typed object:

const result = await RoomPlan.scanRoom();
console.log(result.floorAreaM2); // e.g. 18.34
console.log(result.floors[0].confidence); // "high"

Graceful Degradation

Non-LiDAR devices get isAvailable() → false and fall through to a manual input form. No crashes, no confusing errors. The LiDAR path is an enhancement, not a hard dependency.

Why On-Device Matters Here

For property data, sending floor plans to a cloud service introduces latency, cost, and privacy concerns. Every scan happening fully on-device means no API bills per-scan, no waiting on a round-trip, and no floor plan leaving the phone until the user explicitly submits their report.

The practical result: floor area capture went from ~3 minutes (manual) to under 60 seconds with better accuracy than tape-measure entry. On iPhone Pro hardware, LiDAR readings are accurate to within a few centimetres on a normal rectangular room.

The Limitation Worth Knowing

RoomPlan requires a LiDAR sensor — iPhone 12 Pro and later, or iPad Pro. Most fieldwork happens on phones. For a consumer app targeting everyone, you need a fallback. For a professional tool where you control (or specify) the hardware, worth requiring it outright.

If you're building anything involving physical space measurement on iOS, RoomPlan is dramatically underused. Apple ships a full structured-capture pipeline and most apps still ask users to type numbers into a form.

When Your On-Device Model Decides How Often to Ping You

Todd Sullivan — Mon, 25 May 2026 08:03:36 +0000

Most notification systems are dumb. They fire on a fixed schedule decided by a developer who's never met the user. Every day at 9am. No matter what.

I've been building an iOS health app that does something different: the on-device ML model's training state drives notification frequency. The more the model knows about you, the less it needs to ask.

The Problem With Fixed Schedules

For a health tracking app, consistent data collection is everything — it's the training set. Early on, you need daily check-ins to bootstrap the model. But once the model is trained and the user has established patterns, daily prompts become noise. People turn off notifications. You lose signal entirely.

The naive fix is "let users choose their own frequency." That works until users pick "weekly" in week one when the model is still blind.

The better fix: make the frequency a function of the data itself.

The Logic

I built a NotificationFrequencyManager that recomputes check-in frequency after every engagement cycle and after every model training run. Three signals go in:

Data volume — how many days of logs exist
Logging consistency — what percentage of the last 14 days has at least one entry (≥70% threshold to step down)
ML training state — is the personalised model untrained, training, or trained?

The output maps to three frequencies:

enum NotificationFrequency: Int, Codable, Comparable {
    case high   = 1  // Daily — new users, untrained model
    case medium = 2  // Every 2 days — moderate data, some consistency
    case low    = 4  // Every 4 days — trained model, high consistency
}

There's a hard floor at 4 days. Even the most consistent user with a fully trained model gets nudged at least every 4 days. The model still needs new data to stay fresh. Silence would mean drift.

Closing the Loop

The clever part is that ML training completion directly triggers a frequency recalculation:

func update(logs: [DailyLog], trainingState: TrainingState) -> NotificationFrequency {
    let frequency = compute(logs: logs, trainingState: trainingState)
    currentFrequency = frequency
    storedFrequency = frequency  // persisted to App Group UserDefaults
    return frequency
}

So the flow is: user logs data → engagement manager runs → model retrains → frequency updates → next notification scheduled at new interval. The app literally quiets down as it learns.

Smooth Transitions

One thing I got wrong in early iterations: jumping straight from daily to every-4-days felt jarring in testing. The fix was requiring both consistency and data volume thresholds before stepping down. You can't hit low unless you've been on medium for at least a week.

// Only step from medium → low if model is trained AND consistency is high
if dataVolume >= 30 && consistency >= 0.7 && trainingState == .trained {
    return .low
}

Why This Matters

The insight generalises well beyond health apps. Any time you have an adaptive ML component, the model's confidence or training state is a signal you can use to drive other app behaviour. Frequency of prompts. Depth of UI. Whether to show explanations or trust the user already gets it.

The model knowing you means the app behaves differently for you. That's actually what "personalised AI" should mean — not just personalised outputs, but personalised interactions.

DEV Community: Todd Sullivan

My Local AI Assistant Got Worse When I Remembered Too Much

My Local AI Assistant Got Worse When I Remembered Too Much

I Made My Voice Agent Feel Faster by Streaming Sentences, Not Audio

Silent Data Loss Is Worse Than a Failed Export

The dangerous null

false is not “unanswered”

Do not guess from raw JSON

Audit failures before they escape

The AI-assisted part

The Exporter Was Easy. Making It Deterministic Was the Work.

The Exporter Was Easy. Making It Deterministic Was the Work.

Side effects at the edge

Validation blocks, but still logs

Why this matters for AI-built software

AI Features Need Product Edges, Not Just Better Prompts

One button, several states

Credits should be part of the response

Offline is not a server error

The actual lesson

I Replaced My Old AI Agent Stack with Hermes — and It Finally Feels Like Infrastructure

I Put TestFlight Behind a Preflight Gate Before It Can Touch the Repo

I Replaced Hardcoded Workouts with a Claude-Generated Plan System

The iOS 26 Deep-Link Bug That Only Happened When the App Was Already Open

The Native Part: SceneDelegate Still Matters

The Key Window Trap

The JS Part: Don't Route During Foreground Animation

The Takeaway

The Silent Code Path: When Your AI Runs on Camera But Not on Gallery

What Actually Happened

Why This Pattern Is Easy To Ship

Lessons From The Fix

The Broader Pattern

Wrapping Apple's LiDAR Room Scanner as a Native Expo Module

The Basic Idea

Building the Module

What Comes Back

Graceful Degradation

Why On-Device Matters Here

The Limitation Worth Knowing

When Your On-Device Model Decides How Often to Ping You

The Problem With Fixed Schedules

The Logic

Closing the Loop

Smooth Transitions

Why This Matters

The dangerous `null`

`false` is not “unanswered”