DEV Community: Scott Lee

Building Camrade

Scott Lee — Mon, 31 Aug 2026 23:52:12 +0000

Camrade is a voice-first photo studio. You talk to it about the photograph in front of you, it reads the frame, edits it, and learns your photography style. This is not a tour of how Camrade works but about the failures, because almost every decision worth describing came out of one, and the worst of them were completely silent.

The wine bottle

Early on, Argus told me about a wine bottle. There was no wine bottle.

I had given the voice model a system instruction that amounted to "say what you see." What I had not given it was any way to see. No image was ever sent to it. So it did what a language model does when asked to describe something it knows nothing about: it produced a plausible description, in exactly the confident tone it used for everything else.

That bug was worth more than most features. Not because it was hard to fix but because of what it exposed. A confident wrong answer is not a slightly worse right answer. It is worse than silence, because it contaminates everything around it. Once the agent had described a bottle that was not there, I had no way to know which of its other observations to trust.

Nearly every design decision after that was about making the agent's uncertainty legible.

Blind by design

The fix was not a better prompt. It was making the failure structurally impossible.

The voice model now cannot see. No image is sent to it, ever. To say anything about the photograph it has to call analyze_photo, which returns a critique produced by a different model that did look at the pixels. The system instruction says so in as many words:

Saying "I see a..." without having called analyze_photo is the worst thing you can do here,
because the photographer will believe you.

That split turned out to be load-bearing. gemini-live-2.5-flash handles the conversation and nothing else — it is a transport. It selects a tool and relays the result. Every judgement, every critique, every edit instruction comes from gemini-3.5-flash. The live model is a 2.5-generation model, so it is precisely the thing that should not be forming opinions about anyone's photographs.

The same principle runs through the rest. Capture data the file never recorded stays null rather than being estimated. Every suggestion carries a badge saying whether it rests on measured settings, on what the model can see, or on your profile. If it doesn't know, it says so.

Three failures that made no noise

The wine bottle was loud, but the expensive bugs did the opposite.

A catch block hid a bug for most of the project's life. The voice agent fetched the my profile at session start without an auth header. That request redirects to the sign-in page, which answers 200 with HTML. So res.ok was true, the JSON parse threw, and a
catch commented "No profile is a normal cold start, not a failure" absorbed it. The agent had never once received the profile, not in any session, ever. I only found it while tracing an
unrelated bug; nothing in the app's behavior had ever pointed at it. The real bug was not the missing header. It was a catch that treated broken and empty as the same thing.

I spent four hours debugging code that was never running. The agent would not speak first when prompted. I read the SDK internals twice, reasoned carefully about two different API calls, swapped between them. Nothing changed. The evidence made no sense: the browser was demonstrably sending the right message, and the server behaved as if it hadn't. Vite's hot reload covers the app bundle but not server.mjs or live-bridge.mjs, which Node loads directly at boot. Every fix's browser half reloaded correctly while its server half kept running four-hour-old code. I found it by comparing the process start time against the file's modification time. The dev script now watches those files explicitly, and there is a comment at the top of server.mjs explaining it, because that failure is silent and extremely convincing.

A metric would have counted nothing forever. While wiring up Cloud Monitoring I planned a metric for how often the one-render-at-a-time limit turns someone away. Before writing the filter I checked
whether the app actually emitted that event. It didn't. The filter would have been syntactically perfect and the chart permanently empty and an empty chart looks exactly like a healthy one.

Making "it learns you" falsifiable

Every agent claims it learns you. Almost none can show it.

Camrade has a test that shows the same photograph to the model twice, under two opposite taste profiles, and asserts the advice inverts. It does: "warm up the tones / lift the black point"
versus "crush the shadow tones / cool down the colour balance." Same photo, same model, opposite answers, with only the profile differing.

That test exists because the claim is otherwise unfalsifiable. A model asked "does this feel personalized?" will always say yes.

The mechanism matters too. Every edit you ask for is classified onto an axis (warmth, contrast, blacks, crop) and the profile is counted from those labels rather than summarized. Confidence is arithmetic: how much evidence exists multiplied by how consistent it is. Six contradictory edits cannot read as confident, because the number is not something the model chose.

What I'd tell you

Probe, don't trust. Two of the most expensive bugs came from believing documentation about which models exist and where they are served. gemini-3.1-flash-live-preview does not exist on Vertex AI
at all, it 404s everywhere. Fifteen minutes of probing would have saved hours of reasoning.

Know where your reload boundary is. Hot reload covering some of your code is more dangerous than covering none, because half your changes take effect and the evidence stops making sense.

A catch that cannot distinguish broken from empty will hide a bug indefinitely. Both of the longest-lived bugs here were silent failures, not crashes. Logging the difference costs one line.

The through-line, in the end, was simpler than I expected: being useful and being honest turned out to be the same engineering problem.

This post was written for the #AllThingsAgentic Hackathon. Follow along to see how I do!

How Argus was built

Scott Lee — Mon, 31 Aug 2026 23:49:10 +0000

Argus is an agent that watches San Francisco's civic apparatus and acts on what it finds.

Following one issue through city government is a seven-step chore: watch a dozen publishers that share no format or schedule, notice when an agenda is quietly amended, open the PDF packet, work out whether any of its hundred-odd items touches something you care about, get the hearing into your calendar with the right room number, find out afterwards what was decided, and repeat weekly.

Steps one to four are unpaid research. Steps five to seven are the part anyone actually wants and nobody ever reaches. Argus does all seven on a schedule, and writes to a real Google Calendar and sends real email at the end of it.

The three issues I track are ordinary: road improvements, bike lanes, and schools. Nothing about them is exotic. That's the point, the information isn't secret, it's just that keeping up with it is a part-time job.

The bugs that don't announce themselves

I asked Argus who the mayor of San Francisco was. It didn't know.

The answer was already in its database. The city's department directory, a page Argus had fetched, parsed, embedded and stored contains the sentence "Daniel Lurie is the 46th Mayor of the City and County of San Francisco." I could select it with a SQL query in a few seconds.

The problem was retrieval. Postgres full-text search ANDs its terms, and that directory is 20,000 characters listing roughly a hundred departments, most of which contain the word "Mayor": Mayor's Office of Housing, Mayor's Office for Victims' Rights, and so on. On term frequency alone the catalogue outranked the mayor's own profile page. Worse, the snippet extractor then picked the densest cluster of matches, which was some unrelated office. So the agent received a document that technically contained the answer, along with an excerpt that didn't, and correctly reported that it couldn't find anything.

That is the failure mode I spent most of this project learning to recognize. A missing fact announces itself. An unreachable one looks exactly like a model that doesn't know.

Two searches, both wrong on their own

The fix for the mayor bug was not a better query. It was accepting that keyword search and vector search fail in opposite directions.

Keyword search demands every term appear. "Traffic citations" found nothing in a document headed "Traffic Violations". Vector search understands that those mean the same thing, but has no notion of an exact name, and will happily rank a merely-similar document above the one actually about the thing you asked for.

My first attempt ran keyword search and fell back to vectors only when it returned zero rows. That version was worse than either alone, because it hid the exact case that mattered: a weak but non-empty match blocks the fallback entirely. The department catalogue always matched something, so the vector search, which finds the mayor's profile at a cosine distance of 0.27, never ran at all.

Now both run every time and are merged by reciprocal rank fusion, which needs no shared scale between a ts_rank and a cosine distance because it only uses each result's position in its own list.

The distance cutoff is measured rather than guessed. Across the real corpus, on-topic questions land at 0.27–0.34 and off-topic ones ("weather in Tokyo", "renew a passport in Ireland") at 0.51. The threshold sits at 0.42, in the gap, so an unrelated question returns nothing instead of a confident citation to whatever happened to be nearest.

Some doors are closed on purpose

Two of the sources I wanted refused me. BoardDocs returns 403 to any client that isn't a browser. SFCTA sits behind a Cloudflare challenge.

Both would have taken about ten minutes to defeat with a spoofed user-agent. I decided not to. A 403 to an honest client is a publisher saying no, and the fact that the door opens if you lie about who you are doesn't make it an invitation.

So those sources are supplied by hand and run through the same deterministic parsers as everything else, recorded with kind='manual' so their provenance stays visible. That matters: unlike a polled source, nothing re-checks them, and they can go stale silently.

The invoice you can't see

Argus records what every model call costs as tokens, latency, and dollars.

Building that turned up a trap. Thinking models bill their reasoning tokens at the output rate, but report them in a separate field from the completion. If you cost a call from the two obvious numbers, you undercount. On a real measured call: $0.000557 by the naive arithmetic, $0.001986 actually owed. A 3.6× under-report, on every single call, silently.

The rule that came out of it: an unpriced model records NULL, never 0.00. A zero sums into a total and reports spending that never happened; a NULL means "I don't know" and stays visibly unknown.

And then something unexpected: the total model spend across the
entire project is $0.91. Two hundred and eighty-eight calls, thousands of agenda items judged, dozens of meeting outcomes extracted from transcripts, all under a dollar. The models were never the expensive part. An always-on Postgres instance and one warm container will cost more per month than every Gemini call made during the whole build.

The rule underneath all of it

One principle did more work than any other: the model writes content; code decides actions.

No model call creates, moves, or cancels a calendar event. The model can raise a proposal card; the write happens when a person presses a button, or when a deterministic delivery step runs. A model that decides to "just do it" has nothing to call.

The second half is less obvious and catches a subtler failure: the model names a thing, and the server supplies the facts. A calendar proposal passes a meeting_id and nothing else. The time, the place and the title are read from the database. This isn't about trust, it's that a model asked to restate a date will occasionally restate it slightly wrong, and a hearing on the wrong Tuesday is worse than no hearing at all.

Where it is now

Over a thousand meetings. Thousands of agenda items. Tens of thousands of transcript segments. Hundreds of reference
documents. Thousands of issue matches judged by a model, Over a hundred confirmed with a stored rationale. Tens of meeting outcomes extracted from transcripts. Hundreds of tests. Less than a dollar spent.

The honest open item: that matcher confirms 111 of 3,088 candidates, and I have no idea whether those are the right 111. Until there's a hand-labelled set of examples to measure precision against including deliberate
vocabulary-overlap non-matches "it runs" is not the same as "it's right."

That's the next thing. It's less fun than building the pipeline, and it's the only test that says whether any of this is useful rather than merely working.

This post was written for the #AllThingsAgentic Hackathon. Follow along to see the results!