Argus is an agent that watches San Francisco's civic apparatus and acts on what it finds.
Following one issue through city government is a seven-step chore: watch a dozen publishers that share no format or schedule, notice when an agenda is quietly amended, open the PDF packet, work out whether any of its hundred-odd items touches something you care about, get the hearing into your calendar with the right room number, find out afterwards what was decided, and repeat weekly.
Steps one to four are unpaid research. Steps five to seven are the part anyone actually wants and nobody ever reaches. Argus does all seven on a schedule, and writes to a real Google Calendar and sends real email at the end of it.
The three issues I track are ordinary: road improvements, bike lanes, and schools. Nothing about them is exotic. That's the point, the information isn't secret, it's just that keeping up with it is a part-time job.
The bugs that don't announce themselves
I asked Argus who the mayor of San Francisco was. It didn't know.
The answer was already in its database. The city's department directory, a page Argus had fetched, parsed, embedded and stored contains the sentence "Daniel Lurie is the 46th Mayor of the City and County of San Francisco." I could select it with a SQL query in a few seconds.
The problem was retrieval. Postgres full-text search ANDs its terms, and that directory is 20,000 characters listing roughly a hundred departments, most of which contain the word "Mayor": Mayor's Office of Housing, Mayor's Office for Victims' Rights, and so on. On term frequency alone the catalogue outranked the mayor's own profile page. Worse, the snippet extractor then picked the densest cluster of matches, which was some unrelated office. So the agent received a document that technically contained the answer, along with an excerpt that didn't, and correctly reported that it couldn't find anything.
That is the failure mode I spent most of this project learning to recognize. A missing fact announces itself. An unreachable one looks exactly like a model that doesn't know.
Two searches, both wrong on their own
The fix for the mayor bug was not a better query. It was accepting that keyword search and vector search fail in opposite directions.
Keyword search demands every term appear. "Traffic citations" found nothing in a document headed "Traffic Violations". Vector search understands that those mean the same thing, but has no notion of an exact name, and will happily rank a merely-similar document above the one actually about the thing you asked for.
My first attempt ran keyword search and fell back to vectors only when it returned zero rows. That version was worse than either alone, because it hid the exact case that mattered: a weak but non-empty match blocks the fallback entirely. The department catalogue always matched something, so the vector search, which finds the mayor's profile at a cosine distance of 0.27, never ran at all.
Now both run every time and are merged by reciprocal rank fusion, which needs no shared scale between a ts_rank and a cosine distance because it only uses each result's position in its own list.
The distance cutoff is measured rather than guessed. Across the real corpus, on-topic questions land at 0.27–0.34 and off-topic ones ("weather in Tokyo", "renew a passport in Ireland") at 0.51. The threshold sits at 0.42, in the gap, so an unrelated question returns nothing instead of a confident citation to whatever happened to be nearest.
Some doors are closed on purpose
Two of the sources I wanted refused me. BoardDocs returns 403 to any client that isn't a browser. SFCTA sits behind a Cloudflare challenge.
Both would have taken about ten minutes to defeat with a spoofed user-agent. I decided not to. A 403 to an honest client is a publisher saying no, and the fact that the door opens if you lie about who you are doesn't make it an invitation.
So those sources are supplied by hand and run through the same deterministic parsers as everything else, recorded with kind='manual' so their provenance stays visible. That matters: unlike a polled source, nothing re-checks them, and they can go stale silently.
The invoice you can't see
Argus records what every model call costs as tokens, latency, and dollars.
Building that turned up a trap. Thinking models bill their reasoning tokens at the output rate, but report them in a separate field from the completion. If you cost a call from the two obvious numbers, you undercount. On a real measured call: $0.000557 by the naive arithmetic, $0.001986 actually owed. A 3.6× under-report, on every single call, silently.
The rule that came out of it: an unpriced model records NULL, never 0.00. A zero sums into a total and reports spending that never happened; a NULL means "I don't know" and stays visibly unknown.
And then something unexpected: the total model spend across the
entire project is $0.91. Two hundred and eighty-eight calls, thousands of agenda items judged, dozens of meeting outcomes extracted from transcripts, all under a dollar. The models were never the expensive part. An always-on Postgres instance and one warm container will cost more per month than every Gemini call made during the whole build.
The rule underneath all of it
One principle did more work than any other: the model writes content; code decides actions.
No model call creates, moves, or cancels a calendar event. The model can raise a proposal card; the write happens when a person presses a button, or when a deterministic delivery step runs. A model that decides to "just do it" has nothing to call.
The second half is less obvious and catches a subtler failure: the model names a thing, and the server supplies the facts. A calendar proposal passes a meeting_id and nothing else. The time, the place and the title are read from the database. This isn't about trust, it's that a model asked to restate a date will occasionally restate it slightly wrong, and a hearing on the wrong Tuesday is worse than no hearing at all.
Where it is now
Over a thousand meetings. Thousands of agenda items. Tens of thousands of transcript segments. Hundreds of reference
documents. Thousands of issue matches judged by a model, Over a hundred confirmed with a stored rationale. Tens of meeting outcomes extracted from transcripts. Hundreds of tests. Less than a dollar spent.
The honest open item: that matcher confirms 111 of 3,088 candidates, and I have no idea whether those are the right 111. Until there's a hand-labelled set of examples to measure precision against including deliberate
vocabulary-overlap non-matches "it runs" is not the same as "it's right."
That's the next thing. It's less fun than building the pipeline, and it's the only test that says whether any of this is useful rather than merely working.
This post was written for the #AllThingsAgentic Hackathon. Follow along to see the results!
Top comments (0)