DEV Community: Himanshu Kumar

INKSHIFT: cross out a table, keep the booking

Himanshu Kumar — Thu, 24 Sep 2026 17:03:36 +0000

This is a submission for the Sanity Challenge, Path Two: Vibe-Code Something Strange.

What I Built

Someone has booked Ticket to Ride at Table B. Then Table B becomes unavailable.

Moving the game to Table C sounds easy. But there is already a person attached to that plan, and their booking needs to survive the edit.

I built INKSHIFT for this kind of change. It turns a plan for a games night, workshop or club meetup into a shared signup page. Upload a photo or type the plan, check the sessions, and send the invite link. Guests can book a place without creating an account.

When the plan changes, you review the proposed edits against the people who have already joined. You can correct the reading, check the affected bookings and approve the move. Guests keep their places at the new location.

Sanity gives the gathering continuity: Content Lake stores the linked sessions and registrations, Workflows records each plan review, and App SDK subscribes to the shared schedule. The booking belongs to a session whose location can change.

Open INKSHIFT · Watch the walkthrough · Source code

Demo

The 43-second walkthrough follows a prepared games-night plan through a table change:

You can try the same flow yourself:

Choose Try a sample and open its participant invite.
Join Ticket to Ride, then return to the organiser workspace.
Choose Use the crossed-out example and review the proposed move to Table C.
Approve it, then reopen the participant page.

Here is the organiser's review with a booking already in place:

The review identifies the move and the registration that stays with it. INKSHIFT checks that Table C has enough seats and is available for the full session before allowing approval.

After approval, the guest's existing booking appears under Your places at Table C:

They do not need to sign up again.

The video and screenshots use labelled prepared samples with fixed readings. The screenshots show a saved demo registration and an applied review. Separate photo-reader checks used rendered typed sheets.

Scrub back through the paper

Every approved edit becomes a version. Below the live plan there is a Paper time machine: a slider over every version of the paper, with the sheet on the left and the tables on the right. Move from version 1 to version 2 and the Ticket to Ride card slides from Table B to Table C, carrying Ada's token with it. Move back and it slides home.

Each version shows when it was applied, what changed ("Ticket to Ride: Table B → Table C"), how many bookings were kept and the state of its Workflows run. Readings I discarded appear as a faded note, never as a version.

Version 1 in the sample needs no upload: choose Try a sample, join as a guest, apply the crossed-out example and the time machine has two versions. The rename example adds a third.

It is built from records, not an animation of a guess. Plans only change when a proposal is applied, so each applied proposal's stored preview is exactly the plan it produced. The one gap was the plan before the first edit: nothing stored it. INKSHIFT now records a planBefore snapshot on the proposal at apply time. Older sample gatherings that predate the snapshot rebuild version 1 from the prepared sample definition, and the time machine labels that version Reconstructed. Bookings are placed per version from their createdAt and cancelledAt times; they always point at the same session ID, so nothing has to be guessed about where a person went.

The timeline, photos and booking initials come from an organiser-only server route. A request without the organiser cookie gets a 403, and the App SDK public schedule still carries only sessions and counts.

How Sanity keeps the booking attached

In Content Lake, spaces, sessions and registrations have separate identities. A registration points to a session; the session points to its space.

Registration → Ticket to Ride → Table B
                       ↓ move approved
Registration → Ticket to Ride → Table C

Moving Ticket to Ride changes its space reference. Its session ID stays the same, so the registrations still belong to it. You can inspect these relationships in the Sanity schema.

The app also includes a live content inspector. After the move, it shows session-1-1 at Table C with 1/4 places booked:

App SDK reads this public schedule and its booking counts from Content Lake. Participant names, uploaded photos and organiser access data stay behind authorised server routes. The Sanity write token stays on the server.

A change has a review and a decision

An edited plan becomes a saved proposal. Sanity Workflows tracks it through Reading, Review and either Applied or Discarded, so the organiser can return to a review and see what happened.

There is a timing problem here: someone can join while the organiser is reviewing a move. A proposal that fitted the earlier bookings may no longer fit.

Before applying it, the server checks the event and proposal revisions and recomputes the constraints. If the event has changed, the organiser has to recheck. Once approved, the plan, linked records and public schedule are saved together in one Content Lake transaction.

App SDK subscribes to the public schedule's version. A change prompts the organiser workspace to refresh its private data through an authorised server route. Workflows records the review's progress; the server checks permission and validates the move.

Code

Source code and setup instructions

INKSHIFT uses Next.js and React, Sanity Content Lake, App SDK and Workflows. Photo reading uses Qwen3-VL through Hugging Face Inference Providers. The verification record covers booking preservation, concurrent changes, access recovery and private-data checks.

My Build Process

I built INKSHIFT with two AI-native tools: Codex for the first build and the finish, and Claude Code for an upgrade pass in between. Every step ran against the real Sanity project, so each claim below comes from a test run or a live check.

The pitch, then very short prompts

The idea started as a note I pasted in: "A handwritten plan becomes a working, multiplayer app. Then you change the paper, and the app understands what changed without losing what people already did."

My prompts after that were short. "go on it's for dev.to challenge", then "anything works also if that does not fit the hackathon let me know and switch over to another project". Codex read the Path Two rules before writing code and answered that it fit, because Sanity would hold the model the whole product depends on.

The first correction: a table is not a session

The first model tied each game to its table. Moving Ticket to Ride from Table B to Table C would have replaced the session and dropped its bookings, which is the exact failure the product exists to prevent. Spaces, sessions and registrations became separate records with stable IDs, and a registration points to its session. Most of the later work follows from that decision.

The same day we set a second rule: a region missing from a photo is uncertainty, not evidence of a deletion, and it goes to review.

Where the models got stuck

The vision provider rejected the schema. Qwen3-VL through the Hugging Face router refused the bounding-box format. Codex fixed it by expressing each box as a fixed-length array of numbers.
The reader removed a booked game. On the second photo of an edited plan, the reader decided Ticket to Ride was gone. Review blocked approval until I matched it back to the original session. After approval, the booking appeared at Table C. That run is why review is mandatory, not optional.
The schema deploy was refused. The token Sanity provisioned for the project could write documents but could not deploy a schema. The app doesn't need it at runtime, so it waited until I deployed it with my own Sanity login four days later.
App SDK warned during server rendering in production. Moving the subscription provider behind a browser-only import fixed it.
Vercel picked the wrong framework preset. Committing an explicit Next.js configuration fixed the first deploy.

What I threw away

I asked for a Three.js scroll world on the landing page. It worked: the paper became tables, pawns took their seats and the game moved from B to C. It also looked like a toy, with small pieces and handwriting you couldn't read. I told Claude the 3D looked bad and chose a replacement built from the real interface. The walkthrough on the homepage is now the actual screens.

Reaching into Workflows

Workflows is in early access, so the agent worked from the docs, not from memory. Claude wrote the inkshift-plan-change definition (Reading, then Review, then Applied or Discarded) and its adapter, then hit its session limit before wiring the routes and interface. Codex picked up that working copy, connected proposal creation, readings, corrections, approval and discard to the engine, and deployed definition v1.

One detail from the docs shaped the design: the engine's checks are advisory, and only the Content Lake enforces anything. So the server still rechecks revisions, seats and the time slot before it writes, and a registration that arrives during review invalidates the stale proposal. Codex also added recovery for a plan decision whose workflow follow-up fails.

The time machine

After the first submission I wanted the strange part to be visible: ink on paper edits a live database, and the same people travel with their bookings as the paper changes. I asked Claude Code for a "paper time machine" and gave it one rule: do not fake history.

Before any UI, it checked the rule against the real dataset with a GROQ read. 17 events had applied proposals, and in all 17 the latest applied preview matched the live plan exactly. Two problems came out of that read. 8 of 21 applied proposals were saved before appliedVersion existed, so ordering falls back to the applied time. And nothing stored the plan before the first edit, so version 1 of a sample could only be rebuilt from code. That is why planBefore now exists and why old gatherings say Reconstructed on version 1.

What went wrong: the first test run was flaky. A booking and an approval landed in the same millisecond, and the version-1 frame dropped the guest. The tests now run on a fixed clock, and a booking made at the exact moment of an apply counts as before it. The agent also hit its session limit while writing the component and picked the work up from the last commit.

Writing this post

I pushed back on two drafts: "you've to establish it as completed product not incomplete" and "why'd you talk about how we made it instead of what product and how it uses sanity". That's why the post opens with the product. The build story lives here.

What is still unverified

Real handwriting (every image test used rendered typed sheets) and camera access on physical phones. The domain tests cover relocation, full destinations, identity ambiguity, cropped photos, time conflicts and capacity cuts. With 6 tests for the time machine (ordering, discarded readings left out, bookings followed by session ID, missing photos, reconstructed originals) there are 37 in all. A live race for the last place produced exactly one winner, and five booking IDs survived a relocation.

Sanity Project Details

Project ID: a5xdqsb7

Dataset: production

Workflow: inkshift-plan-change, version 1

To use INKSHIFT for your own gathering, create an event, add its plan and share the participant invite. You can return through Your gatherings, or restore organiser access on another device with your private backup code.

FinePrint: an agent that checks your hackathon entry against the rules it reads

Himanshu Kumar — Thu, 24 Sep 2026 17:01:07 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content.

What I Built

Hackathon rules live in three places: the challenge page, the FAQ and the official contest rules. They don't always agree, and one missed line can disqualify a good project.

FinePrint is an agent you can ask. "I started my app in August. I added Sanity this week. Can I enter Path One?" It reads a Sanity Knowledge Base built from the official rules, proposes facts with exact quotes from your question and calls a typed checker over structured requirement records. The answer shows the rules it read, the facts it accepted and every tool call it made.

A keyword search finds the sentence about the entry period. It can't tell you that "started in August" blocks the development-start rule while reusing components is still allowed, or that the contest rules changed their entry limit halfway through the challenge. That needs structure: each requirement is a record with its conditions, its event and a reference to the source version it came from.

I picked this challenge as the first rule pack, and it changed under me. On September 20 the FAQ said "only one submission per path is allowed" while the contest rules said "There is no limit on the number of Entries you may submit during the Entry Period." FinePrint marked two entries in one path Rules unclear and gave a question for the organizer. By September 24 DEV had replaced that contest-rules line with "Only one submission per path is allowed. This is to encourage quality over quantity." I published a new rule pack, 2026-09-24.1, and a saved two-entry review now shows the check going from Rules unclear to Blocked, with both dated quotes. The old pack stays in Sanity, so the change is visible, not overwritten.

I added GIBC V2’s Open Invention track as a second curated event. FinePrint checks 19 Sanity requirements and 18 GIBC requirements. Compare events carries team size and development date across both, while leaving event-specific answers unknown. An answer about what existed before September 18 cannot establish what existed before July 11. Each check can be Supported, Blocked, Missing fact, Rules unclear or Not applicable. Changing a fact highlights the affected findings. Personal reviews autosave in the browser and can be downloaded as Markdown. A facts-only backup moves work between browsers without importing an unverified verdict.

Two curated events don't cover the hackathon you're entering next, so FinePrint can also read any public rules page. Paste its link and the model proposes requirements. Every requirement must quote the page word for word, or it is dropped and listed. FinePrint maps each quote onto its fixed fact vocabulary (team size, age, build window, entries and so on) and builds the typed condition itself. Anything it can't map, or any number the quote doesn't contain, becomes a Check yourself rule that can never show as Supported. Imported events are labeled "Imported from zero-origin.devpost.com, not reviewed" (with the real host) and stay in your browser, never in the Sanity dataset.

Saved reports retain the requirements and sources used at check time. When a new curated pack is published, FinePrint identifies added, removed or changed requirements and follows source references to the affected findings. A capture-date refresh alone does not pretend that a condition changed. This checks curated Sanity records; it does not monitor organizer websites.

Demo

Try FinePrint · Open the review desk · Source code

A 65-second run on the deployed app, recorded September 25 with nothing sped up: the dated rule change, the agent answering a two-entries question through Sanity Context and an imported Devpost rules page.

Browser capture from September 22. The example also declares that every member is a student. Both reviews retain unanswered questions.

Create a Sanity review, enter a team size of five and a development date of August 23, 2026, then open Compare events. Both checks are blocked for Sanity and supported for GIBC. GIBC still needs answers to its other requirements. Create a separate GIBC review to continue.

Open Try a rule-change rehearsal to lower a hypothetical team limit from six to three. One of the eighteen checks is affected. The rehearsal is labeled and local; it never changes an official source or saved review.

To try the agent, open any Sanity review. Ask FinePrint sits at the top of the desk. Choose Two entries, one path and press Ask FinePrint; the answer now cites the September 24 rule and the typed check is Blocked. Inspect the actual trace and the source interpretation beside the typed result. Add these facts to my review preserves the other answers.

To check another event, start a review and pick Another hackathon (paste its rules link). Try https://zero-origin.devpost.com/rules, then press Read the rules. You get a review of that event's requirements, with the ones FinePrint could not map listed as Check yourself. Ask FinePrint works on it too.

The homepage replays a dated, recorded source explanation. Another example lets you change whether an entire application or only its components existed before the event. The origin check changes, while the August development date remains blocked.

The landing page includes the four recorded comparison runs. Switch between team size and development date, then expand a source read to see which Sanity Knowledge Base entry and structured records were used. The GIBC date example keeps the model disagreement visible.

Live requests share a limit of five runs per ten minutes in a Vercel regional bucket. The question and any form facts explicitly included with it go to the model provider. They are not written to the public Sanity dataset.

How I Used Sanity

The public production dataset in project cxbqxkq6 contains two competitions whose current packs hold 37 requirements, plus the dated source versions each pack quotes. Superseded packs stay in the dataset. Each requirement references its event’s official sources. Event IDs, pack versions and explicit source references keep identical concepts such as team size separate. A GROQ query loads the pack, and Zod validates it before the checker uses it.

These relationships also make the change review possible. A saved finding records the requirement and source versions it used. If a curator changes the team-size requirement, FinePrint follows those references to the team finding. If only the capture date changes, it leaves the condition unchanged. Sanity holds the content and its relationships; the app uses them for retrieval, comparisons and report updates.

Knowledge Base kbyrY7h8fTnL reads the public dataset through the dedicated fineprint MCP endpoint. Its purpose now covers Sanity and GIBC Open Invention, with separate event paths, exact rule titles and source versions. The app uses a server-side Context Viewer token. The browser never receives that credential.

For a question, the agent follows this sequence:

Call initial_context to get the Knowledge Base outline.
Select relevant paths and call knowledge_base_read. The server validates each path against the outline before reading it.
Call check_requirements with quoted project facts and an independent interpretation of the relevant rules.
Validate those proposals and run the typed conditions. Return the model’s interpretation alongside the checker’s result, including disagreements.
Write the answer and cite retrieved entry paths. The question flow removes and discloses citation paths that were never read, and rejects an answer if no valid citations remain.

The question flow allows four model rounds, six entry reads and 45,000 source characters. The trace shows the actual calls and elapsed times. A provider error produces an error state with the completed steps.

The Knowledge Base reads the curated dataset records plus three official pages as website sources: the challenge page, the contest rules and DEV's general hackathon rules. When those sources were added, Context raised two conflicts. One was mine: an entry said development had to start after the opening moment, while the rules say "during, and not prior to, the Entry Period" and the checker accepts the opening moment. I resolved it in favor of the source. The other was the entry limit. The contest rules page Context crawled that day already said one submission per path, while my September 20 dataset record still quoted "no limit". Context flagged the stale record before I had noticed the change myself. I resolved it in favor of the live page, added a standing instruction that dates the old wording as a rule change and published the new pack.

For an imported event, the agent still reads the Knowledge Base for how FinePrint weighs sources, and uses a separate imported_rules_read tool over the quoted requirements. The trace names both, and citations are limited to what was actually read in that run.

What the live runs showed

The two-event test made four real calls to Sanity Context and Modal: team size and development date, once for each event. All four typed checks matched my authored labels. The model agreed on three. Every answer cited a relevant retrieved entry, and the calls took 9.2 to 14.9 seconds.

On the fourth question, the model asked for a timezone even though August 23 was well inside GIBC’s July-to-October build window. It called the result unclear; the typed checker supported the supplied date. FinePrint kept the disagreement visible. That is a failure I want to inspect before relying on the explanation.

The recorded questions, answers and tool traces include that disagreement. Four questions are a small integration check, not an accuracy study.

In a September 22 local run against the real Sanity and Modal services, the August question completed in 12.8 seconds. The agent read four entries and accepted three quoted facts. The date remained a month, with the inferred challenge year disclosed. It refused to treat “started in August” as proof that the whole application already existed. The model still interpreted the prior-work rule as blocked, while the typed check needed that missing fact. Both results were visible.

A second run took 11.4 seconds and read three entries. For “I am planning two entries in Path One,” both the source interpretation and the typed checker kept the submission limit unclear. That was correct against the September 20 rules; against the September 24 pack the same question is Blocked.

After deployment, the same question ran in a browser without an access code in 6.4 seconds. Its trace showed eight actual steps, including three Knowledge Base reads and the requirement check. A separate hosted check verified that the per-finding explanation still worked.

I tested the importer on three real pages:

Page	Rules found	Mapped to a check	Check yourself
Zero Origin (Devpost rules)	18	4	14
This challenge's DEV page	16	8	8
A lablab.ai event plus its guidelines page	9	2	7

The first runs mapped too loosely. "AI tools are permitted" became "AI use disclosed", and a start date quoted without a timezone came out nine hours early. I added two deterministic guards: a date quote must name a timezone, and a quote must contain words for the requirement it's mapped to. Those guards are why most rules end up as Check yourself. On production, the Zero Origin import took 4.3 seconds. In a local run against the real Sanity and Modal services, a question on that imported review ("We are a team of five and the youngest of us is 16") took three model rounds and 18.5 seconds, and team size came out Blocked from both the agent and the checker.

The importer's limits are real. Pages that render their rules with JavaScript expose little text, so a lablab event page gives about 1,000 characters and you need to add its rules page. The model's mappings aren't reviewed and vary between runs; the labels and quote checks keep that visible.

Those runs test particular questions and pages. They do not establish general eligibility accuracy.

Code

FinePrint uses Next.js, TypeScript, Zod, Sanity Content Lake and Context MCP. DeepSeek V4.1 Flash runs on an existing Modal endpoint. Vercel hosts the app.

Start with question-agent.ts for the tool loop, question-facts.ts for quote validation and engine.ts for the checks.

The automated suite includes migration of older reviews, event isolation, unknown facts, changed source dependencies, added and removed rules, deadline changes and bounded API requests. Production smoke checks also exercise anonymous comparisons with cloud providers disabled. The engine matches 42 authored regression scenarios. An earlier recorded model baseline, given the nine Knowledge Base entries that existed at that time, matches 39 of those labels. I wrote the fixtures and the rule pack; that comparison measures agreement with my labels, not independent accuracy or retrieval quality.

Working with coding agents

I started with a broader pitch for a pre-submission checking agent, then chose “Revise the concept together before building.” My next instruction was “uncertain eligibility.” That narrowed the project to one event and a concrete applicability problem.

Codex built the first version. I chose Modal when it asked for a model provider, approved a public rules dataset and connected a read-only Context token. I initially asked for a Three.js landing page. After reviewing it, I chose Claude’s product-focused replacement without 3D, along with a public question flow and visible tool calls. Claude stopped at its session limit partway through that upgrade. Codex recovered the unfinished work and continued it.

I then asked for a product people could keep using and chose multi-event comparison plus rule-change impact. That introduced another boundary: a project fact can travel between events, but a declaration about an event’s requirements usually cannot.

The last addition was mine to push for: "fineprint should be able to work towards all of the hacks if possible". Claude built the importer, and the live runs above are where its first mappings fell short.

The hardest boundary is deciding what a person stated. An English question does not establish an English submission; planning an integration does not establish a working integration. FinePrint rejects several such shortcuts, keeps unknown facts visible and lets the person inspect the quoted facts before applying them to the desk.

Sanity Project Details

Project ID: cxbqxkq6, dataset production (public, read-only rule records)
Public dataset query: competitions with their requirement counts
Document types: competition, requirement, sourceVersion, reviewRubric
Knowledge Base: kbyrY7h8fTnL, served through the named Context MCP endpoint fineprint

No login is needed to try FinePrint. Rule checks run without the model; live agent answers share a limit of five runs per ten minutes.

Pass It On: donate an explanation someone else can use

Himanshu Kumar — Sat, 05 Sep 2026 18:00:09 +0000

This is a submission for Weekend Challenge: Generosity Edition

What I Built

Pass It On lets you donate an explanation. Speak or type what a handbook means, correct it against the source, and leave a reviewed answer that someone else can read, hear, or save.

I built this around giving knowledge and time. A useful explanation should remain available when the next person asks the same question. The source needs to travel with it, too.

The first collection helps someone make their first open-source contribution. It uses selected excerpts from GitHub’s How to Contribute to Open Source, with attribution, a pinned revision, and the CC BY 4.0 license. It covers three questions: whether you need to write code, what to check before starting, and how to prepare a pull request.

You can read a prepared answer in English or Hindi as soon as you arrive, open its supporting excerpt, hear it through ElevenLabs, and save a copy with the source attached. Prepared answers are labelled as Pass It On adaptations. They do not count as community reviews. A volunteer can then contribute an explanation, correct it with Gemini, and leave a source-linked answer for the next visitor.

A student coding club could use this collection during an open-source introduction: a beginner reads or hears an answer, checks the source, and passes the link to a classmate. That is the use case I want to test with a group. I have verified the hosted workflow, but I have not measured adoption or observed an independent participant completing it.

Demo

Read the shared Hindi answer · Read the English starter answer · Open the website

Start with the answer. Open a supporting excerpt, then select Listen to this answer. Copy question link sends the next reader to the same question and language. Save answer downloads the explanation, exact quotes, review or preparation date, attribution, license, source version, and return link for offline reading. The saved copy says that it does not update.

Then try contributing:

Check the deliberately incorrect example. It says only programmers can contribute and documentation does not count. Gemini should identify the contradiction and ask for a correction.
Use the corrected example and check again. Open Review, compare the claims with the quoted source, and complete both acknowledgements.
Return to Questions. Pass its link to a separate browser session and retrieve the shared answer there.

You can also start a spoken interview: hear the question, consent to recording, speak your response, and edit the transcript before continuing. ElevenLabs handles transcription and speech; Gemini checks the explanation and writes the follow-up.

The sponsor step works on Solana devnet. The earlier fictional-grant practice flow supplied the payment evidence: This finalized transfer paid 0.001 test SOL to the demo contributor. Its memo identifies the approved contribution. Test SOL has no monetary value. People can contribute and reuse answers whether or not a sponsor pays the bounty.

The fictional scholarship handbook remains in a separate Practice collection for rehearsing corrections. Its answers and payment evidence are kept separate from the real guide. Visitors can try a clearly labelled demo reviewer role in either collection; it is not independently authenticated review. The real guide is general advice, so the target project’s own contribution instructions still matter.

Code

GitHub repository · Verification and known limits

The project began as Openhand, a giving ledger, and became Pass It On during this challenge. The earlier ledger remains at /ledger. I used Express, browser JavaScript, and Vercel. Claude and Codex helped with implementation, design, testing, and writing.

How I Built It

Each technology handles a specific part of the exchange:

Technology	What it does for the person using the app
ElevenLabs	Lets a volunteer speak an explanation and a reader hear an answer. It also voices the correction question, so the interview can continue aloud.
Google AI / Gemini	Compares the explanation with the handbook, flags contradictions, and asks for corrections. It returns claims paired with exact source excerpts.
Snowflake	Stores approved answers for other sessions and joins question requests with reviewed answers by language and source version. Volunteers can see which questions still need an explanation.
Solana	Links a sponsor's devnet payment to a particular contribution through the transaction memo. The server verifies finalization, sender, recipient, and amount before marking it paid.

I kept the source check and human review separate. The server rejects quotes that do not occur in the handbook. That catches invented excerpts, but an exact quote can still be interpreted badly. A successful AI check therefore opens review; it cannot approve an answer or authorize a payment by itself.

The next reader's experience drove the latest changes. First visits open on available answers. A direct link selects the question and language. Saved handouts carry their source and limitations. Shared links contain no session, review, or payment credentials. Hindi answer text and English source excerpts have separate language labels for assistive technology.

All 40 tests pass. The test suite covers collection isolation, prepared-answer attribution, proof tampering and expiry, review gates, provider failures, exact payment matching, and the sharing/export boundaries. On the hosted real-guide collection, Gemini rejected the incorrect example, accepted the correction, and returned Hindi claims with source excerpts. The demo review saved the answer to Snowflake; a separate request retrieved it without the contributor’s session. ElevenLabs played both the prepared Hindi answer and the newly reviewed answer. The live check exposed the voice key’s credit cap; after raising it within the existing free allowance, the reviewed Hindi answer played successfully. These checks were agent-operated, not independent user research. The project owner confirmed microphone recording; the earlier automated transcription check used generated sample audio.

Before opening community review to a real group, I need authenticated reviewers, correction and removal tools, and durable abuse controls. Adding a group’s own handbook is also still future work. The current bounty uses a fixed devnet recipient and a direct transfer; it is not escrow and does not guarantee exactly-once payment across simultaneous sponsors. The next useful test is a coding club trying this real guide, with a second person finding an answer without the original volunteer present.

Prize Categories

Best Use of Google AI, Best Use of ElevenLabs, Best Use of Snowflake, and Best Use of Solana.

Your MCP Server Says It Is Read-Only. Who Checked?

Himanshu Kumar — Sun, 30 Aug 2026 04:43:19 +0000

MCP servers describe their own tools. A tool can publish readOnlyHint: true, and an agent harness may use that annotation to decide whether a human approval card is needed.

That creates an uncomfortable trust boundary: the server being evaluated is also the server describing how dangerous it is.

I built Airlock to test the declaration against observed behaviour, then enforce the resulting policy on the wire.

Airlock reports what it observed. Absence of a finding is not proof of safety.

The gap

Consider an MCP tool called export_report. It declares itself read-only, returns a normal result and writes a file outside the operator's declared scope.

A harness that trusts the annotation may run it without approval. Airlock opens a case, inventories the declared tools, exercises them under a capped budget and compares the declaration with its observations.

If a tool behaves differently from what it declared, the difference becomes evidence. For an approved case, Airlock emits a connector that points to a per-case enforcing proxy rather than back to the suspect server.

The policy is enforced before the call reaches the upstream tool. It is not a sentence in a system prompt.

A controlled dishonest server

Airlock ships two six-tool fixtures with the same surface.

The dishonest fixture plants five behaviours, including a filesystem write behind readOnlyHint: true, scope escape, undeclared egress, injected instructions and canary exfiltration.

The result:

target      dishonest fixture, controlled_fixture mode
probes      24
result      7 findings of 36 checks, all five planted behaviours

The honest fixture produces zero findings across the same 36 checks.

That contrast is essential. A detector that only finds problems in a fixture designed to look suspicious has not shown much. The honest control shows that the findings follow observed behaviour rather than unfamiliarity.

Four outcomes, not a safety score

Each check resolves to one of four states:

finding
no_finding_observed
not_tested
sensor_failed

Severity is reported separately.

I deliberately avoided a single safety score. An average can hide the one tool that lied. It also encourages readers to treat missing visibility as a passing result.

not_tested is a first-class state. Under transcript_only, MCP traffic cannot reveal server-side filesystem or network activity. Airlock records those questions as untested rather than clean.

It distinguishes capability_absent, where no sensor existed, from evidence_missing, where the sensor could answer but observed nothing.

Testing servers nobody built for Airlock

The owned fixture proves that the detectors can catch planted behaviour. The more interesting question is what happens on external servers.

Airlock audited a deployed ContextFirewall target over HTTPS:

6 tools
30 probes
0 tools declaring any annotation

The server did not lie. It said nothing. A harness that resolves categories such as @write and @destructive from annotations can match nothing either way, so tools such as remember and forget_memory may avoid the approval path.

Airlock also inventoried five public stdio servers:

Server	Tools
`server-filesystem`	14
`server-everything`	13
`mcp-server-git`	12
`server-memory`	9
`server-sequential-thinking`	1

Stdio support matters because many MCP servers are launched as commands rather than exposed at a URL.

Executing what you distrust

Launching a stdio server means executing the code Airlock exists to distrust. The command therefore never comes from a case argument, tool result or model-generated string.

The operator configures a fixed map of names to argument arrays. A case may select a name. Names are looked up, never parsed into shell commands.

The child starts in a throwaway working directory with an explicit environment. Airlock revalidates the whole command binding before each connection, so repointing a configured name revokes an open case instead of silently running a different command.

Airlock documents an important remaining boundary: its HTTP response cap does not apply to the MCP SDK's stdio transport. The audit deadline still applies, but a large line can reach the SDK before Airlock can bound it. Calling that solved would be worse than stating it.

Airlock is itself an MCP server

A TrueForge agent drives the audit through six control tools:

open_case
list_declared_tools
probe_tool
read_evidence
seal_case
emit_policy

Three operations are always approval-gated: probe_tool, seal_case and emit_policy.

The agent inventories the target, asks Airlock to probe opaque tool IDs, reads the aggregate evidence and presents a human choice: Block, Approve selected or Approve all.

For an allowed case, the emitted connector points at the enforcing case proxy. A call to a tool the case did not approve receives:

MCP error -32001: Tool blocked by Airlock policy

The self-audit failed, so the failure is shown

Airlock audited its own control MCP. The server published full annotations, but the probe planner rejected open_case because its schema contains a $ref into $defs, outside Airlock's bounded v1 probe profile.

That case is incomplete. It appears on the hosted page because a product arguing that missing evidence is not proof of safety should not hide its own missing evidence.

Qodo found claim-breaking bugs

Every substantive change went through a Qodo-reviewed pull request.

On the stdio transport, Qodo found that revalidation compared only the target name. Repointing that name could leave an open case running an operator-withdrawn command. It also found that the MCP SDK inherited more host environment variables than the README claimed.

On the approval boundary, Qodo caught a test that claimed to cover every side-effecting control tool while checking three hard-coded names. The first fix introduced a second map that could drift, and Qodo caught that too. The decorators now read from the same policy map the test validates.

Another review found that a passing-suite claim did not survive a clean CI machine. The failures were fixed, re-reviewed and recorded.

The reviews did more than find code defects. They caught statements that were stronger than the observed behaviour, exactly the failure Airlock is designed to expose.

Current verification

Run the complete suite with:

python3 -m venv .venv
.venv/bin/python -m pip install -e '.[dev]'
.venv/bin/python -m pytest -q

The current suite contains 302 passing tests.

Try Airlock

The backend requires persistent state, long-running audits, child processes for stdio targets and a loopback-only operator interface, which makes a serverless deployment the wrong shape. The repository quickstart runs the complete fixture audit locally in two commands.

Airlock is open source under the MIT licence and was built for the WeMakeDevs x TrueFoundry Agent Harness Hackathon.

Do not ask a server whether it is safe. Ask what it did, keep the record and enforce the answer.

Bugs Are Innocent Until Reproduced: Building Verdict, an Evidence-First Agent Harness

Himanshu Kumar — Sun, 30 Aug 2026 04:43:06 +0000

Most flaky bug reports end in one of two places: "cannot reproduce" or a patch nobody can prove fixed the problem.

I built Verdict around a stricter idea:

Bugs are innocent until reproduced.

Verdict turns a GitHub issue into a bounded investigation. It runs an approved command repeatedly under approved conditions, keeps every observation and refuses to claim a reproduction unless the evidence crosses a deterministic threshold.

This is not an autonomous patch generator. It is an evidence-producing agent harness for the difficult step that comes before a patch.

Why another bug investigation tool?

An LLM can read a stack trace and propose a plausible explanation quickly. Plausible is not the same as reproduced.

For an intermittent failure, the questions that matter are concrete:

Which condition actually triggers it?
How often does it fail under that condition?
What happens under a contrasting control?
Which repository range does the evidence support?
What regression test would prevent the same failure from returning?

Verdict treats those as an experiment rather than a conversation.

The three-act investigation

Verdict uses three bounded subagents:

GitHub issue
    |
    v
Hunter: find the trigger
    |
    v
Surgeon: localize the change
    |
    v
Insurance: keep it fixed
    |
    v
Maintainer review

Hunter

Hunter searches only the condition matrix and command budget approved by the maintainer. Successful, failed, partial and unresolved runs all stay in the evidence ledger. An inconvenient result cannot disappear just because it weakens the story.

Surgeon

Surgeon narrows the reproduced condition to the smallest suspect range the records support. Static inspection stays visibly different from a proven execution boundary. Surgeon does not author a patch.

Insurance

Insurance converts the reproduction into a regression plan: the test name, fixture, failing assertion and publication manifest. A draft pull request can only be created through a workflow the maintainer explicitly approves.

Each act is allowed to claim less than the act before it. None can talk the deterministic reducer into a stronger verdict.

The reproduction is a record, not a screenshot

Verdict reproduced TrueForge issue #417, where snapshot registration can wait indefinitely when an upstream request never resolves.

The pinned runtime used @truefoundry/trueforge-core@0.1.4#DaytonaSandboxProvider and ran two conditions:

Condition	Result
`daytona-stalled-endpoint`	10 of 10 runs matched, `REPRODUCTION_PINNED`
`daytona-responsive-endpoint`	0 of 10 matched, `NOT_REPRODUCED`

The control is the important half. A condition that fails every time next to one that never fails is stronger evidence than twenty failures with no contrast.

Anyone can recompute the record:

pnpm --filter @verdict/agent verify:runtime-evidence

The verifier returns:

{
  "verdict": "REPRODUCED",
  "stalledRuns": 10,
  "responsiveControls": 10,
  "provider": "@truefoundry/trueforge-core@0.1.4#DaytonaSandboxProvider",
  "canonicalSha256": "a8bb5dd22e083782bd7782fccb0a1343b59fc77ea8525b6358fecc9b5b8baffa"
}

The evidence binds the observations to the TrueForge session, Hunter thread, repository commit, npm provenance commit and shared source blob.

Exploration and proof are different systems

The model gathers candidate observations. It does not decide what those observations prove.

Verdict's evidence contract is simple:

Issue text and repository content begin as untrusted input.
The investigation may use only approved commands, knobs and budgets.
Every accepted observation must match the evidence schema.
Pure reducers decide which claim the records support.
Missing or conflicting evidence produces an honest partial result.
A maintainer controls the only public write.

The same records always produce the same verdict. That is the boundary between an agent exploring a problem and a system making a claim.

What is live and what is a fixture

The recorded case renders the executed artifact. It includes both conditions, all twenty runs and the recomputable hash.

The interactive workspace is a conceptual fixture. Every generated value is labelled. It is not quietly presented as live runtime evidence.

That distinction matters for a product whose entire argument is that a claim needs a record.

Approval remains a maintainer decision

Verdict also exercised its publication boundary against real GitHub. After explicit approval, a nonce-bound workflow ran, verified the external reproduction reference and created a draft pull request in Verdict's repository. The upstream TrueForge repository remained read-only.

The workflow proof says runtimeReproducedByThisWorkflow: false. That is deliberate. The provider run reproduced the bug. GitHub Actions verified the harness and published the independently checkable proof. Combining those into one vague "verified" flag would erase the boundary.

Qodo reviewed the claims as well as the code

Every substantive change went through a pull request reviewed by Qodo before merge.

The most useful findings were not dramatic crashes. They were mismatches between the implementation and what the project claimed:

A landing card still described the real reproduction as simulated.
CI trusted the recorded verdict instead of recomputing its hash.
README copy claimed coverage on every push while the workflow covered main and pull requests.
A malformed CSS selector silently failed after a cleanup.

Those reviews fit the product philosophy perfectly: do not ship a stronger claim than the evidence supports.

Current verification

The repository runs the same gate locally and in CI:

pnpm lint
pnpm typecheck
pnpm test
pnpm build
pnpm --filter @verdict/agent verify:runtime-evidence

The current suite contains 220 tests across the agent, protocol and web packages.

Try Verdict

Verdict is open source under the MIT licence and was built for the WeMakeDevs x TrueFoundry Agent Harness Hackathon.

The goal is not to make an agent sound certain. The goal is to make certainty inspectable.

Your dog's camera roll is a wellness history. BarkPass makes it speak.

Himanshu Kumar — Sat, 15 Aug 2026 17:36:46 +0000

This is a submission for Weekend Challenge: Dog Days Edition.

Dog owners take hundreds of photos, then rely on memory for the question that matters: has anything changed?

Yesterday's posture, last week's energy, that unusually restless look. Each clue is visible for a moment, then buried between screenshots and receipts. A single photo is a memory. A sequence of photos can become a wellness history.

BarkPass makes that history queryable, audible and portable.

One daily photo becomes a structured visual observation, a short voice update spoken as the dog, a grounded history the owner can question, and an optional on-chain pet passport. BarkPass describes what is visible. It never diagnoses a condition or replaces a veterinarian.

What I Built

BarkPass is a responsive wellness companion for any dog. Its product loop is notice, hear, remember, carry.

Create a profile for your dog.
Add today's photo or a short video.
Gemini returns a structured read of visible mood, energy, posture and flags.
ElevenLabs turns that result into a short first-person voice note.
Snowflake stores the check-in and grounds plain-language trend answers in the rows that actually exist.
Solana turns the dog's identity details into a portable, independently verifiable devnet passport.

Bruno is the editorial example on the landing page, not a hardcoded user. Every new visitor receives a random dog ID. That ID scopes the profile, check-ins, Snowflake history queries and Solana metadata. Start at /app and BarkPass is about your dog, not mine.

Try BarkPass

🔗 Create a BarkPass for your dog: https://barkpass-dog-days.vercel.app/app

🔗 Explore Bruno's optional sample story: https://barkpass-dog-days.vercel.app/app?sample=1

🔗 Read the source: https://github.com/himanshu748/barkpass-dog-days

The clean app route begins with onboarding. Add a real dog profile, choose a photo, inspect the structured observation, play the voice note, ask a history question and open the passport section.

The Bruno route is only an optional sample story. A history product needs several days before its most interesting screen makes sense, so the sample provides seven check-ins immediately. It is not a separate build or a hidden judging route.

The product flow

The first check-in begins in the browser. BarkPass downsizes an image to a maximum edge of 1,600 pixels, or extracts one representative frame from a short video. This reduces upload time and keeps the provider payload bounded. The prepared JPEG is sent only after the owner chooses the file.

Gemini returns a structured observation. BarkPass saves the normalized result as a check-in, then uses the same result to prepare the dog's voice note. Each additional check-in extends the dog's history. The owner can ask a plain-language question such as "Has Luna's energy changed this week?" and receive an answer calculated from Luna's queried Snowflake rows.

The passport is a separate, optional action. It carries identity and care details, not the owner's private image history. Minting returns public Solana Explorer proof. A shelter tip is prepared as an unsigned devnet transaction and still requires the owner to approve it in Phantom.

The product stays useful even when one provider is unavailable. Local profile and check-in history continue to work. Every non-provider result is labelled in the interface so sample data is never presented as live sponsor output.

The architecture, end to end

The four integrations are stages of one pipeline, not four logos attached to a landing page.

The diagram is a static 1,000 × 1,200 portrait image with large type, so it stays readable at DEV's article width without depending on Mermaid support.

The five layers

Layer	Responsibility	What crosses the boundary
React client	Dog profile state, media preparation, check-in UI, audio playback, Phantom connection and labelled local resilience	A prepared JPEG, normalized observation records, dog identity fields and owner-approved wallet actions
Vercel Functions	Validation, provider orchestration, normalization and credential isolation	Only the minimum provider-specific payload; no provider secret is returned to the browser
Google AI + ElevenLabs	Convert one selected frame into structured visible signals, then convert a bounded sentence into audio	Gemini receives the prepared frame; ElevenLabs receives the derived sentence, never the photo
Snowflake	Durable dog profiles, idempotent check-ins and dog-scoped trend queries	Text identity fields and structured observations; the original profile/check-in photo is not stored there
Solana devnet	Public passport metadata, one-of-one mint proof and unsigned shelter-tip preparation	Selected passport identity fields and the connected public wallet address

One check-in request sequence

Prepare in the browser. A chosen photo is resized to a maximum edge of 1,600 pixels and encoded as JPEG at 0.84 quality. For a short video, BarkPass seeks to an early representative frame and sends that frame rather than the whole clip. Nothing leaves the browser before the owner chooses media.
Observe through POST /api/analyze. The function checks the method, media type and payload size before calling gemini-2.5-flash. Gemini is constrained to a JSON schema. The function parses and normalizes the response again, clamps energy and confidence to their allowed ranges, caps visible flags, and derives the bounded voice sentence on the server.
Speak through POST /api/voice. Only that short sentence is sent to ElevenLabs. The function limits it to 500 characters and returns private-cacheable audio/mpeg; the browser creates a temporary object URL for playback.
Remember through POST /api/dogs and POST /api/checkins. Snowflake receives the current random dog_id, profile fields and the normalized observation. MERGE makes profile and check-in retries idempotent instead of duplicating history.
Answer through POST /api/query. The function synchronizes at most 30 recent local check-ins, selects only rows for the requested dog_id, and computes the row count, average, range, direction and most common mood from the returned values. The prose is assembled from those facts; an unrestricted model does not invent the trend answer.
Carry through POST /api/solana/mint. The server validates the current dog, creates a same-origin metadata URL and uses Metaplex to mint a one-of-one BARK token on devnet. The response returns both the mint and transaction Explorer URLs.
Preserve owner control through POST /api/solana/tip. The server builds a 0.01 SOL devnet transfer with the connected wallet as fee payer, but deliberately does not sign it. The browser hands the serialized transaction to Phantom; only the owner can approve and broadcast it.

Identity and data isolation

The same random dog ID is the join key across browser storage, BARKPASS_DOGS, BARKPASS_CHECKINS, history queries and passport metadata. It is generated when a real profile is created and preserved when that profile is edited. Replacing the optional Bruno sample clears every sample-only field and creates a fresh ID, so a new owner cannot inherit Bruno's photo, microchip, vaccination or seven-day history.

Snowflake has two durable entities:

BARKPASS_DOGS     dog_id → name, breed, age, microchip, vaccination
BARKPASS_CHECKINS (checkin_id, dog_id) → date, mood, energy,
                    posture, flags, confidence, summary

The compound check-in key matters: checkin_id makes retries safe, while dog_id prevents one dog's retry from overwriting another dog's record. Query parameters are bound rather than concatenated, and history reads always include WHERE dog_id = ?.

Trust boundaries and failure behavior

The browser is treated as public. Gemini, ElevenLabs, Snowflake and Solana credentials exist only in Vercel's server environment. Functions validate and bound every incoming field before a provider call. Snowflake uses RSA key-pair authentication through a dedicated least-privilege service role, while the Solana authority is limited to devnet. The shelter-tip path is intentionally split: the server can prepare a transaction, but Phantom retains the owner's signing authority.

Each client adapter also has an explicit resilience boundary. If vision is unavailable, a clearly labelled sample observation keeps the interface demonstrable. If voice fails, device speech can keep the ritual audible. If Snowflake is unavailable, the current browser retains local history and computes the same transparent statistics. On-chain actions never pretend to succeed: live responses return Explorer evidence, verified examples are labelled as examples, and a real tip cannot leave the wallet without Phantom.

This separation is what lets BarkPass degrade one capability without collapsing the whole daily check-in—and without presenting fallback output as live sponsor output.

How Google AI observes without pretending to diagnose

Gemini does not receive a vague "how is this dog feeling?" prompt. The server gives it a narrow role: careful veterinary behavior observer. It may describe only visible signals, must avoid diagnosis and must return JSON matching a strict schema.

const schema = {
  type: 'object',
  additionalProperties: false,
  required: [
    'mood',
    'energy_level',
    'posture_notes',
    'health_flags',
    'confidence',
  ],
  properties: {
    mood: {
      type: 'string',
      enum: ['anxious', 'relaxed', 'playful', 'alert', 'tired'],
    },
    energy_level: { type: 'integer', minimum: 1, maximum: 10 },
    posture_notes: { type: 'string' },
    health_flags: { type: 'array', items: { type: 'string' } },
    confidence: { type: 'number', minimum: 0, maximum: 1 },
  },
}

The endpoint uses Gemini's JSON response mode with temperature 0.2, then normalizes every field again before rendering it. The closed mood vocabulary and numeric bounds make daily observations comparable. A paragraph that changes tone every day would be difficult to chart or query. A stable record is useful.

The most important product rule is: describe the frame, do not invent the dog. A low confidence score stays visible. A visible flag becomes a reason to look again, not a medical conclusion.

I ran five licensed dog photographs through the live provider path. Gemini returned four distinct moods and energy values from 4 to 9.

Photo	Mood	Energy	Confidence	API latency
1	Playful	6/10	0.80	10.53 s
2	Playful	9/10	0.90	4.30 s
3	Anxious	4/10	0.40	8.38 s
4	Relaxed	7/10	0.90	6.81 s
5	Alert	7/10	0.80	4.68 s

Average Gemini latency was 6.94 seconds. The test matters because a vision feature that returns "happy dog" for every photograph is only decoration.

Why ElevenLabs is more than a play button

A score is useful. A sentence is memorable.

BarkPass turns the normalized observation into one short first-person line. High energy produces a more active suggestion. Low energy produces a quieter one. A visible flag asks the owner to take another look without claiming a diagnosis. That bounded line is sent to ElevenLabs using eleven_flash_v2_5 and returned as a 44.1 kHz, 128 kbps MP3.

A timed live call produced an 84,889-byte MP3 in 1.55 seconds. The final Production recheck returned HTTP 200 with a playable 52,288-byte MPEG; output size varies with the generated sentence. Combined with the slowest timed Gemini call, the measured provider path was 12.08 seconds, inside the brief's 15-second target.

If the provider cannot respond, BarkPass can use device speech so the daily ritual does not dead-end. The interface labels that output as a fallback. Device speech is resilience, not evidence of ElevenLabs usage. The provider-backed measurement and playable MP3 are the evidence.

That distinction matters because BarkPass is not claiming to translate dogs. The voice is an interface for recall. Owners are more likely to remember "I am taking things slowly today" than a row containing energy_level: 4.

Why Snowflake is the product's memory

One observation is a moment. Several observations become a pattern.

Profiles are merged into BARKPASS_DOGS. Check-ins are merged into BARKPASS_CHECKINS using both checkin_id and dog_id. The compound match makes retries idempotent and prevents a repeated save from creating duplicate history.

MERGE INTO BARKPASS_CHECKINS AS target
USING (...) AS source
ON target.checkin_id = source.checkin_id
AND target.dog_id = source.dog_id
WHEN MATCHED THEN UPDATE SET ...
WHEN NOT MATCHED THEN INSERT (...)

History questions query only the requested dog's rows:

SELECT checkin_date, mood, energy_level, health_flags, summary_text
FROM BARKPASS_CHECKINS
WHERE dog_id = ?
ORDER BY checkin_date DESC
LIMIT 30

The returned sentence is calculated from those energy values and mood counts. BarkPass reports the row count, average and range beside the answer. It cannot invent a trend that is absent from the query result.

The deployment uses an X-Small auto-suspending warehouse and a dedicated service user. Key-pair authentication is preferred over a password, and the runtime role is limited to BarkPass's database, schema, tables and warehouse.

In live Production verification, BarkPass stored Maple and three check-ins with energy values 5, 7 and 8. The API reported an average of 6.7, a range of 5 to 8 and a higher final value. A fresh Snowflake worksheet aggregate independently returned the same row count, average, minimum and maximum.

Snowflake is not an analytics screenshot added after the product. It is why BarkPass can answer "Has her energy changed?" with the supporting numbers beside the sentence.

Why Solana carries the passport

A pet passport is more useful when it is not trapped inside one application's database.

The mint endpoint validates the dog first, then generates a public metadata URL containing name, breed, age, microchip ID and vaccination date. Metaplex creates a one-of-one NFT on Solana devnet with symbol BARK, zero royalties and BarkPass's server vault as the mint authority.

Maple's BarkPass was minted from the public Production route as B3FRp9ndjjiVbFwb7MKruabsHicPQQCnL6JdksErEWwa. The mint transaction finalized without error. Explorer shows the BARK non-fungible token, Maple's public metadata URL and BarkPass's new funded devnet vault as authority.

The shelter-tip endpoint does something deliberately different. It prepares a 0.01 SOL devnet transfer with the connected wallet as fee payer, but the server does not sign for the owner. Phantom receives the serialized transaction and must explicitly approve it before broadcast.

Devnet SOL has no monetary value. The proof is the verifiable lifecycle: generate dog-specific metadata, mint once, return Explorer evidence and preserve wallet approval for a transfer.

The hardest engineering decisions

Making the app about every dog, not Bruno

The landing page needed a memorable dog, but the product could not inherit that dog's identity. Profile creation now generates a random dog ID. Check-ins are stored under that ID in the browser and in Snowflake. Passport metadata is created from the current profile. A contract test specifically asks for Luna's history and fails if Bruno appears anywhere in the response.

Keeping a four-provider flow understandable

It would have been easy to show four independent buttons. That would satisfy a checklist but create no product. BarkPass instead gives each provider one job in a single narrative: Gemini notices, ElevenLabs gives the moment a voice, Snowflake remembers and Solana lets the passport travel.

Designing honest failure states

Provider credentials stay in server environments. The public deployment does not expose billable keys to the browser. If a provider is unavailable, BarkPass keeps local history usable and labels the resulting source. It never paints a fallback response as live AI, Snowflake or on-chain activity.

Making a weekend build feel complete

The app supports photo and short-video input, loading, empty, success and error states, keyboard focus, reduced motion, 390-pixel mobile layouts and desktop layouts. The first-run reveal transition was browser-tested after it briefly produced a blank dashboard. The Solana function bundle also needed rpc-websockets pinned to 9.3.8 so Vercel's CommonJS build received a compatible uuid release.

Privacy, safety and reliability boundaries

Profile photos stay in the owner's browser.
A prepared check-in frame is sent only after the owner selects a file.
Text profile details and check-ins sync to Snowflake when that route is available.
Dog IDs scope every profile, query and passport.
Local profiles and history remain usable when a provider is unavailable.
Fallbacks and sample data are explicitly labelled.
Passport minting uses Solana devnet.
Shelter tips require Phantom approval.
BarkPass is a wellness companion, not veterinary advice.

The public client bundle is intentionally secret-free. All four real provider paths now run through the public Production deployment's same-origin server functions, while the repository contains the complete server implementation and reproducible verification record. No provider key, Snowflake private key or Solana vault secret is shipped to the browser.

How I tested it

The Node test suite covers provider contracts and the isolation boundaries that would be easiest to get wrong:

Gemini JSON normalization and the generated voice line
playable ElevenLabs audio bytes
incomplete Snowflake check-in rejection
Snowflake key-pair authentication without password exposure
integration status with key-pair credentials
dog profile validation
dog-specific history with no Bruno inheritance
dynamic Solana metadata for the requested dog
validation before any mint creates chain state

Run the same checks locally:

npm install
npm test
npm run build

All thirteen contract tests pass. The production Vite build passes. I also verified the first-run flow, sample-to-personal profile isolation, the optional seven-day sample, responsive behavior at 390 pixels, public metadata, the finalized devnet mint, a playable ElevenLabs MP3, the five-photo Gemini set and Snowflake's independent aggregate.

Run it yourself

Clone the repository and copy the environment contract:

git clone https://github.com/himanshu748/barkpass-dog-days.git
cd barkpass-dog-days
cp .env.example .env.local
npm install
npm run dev

The interface works locally with labelled resilience paths. To exercise live providers, configure the values documented in .env.example and point VITE_API_BASE_URL at a running functions deployment. The core server variables are:

GEMINI_API_KEY
ELEVENLABS_API_KEY
SNOWFLAKE_ACCOUNT
SNOWFLAKE_USER
SNOWFLAKE_PRIVATE_KEY_BASE64
SNOWFLAKE_WAREHOUSE
SNOWFLAKE_DATABASE
SNOWFLAKE_SCHEMA
SOLANA_RPC
SOLANA_VAULT_KEY

Real values belong only in local or deployment secrets. They must never be committed.

Prize Categories

I am entering BarkPass for Overall Winner, Best Use of Google AI, Best Use of ElevenLabs, Best Use of Snowflake and Best Use of Solana.

The theme connection is direct: the whole product begins with a dog owner's daily photo. The creative choice is to turn that familiar habit into something the owner can hear, question and carry. The technical execution is the connected pipeline and its source boundaries, not the number of APIs used.

Known limitations

BarkPass is a weekend prototype, not a clinical product. Visual observations can be wrong, which is why confidence stays visible and the language remains non-diagnostic. The Production app has all four provider routes configured, while labelled local resilience keeps a check-in usable during a provider outage. The shelter-tip transaction has been prepared successfully, but final broadcast still requires an owner's Phantom approval. There is no account system or multi-device sync yet.

These are explicit boundaries, not hidden promises.

What comes next

The next version would add authenticated multi-device profiles, owner-controlled sharing with carers or veterinarians, reminders, exportable wellness summaries and rate-limited public provider routes. I would also evaluate Gemini against a larger, more diverse photo set and make the passport updateable through an owner-controlled authority.

Most pet apps begin by asking owners to become better record keepers. BarkPass begins with something they already do: take a photo.

That small habit becomes a story they can hear, question and carry with their dog.

I Bought a ₹6 Share and Learned the Hard Way: Building FinEd Saathi in 10 Days

Himanshu Kumar — Sat, 15 Aug 2026 04:47:37 +0000

I once bought a share at about ₹6 without understanding what the complete transaction would cost. I later sold it around the same price and was surprised to see a loss of roughly ₹50 somewhere in my account. I did not know whether that number came from the contract note, ledger, available funds or P&L view. The important lesson was not that every small trade has the same charge. It was that I had acted without understanding brokerage, taxes or where a number in a broker app actually came from.

That experience became the starting point for FinEd Saathi, the voice-first financial literacy tutor I built during 10 Days of Voice Agents - VoiceForBharat Edition. I chose the Financial Services track because I wanted to solve a problem I had felt myself. The challenge listed Financial Services and Learning & Literacy as two separate tracks. I formally stayed in Financial Services, but I deliberately merged finance with education in the product because access without understanding was the problem I wanted to solve. The tutor is for beginners who want Indian market concepts explained in a patient conversation before they put real money at risk. The public code is in the FinEd Saathi repository.

The problem I wanted to solve

Financial education often assumes that a learner already knows the vocabulary. A person who is still asking what an ETF is may be sent to a fee schedule, a tax circular or a dense product page. Even correct information can be hard to use when several unfamiliar concepts arrive at once.

I wanted the product to begin where the learner is. FinEd can explain stocks, mutual funds and SIPs, ETFs, gold, F&O, IPOs and bonds. It can also unpack a confusing charge without inventing a reason. In my ₹6 share story, the agent treats the price profit or loss as zero until I show where the separate loss appeared, such as a contract note, ledger or P&L view. That is more useful than confidently guessing.

The landing page lets a beginner choose a topic before starting a voice lesson.

Why voice changed the product

Voice makes the lesson feel like a conversation instead of an exam. A learner can ask a short question, interrupt or switch between English, Hindi and a code-mixed register. Deepgram Nova-3 handles multilingual speech recognition. Gemini handles the teaching conversation and tools. LiveKit carries the real-time session.

Murf Falcon is the fastest TTS API. I use Murf Falcon 2 with Nikhil as the FinEd voice because his Indian conversational delivery fits a patient tutor. Murf documents Falcon 2 as a real-time speech model with roughly 100 ms time to first audio. The low-latency path matters because a financial explanation quickly feels unnatural when every turn contains a long pause.

The voice workspace shows what the agent heard, which specialist is active and what the learner can do next.

How FinEd Saathi works

The browser sends audio through LiveKit to a Python worker. Deepgram transcribes the speech, Gemini produces a bounded response and Murf Falcon 2 streams the spoken answer. FinEd owns ordinary lessons. TaxEd is a separate specialist reached only after permission. Optional market data, memory, paper trading, human help, outbound telephony and analytics sit outside the essential voice path so missing optional credentials do not break the core tutor.

The main voice session is deliberately compact:

fined_tts = murf.TTS(
    voice="Nikhil",
    style="Conversational",
    model="falcon-2",
    locale="en-IN",  # Initial locale, selected again before each reply stream.
    tokenizer=tokenize.basic.SentenceTokenizer(min_sentence_len=2),
    text_pacing=True,
)
fined_locale = FinEdTTSLocaleController(fined_tts)

session = AgentSession[SessionState](
    userdata=state,
    stt=fined_stt,
    llm=fined_llm,
    tts=fined_tts,
    turn_detection=inference.TurnDetector(version="v1-mini"),
    vad=ctx.proc.userdata["vad"],
    preemptive_generation=True,
)

Before LiveKit opens each FinEd speech stream, the controller peeks at the actual reply. Devanagari Hindi selects hi-IN; English and Latin-script code-mixed replies select en-IN. That keeps Nikhil fixed while applying a supported locale before Murf snapshots the stream options. TaxEd creates a separate Murf TTS instance using Anusha and a server-normalized en-IN, hi-IN or hi-LATN locale.

What the learner can actually do

A learner can ask for definitions, compare concepts or request a read-only market quote when optional Angel One access is configured. Quote tools can search instruments, return current prices and fetch historical daily closes. They cannot read holdings, positions or account information. If the broker token is missing or expired, the agent says that live data is unavailable instead of guessing a price.

The paper portfolio starts with ₹1,00,000 in virtual cash. It supports simulated NSE EQ cash equity or ETF delivery orders based on a fresh quote. The browser prepares a draft then requires explicit confirmation of that same unexpired draft. The portfolio, virtual cash and fill history stay in browser storage. No real broker order API is called.

Paper trading gives the learner a safe place to practise without connecting a real trading account.

Sessions behave more like a familiar chat product. Meaningful transcripts are stored under local session identifiers in the browser so the learner can switch conversations. Caller memory is separate, private and consent-gated. It saves only learning preferences after a fresh yes. Credentials, government identifiers, account numbers, holdings, income and bank details are excluded.

The analytics page reports anonymous totals such as call count, speaking time, committed handoffs and successful calls. It stores no audio, transcript, utterance text, caller identity or phone number.

Analytics proves whether the experience completed useful actions without turning learners into a surveillance dataset.

Why I built TaxEd as a separate specialist

Investment tax questions need a stricter evidence boundary than a general concept lesson. FinEd first asks whether the learner wants to connect. Only a fresh explicit yes transfers the tax question to TaxEd. Returning to FinEd also requires permission.

TaxEd speaks with Anusha, a Murf Indian voice whose official voice-library entry supports English and Hindi. It searches a packaged registry of official Indian tax rules, states the relevant applicability date and links the source. A rule must be current for the requested date. When a current packaged source is missing, uncertain or past its review date, TaxEd abstains. It does not calculate a person's final liability, file a return or help evade tax.

The handoff is visible, consented and evidence-first.

Safety was part of the architecture

FinEd is education, not investment advice. Deterministic guardrails run before provider inference for real trading, personalized recommendations, guaranteed outcomes, unsafe F&O calls, credential requests and tax evasion. A request to place a real order is refused even when it also mentions paper trading.

The system never asks for a broker password, PIN, OTP, PAN, Aadhaar, bank detail or full account number. F&O mode teaches mechanics, payoff examples and risk only. Human-help requests use a short redacted summary and require fresh consent before storage. Outbound calling is an operator-only optional path that also requires explicit consent. Missing telephony configuration leaves browser voice available and makes the call fail closed.

I documented the deterministic evidence in RED_TEAM.md, including broker identifier redaction during a TaxEd handoff, refusal before model inference and abstention on an unverified tax rule.

The hardest parts

The hardest work was not drawing the interface. It was making state transitions reliable. A handoff that only changes a badge is not a handoff. I had to keep the active specialist, the TTS voice, the transferred question and the return path aligned. I also had to stop repeated permission loops when a clear yes had already been given.

Multilingual speech exposed a similar issue. A multilingual voice identity is not enough when the synthesis locale is fixed. FinEd now routes Nikhil to hi-IN for Devanagari Hindi and en-IN for English before audio begins. TaxEd keeps Anusha while a server-normalized en-IN, hi-IN or hi-LATN locale follows the handoff language. The browser cannot choose arbitrary synthesis settings.

Live market data taught me to design for absence. Broker access tokens expire. The correct fallback is not a stale or invented price. The agent now says the quote is unavailable while concept lessons, the empty paper dashboard and other safe features continue.

I also learned that proof assets are part of product quality. A blurred screenshot can hide the very behavior the article claims to demonstrate. I recaptured the specialist handoff and analytics views so the state, source and privacy boundary are readable.

Run it yourself

You need Python 3.10 through 3.14, uv, Node.js and the repository's pinned pnpm 9 release. Clone the public repository then create local environment files:

cp backend/.env.example backend/.env.local
cp frontend/.env.example frontend/.env.local

Add these required backend values to backend/.env.local:

LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
MURF_API_KEY
DEEPGRAM_API_KEY
GOOGLE_API_KEY

Use the same LiveKit project values in frontend/.env.local and keep the worker name:

LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
AGENT_NAME=my-agent

Keep real values only in .env.local. Never commit them. Install the backend and its local voice models:

cd backend
uv sync
uv run -m livekit.agents download-files
uv run dotenv -f .env.local run -- python src/agent.py start

In a fresh shell, install and run the frontend:

cd frontend
pnpm install
pnpm dev --port 3001

Open http://127.0.0.1:3001, select a learning mode, choose Talk to FinEd Saathi and allow microphone access and browser audio playback. Ask "What is an ETF?" Then ask how an equity ETF is taxed and say "Yes, connect me to TaxEd" when FinEd requests permission.

Common fixes:

Connection fails: confirm frontend and worker use the same LiveKit project and AGENT_NAME.
Voice is silent: confirm MURF_API_KEY, microphone permission and browser audio playback.
Live quote unavailable: refresh the optional Angel One access token. Do not guess a quote.
TaxEd abstains: verify the packaged official rule is current for the requested date.
Outbound call fails: verify explicit consent, the stored LiveKit SIP trunk and Twilio termination setup.

Angel One, Twilio, the local knowledge index and outbound calling are optional. A normal voice lesson works without them.

How I tested it

The backend has a deterministic suite for guardrails, tools, handoffs, tax rules, memory, analytics and outbound consent. The frontend contracts cover the public interface, browser paper portfolio, token route and documentation. I keep the provider-backed evaluation separate because it requires valid external credentials and inference access.

cd backend
uv run pytest -q --ignore=tests/test_agent.py
uv run ruff check .
uv run ruff format --check .

cd frontend
node --test tests/*.test.mjs
pnpm exec tsc --noEmit
pnpm format:check
pnpm build

The red-team record states exactly which deterministic tests support each claim. I prefer that to describing a manual demo as proof of every failure path.

What I would build next

I would make the source registry easier to update with a review workflow, add more broker-independent historical lessons and test interruption latency across a wider set of Indian network conditions. I would also add an export that lets a learner keep a private summary of concepts they understood without exporting a transcript.

Ten days turned one confusing small trade into a product I personally find smooth to use. More importantly, it changed how I think about a finance assistant. The best version is not the one that sounds most confident. It is the one that teaches clearly, proves its sources, asks permission at boundaries and knows when to stop.

Tracing a multi-agent LLM system: otel-swarm and a SigNoz dashboard pack

Himanshu Kumar — Sun, 26 Jul 2026 17:40:05 +0000

The problem

A single LLM call is easy to reason about. You send a prompt, you get tokens back, you log the latency and you move on.

A swarm is not that. Four or five agents run, some in parallel, each with its own model. One of them times out and quietly falls back to a cheaper model. A critic agent reads the output and sends work back for another round. When the whole thing takes 40 seconds instead of 12, or costs three times what you budgeted, you have no idea which agent did it. Your logs are a flat stream of interleaved lines from concurrent tasks, with no parent-child structure and no way to ask "which role burned the tokens".

Traces solve this exactly. The catch is that hand-wiring OpenTelemetry across every role, every provider call and every retry path is boring work that nobody wants to do twice, and if you skip a level the trace tree lies to you.

So I extracted the instrumentation out of DevSwarm (a multi-agent code generator I built for this hackathon) into a standalone MIT library: otel-swarm.

The API

One createSwarm() call, then three verbs.

import { createSwarm } from 'otel-swarm';

const swarm = createSwarm({ service: 'my-swarm', otlpEndpoint: 'http://localhost:4318' });

await swarm.task('generation', { 'my.prompt': prompt }, async (root) => {
  const plan = await swarm.agent('planner', () =>
    swarm.llm('planner', {
      model: 'primary-model-id',
      fallbackModel: 'fallback-model-id',
      call: async (model) => {
        const res = await yourProviderCall(model);
        return { content: res.text, inputTokens: res.usage.in, outputTokens: res.usage.out };
      }
    })
  );

  await swarm.agent('critic', async (span) => {
    swarm.reviewEvents(span, issues);   // each issue becomes a critic_catch span event
  });
});

swarm.events.on('event', (e) => { /* llm_start, llm_end, fallback, critic_catch */ });

task() opens the root span, agent() opens a child span per role and llm() opens a GenAI-semconv span per model call. call(model) is invoked again with fallbackModel if the primary throws, and the switch is recorded as a fallback_promotion span event carrying from, to and the verbatim provider error.

The last line matters more than it looks. swarm.events is an EventEmitter that mirrors the span lifecycle, and LLM events carry the traceId. A live UI reads the emitter, the tracing backend reads the OTLP exporter and both are fed from the same code path, so the dashboard and the UI can never disagree about what happened. The traceId means a row in your UI deep-links to the exact trace in SigNoz.

Running npm run example against a local SigNoz produced 9 spans under service otel-swarm-demo: a root generation span of 1325ms, then agent.planner/llm.planner, agent.frontend/llm.frontend, agent.backend/llm.backend and agent.critic/llm.critic. llm.frontend carried a fallback_promotion event and agent.critic carried a critic_catch event. The whole story is one trace.

In production use inside DevSwarm the same library has traced 29 generations, 243 model calls across 8 models, 3.78 million tokens and 257 review catches.

Lesson 1: put the fallback in a span event, never on the attribute you group by

This one cost me an afternoon.

An earlier version of the library did the obvious thing when a primary model failed: it overwrote gen_ai.request.model with the fallback's name, so the span would "tell the truth" about which model actually answered.

That is wrong, and it is wrong in a way that is hard to see. Every dashboard panel that groups by model then attributes the primary's failure, its timeout and all of its wasted latency to the fallback that cleaned up after it. My "tokens and latency by model" table showed a cheap fallback model with terrible p95 latency and my expensive primary looking flawless, because every time the primary blew up its cost was silently reassigned to whoever picked up the pieces. I spent an afternoon convinced the wrong model was slow.

The rule that comes out of it: the attribute you group by must record the model you attempted, not the model that ended up answering. The promotion is a discrete thing that happened during the span, and a discrete thing that happened during a span is what span events are for:

span.addEvent('fallback_promotion', { from: model, to: fallbackModel, reason: String(err.message || err) });

Now the grouped panel keeps attributing failure to whoever caused it, and "how often does this role get promoted" is a separate query over events. Two different questions, two different storage locations, no cross-contamination.

The general form: if a value can change mid-span, it does not belong on an attribute you aggregate over.

Lesson 2: reading inside span events in ClickHouse is not obvious

The consequence of putting things in events is that you now have to query events, and SigNoz stores them as an array of JSON strings on the span row. There is no autocomplete that will lead you here. The pattern is ARRAY JOIN to flatten the array into one row per event, then JSONExtractString to reach into the event's attributeMap:

SELECT JSONExtractString(e, 'attributeMap', 'severity') AS severity,
       JSONExtractString(e, 'attributeMap', 'target') AS target,
       count() AS catches
FROM signoz_traces.distributed_signoz_index_v3
ARRAY JOIN events AS e
WHERE serviceName = '{{.service}}' AND name = 'agent.critic'
  AND JSONExtractString(e, 'name') = 'critic_catch'
GROUP BY severity, target ORDER BY catches DESC

That gives you review findings broken down by severity and by what they were aimed at, straight out of span events, with no separate metrics pipeline.

When you only need to know whether an event fired at all, do not pay for the ARRAY JOIN. arrayExists with a substring test over the raw array is enough and it keeps one row per span, which is what you want for a time series:

SELECT toStartOfInterval(timestamp, INTERVAL 15 MINUTE) AS ts,
       attributes_string['swarm.role'] AS role,
       count() AS value
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = '{{.service}}'
  AND arrayExists(e -> e LIKE '%fallback_promotion%', events)
  AND timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}
GROUP BY ts, role ORDER BY ts

The dashboard pack

The repo ships 3 importable SigNoz dashboards (POST each JSON to /api/v1/dashboards). Every query uses a {{.service}} dashboard variable that defaults to otel-swarm-demo, so you point the pack at your own service by editing one dropdown instead of doing find-and-replace across query strings.

Generation Overview answers "is the swarm healthy right now": generation count, review-gate pass rate, seconds per generation over time and a per-role table of calls, average latency, p95 and errors.
LLM Economics answers "where did the tokens go": total tokens, tokens over time by role, a tokens-and-latency table by model and fallback promotions by role (the arrayExists query above).
Review-Gate Funnel answers "is the reviewer doing its job": total catches, regeneration rounds, catches per generation over time and a table of recent generations with verdict, catches and regenerations.

The alert rules

Two rules ship in the v2alpha1 schema (POST to /api/v2/rules).

The first is a fallback-promotion spike. If promotions jump, a provider is degrading and you are silently paying a different bill than you planned.

The second is the one I care about more: a review catch-rate flatline. If your critic agent suddenly stops finding anything, the tempting read is that the generators got better. In practice the reviewer broke: a prompt change made it answer in a shape the parser drops, or its model started returning empty content and the failure is being swallowed as "no issues found". A quality gate that passes everything is indistinguishable from no quality gate, and it fails silently by construction. Alert on the absence.

The repo also includes casting.yaml and casting.yaml.lock, the Foundry config the SigNoz instance behind these dashboards was installed from, so the backend the pack targets is reproducible rather than assumed.

Install and run

npm install github:himanshu748/otel-swarm

To see the example trace end to end against a local SigNoz:

git clone https://github.com/himanshu748/otel-swarm && cd otel-swarm
npm install
npm run example                                                     # spans to console
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npm run example   # spans to SigNoz

With no endpoint set it exports to the console, so you can check the span tree before wiring a backend.

Closing

The library is about 130 lines. Most of the value is not code, it is the two decisions above: keep mutable facts in span events, and treat "the reviewer found nothing" as a symptom rather than a result. Both were learned by getting them wrong first.

I built otel-swarm with Claude Code, including the dashboard JSON and the ClickHouse queries in this post.

Repo: github.com/himanshu748/otel-swarm, MIT.

We instrumented an AI agent swarm with SigNoz, and its own telemetry told us we were wrong about almost everything

Himanshu Kumar — Sat, 25 Jul 2026 15:01:20 +0000

Built for the WeMakeDevs Agents of SigNoz hackathon, July 2026.

Mission Control. The graph is the swarm, the river underneath it is the live span stream, and every bar deep-links into that trace in SigNoz.

DevSwarm turns one prompt into a working full-stack app. Five open-weight models plan it, build it, review it and repair their own routing. Nothing it produces is trusted blindly, and every step is an OpenTelemetry span in SigNoz, including the steps that go wrong.

We built the observability first, expecting it to prove the thing worked.

It did something more useful. It spent a week proving that almost everything we believed about our own system was wrong. We blamed a model for a limit we had set ourselves. We blamed a provider outage on the model. We assumed our review agent was our strongest link when it was measurably the weakest. And we spent days writing a design system that, when we finally measured it, was making the output worse.

Not one of those was found by reading the code again. Every single one came off a span event, a dashboard row or a benchmark that disagreed with us.

So this is not an architecture post. It is six times the telemetry told us we were wrong, with the queries.

The current numbers, all read live out of SigNoz rather than typed into a slide: 22 generations, 189 traced model calls across 8 models, 2.85 million tokens, 225 critic catches, 19 fallback promotions and 24 generated apps each reporting under their own service name.

What the swarm actually is

Five roles, each on the open-weight model that measured best for that job:

role	model	job
planner	GLM-5.2	turn a prompt into a typed build plan and a locked API contract
frontend	GLM-5.2	one self-contained index.html against that contract
backend	Qwen3-Coder-480B	one Express server against the same contract
critic	Kimi-K2.7-Code	review both, gate the merge, route catches back to their owner
doctor	GLM-5.2	read the swarm's own traces and repair its model routing

Every number on our own landing page is a live ClickHouse query against the trace store. Marketing copy that drifts from the telemetry is impossible by construction.

Everything is served through Hugging Face Inference Providers. There are zero closed-model API calls in the system, which turned out to matter for reasons we did not anticipate (see the provider section below).

The critic is the load-bearing part. Frontend and backend are generated in parallel from the same contract, then an independent model reviews both for contract conformance, security and runtime bugs. Real catches route back to the agent that owns them, that agent patches its own file and the critic re-reviews only the delta. Two regeneration rounds, then it ships with an honest verdict either way.

The same gate runs on changes. Asking a finished app for "a star rating on each book, settable when adding one" re-plans against the stored contract, re-runs only the agents that instruction touches, and puts the result through the identical review. A refinement is its own root span, so a change request is as traceable after the fact as the build that preceded it.

Why we instrumented before we polished

A multi-agent system fails in ways a single-model tool does not. A call can succeed while producing an unusable artifact. A fallback can rescue a request so smoothly that nobody notices the primary is dead. Latency can triple because one role silently started thinking twice as long. None of that shows up in a request log.

So the very first thing that worked in this project was not code generation. It was a trace.

SigNoz is self-hosted through Foundry, which is a single-config install. Our casting.yaml and casting.yaml.lock are committed to the repo so the deployment is reproducible by anyone, judges included.

Three signals, and what each one carries

Traces. Every model call is a span named llm.<role> carrying GenAI semantic conventions:

span.setAttributes({
  'gen_ai.operation.name': 'chat',
  'gen_ai.request.model': model,
  'gen_ai.usage.input_tokens': usage.prompt_tokens,
  'gen_ai.usage.output_tokens': usage.completion_tokens,
  'devswarm.role': role
});

Two span events do the heavy diagnostic lifting. fallback_promotion records that a primary failed, which model took over and the verbatim reason. critic_catch records every issue the review agent found, with its target and severity. Both are events rather than separate spans on purpose: they belong to the call they describe, and they survive in the trace even when the call ultimately succeeds.

Metrics. Six counters and a histogram, because some questions are time series questions rather than trace questions: devswarm.tokens, devswarm.llm.calls, devswarm.llm.duration, devswarm.fallback.promotions, devswarm.critic.catches, devswarm.generations, devswarm.refinements. Labelled by role, model and outcome.

Logs. Structured records for the things a human reads during an incident: a fallback promoting, a doctor diagnosis, a generation completing with its verdict and catch count. Same resource attributes as the traces, so a log line and a span line up.

The whole trace layer is now extracted into a small library, otel-swarm, that any multi-agent system can drop in. DevSwarm consumes it as a real dependency, which means if the library breaks, our own dashboards go dark first. That felt like the honest way to ship it.

One generation as a flame graph. Planner, then frontend and backend in parallel, then the critic.

The log stream during a rough run. WARN lines are fallback promotions, each naming the model that failed and why.

Reading the swarm out of ClickHouse

Five dashboards, all committed as JSON in observability/dashboards/. The one we actually live in is Command Center: is the swarm healthy, and if not, which role. The odd one out is Born Observable, which contains no swarm data at all, only the generated apps reporting under their own service names.

Three queries worth sharing, because span events in ClickHouse are not obvious the first time.

Role health, straight off the trace store:

SELECT attributes_string['devswarm.role'] AS role,
       count() AS calls,
       countIf(statusCode = 2) AS errors,
       round(quantile(0.95)(durationNano) / 1e9, 1) AS p95_s,
       sum(attributes_number['gen_ai.usage.input_tokens']
         + attributes_number['gen_ai.usage.output_tokens']) AS tokens
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = 'devswarm' AND name LIKE 'llm.%'
GROUP BY role ORDER BY calls DESC

Fallback promotions over time, which requires reaching into the events array:

SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS ts,
       attributes_string['devswarm.role'] AS role,
       count() AS value
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = 'devswarm'
  AND arrayExists(e -> e LIKE '%fallback_promotion%', events)
GROUP BY ts, role ORDER BY ts

The one that took longest to work out reads inside the events, so you can ask what the review gate actually caught rather than how many times it caught something:

SELECT JSONExtractString(e, 'attributeMap', 'severity') AS severity,
       JSONExtractString(e, 'attributeMap', 'target') AS target,
       count() AS catches
FROM signoz_traces.distributed_signoz_index_v3
ARRAY JOIN events AS e
WHERE serviceName = 'devswarm' AND name = 'agent.critic'
  AND JSONExtractString(e, 'name') = 'critic_catch'
GROUP BY severity, target ORDER BY catches DESC

The answer, over the week: 221 high, 46 medium, 10 low, and the frontend agent is on the receiving end of 70 percent of them. That single row changed how we think about the swarm. The half of the system generating markup and client-side state is where the bugs live, not the half touching the database.

One design decision we are glad about: the marketing numbers on our own landing page are fetched from these same queries at request time. The page cannot drift from the telemetry, because there is only one source of both.

Command Center. Top row answers "is the swarm healthy", the role-health table answers "which role", while the fallback chart should trend to zero.

LLM Economics: tokens and latency per role and per model, which is how we caught the frontend role burning two thirds of its budget on reasoning.

What one app actually costs

Every token in the table below came off a span. This is one real run, the letterpress site above, priced at the rates the Hugging Face router itself reports for the providers we use.

role	model	in	out	cost
frontend	GLM-5.2	27,855	38,888	$0.2101
critic	Kimi-K2.7-Code	50,996	14,539	$0.1066
planner	GLM-5.2	670	4,090	$0.0189
backend	Qwen3-Coder-480B	2,800	3,778	$0.0069
		82,321	61,295	$0.34

Thirty four cents for a designed marketing site with a working Express backend, a waitlist that validates email and rejects duplicates, plus its own OpenTelemetry wiring. Across passing runs the range is about 18 cents to 56 cents.

Two things in that table surprised us.

The critic costs fifteen times what the backend author costs. Reviewing the code is dramatically more expensive than writing it, because review means reading both artifacts in full, twice, while the backend agent writes one file once. Nobody budgets for that. If you are building a review gate into an agent system, it is not a rounding error on top of generation, it is a third of your bill.

And a failed run costs more than a successful one. Our worst generations burned 232,000 tokens hitting the regeneration ceiling, against 74,000 for the cleanest pass. So convergence is not only a quality metric, it is the cost metric. Fixing the contract-format bug in finding five did more for our unit economics than any model swap we made.

Alerts that wake an agent instead of a human

This is the part of the build we are proudest of, and it is a genuinely small amount of code.

Two alert rules live in observability/alerts/: a fallback-usage spike and a critic catch-rate flatline. Both notify a webhook channel called swarm-doctor, which points at POST /api/doctor/webhook on the swarm itself.

When an alert fires, the Doctor wakes up, queries the swarm's own traces for the last hour and decides what to do about the routing table. It promotes a backup model, resets a recovered primary or does nothing, then explains itself in plain English using the numbers it just read. Its first real diagnosis, verbatim:

"The critic role is the clear problem area: its primary triggered 6 fallback promotions out of 9 calls (67%) with a 22% error rate and p95 latency of 242s. Backend, planner and doctor are healthy."

The Doctor's own model calls are traced too, so the healer is exactly as observable as the patient.

The Doctor reporting a healthy swarm. It read 180 minutes of its own traces to say so, and it is honest about sample size: "call volume is very low, so latency figures are not statistically meaningful".

Two hard-won SigNoz API notes, since we lost hours to both:

Alert rules must be created against /api/v2/rules with schemaVersion: v2alpha1, a notificationSettings block and at least one channel. The v1 endpoint accepts the request and returns "alert rule is not valid" with no indication of which field is wrong. Dashboards, by contrast, go to /api/v1/dashboards with a SIGNOZ-API-KEY header and behave exactly as documented.

Also: a cold Docker restart can leave ClickHouse replicas read-only until Keeper reconnects. It usually self-heals within a minute. If it does not, SYSTEM RESTORE REPLICA per table clears it.

Apps that are born observable

Every app the swarm generates ships instrumented. Alongside index.html and server.js, each generated folder gets an otel.mjs bootstrap, a package.json, and a signoz-dashboard.json scoped to that app's own service name. If a SIGNOZ_API_TOKEN is configured, the dashboard is created in SigNoz at generation time, before the user has opened the preview.

So the generated app appears in SigNoz as its own service, with RED metrics and a routes table, seconds after it exists. Twenty four of them are in our instance right now.

Worth saying because people assume otherwise: there is no image model anywhere in this pipeline. The swarm generates 227 inline SVG elements across the 26 apps it has built, an average of 8.7 per app, and every one of them was written as markup by a language model. The Vandercook press in the screenshot above is hand-drawn SVG, not a generated image. The only assets we ever image-generated are DevSwarm's own favicon and social card, which are branding for the tool rather than anything the swarm produces.

One of the outputs. The Vandercook press is inline SVG the model drew itself, and this app reports to SigNoz under its own service name from the moment it boots.

One caveat we had to learn the hard way: an app only emits spans while its backend is actually running. Early on our preview served the frontend statically, so the generated Express server never booted and the app silently fell back to localStorage. The preview looked perfect and the service page was almost empty. That mismatch, two spans where there should have been dozens, is what gave the bug away. Previews now spawn the real server as a child process and proxy to it.

Six things our own telemetry told us we had wrong

This is the section I would want to read, so it is the longest one.

1. A "model limitation" was a stale constant we wrote ourselves.

Traces showed every frontend failure as finish: length, truncating full-page HTML. We concluded GLM-5.2's provider capped completions at 16384 tokens and moved the role to another model. The cap was real when we found it. It was also in our own config, and when the provider limit later lifted, our constant kept enforcing a limit that no longer existed. The average frontend artifact needs about 19,000 output tokens. We had guaranteed truncation and blamed the model for a fortnight.

2. The real cause was provider roulette, visible only in the span event text.

After removing our own cap, GLM still failed intermittently with a 400: max_completion_tokens is limited to 16384 for glm-5.2. The Hugging Face router load-balances a model across every provider serving it, and their limits disagree. We probed all seven: scaleway caps at 16384, featherless at 32768, novita and zai-org at 131072, while together, fireworks-ai and deepinfra accept 200000 or more. Unpinned, roughly one request in seven hit the strict provider and died instantly. Pinning the model to one provider produced our first ever generation with zero fallbacks. That entire diagnosis came out of the reason attribute on a fallback_promotion event.

3. Our span attribute was hiding the failures we most wanted to see.

When a primary failed, our code overwrote gen_ai.request.model on the span with the fallback's name. It seemed tidy. It meant every dashboard row attributed the primary's failure, and its wasted latency, to the fallback that cleaned up after it. We spent an afternoon convinced the critic's backup model was slow and error-prone. Isolating them in a benchmark showed the opposite: the backup was fine at 7 seconds, and the primary was the problem. If you take one implementation detail from this post, take this one. Record the model you attempted, and put the promotion in an event, not on top of the attribute you will later group by.

4. Our review gate was the weakest model in the swarm.

We had never benchmarked the critic, so we built one: a generated app with three documented contract defects, three runs per model, scored on defect recall. DeepSeek-V4-Pro, our incumbent primary, found 2 of 9. One run burned its entire 32768-token budget and returned nothing parseable. Kimi-K2.7-Code found 8 of 9 and was consistent across runs. The clearest pattern in the data was that on a review task, reasoning volume tracks defect recall: the terse models answered in under 250 output tokens and missed real bugs.

5. A 20 percent pass rate was one missing sentence in the contract.

Our plans specified field names and types but never formats, ranges or nullability. So the backend rejected rating: 0 while the frontend sent 0 as its default, and the backend demanded YYYY-MM-DD while the frontend sent full ISO strings. Three consecutive generations hit the regeneration ceiling on exactly this class of disagreement. The fix was making the planner write a binding rules string per field, for example "integer 0 to 5 inclusive, where 0 means unrated and is a valid value". Both builders now read the same sentence. Catches dropped from 9 to 3 and the next run passed.

6. Our design system was making the output worse.

We wrote a careful design guide so generated apps would not look like generated apps. Then we measured it: same model, same prompt, the only variable being whether the guide was attached. Without it, 14 inline SVGs, 3 animations and a deliberate typeface pairing. With it, 5 SVGs, 1 animation and Courier New. Three of our own rules did that. "System font stack is fine" told the model not to bother choosing type. "Cut any animation that does not serve the subject" read as licence to strip ornament. And our frontend prompt banned all external requests, which silently banned Google Fonts, so it could not have chosen a real typeface even if it wanted to. We had written a list of prohibitions, which is good at preventing bad output and bad at producing good output.

Before. Mono labels from a system stack, and card colours the backend invented at random.

After. Same model, same prompt, three rules removed from our guide: Fraunces display type, a drawn logo mark, filter chips carrying live counts, plus the books rendered as spines on a shelf.

There was a second layer to that one. The guide had a generous section for marketing sites, licensing scroll reveals, entrance sequences and layered depth, plus a stingy section for apps. Every app the swarm built was being held to a deliberately plainer standard than every site, and nobody had noticed because the sites looked great.

How this was built

DevSwarm was built with Claude Code, which is worth stating plainly rather than leaving as an inference. An AI coding agent helped build an AI coding agent, and the hackathon rules ask entrants to declare assistant use, so here it is.

It is also relevant to the point of this post. Every finding in the section above started as a confident, wrong belief held by both of us, human and assistant alike. The stale token cap, the model we blamed for a provider's limit, the review gate we assumed was our strongest link, the design system we were sure was helping. None of those were resolved by reasoning harder about the code. They were resolved by a span event, a dashboard row or a benchmark disagreeing with us.

That is the argument for instrumenting an agent system early. When you are building with agents, and with an agent, the telemetry is the only participant in the conversation with no opinion to defend.

What is worth copying from this build

If you are instrumenting an agent system, three things paid for themselves immediately.

Put the reason text in the span event. Not a code, not an enum, the actual provider error string. Both of our worst bugs were solved by reading that field, and neither would have been visible in a metric.

Never overwrite an attribute you intend to group by. Add, do not replace.

Make your product read its own telemetry. Our landing page statistics, our Doctor's diagnosis and our dashboards all run the same queries against the same trace store. It removes a whole category of drift, and it turns your observability stack from a debugging tool into a feature.

The uncomfortable part

The through line of every finding above is the same, and I have thought about it more than I expected to.

Not one of these was a hard problem. A stale constant. A provider with a different limit. An attribute overwritten in the wrong place. A missing sentence in a contract. Three over-cautious lines in a style guide. Any of them would have been a five minute fix if we had known. Together they cost us most of a week and made the system look, from the outside, like the models were letting us down.

They were not. Every single time, the model did exactly what our configuration told it to do. The failure was always upstream of the model, in something we had written and then stopped looking at.

I think that is the actual lesson of building with agents, and it is not a comfortable one. The debugging skill is not prompt engineering. It is being willing to believe your instrumentation over your own memory of what you configured three days ago. We only got there because the telemetry kept producing numbers that made our explanations impossible.

If you are building something similar, I would genuinely like to know whether your experience matches. My suspicion is that a lot of "the model is not good enough" is actually "my config is stale and I have no way to see it".

The bug that never crashed: how I fuzzed an AI's own code sandbox and found it lying to its model

Himanshu Kumar — Wed, 22 Jul 2026 13:26:43 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The bug that never crashed

The scariest bug I caught this month never threw a stack trace. It never paged anyone. It just quietly made an AI agent dumber, on repeat, and handed the blame to the model.

Here is the whole story, because the way I found it turned out to matter more than any single fix.

Where it started

smolagents is Hugging Face's agent framework, 28k+ stars, and it has an unusual design: the agent writes Python as its reasoning, and a sandboxed interpreter (LocalPythonExecutor) runs that code. The model thinks by coding. So the sandbox is not a side feature. It is the surface the model lives on.

I went in looking for one bug to fix for this challenge. Every obvious open issue was already claimed or had a competing PR. So instead of reading the issue tracker, I did something almost stupid: I fed the sandbox ordinary, boring, valid Python and watched what it refused to run.

The rule I gave myself: if it is valid Python and the sandbox rejects it, that is a bug. The model writes valid Python. If the sandbox chokes on it, the agent pays.

The first false rejection

config = {**{"temperature": 0.7}, "top_p": 0.9}

Merging dicts with **. Every LLM writes this. The sandbox's answer:

InterpreterError: NoneType is not supported.

There is no None in that line. I stared at it, then went to the source. In Python's AST, a **spread entry inside a dict literal has None where the key would be, a marker that says "this is a spread." smolagents walked the keys and tried to evaluate that None as if it were an expression. So the spread marker got blamed on a value the developer never wrote.

That is when it clicked. This was not a crash bug. It was a lying bug.

Why lying is worse than crashing

Here is the part that turned a one-off fix into an obsession.

The error was handled. smolagents catches it and feeds it back to the model as guidance: "here is what went wrong, try again." Good design, normally. But the guidance was wrong. The message said NoneType, and the model's code had no None, so the model could not act on it. It did the only thing a faithful agent can do with a message that is already a lie: it retried the exact same valid code.

And again. And again.

I only saw the loop because I had wired the demo to Sentry. One issue. Three events. Three identical failures on one line, each burning a real LLM call and a slot in the step budget, until the run gave up and produced a worse answer than it should have. Without something counting the events, you would never see the loop. You would just see an agent that "isn't very good," and you would go blame the model.

A red stack trace is mercy. It tells you where to look. The polite, well-handled, misleading error is the one that eats your afternoon, or in this case, eats the agent's entire budget while looking like helpful feedback.

Sentry's Seer read the same event and reached the same root cause I did, independently: None keys in ast.Dict fed to the evaluator, agent retries in a loop. That was the moment I trusted the pattern enough to go hunting.

The pattern held everywhere I looked

Same fuzzer, same rule, more boring valid Python. The sandbox kept lying:

10 ** 10 ** 8 did not error at all. It froze the entire process, forever, and the execution timeout never fired, because a single big-integer operation runs as one uninterruptible C call that holds the GIL and never lets the timeout thread wake up. The faulthandler dump showed the main thread stuck before the timer even armed.
best, *rest = scores failed with "Cannot unpack tuple of wrong size." There was no wrong size. The sandbox simply never implemented starred unpacking, a feature Python shipped in 2008.
a, b = "hi" was rejected outright. Strings unpack fine in real Python.
[a, b] = [1, 2] silently assigned nothing. No error, no values. The quietest failure of all.

Four different corners of the language. One personality: confidently wrong, politely delivered, and invisible unless you were counting.

The fixes, briefly

Each one came down to matching CPython instead of guessing:

Dict spread: evaluate pairwise, a None key means merge a mapping. And gate it on hasattr(value, "keys"), not the Mapping ABC, so duck-typed mappings work too.
The freeze: you cannot interrupt a GIL-holding C call, so refuse to start it. Estimate the result's bit length from the operands in O(1) and raise a real error above a cap, pointing the model at pow(base, exp, mod).
Unpacking: accept any iterable, support one starred target that absorbs the surplus into a list, handle list-pattern targets, and use CPython's exact error messages so that when the model does pass the wrong count, it gets an actionable message instead of a dead end.

Four fixes, four PRs, sixty-plus new tests, all verified failing on main first.

The twist I did not expect

I opened the PRs. Minutes later, OpenAI's Codex reviewer commented on two of them, and it was right both times. On the big-int fix it caught that I had checked type(x) is int, which lets bool and int subclasses slip through the guard. On the dict fix it caught that my Mapping ABC check was stricter than CPython.

So the final scoreboard was: an AI wrote the buggy code, a different AI reviewed my AI-assisted fix and found the hole, and a third AI (Seer) had already confirmed the root cause. I was the one steering, but I spent a good chunk of this project as the human in a loop of machines checking each other. Fitting, for a bug about an agent that could not tell it was stuck.

What I actually learned

Fuzz the boundary with valid input. Everyone fuzzes with garbage to find crashes. The higher-value bugs in an AI system are the false rejections: valid input the system refuses, because that is what silently degrades a model that is doing everything right.
A handled error is not a safe error. If the message is wrong, "handled" just means the failure is quiet instead of loud. Quiet is worse.
Instrument for absence and repetition. The tell for this whole bug class was not an exception. It was the same event, three times. An agent getting a different error each step is exploring. An agent getting the same error from the same input is stuck, and the stuckness is invisible to the agent because the message looks like feedback. Count your events.

The bugs are fixed. The lesson I am keeping is simpler than any of them: the dangerous failures in an agent stack are not the loud crashes. They are the polite, confident, wrong messages that let the model fail on repeat while everyone blames the model.

The evidence

Each one has its own full write-up with Sentry before/after evidence, if you want the deep dives. This was the story of how they all turned out to be the same bug wearing four different masks.

The smolagents sandbox broke 'a, *b = list', one of Python's most common lines

Himanshu Kumar — Tue, 21 Jul 2026 05:46:04 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Fourth entry, and the fuzzer that found entry 3 is still paying out. Same failure family, different corner of the language: valid Python the sandbox refuses to run, with an error that lies about why.

The one-liner every developer writes

best, *rest = scores

Splitting a list into "the first one" and "the rest" is about as ordinary as Python gets. It is PEP 3132, shipped in 2008. Under smolagents' sandbox it fails with:

InterpreterError: Cannot unpack tuple of wrong size

There is no wrong size. scores has four items and the pattern accepts any length of two or more. The message describes a problem that does not exist, so the agent does the only thing a faithful agent can do with a message that is already a lie: it retries the identical, valid code.

And it is not just the starred form. All of these are standard Python and all of them were broken:

first, *rest = "hello"   # Cannot unpack tuple of wrong size
a, b = "hi"              # Cannot unpack non-tuple value
[a, b] = [1, 2]          # silently assigns nothing at all

Why it happens

smolagents runs model-generated code in its own AST interpreter, and assignment targets go through one function, set_value. It handled exactly one shape: a fixed-size ast.Tuple.

elif isinstance(target, ast.Tuple):
    if not isinstance(value, tuple):
        if hasattr(value, "__iter__") and not isinstance(value, (str, bytes)):
            value = tuple(value)
        else:
            raise InterpreterError("Cannot unpack non-tuple value")
    if len(target.elts) != len(value):
        raise InterpreterError("Cannot unpack tuple of wrong size")
    ...

Three separate holes hide in those few lines:

Starred targets. a, *b has two target elements but the value has three items, so len(target.elts) != len(value) fires. The * marker is never even looked at.
Strings and bytes. They are explicitly excluded from the iterable path, so a, b = "hi" is rejected even though CPython unpacks strings happily.
List-pattern targets. [a, b] = ... is an ast.List, not an ast.Tuple, so it matches no branch, falls through, and silently assigns nothing. No error, no values, the worst kind of quiet.

The Sentry angle: count the retries

Same lesson as the last two entries, and Sentry keeps making it visible. The error is handled: the agent catches it and feeds it back to the model as guidance. But the guidance is wrong, so the model cannot act on it, so it loops. One bug, one misleading message, three identical failures burning three steps:

Three events on one issue is the retry loop made countable. Without it you would see a slow run, not a stuck one. Sentry's Seer read the same event and landed on the exact cause:

smolagents' custom Python interpreter does not support starred unpacking (e.g. best, *rest = scores), treating it as a fixed-size tuple unpack. set_value checks len(target.elts) != len(value) and raises, without handling ast.Starred targets.

The fix

Rewrite the branch to match CPython instead of guessing:

elif isinstance(target, (ast.Tuple, ast.List)):
    elts = target.elts
    starred = [i for i, e in enumerate(elts) if isinstance(e, ast.Starred)]
    if len(starred) > 1:
        raise InterpreterError("multiple starred expressions in assignment")
    if not hasattr(value, "__iter__"):
        raise InterpreterError(f"cannot unpack non-iterable {type(value).__name__} object")
    values = list(value)
    if starred:
        i = starred[0]
        n_after = len(elts) - i - 1
        if len(values) < i + n_after:
            raise InterpreterError(
                f"not enough values to unpack (expected at least {i + n_after}, got {len(values)})"
            )
        split = len(values) - n_after
        # assign head, then the starred target gets the middle as a list, then the tail
        ...
    else:
        if len(values) < len(elts):
            raise InterpreterError(f"not enough values to unpack (expected {len(elts)}, got {len(values)})")
        if len(values) > len(elts):
            raise InterpreterError(f"too many values to unpack (expected {len(elts)})")
        ...

Any iterable now unpacks, a single starred target absorbs the surplus into a list in any position (a, *b, *a, b, a, *b, c), list-pattern targets work, and the size errors read exactly like CPython's, so when the model genuinely does pass the wrong number of values it gets an actionable message instead of a dead end.

After

app: step 1 ok, output = (90, [82, 71, 65])

One step. No loop. best is 90, rest is the tail, the way the model expected all along.

Numbers

4 valid unpacking forms fixed: starred targets, string unpacking, list-pattern targets, and the CPython error messages
Reproduced on current main and 1.26.0; 3 wasted agent steps per occurrence, visible only because Sentry counts events
14 new tests plus one existing test updated to the improved message
411 passing, ruff clean

Links

The through-line across all four entries has not changed: the dangerous agent bugs are not the loud crashes, they are the polite, well-handled, wrong messages that let the model fail on repeat. Fuzz the sandbox with ordinary valid code, and count your events.

The smolagents bug that made my agent retry the same valid code three times

Himanshu Kumar — Mon, 20 Jul 2026 07:19:13 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Third entry in the DEV x Sentry Bug Smash. Entry 1 was a crash with a confusing message. Entry 2 was a freeze the timeout could not catch. This one is quieter and sneakier: valid Python that the sandbox rejects with an error pointing at the wrong thing entirely.

When the open issues run out, fuzz

By entry 3 every obvious open smolagents bug was already claimed or had a competing PR. So instead of reading the issue tracker I pointed a small fuzzer at the piece of smolagents that runs the most untrusted code: LocalPythonExecutor, the sandbox that executes model-generated Python.

The method is boring and effective: feed it ordinary, valid Python one snippet at a time, and flag anything that raises InterpreterError. Valid Python that the sandbox refuses to run is, by definition, a bug, because the model writes valid Python and expects it to work.

That surfaced four unreported bugs in one afternoon. This post is about the one I shipped: dict unpacking.

The bug

config = {**{"temperature": 0.7, "max_tokens": 512}, "top_p": 0.9}

Merging dicts with ** is one of the most common things an LLM writes. Under smolagents it fails with:

InterpreterError: NoneType is not supported.

There is no None anywhere in that line. The message sends you looking for a null value that does not exist.

Why it happens

In Python's AST, a dict literal keeps its keys and values in two parallel lists. For a normal entry the key is an AST node. For a **mapping spread entry, the key is literally None, a signal that says "this is a spread, not a key/value pair."

smolagents evaluated every key by walking expression.keys and calling evaluate_ast(key, ...) on each one. When the key is None, that call falls through every isinstance branch to the catch-all raise InterpreterError(f"{type} is not supported"). So the spread marker got evaluated as if it were an expression, and the model got blamed for a None it never wrote.

Why silence is the expensive part

Here is the part the Sentry view made obvious. The error is handled: the agent catches it and feeds it back to the model as "here is what went wrong, try again." But the message names NoneType, and the model's code has no None, so the model cannot act on it. It retries the exact same valid syntax. And again. Every step burns a real LLM call and a slot in the step budget until the run gives up.

One bug, one misleading message, three identical failures:

Three events on a single issue is not noise. It is the agent stuck in a loop, and without Sentry counting the events you would never see the loop, only a run that quietly underperformed. Sentry's Seer read the same event and reached the exact root cause:

smolagents' LocalPythonExecutor doesn't handle dict unpacking (**) syntax: None keys in ast.Dict cause an unsupported type error. [...] evaluate_ast(None, ...) matches no isinstance branch and falls to the else clause. The interpreter raises InterpreterError: NoneType is not supported, the agent retries with identical code, burning steps in a loop.

The fix

Evaluate the dict pairwise instead of evaluating keys blindly. A None key means "merge this mapping":

result = {}
for key_node, value_node in zip(expression.keys, expression.values):
    if key_node is None:
        value = evaluate_ast(value_node, *common_params)
        if not hasattr(value, "keys"):
            raise InterpreterError(f"'{type(value).__name__}' object is not a mapping")
        result.update(value)
    else:
        key = evaluate_ast(key_node, *common_params)
        result[key] = evaluate_ast(value_node, *common_params)
return result

This matches CPython exactly: spreads merge in order, later keys win, and unpacking a non-mapping raises 'list' object is not a mapping.

A reviewer caught my fix being too strict

I first gated the spread on isinstance(value, Mapping). Minutes after the PR opened, OpenAI's Codex reviewer flagged it (P2): CPython does not require the Mapping ABC, it only requires an object with a keys() method. Since the sandbox lets users define their own classes, a duck-typed mapping with keys() and __getitem__() would have been wrongly rejected. I switched the check to hasattr(value, "keys") and added a test for exactly that case. AI wrote the code, AI reviewed the code, I kept score.

After

On the patched build the same line just runs:

app: step 1 ok, config = {'temperature': 0.7, 'max_tokens': 512, 'top_p': 0.9}

One step, no loop, no phantom None.

Numbers

4 unreported bugs found by fuzzing valid Python through the sandbox; this is the first fix
Misleading NoneType error reproduced on current main and 1.26.0
3 wasted agent steps per occurrence, visible only because Sentry counts events
9 new tests: spreads, double spreads, override order both ways, a duck-typed mapping class, empty spread, non-mapping rejection
406 passing, ruff clean

Links

The pattern across all three entries: the worst agent bugs do not throw a red stack trace at you. They hand the model a plausible-but-wrong message and let it fail politely, on repeat. Count your events.